[PATCH blktests 0/3] Three SRP patches

[PATCH blktests 0/3] Three SRP patches

[Bart Van Assche]

Hi Omar,

The three patches in this series improve the SRP tests. Please consider these
patches for inclusion in the upstream blktests repository.

Thanks,

Bart.

Bart Van Assche (3):
  tests/block/030: Make this test less noisy
  tests/srp/rc: Improve reliability of stop_lio_srpt()
  rdma: Use rdma link instead of /sys/class/infiniband/*/parent

 common/multipath-over-rdma | 111 +++++++++++--------------------------
 tests/block/030            |   2 +-
 tests/srp/rc               |  52 ++++++-----------
 3 files changed, 48 insertions(+), 117 deletions(-)

[PATCH blktests 1/3] tests/block/030: Make this test less noisy

[Bart Van Assche]

Since test block/030 injects blk_mq_realloc_hw_ctxs() failures, it is
expected that writes into the 'submit_queues' attribute can fail. Send
the 'nproc: write error: Cannot allocate memory' failures to $FULL instead
of stderr. See also commit a668c61064f2 ("Add a test that triggers the
blk_mq_realloc_hw_ctxs() error path").

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 tests/block/030 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/block/030 b/tests/block/030
index 84302c109c0c..d2e528697873 100755
--- a/tests/block/030
+++ b/tests/block/030
@@ -42,7 +42,7 @@ test() {
 	if { echo "$(<"$sq")" >$sq; } 2>/dev/null; then
 		for ((i = 0; i < 100; i++)); do
 			echo 1 > $sq
-			nproc > $sq
+			{ nproc > $sq; } >>"$FULL" 2>&1
 		done
 	else
 		SKIP_REASON="Skipping test because $sq cannot be modified"

[PATCH blktests 2/3] tests/srp/rc: Improve reliability of stop_lio_srpt()

[Bart Van Assche]

Remove the 'np' directory if it exists. Unload the iscsi_target_mod kernel
module if it has been loaded.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 tests/srp/rc | 43 +++++++++++++++----------------------------
 1 file changed, 15 insertions(+), 28 deletions(-)

diff --git a/tests/srp/rc b/tests/srp/rc
index 1f665a28db66..700cd71ea155 100755
--- a/tests/srp/rc
+++ b/tests/srp/rc
@@ -545,34 +545,21 @@ stop_lio_srpt() {
 		fi
 	done
 
-	if [ -e /sys/kernel/config/target/srpt ]; then
-		(
-			cd /sys/kernel/config/target/srpt && (
-				for d in */*/acls/*/*/lun*; do [ -L "$d" ] && rm "$d"; done
-				for d in */*/acls/*/lun*; do [ -d "$d" ] && rmdir "$d"; done
-				for d in */*/acls/*; do [ -d "$d" ] && rmdir "$d"; done
-				for d in */*/lun/lun*/*; do [ -L "$d" ] && rm "$d"; done
-				for d in */*/lun/lun*; do [ -d "$d" ] && rmdir "$d"; done
-				for d in */*; do [ -e "$d/lun" ] && rmdir "$d"; done
-				for d in *; do [ -e "$d/fabric_statistics" ] && rmdir "$d"; done
-				true
-			) &&
-				cd .. &&
-				for ((i=0;i<10;i++)); do
-					rmdir srpt
-					[ -e srpt ] || break
-					sleep .1
-				done &&
-				[ ! -e srpt ] &&
-				unload_module ib_srpt 10
-		) || return $?
-	fi
-
-	rmdir /sys/kernel/config/target/core/*/* >&/dev/null
-	rmdir /sys/kernel/config/target/core/* >&/dev/null
-
-	for m in ib_srpt target_core_pscsi target_core_iblock target_core_file \
-			 target_core_stgt target_core_user target_core_mod
+	rmdir /sys/kernel/config/target/*/*/*/np/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/np >&/dev/null
+	rm -f /sys/kernel/config/target/*/*/*/acls/*/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/acls/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/acls/* >&/dev/null
+	rm -f /sys/kernel/config/target/*/*/*/lun/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/lun/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/*/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/*/* >&/dev/null
+	rmdir /sys/kernel/config/target/* >&/dev/null
+
+	for m in ib_srpt iscsi_target_mod target_core_pscsi target_core_iblock \
+			 target_core_file target_core_stgt target_core_user \
+			 target_core_mod
 	do
 		unload_module $m 10 || return $?
 	done

[PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

[Bart Van Assche]

The approach of verifying whether or not an RDMA interface is associated
with the rdma_rxe interface by looking up its parent device is deprecated
and will be removed soon from the Linux kernel. Hence this patch that uses
the rdma link command instead.

Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Yi Zhang <yi.zhang@redhat.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 common/multipath-over-rdma | 111 +++++++++++--------------------------
 tests/srp/rc               |   9 +--
 2 files changed, 32 insertions(+), 88 deletions(-)

diff --git a/common/multipath-over-rdma b/common/multipath-over-rdma
index 9d9d2b27af83..9e68189b56fd 100644
--- a/common/multipath-over-rdma
+++ b/common/multipath-over-rdma
@@ -57,6 +57,9 @@ _multipathd_version_ge() {
 }
 
 get_ipv4_addr() {
+	if [ ! -e "/sys/class/net/$1" ]; then
+		echo "get_ipv4_addr(): $1 is not a network interface" 1>&2
+	fi
 	ip -4 -o addr show dev "$1" |
 		sed -n 's/.*[[:blank:]]inet[[:blank:]]*\([^[:blank:]/]*\).*/\1/p'
 }
@@ -76,41 +79,11 @@ is_number() {
 	[ "$1" -eq "0$1" ] 2>/dev/null
 }
 
-# Check whether a device is an RDMA device. An example argument:
-# /sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0
-is_rdma_device() {
-	local d i inode1 inode2
-
-	inode1=$(stat -c %i "$1")
-	# echo "inode1 = $inode1"
-	for i in /sys/class/infiniband/*; do
-		d=/sys/class/infiniband/"$(readlink "$i")"
-		d=$(dirname "$(dirname "$d")")
-		inode2=$(stat -c %i "$d")
-		# echo "inode2 = $inode2"
-		if [ "$inode1" = "$inode2" ]; then
-			return
-		fi
-	done
-	false
-}
-
 # Lists RDMA capable network interface names, e.g. ib0 ib1.
 rdma_network_interfaces() {
-	(
-		cd /sys/class/net &&
-			for i in *; do
-				[ -e "$i" ] || continue
-				# Skip IPoIB (ARPHRD_INFINIBAND) network
-				# interfaces.
-				[ "$(<"$i"/type)" = 32 ] && continue
-				[ -L "$i/device" ] || continue
-				d=$(readlink "$i/device" 2>/dev/null)
-				if [ -n "$d" ] && is_rdma_device "$i/$d"; then
-					echo "$i"
-				fi
-			done
-	)
+	rdma link show |
+		sed -n 's/^.*[[:blank:]]netdev[[:blank:]]\+\([^[:blank:]]*\)[[:blank:]]*/\1/p' |
+		sort -u
 }
 
 # Check whether any stacked block device holds block device $1. If so, echo
@@ -411,47 +384,36 @@ all_primary_gids() {
 	done
 }
 
-# Check whether or not an rdma_rxe instance has been associated with network
-# interface $1.
-has_rdma_rxe() {
-	local f
-
-	for f in /sys/class/infiniband/*/parent; do
-		if [ -e "$f" ] && [ "$(<"$f")" = "$1" ]; then
-			return 0
-		fi
-	done
-
-	return 1
+# Check whether or not an rdma_rxe or siw instance has been associated with
+# network interface $1.
+has_soft_rdma() {
+	rdma link | grep -q " netdev $1[[:blank:]]*\$"
 }
 
 # Load the rdma_rxe or siw kernel module and associate it with all network
 # interfaces.
 start_soft_rdma() {
+	local type
+
 	{
 	if [ -n "$use_siw" ]; then
 		modprobe siw || return $?
-		(
-			cd /sys/class/net &&
-				for i in *; do
-					[ -e "$i" ] || continue
-					[ -e "/sys/class/infiniband/${i}_siw" ] && continue
-					rdma link add "${i}_siw" type siw netdev "$i" ||
-						echo "Failed to bind the siw driver to $i"
-				done
-		)
+		type=siw
 	else
 		modprobe rdma_rxe || return $?
-		(
-			cd /sys/class/net &&
-				for i in *; do
-					if [ -e "$i" ] && ! has_rdma_rxe "$i"; then
-						echo "$i" > /sys/module/rdma_rxe/parameters/add ||
-							echo "Failed to bind the rdma_rxe driver to $i"
-					fi
-				done
-		)
+		type=rxe
 	fi
+	(
+		cd /sys/class/net &&
+			for i in *; do
+				[ -e "$i" ] || continue
+				[ "$i" = "lo" ] && continue
+				[ "$(<"$i/addr_len")" = 6 ] || continue
+				has_soft_rdma "$i" && continue
+				rdma link add "${i}_$type" type $type netdev "$i" ||
+				echo "Failed to bind the $type driver to $i"
+			done
+	)
 	} >>"$FULL"
 }
 
@@ -459,27 +421,16 @@ start_soft_rdma() {
 # unload the rdma_rxe kernel module.
 stop_soft_rdma() {
 	{
-	(
-		cd /sys/class/net &&
-			for i in *; do
-				if [ -e "$i" ] && has_rdma_rxe "$i"; then
-					{ echo "$i" > /sys/module/rdma_rxe/parameters/remove; } \
-						2>/dev/null
-				fi
-			done
-	)
+	rdma link |
+		sed -n 's,^link[[:blank:]]*\([^/]*\)/.* netdev .*,\1,p' |
+		while read -r i; do
+		      echo "$i ..."
+		      rdma link del "${i}" || echo "Failed to remove ${i}"
+		done
 	if ! unload_module rdma_rxe 10; then
 		echo "Unloading rdma_rxe failed"
 		return 1
 	fi
-	(
-		cd /sys/class/net &&
-			for i in *_siw; do
-				[ -e "$i" ] || continue
-				rdma link del "${i}" ||
-					echo "Failed to unbind the siw driver from ${i%_siw}"
-			done
-	)
 	if ! unload_module siw 10; then
 		echo "Unloading siw failed"
 		return 1
diff --git a/tests/srp/rc b/tests/srp/rc
index 700cd71ea155..07378fab2f2c 100755
--- a/tests/srp/rc
+++ b/tests/srp/rc
@@ -142,14 +142,7 @@ do_ib_cm_login() {
 }
 
 rdma_dev_to_net_dev() {
-	local b d rdma_dev=$1
-
-	b=/sys/class/infiniband/$rdma_dev/parent
-	if [ -e "$b" ]; then
-		echo "$(<"$b")"
-	else
-		echo "${rdma_dev%_siw}"
-	fi
+	rdma link show "$1/1" | sed 's/.* netdev //;s/[[:blank:]]*$//'
 }
 
 # Tell the SRP initiator to log in to an SRP target using the RDMA/CM.

Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

[Omar Sandoval]

On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote:
> The approach of verifying whether or not an RDMA interface is associated
> with the rdma_rxe interface by looking up its parent device is deprecated
> and will be removed soon from the Linux kernel. Hence this patch that uses
> the rdma link command instead.
> 
> Cc: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Yi Zhang <yi.zhang@redhat.com>
> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
> ---
>  common/multipath-over-rdma | 111 +++++++++++--------------------------
>  tests/srp/rc               |   9 +--
>  2 files changed, 32 insertions(+), 88 deletions(-)

I think we need to add _have_program rdma checks to srp and nvmeof-mp,
right? The first two patches look fine, I'll merge those.

Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

[Bart Van Assche]

On 2/3/21 8:40 AM, Omar Sandoval wrote:
> On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote:
>> The approach of verifying whether or not an RDMA interface is associated
>> with the rdma_rxe interface by looking up its parent device is deprecated
>> and will be removed soon from the Linux kernel. Hence this patch that uses
>> the rdma link command instead.
>>
>> Cc: Jason Gunthorpe <jgg@nvidia.com>
>> Cc: Yi Zhang <yi.zhang@redhat.com>
>> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
>> ---
>>  common/multipath-over-rdma | 111 +++++++++++--------------------------
>>  tests/srp/rc               |   9 +--
>>  2 files changed, 32 insertions(+), 88 deletions(-)
> 
> I think we need to add _have_program rdma checks to srp and nvmeof-mp,
> right? The first two patches look fine, I'll merge those.

Sure, I will add a _have_program rdma check and repost this patch.

Thanks,

Bart.

Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

[Yi Zhang]

Hello Bart

Currently blktests was broken on latest kernel, could you resend this 
updated patch. :)

Thanks
Yi

On 2/4/21 11:58 AM, Bart Van Assche wrote:
> On 2/3/21 8:40 AM, Omar Sandoval wrote:
>> On Mon, Jan 25, 2021 at 08:45:19PM -0800, Bart Van Assche wrote:
>>> The approach of verifying whether or not an RDMA interface is associated
>>> with the rdma_rxe interface by looking up its parent device is deprecated
>>> and will be removed soon from the Linux kernel. Hence this patch that uses
>>> the rdma link command instead.
>>>
>>> Cc: Jason Gunthorpe <jgg@nvidia.com>
>>> Cc: Yi Zhang <yi.zhang@redhat.com>
>>> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
>>> ---
>>>   common/multipath-over-rdma | 111 +++++++++++--------------------------
>>>   tests/srp/rc               |   9 +--
>>>   2 files changed, 32 insertions(+), 88 deletions(-)
>> I think we need to add _have_program rdma checks to srp and nvmeof-mp,
>> right? The first two patches look fine, I'll merge those.
> Sure, I will add a _have_program rdma check and repost this patch.
>
> Thanks,
>
> Bart.
>
>

Re: [PATCH blktests 3/3] rdma: Use rdma link instead of /sys/class/infiniband/*/parent

[Bart Van Assche]

On 2/28/21 6:45 AM, Yi Zhang wrote:
> Currently blktests was broken on latest kernel, could you resend this
> updated patch. :)
Thanks for the reminder. A new version of this patch has just been posted.

Bart.