[PATCH 0/4] Add rxe as a second software RDMA option

[PATCH 0/4] Add rxe as a second software RDMA option

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

Zhu would like to have the ability to run file system tests with
software RoCE. Add rxe as an alternative to siw.

I've tested that the Kconfig and Ansible infrastructure behaves as
expected, but rxe itself might not be robust enough for the existing
workflows to pass cleanly. But hey, that's why we're running these
tests, right?

Chuck Lever (4):
  bringup.goals: Make siw settings more generic
  playbooks: Add a playbook to set up rxe on target nodes
  Kconfig: Add support for testing RDMA with software RoCE
  Update help text that mentions SIW

 Makefile                                   |  2 +-
 kconfigs/Kconfig.bringup.goals             | 36 ++++++++++++++++++----
 playbooks/roles/rxe/tasks/main.yml         | 29 +++++++++++++++++
 playbooks/roles/rxe/templates/udev-rule.j2 |  2 ++
 playbooks/rxe.yml                          |  4 +++
 scripts/rdma.Makefile                      | 23 ++++++++++++++
 scripts/siw.Makefile                       | 11 -------
 workflows/fstests/nfs/Kconfig              |  6 ++--
 workflows/gitr/nfs/Kconfig                 |  3 +-
 9 files changed, 93 insertions(+), 23 deletions(-)
 create mode 100644 playbooks/roles/rxe/tasks/main.yml
 create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2
 create mode 100644 playbooks/rxe.yml
 create mode 100644 scripts/rdma.Makefile
 delete mode 100644 scripts/siw.Makefile

-- 
2.45.1

[PATCH 1/4] bringup.goals: Make siw settings more generic

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

I'm about to introduce another option for software RDMA. Rename
the siw Makefile and Kconfig options to make way for this new
option.

Since only one type of emulated RDMA device can be attached to an
Ethernet device at a time, make the Kconfig logic choose between
no device, siw, or, when it's added, rxe.

NB: This patch replaces CONFIG_KDEVOPS_SETUP_SIW. If you have
enabled siw in your configuration, please visit the config setting
for software RDMA after applying this patch.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 Makefile                                |  2 +-
 kconfigs/Kconfig.bringup.goals          | 26 +++++++++++++++++++------
 scripts/{siw.Makefile => rdma.Makefile} |  2 +-
 3 files changed, 22 insertions(+), 8 deletions(-)
 rename scripts/{siw.Makefile => rdma.Makefile} (79%)

diff --git a/Makefile b/Makefile
index 266eccd90111..20c2081118a9 100644
--- a/Makefile
+++ b/Makefile
@@ -104,7 +104,7 @@ ifeq (y,$(CONFIG_WORKFLOWS))
 include workflows/Makefile
 endif # CONFIG_WORKFLOWS
 
-include scripts/siw.Makefile
+include scripts/rdma.Makefile
 include scripts/ktls.Makefile
 include scripts/nfsd.Makefile
 include scripts/smbd.Makefile
diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals
index e6d3b1815f3b..e8629ebea5e0 100644
--- a/kconfigs/Kconfig.bringup.goals
+++ b/kconfigs/Kconfig.bringup.goals
@@ -77,16 +77,30 @@ config KDEVOPS_TRY_INSTALL_KDEV_TOOLS
 	  most kernel developers might prefer to have installed on target
 	  systems.
 
-config KDEVOPS_SETUP_SIW
-	bool "Configure soft iWARP on all test systems"
-	default n
+choice
+	prompt "Configure an RDMA device"
+	default KDEVOPS_SETUP_RDMA_NONE
 	help
-	  Configure software-emulated RDMA on all target hosts via the
+	  Provision an RDMA device on target nodes. The default is to
+	  provision no additional RDMA devices.
+
+config KDEVOPS_SETUP_RDMA_NONE
+	bool "none"
+	help
+	  Do not provision an additional RDMA device on target nodes.
+	  This is the default behavior.
+
+config KDEVOPS_SETUP_RDMA_SIW
+	bool "siw"
+	help
+	  Provision software-emulated RDMA on target nodes via the
 	  kernel siw.ko driver. siw can be used with any standard
 	  Ethernet device, including virtio-net devices.
 
-	  Because siw is currently not enabled in most Linux distro
-	  kernels, this option defaults to N.
+	  To use this device driver, enable the CONFIG_RDMA_SIW
+	  option in your test kernels.
+
+endchoice
 
 config KDEVOPS_SETUP_KTLS
 	bool "Configure ktls on the hosts with self-signed CA"
diff --git a/scripts/siw.Makefile b/scripts/rdma.Makefile
similarity index 79%
rename from scripts/siw.Makefile
rename to scripts/rdma.Makefile
index d09d1a2f82ac..d7cdf0062638 100644
--- a/scripts/siw.Makefile
+++ b/scripts/rdma.Makefile
@@ -1,4 +1,4 @@
-ifeq (y,$(CONFIG_KDEVOPS_SETUP_SIW))
+ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_SIW))
 
 siw:
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
-- 
2.45.1

[PATCH 2/4] playbooks: Add a playbook to set up rxe on target nodes

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

The playbook adds a udev rule to provision an rxe device when the
node is booted.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 playbooks/roles/rxe/tasks/main.yml         | 29 ++++++++++++++++++++++
 playbooks/roles/rxe/templates/udev-rule.j2 |  2 ++
 playbooks/rxe.yml                          |  4 +++
 3 files changed, 35 insertions(+)
 create mode 100644 playbooks/roles/rxe/tasks/main.yml
 create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2
 create mode 100644 playbooks/rxe.yml

diff --git a/playbooks/roles/rxe/tasks/main.yml b/playbooks/roles/rxe/tasks/main.yml
new file mode 100644
index 000000000000..88e655981f7f
--- /dev/null
+++ b/playbooks/roles/rxe/tasks/main.yml
@@ -0,0 +1,29 @@
+---
+- name: Include optional extra_vars
+  ansible.builtin.include_vars:
+    file: "{{ item }}"
+  with_first_found:
+    - files:
+        - "../extra_vars.yml"
+        - "../extra_vars.yaml"
+        - "../extra_vars.json"
+      skip: true
+  failed_when: false
+  tags: vars
+
+- name: Insert a udev rule to create an rxe device
+  become: true
+  become_method: ansible.builtin.sudo
+  ansible.builtin.template:
+    src: "udev-rule.j2"
+    dest: "/usr/lib/udev/rules.d/99-rxe.rules"
+    owner: "root"
+    group: "root"
+    mode: "0644"
+    force: false
+
+- name: Reload the udev ruleset
+  become: true
+  become_method: ansible.builtin.sudo
+  ansible.builtin.shell: "udevadm control --reload && udevadm trigger"
+  changed_when: true
diff --git a/playbooks/roles/rxe/templates/udev-rule.j2 b/playbooks/roles/rxe/templates/udev-rule.j2
new file mode 100644
index 000000000000..9f5b5e4a68f3
--- /dev/null
+++ b/playbooks/roles/rxe/templates/udev-rule.j2
@@ -0,0 +1,2 @@
+SUBSYSTEM=="net", KERNEL=="{{ ansible_default_ipv4.interface }}", \
+   PROGRAM="/usr/sbin/rdma link add rxe0 type rxe netdev {{ ansible_default_ipv4.interface }}"
diff --git a/playbooks/rxe.yml b/playbooks/rxe.yml
new file mode 100644
index 000000000000..dc51c0347628
--- /dev/null
+++ b/playbooks/rxe.yml
@@ -0,0 +1,4 @@
+---
+- hosts: all
+  roles:
+    - role: rxe
-- 
2.45.1

[PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

Add kdevops config option to use rxe rather than siw for RDMA. rxe
is the kernel's software RoCE driver, and like siw, it utilizes a
standard Ethernet device.

Suggested-by: Zhu Yanjun <zyjzyj2000@gmail.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 kconfigs/Kconfig.bringup.goals | 10 ++++++++++
 scripts/rdma.Makefile          | 12 ++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals
index e8629ebea5e0..794742530e9e 100644
--- a/kconfigs/Kconfig.bringup.goals
+++ b/kconfigs/Kconfig.bringup.goals
@@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
 	  To use this device driver, enable the CONFIG_RDMA_SIW
 	  option in your test kernels.
 
+config KDEVOPS_SETUP_RDMA_RXE
+	bool "rxe"
+	help
+	  Provision software-emulated RDMA on target nodes via the
+	  kernel rxe.ko driver. rxe can be used with any standard
+	  Ethernet device, including virtio-net devices.
+
+	  To use this device driver, enable the CONFIG_RDMA_RXE
+	  option in your test kernels.
+
 endchoice
 
 config KDEVOPS_SETUP_KTLS
diff --git a/scripts/rdma.Makefile b/scripts/rdma.Makefile
index d7cdf0062638..dd2a70120eb2 100644
--- a/scripts/rdma.Makefile
+++ b/scripts/rdma.Makefile
@@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
 PHONY += siw
 
 endif
+
+ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
+
+rxe:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
+		-f 30 -i hosts playbooks/rxe.yml
+
+KDEVOPS_BRING_UP_DEPS += rxe
+
+PHONY += rxe
+
+endif
-- 
2.45.1

[PATCH 4/4] Update help text that mentions SIW

[cel]

From: Chuck Lever <chuck.lever@oracle.com>

Generalize the help text, now that kdevops has two software RDMA
options.

Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 workflows/fstests/nfs/Kconfig | 6 +++---
 workflows/gitr/nfs/Kconfig    | 3 +--
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/workflows/fstests/nfs/Kconfig b/workflows/fstests/nfs/Kconfig
index 9f47761712b2..926bc40b4559 100644
--- a/workflows/fstests/nfs/Kconfig
+++ b/workflows/fstests/nfs/Kconfig
@@ -50,9 +50,9 @@ config FSTESTS_NFS_SECTION_RDMA
 	bool "Enable testing section: nfs_rdma"
 	default n
 	help
-	  Enabling this will test with the "rdma" mount option. Unless the
-	  hosts have RDMA hardware already, you probably also want to enable
-	  CONFIG_KDEVOPS_SETUP_SIW as well.
+	  Enabling this will test with the "rdma" mount option. Unless
+	  the target nodes have RDMA hardware already, enable one of
+	  the software RDMA options as well.
 
 config FSTESTS_NFS_SECTION_TLS
 	bool "Enable testing section: nfs_tls"
diff --git a/workflows/gitr/nfs/Kconfig b/workflows/gitr/nfs/Kconfig
index a8c81b098957..1b1e53316ac7 100644
--- a/workflows/gitr/nfs/Kconfig
+++ b/workflows/gitr/nfs/Kconfig
@@ -38,8 +38,7 @@ config GITR_NFS_SECTION_RDMA
 	help
 	  Provision a target node to run the gitr workflow on a mount
 	  using NFS/RDMA. Unless your target nodes have RDMA hardware
-	  already, you are likely to need to enable the
-	  CONFIG_KDEVOPS_SETUP_SIW option as well.
+	  already, enable one of the software RDMA options as well.
 
 config GITR_NFS_SECTION_V42
 	bool "Enable testing section: nfs_v42"
-- 
2.45.1

Re: [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE

[Zhu Yanjun]

On Mon, Jun 3, 2024 at 6:07 PM <cel@kernel.org> wrote:
>
> From: Chuck Lever <chuck.lever@oracle.com>
>
> Add kdevops config option to use rxe rather than siw for RDMA. rxe
> is the kernel's software RoCE driver, and like siw, it utilizes a
> standard Ethernet device.
>
> Suggested-by: Zhu Yanjun <zyjzyj2000@gmail.com>

I am fine with this commit.
Thanks a lot.

Zhu Yanjun

> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>  kconfigs/Kconfig.bringup.goals | 10 ++++++++++
>  scripts/rdma.Makefile          | 12 ++++++++++++
>  2 files changed, 22 insertions(+)
>
> diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals
> index e8629ebea5e0..794742530e9e 100644
> --- a/kconfigs/Kconfig.bringup.goals
> +++ b/kconfigs/Kconfig.bringup.goals
> @@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
>           To use this device driver, enable the CONFIG_RDMA_SIW
>           option in your test kernels.
>
> +config KDEVOPS_SETUP_RDMA_RXE
> +       bool "rxe"
> +       help
> +         Provision software-emulated RDMA on target nodes via the
> +         kernel rxe.ko driver. rxe can be used with any standard
> +         Ethernet device, including virtio-net devices.
> +
> +         To use this device driver, enable the CONFIG_RDMA_RXE
> +         option in your test kernels.
> +
>  endchoice
>
>  config KDEVOPS_SETUP_KTLS
> diff --git a/scripts/rdma.Makefile b/scripts/rdma.Makefile
> index d7cdf0062638..dd2a70120eb2 100644
> --- a/scripts/rdma.Makefile
> +++ b/scripts/rdma.Makefile
> @@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
>  PHONY += siw
>
>  endif
> +
> +ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
> +
> +rxe:
> +       $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
> +               -f 30 -i hosts playbooks/rxe.yml
> +
> +KDEVOPS_BRING_UP_DEPS += rxe
> +
> +PHONY += rxe
> +
> +endif
> --
> 2.45.1
>

Re: [PATCH 0/4] Add rxe as a second software RDMA option

[Chuck Lever III]

> On Jun 3, 2024, at 12:06 PM, cel@kernel.org wrote:
> 
> From: Chuck Lever <chuck.lever@oracle.com>
> 
> Zhu would like to have the ability to run file system tests with
> software RoCE. Add rxe as an alternative to siw.

Turns out Red Hat is also interested in testing NFS/RDMA
with RoCE. I've pushed these to linux-kdevops/kdevops.git.


> Chuck Lever (4):
>  bringup.goals: Make siw settings more generic
>  playbooks: Add a playbook to set up rxe on target nodes
>  Kconfig: Add support for testing RDMA with software RoCE
>  Update help text that mentions SIW
> 
> Makefile                                   |  2 +-
> kconfigs/Kconfig.bringup.goals             | 36 ++++++++++++++++++----
> playbooks/roles/rxe/tasks/main.yml         | 29 +++++++++++++++++
> playbooks/roles/rxe/templates/udev-rule.j2 |  2 ++
> playbooks/rxe.yml                          |  4 +++
> scripts/rdma.Makefile                      | 23 ++++++++++++++
> scripts/siw.Makefile                       | 11 -------
> workflows/fstests/nfs/Kconfig              |  6 ++--
> workflows/gitr/nfs/Kconfig                 |  3 +-
> 9 files changed, 93 insertions(+), 23 deletions(-)
> create mode 100644 playbooks/roles/rxe/tasks/main.yml
> create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2
> create mode 100644 playbooks/rxe.yml
> create mode 100644 scripts/rdma.Makefile
> delete mode 100644 scripts/siw.Makefile
> 
> -- 
> 2.45.1
> 

--
Chuck Lever

Re: [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE

[Zhu Yanjun]

On 03.06.24 18:06, cel@kernel.org wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
>
> Add kdevops config option to use rxe rather than siw for RDMA. rxe
> is the kernel's software RoCE driver, and like siw, it utilizes a
> standard Ethernet device.

Hi, Chuck

Thanks a lot. I am very glad that rxe is used in kdevops. I looked 
through this software.

It is very powerful. But because I have not an appropriate host at hand, 
I can not install

kdevops in the local hosts successfully.

After I can find a good host, I will install kdevops in the host and 
verify the functionality of rxe

of the new linux kernel release.

Chuck, thanks a lot for your efforts to bring rxe into kdevops (a 
powerful tool).

Zhu Yanjun

>
> Suggested-by: Zhu Yanjun <zyjzyj2000@gmail.com>
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>   kconfigs/Kconfig.bringup.goals | 10 ++++++++++
>   scripts/rdma.Makefile          | 12 ++++++++++++
>   2 files changed, 22 insertions(+)
>
> diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals
> index e8629ebea5e0..794742530e9e 100644
> --- a/kconfigs/Kconfig.bringup.goals
> +++ b/kconfigs/Kconfig.bringup.goals
> @@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
>   	  To use this device driver, enable the CONFIG_RDMA_SIW
>   	  option in your test kernels.
>   
> +config KDEVOPS_SETUP_RDMA_RXE
> +	bool "rxe"
> +	help
> +	  Provision software-emulated RDMA on target nodes via the
> +	  kernel rxe.ko driver. rxe can be used with any standard
> +	  Ethernet device, including virtio-net devices.
> +
> +	  To use this device driver, enable the CONFIG_RDMA_RXE
> +	  option in your test kernels.
> +
>   endchoice
>   
>   config KDEVOPS_SETUP_KTLS
> diff --git a/scripts/rdma.Makefile b/scripts/rdma.Makefile
> index d7cdf0062638..dd2a70120eb2 100644
> --- a/scripts/rdma.Makefile
> +++ b/scripts/rdma.Makefile
> @@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
>   PHONY += siw
>   
>   endif
> +
> +ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
> +
> +rxe:
> +	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
> +		-f 30 -i hosts playbooks/rxe.yml
> +
> +KDEVOPS_BRING_UP_DEPS += rxe
> +
> +PHONY += rxe
> +
> +endif

-- 
Best