* [PATCH 1/4] bringup.goals: Make siw settings more generic
2024-06-03 16:06 [PATCH 0/4] Add rxe as a second software RDMA option cel
@ 2024-06-03 16:06 ` cel
2024-06-03 16:06 ` [PATCH 2/4] playbooks: Add a playbook to set up rxe on target nodes cel
` (3 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: cel @ 2024-06-03 16:06 UTC (permalink / raw)
To: kdevops; +Cc: Chuck Lever
From: Chuck Lever <chuck.lever@oracle.com>
I'm about to introduce another option for software RDMA. Rename
the siw Makefile and Kconfig options to make way for this new
option.
Since only one type of emulated RDMA device can be attached to an
Ethernet device at a time, make the Kconfig logic choose between
no device, siw, or, when it's added, rxe.
NB: This patch replaces CONFIG_KDEVOPS_SETUP_SIW. If you have
enabled siw in your configuration, please visit the config setting
for software RDMA after applying this patch.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
Makefile | 2 +-
kconfigs/Kconfig.bringup.goals | 26 +++++++++++++++++++------
scripts/{siw.Makefile => rdma.Makefile} | 2 +-
3 files changed, 22 insertions(+), 8 deletions(-)
rename scripts/{siw.Makefile => rdma.Makefile} (79%)
diff --git a/Makefile b/Makefile
index 266eccd90111..20c2081118a9 100644
--- a/Makefile
+++ b/Makefile
@@ -104,7 +104,7 @@ ifeq (y,$(CONFIG_WORKFLOWS))
include workflows/Makefile
endif # CONFIG_WORKFLOWS
-include scripts/siw.Makefile
+include scripts/rdma.Makefile
include scripts/ktls.Makefile
include scripts/nfsd.Makefile
include scripts/smbd.Makefile
diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals
index e6d3b1815f3b..e8629ebea5e0 100644
--- a/kconfigs/Kconfig.bringup.goals
+++ b/kconfigs/Kconfig.bringup.goals
@@ -77,16 +77,30 @@ config KDEVOPS_TRY_INSTALL_KDEV_TOOLS
most kernel developers might prefer to have installed on target
systems.
-config KDEVOPS_SETUP_SIW
- bool "Configure soft iWARP on all test systems"
- default n
+choice
+ prompt "Configure an RDMA device"
+ default KDEVOPS_SETUP_RDMA_NONE
help
- Configure software-emulated RDMA on all target hosts via the
+ Provision an RDMA device on target nodes. The default is to
+ provision no additional RDMA devices.
+
+config KDEVOPS_SETUP_RDMA_NONE
+ bool "none"
+ help
+ Do not provision an additional RDMA device on target nodes.
+ This is the default behavior.
+
+config KDEVOPS_SETUP_RDMA_SIW
+ bool "siw"
+ help
+ Provision software-emulated RDMA on target nodes via the
kernel siw.ko driver. siw can be used with any standard
Ethernet device, including virtio-net devices.
- Because siw is currently not enabled in most Linux distro
- kernels, this option defaults to N.
+ To use this device driver, enable the CONFIG_RDMA_SIW
+ option in your test kernels.
+
+endchoice
config KDEVOPS_SETUP_KTLS
bool "Configure ktls on the hosts with self-signed CA"
diff --git a/scripts/siw.Makefile b/scripts/rdma.Makefile
similarity index 79%
rename from scripts/siw.Makefile
rename to scripts/rdma.Makefile
index d09d1a2f82ac..d7cdf0062638 100644
--- a/scripts/siw.Makefile
+++ b/scripts/rdma.Makefile
@@ -1,4 +1,4 @@
-ifeq (y,$(CONFIG_KDEVOPS_SETUP_SIW))
+ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_SIW))
siw:
$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
--
2.45.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 2/4] playbooks: Add a playbook to set up rxe on target nodes
2024-06-03 16:06 [PATCH 0/4] Add rxe as a second software RDMA option cel
2024-06-03 16:06 ` [PATCH 1/4] bringup.goals: Make siw settings more generic cel
@ 2024-06-03 16:06 ` cel
2024-06-03 16:06 ` [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE cel
` (2 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: cel @ 2024-06-03 16:06 UTC (permalink / raw)
To: kdevops; +Cc: Chuck Lever
From: Chuck Lever <chuck.lever@oracle.com>
The playbook adds a udev rule to provision an rxe device when the
node is booted.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
playbooks/roles/rxe/tasks/main.yml | 29 ++++++++++++++++++++++
playbooks/roles/rxe/templates/udev-rule.j2 | 2 ++
playbooks/rxe.yml | 4 +++
3 files changed, 35 insertions(+)
create mode 100644 playbooks/roles/rxe/tasks/main.yml
create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2
create mode 100644 playbooks/rxe.yml
diff --git a/playbooks/roles/rxe/tasks/main.yml b/playbooks/roles/rxe/tasks/main.yml
new file mode 100644
index 000000000000..88e655981f7f
--- /dev/null
+++ b/playbooks/roles/rxe/tasks/main.yml
@@ -0,0 +1,29 @@
+---
+- name: Include optional extra_vars
+ ansible.builtin.include_vars:
+ file: "{{ item }}"
+ with_first_found:
+ - files:
+ - "../extra_vars.yml"
+ - "../extra_vars.yaml"
+ - "../extra_vars.json"
+ skip: true
+ failed_when: false
+ tags: vars
+
+- name: Insert a udev rule to create an rxe device
+ become: true
+ become_method: ansible.builtin.sudo
+ ansible.builtin.template:
+ src: "udev-rule.j2"
+ dest: "/usr/lib/udev/rules.d/99-rxe.rules"
+ owner: "root"
+ group: "root"
+ mode: "0644"
+ force: false
+
+- name: Reload the udev ruleset
+ become: true
+ become_method: ansible.builtin.sudo
+ ansible.builtin.shell: "udevadm control --reload && udevadm trigger"
+ changed_when: true
diff --git a/playbooks/roles/rxe/templates/udev-rule.j2 b/playbooks/roles/rxe/templates/udev-rule.j2
new file mode 100644
index 000000000000..9f5b5e4a68f3
--- /dev/null
+++ b/playbooks/roles/rxe/templates/udev-rule.j2
@@ -0,0 +1,2 @@
+SUBSYSTEM=="net", KERNEL=="{{ ansible_default_ipv4.interface }}", \
+ PROGRAM="/usr/sbin/rdma link add rxe0 type rxe netdev {{ ansible_default_ipv4.interface }}"
diff --git a/playbooks/rxe.yml b/playbooks/rxe.yml
new file mode 100644
index 000000000000..dc51c0347628
--- /dev/null
+++ b/playbooks/rxe.yml
@@ -0,0 +1,4 @@
+---
+- hosts: all
+ roles:
+ - role: rxe
--
2.45.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE
2024-06-03 16:06 [PATCH 0/4] Add rxe as a second software RDMA option cel
2024-06-03 16:06 ` [PATCH 1/4] bringup.goals: Make siw settings more generic cel
2024-06-03 16:06 ` [PATCH 2/4] playbooks: Add a playbook to set up rxe on target nodes cel
@ 2024-06-03 16:06 ` cel
2024-06-03 16:54 ` Zhu Yanjun
2024-06-07 13:10 ` Zhu Yanjun
2024-06-03 16:06 ` [PATCH 4/4] Update help text that mentions SIW cel
2024-06-05 17:50 ` [PATCH 0/4] Add rxe as a second software RDMA option Chuck Lever III
4 siblings, 2 replies; 8+ messages in thread
From: cel @ 2024-06-03 16:06 UTC (permalink / raw)
To: kdevops; +Cc: Chuck Lever, Zhu Yanjun
From: Chuck Lever <chuck.lever@oracle.com>
Add kdevops config option to use rxe rather than siw for RDMA. rxe
is the kernel's software RoCE driver, and like siw, it utilizes a
standard Ethernet device.
Suggested-by: Zhu Yanjun <zyjzyj2000@gmail.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
kconfigs/Kconfig.bringup.goals | 10 ++++++++++
scripts/rdma.Makefile | 12 ++++++++++++
2 files changed, 22 insertions(+)
diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals
index e8629ebea5e0..794742530e9e 100644
--- a/kconfigs/Kconfig.bringup.goals
+++ b/kconfigs/Kconfig.bringup.goals
@@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
To use this device driver, enable the CONFIG_RDMA_SIW
option in your test kernels.
+config KDEVOPS_SETUP_RDMA_RXE
+ bool "rxe"
+ help
+ Provision software-emulated RDMA on target nodes via the
+ kernel rxe.ko driver. rxe can be used with any standard
+ Ethernet device, including virtio-net devices.
+
+ To use this device driver, enable the CONFIG_RDMA_RXE
+ option in your test kernels.
+
endchoice
config KDEVOPS_SETUP_KTLS
diff --git a/scripts/rdma.Makefile b/scripts/rdma.Makefile
index d7cdf0062638..dd2a70120eb2 100644
--- a/scripts/rdma.Makefile
+++ b/scripts/rdma.Makefile
@@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
PHONY += siw
endif
+
+ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
+
+rxe:
+ $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
+ -f 30 -i hosts playbooks/rxe.yml
+
+KDEVOPS_BRING_UP_DEPS += rxe
+
+PHONY += rxe
+
+endif
--
2.45.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE
2024-06-03 16:06 ` [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE cel
@ 2024-06-03 16:54 ` Zhu Yanjun
2024-06-07 13:10 ` Zhu Yanjun
1 sibling, 0 replies; 8+ messages in thread
From: Zhu Yanjun @ 2024-06-03 16:54 UTC (permalink / raw)
To: cel; +Cc: kdevops, Chuck Lever
On Mon, Jun 3, 2024 at 6:07 PM <cel@kernel.org> wrote:
>
> From: Chuck Lever <chuck.lever@oracle.com>
>
> Add kdevops config option to use rxe rather than siw for RDMA. rxe
> is the kernel's software RoCE driver, and like siw, it utilizes a
> standard Ethernet device.
>
> Suggested-by: Zhu Yanjun <zyjzyj2000@gmail.com>
I am fine with this commit.
Thanks a lot.
Zhu Yanjun
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
> kconfigs/Kconfig.bringup.goals | 10 ++++++++++
> scripts/rdma.Makefile | 12 ++++++++++++
> 2 files changed, 22 insertions(+)
>
> diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals
> index e8629ebea5e0..794742530e9e 100644
> --- a/kconfigs/Kconfig.bringup.goals
> +++ b/kconfigs/Kconfig.bringup.goals
> @@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
> To use this device driver, enable the CONFIG_RDMA_SIW
> option in your test kernels.
>
> +config KDEVOPS_SETUP_RDMA_RXE
> + bool "rxe"
> + help
> + Provision software-emulated RDMA on target nodes via the
> + kernel rxe.ko driver. rxe can be used with any standard
> + Ethernet device, including virtio-net devices.
> +
> + To use this device driver, enable the CONFIG_RDMA_RXE
> + option in your test kernels.
> +
> endchoice
>
> config KDEVOPS_SETUP_KTLS
> diff --git a/scripts/rdma.Makefile b/scripts/rdma.Makefile
> index d7cdf0062638..dd2a70120eb2 100644
> --- a/scripts/rdma.Makefile
> +++ b/scripts/rdma.Makefile
> @@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
> PHONY += siw
>
> endif
> +
> +ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
> +
> +rxe:
> + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
> + -f 30 -i hosts playbooks/rxe.yml
> +
> +KDEVOPS_BRING_UP_DEPS += rxe
> +
> +PHONY += rxe
> +
> +endif
> --
> 2.45.1
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE
2024-06-03 16:06 ` [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE cel
2024-06-03 16:54 ` Zhu Yanjun
@ 2024-06-07 13:10 ` Zhu Yanjun
1 sibling, 0 replies; 8+ messages in thread
From: Zhu Yanjun @ 2024-06-07 13:10 UTC (permalink / raw)
To: cel, kdevops; +Cc: Chuck Lever
On 03.06.24 18:06, cel@kernel.org wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
>
> Add kdevops config option to use rxe rather than siw for RDMA. rxe
> is the kernel's software RoCE driver, and like siw, it utilizes a
> standard Ethernet device.
Hi, Chuck
Thanks a lot. I am very glad that rxe is used in kdevops. I looked
through this software.
It is very powerful. But because I have not an appropriate host at hand,
I can not install
kdevops in the local hosts successfully.
After I can find a good host, I will install kdevops in the host and
verify the functionality of rxe
of the new linux kernel release.
Chuck, thanks a lot for your efforts to bring rxe into kdevops (a
powerful tool).
Zhu Yanjun
>
> Suggested-by: Zhu Yanjun <zyjzyj2000@gmail.com>
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
> kconfigs/Kconfig.bringup.goals | 10 ++++++++++
> scripts/rdma.Makefile | 12 ++++++++++++
> 2 files changed, 22 insertions(+)
>
> diff --git a/kconfigs/Kconfig.bringup.goals b/kconfigs/Kconfig.bringup.goals
> index e8629ebea5e0..794742530e9e 100644
> --- a/kconfigs/Kconfig.bringup.goals
> +++ b/kconfigs/Kconfig.bringup.goals
> @@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
> To use this device driver, enable the CONFIG_RDMA_SIW
> option in your test kernels.
>
> +config KDEVOPS_SETUP_RDMA_RXE
> + bool "rxe"
> + help
> + Provision software-emulated RDMA on target nodes via the
> + kernel rxe.ko driver. rxe can be used with any standard
> + Ethernet device, including virtio-net devices.
> +
> + To use this device driver, enable the CONFIG_RDMA_RXE
> + option in your test kernels.
> +
> endchoice
>
> config KDEVOPS_SETUP_KTLS
> diff --git a/scripts/rdma.Makefile b/scripts/rdma.Makefile
> index d7cdf0062638..dd2a70120eb2 100644
> --- a/scripts/rdma.Makefile
> +++ b/scripts/rdma.Makefile
> @@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
> PHONY += siw
>
> endif
> +
> +ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
> +
> +rxe:
> + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
> + -f 30 -i hosts playbooks/rxe.yml
> +
> +KDEVOPS_BRING_UP_DEPS += rxe
> +
> +PHONY += rxe
> +
> +endif
--
Best
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 4/4] Update help text that mentions SIW
2024-06-03 16:06 [PATCH 0/4] Add rxe as a second software RDMA option cel
` (2 preceding siblings ...)
2024-06-03 16:06 ` [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE cel
@ 2024-06-03 16:06 ` cel
2024-06-05 17:50 ` [PATCH 0/4] Add rxe as a second software RDMA option Chuck Lever III
4 siblings, 0 replies; 8+ messages in thread
From: cel @ 2024-06-03 16:06 UTC (permalink / raw)
To: kdevops; +Cc: Chuck Lever
From: Chuck Lever <chuck.lever@oracle.com>
Generalize the help text, now that kdevops has two software RDMA
options.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
workflows/fstests/nfs/Kconfig | 6 +++---
workflows/gitr/nfs/Kconfig | 3 +--
2 files changed, 4 insertions(+), 5 deletions(-)
diff --git a/workflows/fstests/nfs/Kconfig b/workflows/fstests/nfs/Kconfig
index 9f47761712b2..926bc40b4559 100644
--- a/workflows/fstests/nfs/Kconfig
+++ b/workflows/fstests/nfs/Kconfig
@@ -50,9 +50,9 @@ config FSTESTS_NFS_SECTION_RDMA
bool "Enable testing section: nfs_rdma"
default n
help
- Enabling this will test with the "rdma" mount option. Unless the
- hosts have RDMA hardware already, you probably also want to enable
- CONFIG_KDEVOPS_SETUP_SIW as well.
+ Enabling this will test with the "rdma" mount option. Unless
+ the target nodes have RDMA hardware already, enable one of
+ the software RDMA options as well.
config FSTESTS_NFS_SECTION_TLS
bool "Enable testing section: nfs_tls"
diff --git a/workflows/gitr/nfs/Kconfig b/workflows/gitr/nfs/Kconfig
index a8c81b098957..1b1e53316ac7 100644
--- a/workflows/gitr/nfs/Kconfig
+++ b/workflows/gitr/nfs/Kconfig
@@ -38,8 +38,7 @@ config GITR_NFS_SECTION_RDMA
help
Provision a target node to run the gitr workflow on a mount
using NFS/RDMA. Unless your target nodes have RDMA hardware
- already, you are likely to need to enable the
- CONFIG_KDEVOPS_SETUP_SIW option as well.
+ already, enable one of the software RDMA options as well.
config GITR_NFS_SECTION_V42
bool "Enable testing section: nfs_v42"
--
2.45.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH 0/4] Add rxe as a second software RDMA option
2024-06-03 16:06 [PATCH 0/4] Add rxe as a second software RDMA option cel
` (3 preceding siblings ...)
2024-06-03 16:06 ` [PATCH 4/4] Update help text that mentions SIW cel
@ 2024-06-05 17:50 ` Chuck Lever III
4 siblings, 0 replies; 8+ messages in thread
From: Chuck Lever III @ 2024-06-05 17:50 UTC (permalink / raw)
To: Chuck Lever; +Cc: kdevops@lists.linux.dev, Zhu Yanjun
> On Jun 3, 2024, at 12:06 PM, cel@kernel.org wrote:
>
> From: Chuck Lever <chuck.lever@oracle.com>
>
> Zhu would like to have the ability to run file system tests with
> software RoCE. Add rxe as an alternative to siw.
Turns out Red Hat is also interested in testing NFS/RDMA
with RoCE. I've pushed these to linux-kdevops/kdevops.git.
> Chuck Lever (4):
> bringup.goals: Make siw settings more generic
> playbooks: Add a playbook to set up rxe on target nodes
> Kconfig: Add support for testing RDMA with software RoCE
> Update help text that mentions SIW
>
> Makefile | 2 +-
> kconfigs/Kconfig.bringup.goals | 36 ++++++++++++++++++----
> playbooks/roles/rxe/tasks/main.yml | 29 +++++++++++++++++
> playbooks/roles/rxe/templates/udev-rule.j2 | 2 ++
> playbooks/rxe.yml | 4 +++
> scripts/rdma.Makefile | 23 ++++++++++++++
> scripts/siw.Makefile | 11 -------
> workflows/fstests/nfs/Kconfig | 6 ++--
> workflows/gitr/nfs/Kconfig | 3 +-
> 9 files changed, 93 insertions(+), 23 deletions(-)
> create mode 100644 playbooks/roles/rxe/tasks/main.yml
> create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2
> create mode 100644 playbooks/rxe.yml
> create mode 100644 scripts/rdma.Makefile
> delete mode 100644 scripts/siw.Makefile
>
> --
> 2.45.1
>
--
Chuck Lever
^ permalink raw reply [flat|nested] 8+ messages in thread