* [PATCH v2 0/7] Add a git regression workflow
@ 2024-01-03 15:08 Chuck Lever
2024-01-03 15:08 ` [PATCH v2 1/7] nfsd_add_export: Add an export_options variable Chuck Lever
` (7 more replies)
0 siblings, 8 replies; 23+ messages in thread
From: Chuck Lever @ 2024-01-03 15:08 UTC (permalink / raw)
To: kdevops
Since the last series, I've added support for tmpfs and NFS mounts,
in addition to a few other refinements and loads of testing. I'd
like a little review before I push these.
We had talked before about setting up multiple concurrent mounts for
running tests of different NFS versions in parallel. +1 for running
tests in parallel, but I think we want to make this a more generic
mechanism that can test other mount option combinations as well
multiple NFS versions (and perhaps could be reused with other
workflows).
I don't have a specific proposal for that yet, but I'm interested in
discussion.
---
Chuck Lever (7):
nfsd_add_export: Add an export_options variable
nfsd_add_export: Set fsid= on tmpfs exports
create_tmpfs: Enable mounting tmpfs file systems for testing
create_nfs_mount: Enable testing arbitrary NFS mounts
Add a playbook for the git regression workflow
Add a workflow for the git regression suite
gitr: Adjust the gen_hosts and gen_nodes playbooks
.gitignore | 2 +
README.md | 12 +-
docs/gitr.md | 22 ++
kconfigs/workflows/Kconfig | 25 ++
playbooks/gitr.yml | 4 +
playbooks/roles/create_nfs_mount/README.md | 51 +++++
.../roles/create_nfs_mount/defaults/main.yml | 6 +
.../tasks/install-deps/debian/main.yml | 10 +
.../tasks/install-deps/main.yml | 12 +
.../tasks/install-deps/redhat/main.yml | 14 ++
.../tasks/install-deps/suse/main.yml | 8 +
.../roles/create_nfs_mount/tasks/main.yml | 47 ++++
playbooks/roles/create_tmpfs/README.md | 50 ++++
.../roles/create_tmpfs/defaults/main.yml | 6 +
playbooks/roles/create_tmpfs/tasks/main.yml | 54 +++++
playbooks/roles/fstests/tasks/main.yml | 2 +
playbooks/roles/gen_hosts/defaults/main.yml | 1 +
playbooks/roles/gen_hosts/tasks/main.yml | 13 ++
playbooks/roles/gen_hosts/templates/gitr.j2 | 23 ++
playbooks/roles/gen_nodes/defaults/main.yml | 1 +
playbooks/roles/gen_nodes/tasks/main.yml | 15 ++
playbooks/roles/gitr/defaults/main.yml | 9 +
.../gitr/tasks/install-deps/debian/main.yml | 28 +++
.../roles/gitr/tasks/install-deps/main.yml | 12 +
.../gitr/tasks/install-deps/redhat/main.yml | 38 ++++
.../gitr/tasks/install-deps/suse/main.yml | 24 ++
playbooks/roles/gitr/tasks/main.yml | 214 ++++++++++++++++++
playbooks/roles/nfsd_add_export/README.md | 1 +
.../roles/nfsd_add_export/tasks/main.yml | 24 ++
.../nfsd_add_export/templates/exports.j2 | 2 +-
playbooks/roles/pynfs/tasks/main.yml | 2 +
scripts/workflows/gitr/run_kernel_ci.sh | 1 +
scripts/workflows/gitr/run_kernel_ci_kotd.sh | 1 +
scripts/workflows/gitr/run_loop.sh | 58 +++++
workflows/Makefile | 5 +
workflows/gitr/Kconfig | 82 +++++++
workflows/gitr/Makefile | 82 +++++++
workflows/gitr/btrfs/Kconfig | 20 ++
workflows/gitr/btrfs/Makefile | 5 +
workflows/gitr/ext4/Kconfig | 20 ++
workflows/gitr/ext4/Makefile | 5 +
workflows/gitr/nfs/Kconfig | 31 +++
workflows/gitr/nfs/Makefile | 14 ++
workflows/gitr/tmpfs/Kconfig | 5 +
workflows/gitr/tmpfs/Makefile | 4 +
workflows/gitr/xfs/Kconfig | 20 ++
workflows/gitr/xfs/Makefile | 6 +
47 files changed, 1089 insertions(+), 2 deletions(-)
create mode 100644 docs/gitr.md
create mode 100644 playbooks/gitr.yml
create mode 100644 playbooks/roles/create_nfs_mount/README.md
create mode 100644 playbooks/roles/create_nfs_mount/defaults/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/install-deps/debian/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/install-deps/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/install-deps/redhat/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/install-deps/suse/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/main.yml
create mode 100644 playbooks/roles/create_tmpfs/README.md
create mode 100644 playbooks/roles/create_tmpfs/defaults/main.yml
create mode 100644 playbooks/roles/create_tmpfs/tasks/main.yml
create mode 100644 playbooks/roles/gen_hosts/templates/gitr.j2
create mode 100644 playbooks/roles/gitr/defaults/main.yml
create mode 100644 playbooks/roles/gitr/tasks/install-deps/debian/main.yml
create mode 100644 playbooks/roles/gitr/tasks/install-deps/main.yml
create mode 100644 playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
create mode 100644 playbooks/roles/gitr/tasks/install-deps/suse/main.yml
create mode 100644 playbooks/roles/gitr/tasks/main.yml
create mode 120000 scripts/workflows/gitr/run_kernel_ci.sh
create mode 120000 scripts/workflows/gitr/run_kernel_ci_kotd.sh
create mode 100755 scripts/workflows/gitr/run_loop.sh
create mode 100644 workflows/gitr/Kconfig
create mode 100644 workflows/gitr/Makefile
create mode 100644 workflows/gitr/btrfs/Kconfig
create mode 100644 workflows/gitr/btrfs/Makefile
create mode 100644 workflows/gitr/ext4/Kconfig
create mode 100644 workflows/gitr/ext4/Makefile
create mode 100644 workflows/gitr/nfs/Kconfig
create mode 100644 workflows/gitr/nfs/Makefile
create mode 100644 workflows/gitr/tmpfs/Kconfig
create mode 100644 workflows/gitr/tmpfs/Makefile
create mode 100644 workflows/gitr/xfs/Kconfig
create mode 100644 workflows/gitr/xfs/Makefile
--
Chuck Lever
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH v2 1/7] nfsd_add_export: Add an export_options variable
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
@ 2024-01-03 15:08 ` Chuck Lever
2024-01-03 15:08 ` [PATCH v2 2/7] nfsd_add_export: Set fsid= on tmpfs exports Chuck Lever
` (6 subsequent siblings)
7 siblings, 0 replies; 23+ messages in thread
From: Chuck Lever @ 2024-01-03 15:08 UTC (permalink / raw)
To: kdevops
From: Chuck Lever <chuck.lever@oracle.com>
Enable consumers of the nfsd_add_export role to set unique export
options for each created export. The default set of options,
nfsd_export_options, is used if the consumer does not set this
variable.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
playbooks/roles/fstests/tasks/main.yml | 2 ++
playbooks/roles/nfsd_add_export/README.md | 1 +
.../roles/nfsd_add_export/templates/exports.j2 | 2 +-
playbooks/roles/pynfs/tasks/main.yml | 2 ++
4 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/playbooks/roles/fstests/tasks/main.yml b/playbooks/roles/fstests/tasks/main.yml
index a33844aafd15..78b2eafbc756 100644
--- a/playbooks/roles/fstests/tasks/main.yml
+++ b/playbooks/roles/fstests/tasks/main.yml
@@ -633,6 +633,7 @@
vars:
server_host: "{{ fstests_nfs_server_host }}"
export_volname: "{{ volname_prefix }}-t"
+ export_options: "{{ nfsd_export_options }}"
export_fstype: "{{ nfsd_export_fstype }}"
export_size: 20g
when:
@@ -645,6 +646,7 @@
vars:
server_host: "{{ fstests_nfs_server_host }}"
export_volname: "{{ volname_prefix }}-s"
+ export_options: "{{ nfsd_export_options }}"
export_fstype: "{{ nfsd_export_fstype }}"
export_size: 20g
when:
diff --git a/playbooks/roles/nfsd_add_export/README.md b/playbooks/roles/nfsd_add_export/README.md
index 94ee7c8e82af..8e9e21e02e82 100644
--- a/playbooks/roles/nfsd_add_export/README.md
+++ b/playbooks/roles/nfsd_add_export/README.md
@@ -19,6 +19,7 @@ Role Variables
the new export is to be created
* export_volname: the name of the new export, to be created under
the /exports directory on the NFS server
+ * export_options: the export options for the new export
* export_fstype: the file system type of the new export
* export_size: the maximum size of the new export
* export_user: the owner of the new export
diff --git a/playbooks/roles/nfsd_add_export/templates/exports.j2 b/playbooks/roles/nfsd_add_export/templates/exports.j2
index 3005f4ba62c3..40a6af8beb4e 100644
--- a/playbooks/roles/nfsd_add_export/templates/exports.j2
+++ b/playbooks/roles/nfsd_add_export/templates/exports.j2
@@ -1 +1 @@
-{{ nfsd_export_path }}/{{ export_volname }} *({{ nfsd_export_options }})
+{{ nfsd_export_path }}/{{ export_volname }} *({{ export_options }})
diff --git a/playbooks/roles/pynfs/tasks/main.yml b/playbooks/roles/pynfs/tasks/main.yml
index 096b08461572..749306a271e6 100644
--- a/playbooks/roles/pynfs/tasks/main.yml
+++ b/playbooks/roles/pynfs/tasks/main.yml
@@ -72,6 +72,7 @@
vars:
server_host: "{{ nfsd_hostname }}"
export_volname: "{{ volname_base }}-4.0"
+ export_options: "{{ nfsd_export_options }}"
export_fstype: "{{ nfsd_export_fstype }}"
export_size: 10g
@@ -82,6 +83,7 @@
vars:
server_host: "{{ nfsd_hostname }}"
export_volname: "{{ volname_base }}-4.1"
+ export_options: "{{ nfsd_export_options }}"
export_fstype: "{{ nfsd_export_fstype }}"
export_size: 10g
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 2/7] nfsd_add_export: Set fsid= on tmpfs exports
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
2024-01-03 15:08 ` [PATCH v2 1/7] nfsd_add_export: Add an export_options variable Chuck Lever
@ 2024-01-03 15:08 ` Chuck Lever
2024-01-03 15:08 ` [PATCH v2 3/7] create_tmpfs: Enable mounting tmpfs file systems for testing Chuck Lever
` (5 subsequent siblings)
7 siblings, 0 replies; 23+ messages in thread
From: Chuck Lever @ 2024-01-03 15:08 UTC (permalink / raw)
To: kdevops
From: Chuck Lever <chuck.lever@oracle.com>
Local file systems backed by durable storage generally get a unique
FSID when they are created. With tmpfs, there is no durable storage
so there is no FSID unless one is provided in the export options.
It's a best practice to use fsid=`uuidgen` for this purpose.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
playbooks/roles/nfsd_add_export/tasks/main.yml | 24 ++++++++++++++++++++
.../roles/nfsd_add_export/templates/exports.j2 | 2 +-
2 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/playbooks/roles/nfsd_add_export/tasks/main.yml b/playbooks/roles/nfsd_add_export/tasks/main.yml
index 2ab289c629db..ec456dda622a 100644
--- a/playbooks/roles/nfsd_add_export/tasks/main.yml
+++ b/playbooks/roles/nfsd_add_export/tasks/main.yml
@@ -88,6 +88,30 @@
group: "root"
mode: '644'
+- name: Copy export options to a template variable
+ set_fact:
+ template_export_options: "{{ export_options }}"
+
+- name: Check for an fsid= export option
+ set_fact:
+ fsid_is_present: '{{ "fsid=" in export_options }}'
+ when:
+ - export_fstype == "tmpfs"
+
+- name: Generate a unique fsid
+ command: "uuidgen -r"
+ register: export_fsid
+ when:
+ - export_fstype == "tmpfs"
+ - not fsid_is_present|bool
+
+- name: Add an fsid= export option
+ set_fact:
+ template_export_options: "{{ export_options }},fsid={{ export_fsid.stdout }}"
+ when:
+ - export_fstype == "tmpfs"
+ - not fsid_is_present|bool
+
- name: Create an exports.d file for the new fs
become: yes
become_flags: 'su - -c'
diff --git a/playbooks/roles/nfsd_add_export/templates/exports.j2 b/playbooks/roles/nfsd_add_export/templates/exports.j2
index 40a6af8beb4e..a0ed3284b162 100644
--- a/playbooks/roles/nfsd_add_export/templates/exports.j2
+++ b/playbooks/roles/nfsd_add_export/templates/exports.j2
@@ -1 +1 @@
-{{ nfsd_export_path }}/{{ export_volname }} *({{ export_options }})
+{{ nfsd_export_path }}/{{ export_volname }} *({{ template_export_options }})
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 3/7] create_tmpfs: Enable mounting tmpfs file systems for testing
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
2024-01-03 15:08 ` [PATCH v2 1/7] nfsd_add_export: Add an export_options variable Chuck Lever
2024-01-03 15:08 ` [PATCH v2 2/7] nfsd_add_export: Set fsid= on tmpfs exports Chuck Lever
@ 2024-01-03 15:08 ` Chuck Lever
2024-01-25 14:26 ` Luis Chamberlain
2024-01-03 15:08 ` [PATCH v2 4/7] create_nfs_mount: Enable testing arbitrary NFS mounts Chuck Lever
` (4 subsequent siblings)
7 siblings, 1 reply; 23+ messages in thread
From: Chuck Lever @ 2024-01-03 15:08 UTC (permalink / raw)
To: kdevops
From: Chuck Lever <chuck.lever@oracle.com>
To enable tmpfs as a generic test target, add a playbook for
mounting a tmpfs file system. The new playbook is similar to
the create_partition playbook: it creates a persistent fstab
entry, but does not need to reserve a block device for the
new file system.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
playbooks/roles/create_tmpfs/README.md | 50 ++++++++++++++++++++++
playbooks/roles/create_tmpfs/defaults/main.yml | 6 +++
playbooks/roles/create_tmpfs/tasks/main.yml | 54 ++++++++++++++++++++++++
3 files changed, 110 insertions(+)
create mode 100644 playbooks/roles/create_tmpfs/README.md
create mode 100644 playbooks/roles/create_tmpfs/defaults/main.yml
create mode 100644 playbooks/roles/create_tmpfs/tasks/main.yml
diff --git a/playbooks/roles/create_tmpfs/README.md b/playbooks/roles/create_tmpfs/README.md
new file mode 100644
index 000000000000..fe8736229788
--- /dev/null
+++ b/playbooks/roles/create_tmpfs/README.md
@@ -0,0 +1,50 @@
+create_tmpfs
+================
+
+The create_tmpfs role mounts a tmpfs file system.
+
+Requirements
+------------
+
+None.
+
+Role Variables
+--------------
+
+ * tmpfs_mount_options: extra mount options to use for /etc/fstab.
+ This variable should never be empty. If you want to use the
+ default mount options, do not override the defaults, which
+ is "defaults"
+ * tmpfs_mounted_on: the directory on which to mount the new file system
+ * tmpfs_user: the user to assign the directory path to
+ * tmpfs_group: the group to assign the directory path to
+
+Dependencies
+------------
+
+None.
+
+Example Playbook
+----------------
+
+Below is an example playbook task:
+
+```
+- name: Create /test/tmpfs if needed
+ include_role:
+ name: create_tmpfs
+ vars:
+ tmpfs_mount_options: "size=75%"
+ tmpfs_mounted_on: "/test/tmpfs"
+ tmpfs_user: "vagrant"
+ tmpfs_group: "vagrant"
+ tags: [ 'oscheck', 'test_tmpfs_enable' ]
+```
+
+For further examples refer to one of this role's users, the
+[https://github.com/linux-kdevops/kdevops](kdevops) project.
+
+License
+-------
+
+copyleft-next-0.3.1
diff --git a/playbooks/roles/create_tmpfs/defaults/main.yml b/playbooks/roles/create_tmpfs/defaults/main.yml
new file mode 100644
index 000000000000..cb04cf4ac154
--- /dev/null
+++ b/playbooks/roles/create_tmpfs/defaults/main.yml
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier copyleft-next-0.3.1
+---
+tmpfs_mount_options: "defaults"
+tmpfs_mounted_on: "/data"
+tmpfs_user: "root"
+tmpfs_group: "root"
diff --git a/playbooks/roles/create_tmpfs/tasks/main.yml b/playbooks/roles/create_tmpfs/tasks/main.yml
new file mode 100644
index 000000000000..cf96d0b1304d
--- /dev/null
+++ b/playbooks/roles/create_tmpfs/tasks/main.yml
@@ -0,0 +1,54 @@
+---
+- name: Import optional extra_args file
+ include_vars: "{{ item }}"
+ ignore_errors: yes
+ with_first_found:
+ - files:
+ - "../extra_vars.yml"
+ - "../extra_vars.yaml"
+ - "../extra_vars.json"
+ skip: true
+ tags: vars
+
+- name: Check if {{ tmpfs_mounted_on }} is mounted
+ become: yes
+ become_method: sudo
+ ansible.builtin.command: mountpoint -q {{ tmpfs_mounted_on }}
+ register: mountpoint_stat
+ failed_when: False
+ changed_when: False
+
+- name: Check if {{ tmpfs_mounted_on }} is present in /etc/fstab
+ become: yes
+ become_method: sudo
+ ansible.builtin.shell: |
+ set -o pipefail
+ awk '{ print $2 }' /etc/fstab | grep -q ^"{{ tmpfs_mounted_on }}"
+ register: fstab_grep_stat
+ failed_when: False
+ changed_when: False
+ args:
+ executable: /bin/bash
+
+- name: Ensure {{ tmpfs_mounted_on }} is mounted
+ become: yes
+ become_flags: 'su - -c'
+ become_method: sudo
+ throttle: 1
+ ansible.builtin.mount:
+ path: "{{ tmpfs_mounted_on }}"
+ src: "tmpfs"
+ fstype: "tmpfs"
+ opts: "{{ tmpfs_mount_options }}"
+ state: mounted
+ when:
+ mountpoint_stat != 0
+
+- name: Ensure {{ tmpfs_mounted_on }} has the correct permissions
+ become: yes
+ become_method: sudo
+ ansible.builtin.file:
+ state: directory
+ path: "{{ tmpfs_mounted_on }}"
+ owner: "{{ tmpfs_user }}"
+ group: "{{ tmpfs_group }}"
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 4/7] create_nfs_mount: Enable testing arbitrary NFS mounts
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
` (2 preceding siblings ...)
2024-01-03 15:08 ` [PATCH v2 3/7] create_tmpfs: Enable mounting tmpfs file systems for testing Chuck Lever
@ 2024-01-03 15:08 ` Chuck Lever
2024-01-03 15:08 ` [PATCH v2 5/7] Add a playbook for the git regression workflow Chuck Lever
` (3 subsequent siblings)
7 siblings, 0 replies; 23+ messages in thread
From: Chuck Lever @ 2024-01-03 15:08 UTC (permalink / raw)
To: kdevops
From: Chuck Lever <chuck.lever@oracle.com>
To enable an NFS mount as a generic test target, add a playbook for
mounting an NFS server.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
playbooks/roles/create_nfs_mount/README.md | 51 ++++++++++++++++++++
playbooks/roles/create_nfs_mount/defaults/main.yml | 6 ++
.../tasks/install-deps/debian/main.yml | 10 ++++
.../create_nfs_mount/tasks/install-deps/main.yml | 12 +++++
.../tasks/install-deps/redhat/main.yml | 14 +++++
.../tasks/install-deps/suse/main.yml | 8 +++
playbooks/roles/create_nfs_mount/tasks/main.yml | 47 ++++++++++++++++++
7 files changed, 148 insertions(+)
create mode 100644 playbooks/roles/create_nfs_mount/README.md
create mode 100644 playbooks/roles/create_nfs_mount/defaults/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/install-deps/debian/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/install-deps/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/install-deps/redhat/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/install-deps/suse/main.yml
create mode 100644 playbooks/roles/create_nfs_mount/tasks/main.yml
diff --git a/playbooks/roles/create_nfs_mount/README.md b/playbooks/roles/create_nfs_mount/README.md
new file mode 100644
index 000000000000..8a719a8ae0e0
--- /dev/null
+++ b/playbooks/roles/create_nfs_mount/README.md
@@ -0,0 +1,51 @@
+create_nfs_mount
+================
+
+The create_nfs_mount role mounts an NFS export.
+
+Requirements
+------------
+
+None. This role installs any necessary NFS client administrative
+tools automatically.
+
+Role Variables
+--------------
+
+ * nfs_mount_options: extra mount options to use for /etc/fstab.
+ This variable should never be empty. If you want to use the
+ default mount options, do not override the defaults, which
+ is "defaults"
+ * nfs_mounted_on: the directory on which to mount the new file system
+ * nfs_server_hostname: the NFS server's hostname or IP address
+ * nfs_server_export: the pathname of the export to be mounted
+
+Dependencies
+------------
+
+None.
+
+Example Playbook
+----------------
+
+Below is an example playbook task:
+
+```
+- name: Create /test/nfs if needed
+ include_role:
+ name: create_nfs_mount
+ vars:
+ nfs_mounted_on: "/test/nfs"
+ nfs_server_hostname: "nfs.example.com"
+ nfs_server_export: "/export/example"
+ nfs_mount_options: "sec=sys,vers=4.1"
+ tags: [ 'test_nfs_enable' ]
+```
+
+For further examples refer to one of this role's users, the
+[https://github.com/linux-kdevops/kdevops](kdevops) project.
+
+License
+-------
+
+copyleft-next-0.3.1
diff --git a/playbooks/roles/create_nfs_mount/defaults/main.yml b/playbooks/roles/create_nfs_mount/defaults/main.yml
new file mode 100644
index 000000000000..7e15b93abe8c
--- /dev/null
+++ b/playbooks/roles/create_nfs_mount/defaults/main.yml
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier copyleft-next-0.3.1
+---
+nfs_mount_options: "defaults"
+nfs_mounted_on: "/data"
+nfs_server_hostname: "kdevops-nfsd"
+nfs_server_export: "/export"
diff --git a/playbooks/roles/create_nfs_mount/tasks/install-deps/debian/main.yml b/playbooks/roles/create_nfs_mount/tasks/install-deps/debian/main.yml
new file mode 100644
index 000000000000..d05919ac4606
--- /dev/null
+++ b/playbooks/roles/create_nfs_mount/tasks/install-deps/debian/main.yml
@@ -0,0 +1,10 @@
+---
+- name: Install NFS client administrative tools
+ become: yes
+ become_method: sudo
+ apt:
+ name:
+ - nfs-common
+ state: present
+ update_cache: yes
+ tags: [ 'deps' ]
diff --git a/playbooks/roles/create_nfs_mount/tasks/install-deps/main.yml b/playbooks/roles/create_nfs_mount/tasks/install-deps/main.yml
new file mode 100644
index 000000000000..801ffb44fa00
--- /dev/null
+++ b/playbooks/roles/create_nfs_mount/tasks/install-deps/main.yml
@@ -0,0 +1,12 @@
+---
+- name: Debian-specific setup
+ ansible.builtin.include_tasks: roles/create_nfs_mount/tasks/install-deps/debian/main.yml
+ when: ansible_os_family == 'Debian'
+
+- name: SuSE-specific setup
+ ansible.builtin.include_tasks: roles/create_nfs_mount/tasks/install-deps/suse/main.yml
+ when: ansible_os_family == 'Suse'
+
+- name: Red Hat-specific setup
+ ansible.builtin.include_tasks: roles/create_nfs_mount/tasks/install-deps/redhat/main.yml
+ when: ansible_os_family == 'RedHat'
diff --git a/playbooks/roles/create_nfs_mount/tasks/install-deps/redhat/main.yml b/playbooks/roles/create_nfs_mount/tasks/install-deps/redhat/main.yml
new file mode 100644
index 000000000000..5fb04bcb4903
--- /dev/null
+++ b/playbooks/roles/create_nfs_mount/tasks/install-deps/redhat/main.yml
@@ -0,0 +1,14 @@
+---
+- name: Install NFS client administrative tools
+ become: yes
+ become_method: sudo
+ yum:
+ update_cache: yes
+ name: "{{ packages }}"
+ retries: 3
+ delay: 5
+ register: result
+ until: result.rc == 0
+ vars:
+ packages:
+ - nfs-utils
diff --git a/playbooks/roles/create_nfs_mount/tasks/install-deps/suse/main.yml b/playbooks/roles/create_nfs_mount/tasks/install-deps/suse/main.yml
new file mode 100644
index 000000000000..f84f03c83f9d
--- /dev/null
+++ b/playbooks/roles/create_nfs_mount/tasks/install-deps/suse/main.yml
@@ -0,0 +1,8 @@
+---
+- name: Install NFS client administrative tools
+ become: yes
+ become_method: sudo
+ zypper:
+ name:
+ - nfs-utils
+ state: present
diff --git a/playbooks/roles/create_nfs_mount/tasks/main.yml b/playbooks/roles/create_nfs_mount/tasks/main.yml
new file mode 100644
index 000000000000..1ce7f4f6f38b
--- /dev/null
+++ b/playbooks/roles/create_nfs_mount/tasks/main.yml
@@ -0,0 +1,47 @@
+---
+- name: Import optional extra_args file
+ include_vars: "{{ item }}"
+ ignore_errors: yes
+ with_first_found:
+ - files:
+ - "../extra_vars.yml"
+ - "../extra_vars.yaml"
+ - "../extra_vars.json"
+ skip: true
+ tags: vars
+
+- name: Check if {{ nfs_mounted_on }} is mounted
+ become: yes
+ become_method: sudo
+ ansible.builtin.command: mountpoint -q {{ nfs_mounted_on }}
+ register: mountpoint_stat
+ failed_when: False
+ changed_when: False
+
+#- name: Check if {{ nfs_mounted_on }} is present in /etc/fstab
+# become: yes
+# become_method: sudo
+# ansible.builtin.shell: |
+# set -o pipefail
+# awk '{ print $2 }' /etc/fstab | grep -q ^"{{ nfs_mounted_on }}"
+# register: fstab_grep_stat
+# failed_when: False
+# changed_when: False
+# args:
+# executable: /bin/bash
+
+- include: tasks/install-deps/main.yml
+
+- name: Ensure {{ nfs_mounted_on }} is mounted
+ become: yes
+ become_flags: 'su - -c'
+ become_method: sudo
+ throttle: 1
+ ansible.builtin.mount:
+ path: "{{ nfs_mounted_on }}"
+ src: "{{ nfs_server_hostname }}:{{ nfs_server_export }}"
+ fstype: "nfs"
+ opts: "{{ nfs_mount_options }}"
+ state: mounted
+ when:
+ mountpoint_stat != 0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 5/7] Add a playbook for the git regression workflow
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
` (3 preceding siblings ...)
2024-01-03 15:08 ` [PATCH v2 4/7] create_nfs_mount: Enable testing arbitrary NFS mounts Chuck Lever
@ 2024-01-03 15:08 ` Chuck Lever
2024-01-25 14:34 ` Luis Chamberlain
2024-01-25 15:14 ` Luis Chamberlain
2024-01-03 15:08 ` [PATCH v2 6/7] Add a workflow for the git regression suite Chuck Lever
` (2 subsequent siblings)
7 siblings, 2 replies; 23+ messages in thread
From: Chuck Lever @ 2024-01-03 15:08 UTC (permalink / raw)
To: kdevops
From: Chuck Lever <chuck.lever@oracle.com>
This workflow builds the git tool and runs its built-in regression
suite on the target file system. The git regression suite acts as an
extensive file system function test. This workflow runs the suite in
multiple threads to add timing stress.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
playbooks/gitr.yml | 4
playbooks/roles/gitr/defaults/main.yml | 9 +
.../roles/gitr/tasks/install-deps/debian/main.yml | 28 +++
playbooks/roles/gitr/tasks/install-deps/main.yml | 12 +
.../roles/gitr/tasks/install-deps/redhat/main.yml | 38 ++++
.../roles/gitr/tasks/install-deps/suse/main.yml | 24 ++
playbooks/roles/gitr/tasks/main.yml | 214 ++++++++++++++++++++
7 files changed, 329 insertions(+)
create mode 100644 playbooks/gitr.yml
create mode 100644 playbooks/roles/gitr/defaults/main.yml
create mode 100644 playbooks/roles/gitr/tasks/install-deps/debian/main.yml
create mode 100644 playbooks/roles/gitr/tasks/install-deps/main.yml
create mode 100644 playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
create mode 100644 playbooks/roles/gitr/tasks/install-deps/suse/main.yml
create mode 100644 playbooks/roles/gitr/tasks/main.yml
diff --git a/playbooks/gitr.yml b/playbooks/gitr.yml
new file mode 100644
index 000000000000..4c3e3ba919d7
--- /dev/null
+++ b/playbooks/gitr.yml
@@ -0,0 +1,4 @@
+---
+- hosts: all
+ roles:
+ - role: gitr
diff --git a/playbooks/roles/gitr/defaults/main.yml b/playbooks/roles/gitr/defaults/main.yml
new file mode 100644
index 000000000000..fe2763aa8f73
--- /dev/null
+++ b/playbooks/roles/gitr/defaults/main.yml
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier GPL-2.0+
+---
+# Our sensible defaults for the gitr role.
+#
+# The default is to not run the gitr tests.
+kdevops_run_gitr: False
+
+gitr_uses_no_devices: False
+gitr_nfs_use_kdevops_nfsd: False
diff --git a/playbooks/roles/gitr/tasks/install-deps/debian/main.yml b/playbooks/roles/gitr/tasks/install-deps/debian/main.yml
new file mode 100644
index 000000000000..50d4ba1037f3
--- /dev/null
+++ b/playbooks/roles/gitr/tasks/install-deps/debian/main.yml
@@ -0,0 +1,28 @@
+---
+- name: Install build dependencies for git
+ become: yes
+ become_method: sudo
+ apt:
+ name:
+ - gcc
+ - git-core
+ - gettext
+ - libcurl4-openssl-dev
+ - libexpat1-dev
+ - libssl-dev
+ - libz-dev
+ state: present
+ update_cache: yes
+ tags: [ 'gitr', 'deps' ]
+
+- name: Install run-time dependencies for git regression tests
+ become: yes
+ become_method: sudo
+ apt:
+ name:
+ - cvs
+ - cvsps
+ - sqlite3
+ state: present
+ update_cache: yes
+ tags: [ 'gitr', 'deps' ]
diff --git a/playbooks/roles/gitr/tasks/install-deps/main.yml b/playbooks/roles/gitr/tasks/install-deps/main.yml
new file mode 100644
index 000000000000..c4f0bc1f2192
--- /dev/null
+++ b/playbooks/roles/gitr/tasks/install-deps/main.yml
@@ -0,0 +1,12 @@
+---
+- name: Debian-specific set up
+ ansible.builtin.include_tasks: roles/gitr/tasks/install-deps/debian/main.yml
+ when: ansible_os_family == 'Debian'
+
+- name: SuSE-specific set up
+ ansible.builtin.include_tasks: roles/gitr/tasks/install-deps/suse/main.yml
+ when: ansible_os_family == 'Suse'
+
+- name: Red Hat-specific set up
+ ansible.builtin.include_tasks: roles/gitr/tasks/install-deps/redhat/main.yml
+ when: ansible_os_family == 'RedHat'
diff --git a/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml b/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
new file mode 100644
index 000000000000..30fb0f9b16d5
--- /dev/null
+++ b/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
@@ -0,0 +1,38 @@
+---
+- name: Install build dependencies for git
+ become: yes
+ become_method: sudo
+ yum:
+ update_cache: yes
+ name: "{{ packages }}"
+ retries: 3
+ delay: 5
+ register: result
+ until: result.rc == 0
+ vars:
+ packages:
+ - expat-devel
+ - gcc
+ - gettext
+ - git-core
+ - libcurl-devel
+ - openssl-devel
+ - zlib-devel
+
+- name: Install run-time dependencies for git regression tests
+ become: yes
+ become_method: sudo
+ yum:
+ update_cache: yes
+ name: "{{ packages }}"
+ retries: 3
+ delay: 5
+ register: result
+ until: result.rc == 0
+ vars:
+ packages:
+ - cvs
+ - cvsps
+ - git-email
+ - subversion
+# - subversion-perl
diff --git a/playbooks/roles/gitr/tasks/install-deps/suse/main.yml b/playbooks/roles/gitr/tasks/install-deps/suse/main.yml
new file mode 100644
index 000000000000..0db06c5b66d0
--- /dev/null
+++ b/playbooks/roles/gitr/tasks/install-deps/suse/main.yml
@@ -0,0 +1,24 @@
+---
+- name: Install build dependencies for git
+ become: yes
+ become_method: sudo
+ zypper:
+ name:
+ - gcc
+ - gettext
+ - git-core
+ - libcurl-devel
+ - libexpat-devel
+ - libopenssl-1_1-devel
+ state: present
+
+- name: Install run-time dependencies for git regression tests
+ become: yes
+ become_method: sudo
+ zypper:
+ name:
+ - cvs
+ - cvsps
+ - subversion
+ - subversion-perl
+ state: present
diff --git a/playbooks/roles/gitr/tasks/main.yml b/playbooks/roles/gitr/tasks/main.yml
new file mode 100644
index 000000000000..f225d547dd65
--- /dev/null
+++ b/playbooks/roles/gitr/tasks/main.yml
@@ -0,0 +1,214 @@
+---
+- name: Import optional extra_args file
+ include_vars: "{{ item }}"
+ ignore_errors: yes
+ with_first_found:
+ - files:
+ - "../extra_vars.yml"
+ - "../extra_vars.yaml"
+ - "../extra_vars.json"
+ skip: true
+ tags: vars
+
+- name: Set the path where we collect gitr test results
+ tags: [ 'vars' ]
+ set_fact:
+ gitr_workflow_dir: "../workflows/gitr"
+
+- name: Clean up localhost results directory and files
+ tags: [ 'clean_local_results' ]
+ local_action: file path="{{ gitr_workflow_dir }}/results/" state=absent
+ run_once: true
+
+- name: Create the results directory
+ tags: [ 'first_run' ]
+ local_action: file path="{{ gitr_workflow_dir }}/results/" state=directory
+ run_once: true
+
+- name: Remove the old test file system, if needed
+ tags: [ 'gitr' ]
+ become: yes
+ become_flags: 'su - -c'
+ become_method: sudo
+ ansible.posix.mount:
+ path: "{{ gitr_mnt }}"
+ state: unmounted
+
+- name: Create the file system under test, if needed
+ tags: [ 'gitr' ]
+ include_role:
+ name: create_partition
+ vars:
+ disk_setup_device: "{{ gitr_device }}"
+ disk_setup_fstype: "{{ gitr_fstype }}"
+ disk_setup_label : "{{ gitr_label }}"
+ disk_setup_fs_opts: "-L {{ gitr_label }}"
+ disk_setup_path: "{{ gitr_mnt }}"
+ disk_setup_mount_opts: "{{ gitr_mount_opts }}"
+ disk_setup_user: "{{ data_user }}"
+ disk_setup_group: "{{ data_group }}"
+ when:
+ - not gitr_uses_no_devices|bool
+
+- name: Create the tmpfs file system under test, if needed
+ tags: [ 'gitr' ]
+ include_role:
+ name: create_tmpfs
+ vars:
+ tmpfs_mnt_options: "{{ gitr_mount_opts }}"
+ tmpfs_mounted_on: "{{ gitr_mnt }}"
+ tmpfs_user: "{{ data_user }}"
+ tmpfs_group: "{{ data_group }}"
+ when:
+ - gitr_fstype == "tmpfs"
+
+- name: Create an NFSD export for testing
+ include_role:
+ name: nfsd_add_export
+ vars:
+ server_host: "{{ gitr_nfs_server_host }}"
+ export_volname: "{{ gitr_nfs_server_export }}"
+ export_options: "{{ nfsd_export_options }}"
+ export_fstype: "{{ nfsd_export_fstype }}"
+ export_size: 5g
+ export_user: "{{ data_user }}"
+ export_group: "{{ data_group }}"
+ when:
+ - gitr_nfs_use_kdevops_nfsd|bool
+ - gitr_fstype == "nfs"
+
+- name: Create the NFS mount under test, if needed
+ tags: [ 'gitr' ]
+ include_role:
+ name: create_nfs_mount
+ vars:
+ nfs_mounted_on: "{{ gitr_mnt }}"
+ nfs_server_hostname: "{{ gitr_nfs_server_host }}"
+ nfs_server_export: "{{ nfsd_export_path }}/{{ gitr_nfs_server_export }}"
+ nfs_mount_options: "{{ gitr_mount_opts }}"
+ when:
+ - gitr_fstype == "nfs"
+
+- include: tasks/install-deps/main.yml
+
+- name: Remove old git dir
+ tags: [ 'gitr' ]
+ become: yes
+ become_flags: 'su - -c'
+ become_method: sudo
+ ansible.builtin.file:
+ path: "{{ gitr_mnt }}/git"
+ state: absent
+
+- name: git clone git
+ tags: [ 'gitr' ]
+ ansible.builtin.git:
+ repo: "{{ gitr_repo }}"
+ dest: "{{ gitr_mnt }}/git"
+ update: yes
+ version: "{{ gitr_repo_commit }}"
+ retries: 3
+ delay: 5
+ register: result
+ until: not result.failed
+
+- name: Get nproc
+ tags: [ 'gitr', 'build', 'vars' ]
+ ansible.builtin.command: "{{ num_jobs }}"
+ register: nproc
+ changed_when: False
+
+- name: Build git
+ tags: [ 'gitr', 'build' ]
+ community.general.make:
+ chdir: "{{ gitr_mnt }}/git"
+ jobs: "{{ nproc.stdout }}"
+
+- name: Run the full git regression suite
+ tags: [ 'run_tests' ]
+ community.general.make:
+ chdir: "{{ gitr_mnt }}/git"
+ target: "test"
+ jobs: "{{ nproc.stdout }}"
+ register: gitr_results
+ failed_when: False
+ changed_when: False
+
+- name: Capture RPC statistics
+ tags: [ 'copy_results' ]
+ ansible.builtin.command: "mountstats --rpc"
+ register: rpc_results
+ when:
+ - gitr_fstype == "nfs"
+
+- name: Capture transport statistics
+ tags: [ 'copy_results' ]
+ ansible.builtin.command: "mountstats --xprt"
+ register: xprt_results
+ when:
+ - gitr_fstype == "nfs"
+
+- name: Save git regression test output to a file
+ tags: [ 'copy_results' ]
+ ansible.builtin.copy:
+ content: "{{ gitr_results.stdout_lines|join('\n') }}"
+ dest: "{{ gitr_mnt }}/gitr.stdout"
+
+- name: Save RPC statistics to a file
+ tags: [ 'copy_results' ]
+ ansible.builtin.copy:
+ content: "{{ rpc_results.stdout_lines|join('\n') }}"
+ dest: "{{ gitr_mnt }}/gitr.rpc"
+ when:
+ - gitr_fstype == "nfs"
+
+- name: Save transport statistics to a file
+ tags: [ 'copy_results' ]
+ ansible.builtin.copy:
+ content: "{{ xprt_results.stdout_lines|join('\n') }}"
+ dest: "{{ gitr_mnt }}/gitr.xprt"
+ when:
+ - gitr_fstype == "nfs"
+
+- name: Get used target kernel version
+ tags: [ 'copy_results' ]
+ ansible.builtin.command: "uname -r"
+ register: uname_cmd
+
+- name: Store kernel_rev variable
+ tags: [ 'copy_results' ]
+ set_fact:
+ kernel_rev: "{{ uname_cmd.stdout_lines | regex_replace('\\]') | regex_replace('\\[') | replace(\"'\",'') }}"
+ run_once: true
+
+- name: Copy regression results over
+ tags: [ 'copy_results' ]
+ ansible.builtin.fetch:
+ src: "{{ gitr_mnt }}/gitr.stdout"
+ dest: "{{ gitr_workflow_dir }}/results/{{ kernel_rev }}.stdout"
+ flat: yes
+
+- name: Copy RPC statistics over
+ tags: [ 'copy_results' ]
+ ansible.builtin.fetch:
+ src: "{{ gitr_mnt }}/gitr.rpc"
+ dest: "{{ gitr_workflow_dir }}/results/{{ kernel_rev }}.rpc"
+ flat: yes
+ when:
+ - gitr_fstype == "nfs"
+
+- name: Copy transport statistics over
+ tags: [ 'copy_results' ]
+ ansible.builtin.fetch:
+ src: "{{ gitr_mnt }}/gitr.xprt"
+ dest: "{{ gitr_workflow_dir }}/results/{{ kernel_rev }}.xprt"
+ flat: yes
+ when:
+ - gitr_fstype == "nfs"
+
+# XXX: This is working now, but it should copy the failures into a log file
+- name: Report failures
+ tags: [ 'copy_results' ]
+ debug:
+ msg: "{{ gitr_results.stderr_lines }}"
+ when: gitr_results.stderr | length > 0
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 6/7] Add a workflow for the git regression suite
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
` (4 preceding siblings ...)
2024-01-03 15:08 ` [PATCH v2 5/7] Add a playbook for the git regression workflow Chuck Lever
@ 2024-01-03 15:08 ` Chuck Lever
2024-01-25 15:39 ` Luis Chamberlain
2024-01-03 15:09 ` [PATCH v2 7/7] gitr: Adjust the gen_hosts and gen_nodes playbooks Chuck Lever
2024-01-03 17:10 ` [PATCH v2 0/7] Add a git regression workflow Jeff Layton
7 siblings, 1 reply; 23+ messages in thread
From: Chuck Lever @ 2024-01-03 15:08 UTC (permalink / raw)
To: kdevops
From: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
.gitignore | 2 +
README.md | 12 +++-
docs/gitr.md | 22 +++++++
kconfigs/workflows/Kconfig | 25 ++++++++
scripts/workflows/gitr/run_kernel_ci.sh | 1
scripts/workflows/gitr/run_kernel_ci_kotd.sh | 1
scripts/workflows/gitr/run_loop.sh | 58 ++++++++++++++++++
workflows/Makefile | 5 ++
workflows/gitr/Kconfig | 82 ++++++++++++++++++++++++++
workflows/gitr/Makefile | 82 ++++++++++++++++++++++++++
workflows/gitr/btrfs/Kconfig | 20 ++++++
workflows/gitr/btrfs/Makefile | 5 ++
workflows/gitr/ext4/Kconfig | 20 ++++++
workflows/gitr/ext4/Makefile | 5 ++
workflows/gitr/nfs/Kconfig | 31 ++++++++++
workflows/gitr/nfs/Makefile | 14 ++++
workflows/gitr/tmpfs/Kconfig | 5 ++
workflows/gitr/tmpfs/Makefile | 4 +
workflows/gitr/xfs/Kconfig | 20 ++++++
workflows/gitr/xfs/Makefile | 6 ++
20 files changed, 419 insertions(+), 1 deletion(-)
create mode 100644 docs/gitr.md
create mode 120000 scripts/workflows/gitr/run_kernel_ci.sh
create mode 120000 scripts/workflows/gitr/run_kernel_ci_kotd.sh
create mode 100755 scripts/workflows/gitr/run_loop.sh
create mode 100644 workflows/gitr/Kconfig
create mode 100644 workflows/gitr/Makefile
create mode 100644 workflows/gitr/btrfs/Kconfig
create mode 100644 workflows/gitr/btrfs/Makefile
create mode 100644 workflows/gitr/ext4/Kconfig
create mode 100644 workflows/gitr/ext4/Makefile
create mode 100644 workflows/gitr/nfs/Kconfig
create mode 100644 workflows/gitr/nfs/Makefile
create mode 100644 workflows/gitr/tmpfs/Kconfig
create mode 100644 workflows/gitr/tmpfs/Makefile
create mode 100644 workflows/gitr/xfs/Kconfig
create mode 100644 workflows/gitr/xfs/Makefile
diff --git a/.gitignore b/.gitignore
index 05b25d3a1fc1..738d073e889a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -69,6 +69,8 @@ workflows/blktests/results/[0-9]*
workflows/demos/reboot-limit/results/
+workflows/gitr/results/
+
# For now we ignore all results as we have work to do:
# - copy only data for failed tests
# - envision a way so that regressions are clearly visible with a git diff
diff --git a/README.md b/README.md
index 16a91b3608cb..d5d144f971e9 100644
--- a/README.md
+++ b/README.md
@@ -105,7 +105,7 @@ workflow you can just run:
For more details see [kdevops blktests docs](docs/blktests.md)
-### Start testing NFS with in 2 commands
+### Start testing NFS in 2 commands
To test the kernel's nfs server with the pynfs testsuite, enable the pynfs
workflow and then run:
@@ -115,6 +115,16 @@ workflow and then run:
For more details see [kdevops nfs docs](docs/nfs.md)
+### Start running the git regression suite in 2 commands
+
+To test a kernel using the git regression suite, enable the gitr workflow
+and then run:
+
+ * `make gitr`
+ * `make gitr-baseline`
+
+For more details see [kdevops gitr docs](docs/gitr.md)
+
### Runs some kernel selftests in a parallel manner
kdevops supports running Linux kernel selftests in parallel, this is as easy as:
diff --git a/docs/gitr.md b/docs/gitr.md
new file mode 100644
index 000000000000..73e5c42ec0e4
--- /dev/null
+++ b/docs/gitr.md
@@ -0,0 +1,22 @@
+# kdevops git regression suite
+
+kdevops can run the git tool's internal regression suite against
+a target file system.
+
+Run `make menuconfig` and select:
+
+ Target workflows -> Dedicated target Linux test workflow ->gitr
+
+Then configure the test parameters by going to:
+
+ Target workflows -> Configure and run the git regression suite
+
+Choose the file system type to test and the location of the repo
+that contains the version of git you want to use for the test.
+
+Then, run:
+
+ * `make`
+ * `make bringup`
+ * `make gitr`
+ * `make gitr-baseline`
diff --git a/kconfigs/workflows/Kconfig b/kconfigs/workflows/Kconfig
index 652f65de7647..1a904258734f 100644
--- a/kconfigs/workflows/Kconfig
+++ b/kconfigs/workflows/Kconfig
@@ -152,6 +152,13 @@ config KDEVOPS_WORKFLOW_DEDICATE_SELFTESTS
selftests work. Note that you will need at least 8 GiB of RAM
for your target guests.
+config KDEVOPS_WORKFLOW_DEDICATE_GITR
+ bool "gitr"
+ select KDEVOPS_WORKFLOW_ENABLE_GITR
+ help
+ This will dedicate your configuration to running only the
+ multi-threaded git regression workflow.
+
endchoice
endif
@@ -209,6 +216,14 @@ config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_SELFTESTS
Select this option if you are doing Linux kernel developent and
testing with sefltests.
+config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_GITR
+ bool "gitr"
+ select KDEVOPS_WORKFLOW_ENABLE_GITR
+ depends on VAGRANT || TERRAFORM_PRIVATE_NET
+ help
+ Select this option if you want to configure the deployment for
+ testing via the multi-threaded git regression workflow.
+
endif # !WORKFLOWS_DEDICATED_WORKFLOW
config KDEVOPS_WORKFLOW_ENABLE_FSTESTS
@@ -262,6 +277,16 @@ source "workflows/selftests/Kconfig"
endmenu
endif # KDEVOPS_WORKFLOW_ENABLE_SELFTESTS
+config KDEVOPS_WORKFLOW_ENABLE_GITR
+ bool
+ default y if KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_GITR || KDEVOPS_WORKFLOW_DEDICATE_GITR
+
+if KDEVOPS_WORKFLOW_ENABLE_GITR
+menu "Configure and run the git regression suite"
+source "workflows/gitr/Kconfig"
+endmenu
+endif # KDEVOPS_WORKFLOW_ENABLE_GITR
+
config KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT
bool
default y if KDEVOPS_WORKFLOW_ENABLE_FSTESTS || KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
diff --git a/scripts/workflows/gitr/run_kernel_ci.sh b/scripts/workflows/gitr/run_kernel_ci.sh
new file mode 120000
index 000000000000..4fd5dc5e8e13
--- /dev/null
+++ b/scripts/workflows/gitr/run_kernel_ci.sh
@@ -0,0 +1 @@
+../generic/run_kernel_ci.sh
\ No newline at end of file
diff --git a/scripts/workflows/gitr/run_kernel_ci_kotd.sh b/scripts/workflows/gitr/run_kernel_ci_kotd.sh
new file mode 120000
index 000000000000..8f94d6ba4f52
--- /dev/null
+++ b/scripts/workflows/gitr/run_kernel_ci_kotd.sh
@@ -0,0 +1 @@
+../kotd/run_kernel_ci_kotd.sh
\ No newline at end of file
diff --git a/scripts/workflows/gitr/run_loop.sh b/scripts/workflows/gitr/run_loop.sh
new file mode 100755
index 000000000000..2ba961e2f4cb
--- /dev/null
+++ b/scripts/workflows/gitr/run_loop.sh
@@ -0,0 +1,58 @@
+#!/bin/bash
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+# Part of kdevops kernel-ci, this is the script which will run the test workflow
+# as many times as indicated up to CONFIG_KERNEL_CI_STEADY_STATE_GOAL
+
+source ${TOPDIR}/.config
+source ${TOPDIR}/scripts/lib.sh
+
+WORKFLOWDIR=${TOPDIR}/workflows/gitr
+
+COUNT=1
+
+run_loop()
+{
+ while true; do
+ echo "== kernel-ci gitr test loop $COUNT start: $(date)" > $KERNEL_CI_FAIL_LOG
+ echo "/usr/bin/time -f %E make gitr-baseline" >> $KERNEL_CI_FAIL_LOG
+ /usr/bin/time -p -o $KERNEL_CI_LOGTIME make gitr-baseline >> $KERNEL_CI_FAIL_LOG
+ ANSIBLE_CALL_RET=$?
+ echo "End $COUNT: $(date)" >> $KERNEL_CI_FAIL_LOG
+ cat $KERNEL_CI_LOGTIME >> $KERNEL_CI_FAIL_LOG
+ echo "git status:" >> $KERNEL_CI_FAIL_LOG
+ git status >> $KERNEL_CI_FAIL_LOG
+ echo "Results:" >> $KERNEL_CI_FAIL_LOG
+
+ rm -f $KERNEL_CI_DIFF_LOG
+
+ if [[ "$ANSIBLE_CALL_RET" -ne 0 ]]; then
+ echo "Test $COUNT: FAILED!" >> $KERNEL_CI_DIFF_LOG
+ echo "== Test loop count $COUNT" >> $KERNEL_CI_DIFF_LOG
+ echo "$(git describe)" >> $KERNEL_CI_DIFF_LOG
+ git diff >> $KERNEL_CI_DIFF_LOG
+ cat $KERNEL_CI_DIFF_LOG >> $KERNEL_CI_FAIL_LOG
+ cat $KERNEL_CI_FAIL_LOG >> $KERNEL_CI_FULL_LOG
+ echo $COUNT > $KERNEL_CI_FAIL_FILE
+ exit 1
+ else
+ echo "Test $COUNT: OK!" >> $KERNEL_CI_FAIL_LOG
+ echo "----------------------------------------------------------------" >> $KERNEL_CI_FAIL_LOG
+ cat $KERNEL_CI_FAIL_LOG >> $KERNEL_CI_FULL_LOG
+ fi
+
+ # This let's us keep track of which loop count was last successful
+ echo $COUNT > $KERNEL_CI_OK_FILE
+
+ let COUNT=$COUNT+1
+ if [[ "$CONFIG_KERNEL_CI_ENABLE_STEADY_STATE" == "y" &&
+ "$COUNT" -gt "$CONFIG_KERNEL_CI_STEADY_STATE_GOAL" ]]; then
+ exit 0
+ fi
+ sleep 1
+ done
+}
+
+rm -f $KERNEL_CI_FAIL_FILE $KERNEL_CI_OK_FILE
+echo "= kernel-ci full log" > $KERNEL_CI_FULL_LOG
+run_loop
diff --git a/workflows/Makefile b/workflows/Makefile
index 1c0e13a93f38..122eb98bb66b 100644
--- a/workflows/Makefile
+++ b/workflows/Makefile
@@ -43,6 +43,11 @@ ifeq (y,$(CONFIG_WORKFLOWS_REBOOT_LIMIT))
include workflows/demos/reboot-limit/Makefile
endif # CONFIG_WORKFLOWS_REBOOT_LIMIT == y
+ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_GITR))
+WORKFLOW_ARGS += kdevops_workflow_enable_gitr='True'
+include workflows/gitr/Makefile
+endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_GITR == y
+
ANSIBLE_EXTRA_ARGS += $(WORKFLOW_ARGS)
ANSIBLE_EXTRA_ARGS_SEPARATED += $(WORKFLOW_ARGS_SEPARATED)
ANSIBLE_EXTRA_ARGS_DIRECT += $(WORKFLOW_ARGS_DIRECT)
diff --git a/workflows/gitr/Kconfig b/workflows/gitr/Kconfig
new file mode 100644
index 000000000000..14f69676c5e8
--- /dev/null
+++ b/workflows/gitr/Kconfig
@@ -0,0 +1,82 @@
+if KDEVOPS_WORKFLOW_ENABLE_GITR
+
+choice
+ prompt "The target file system type to be tested"
+ default GITR_BTRFS
+
+config GITR_XFS
+ bool "xfs"
+ help
+ Run the git regression suite on an xfs file system.
+
+config GITR_BTRFS
+ bool "btrfs"
+ help
+ Run the git regression suite on a btrfs file system.
+
+config GITR_EXT4
+ bool "ext4"
+ help
+ Run the git regression suite on an ext4 file system.
+
+config GITR_NFS
+ bool "nfs"
+ help
+ Run the git regression suite on an NFS mount.
+
+config GITR_TMPFS
+ bool "tmpfs"
+ help
+ Run the git regression suite on a tmpfs file system.
+
+endchoice
+
+if GITR_XFS
+menu "Configure how to test XFS"
+source "workflows/gitr/xfs/Kconfig"
+endmenu
+endif # GITR_XFS
+
+if GITR_BTRFS
+menu "Configure how to test btrfs"
+source "workflows/gitr/btrfs/Kconfig"
+endmenu
+endif # GITR_BTRFS
+
+if GITR_EXT4
+menu "Configure how to test ext4"
+source "workflows/gitr/ext4/Kconfig"
+endmenu
+endif # GITR_EXT4
+
+if GITR_NFS
+menu "Configure how nfs should be tested"
+source "workflows/gitr/nfs/Kconfig"
+endmenu
+endif # GITR_NFS
+
+if GITR_TMPFS
+menu "Configure how tmpfs should be tested"
+source "workflows/gitr/tmpfs/Kconfig"
+endmenu
+endif # GITR_TMPFS
+
+config GITR_MNT
+ string "Where to mount the file system under test"
+ default "/gitr"
+ help
+ The directory over which to mount the file system under test.
+
+config GITR_REPO
+ string "The git tool repository to clone"
+ default "https://git.kernel.org/pub/scm/git/git.git"
+ help
+ The git tool repository to clone.
+
+config GITR_REPO_COMMIT
+ string "The version of git to check out"
+ default "maint"
+ help
+ The commit to be used for the test.
+
+endif # KDEVOPS_WORKFLOW_ENABLE_GITR
diff --git a/workflows/gitr/Makefile b/workflows/gitr/Makefile
new file mode 100644
index 000000000000..0389c3132e1a
--- /dev/null
+++ b/workflows/gitr/Makefile
@@ -0,0 +1,82 @@
+ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
+export KDEVOPS_HOSTS_TEMPLATE := gitr.j2
+endif
+
+GITR_MNT:=$(subst ",,$(CONFIG_GITR_MNT))
+GITR_ARGS += gitr_mnt=$(GITR_MNT)
+
+ifeq (y,$(CONFIG_GITR_XFS))
+include $(TOPDIR)/workflows/gitr/xfs/Makefile
+endif
+
+ifeq (y,$(CONFIG_GITR_BTRFS))
+include $(TOPDIR)/workflows/gitr/btrfs/Makefile
+endif
+
+ifeq (y,$(CONFIG_GITR_EXT4))
+include $(TOPDIR)/workflows/gitr/ext4/Makefile
+endif
+
+ifeq (y,$(CONFIG_GITR_NFS))
+include $(TOPDIR)/workflows/gitr/nfs/Makefile
+endif
+
+ifeq (y,$(CONFIG_GITR_TMPFS))
+include $(TOPDIR)/workflows/gitr/tmpfs/Makefile
+endif
+
+GITR_REPO:=$(subst ",,$(CONFIG_GITR_REPO))
+GITR_ARGS += gitr_repo=$(GITR_REPO)
+
+GITR_REPO_COMMIT:=$(subst ",,$(CONFIG_GITR_REPO_COMMIT))
+GITR_ARGS += gitr_repo_commit=$(GITR_REPO_COMMIT)
+
+WORKFLOW_ARGS_DIRECT += $(GITR_ARGS)
+
+GITR_KERNEL_CI_LOOP := false
+GITR_KERNEL_CI_LOOP_KOTD := false
+
+ifeq (y,$(CONFIG_KERNEL_CI))
+GITR_KERNEL_CI_LOOP := scripts/workflows/gitr/run_kernel_ci.sh
+GITR_KERNEL_CI_LOOP_KOTD := scripts/workflows/gitr/run_kernel_ci_kotd.sh
+endif # CONFIG_KERNEL_CI
+
+gitr:
+ $(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
+ -f 30 -i hosts playbooks/gitr.yml --skip-tags run_tests,copy_results
+
+gitr-baseline:
+ $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+ -f 30 -i hosts -l baseline playbooks/gitr.yml \
+ --tags vars,run_tests,copy_results \
+ --extra-vars=@./extra_vars.yaml
+
+gitr-baseline-loop:
+ $(Q)$(GITR_KERNEL_CI_LOOP) baseline
+
+gitr-baseline-kernelci:
+ $(Q)$(GITR_KERNEL_CI_LOOP_KOTD) baseline
+
+gitr-dev-baseline:
+ $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+ -f 30 -i hosts -l dev playbooks/gitr.yml --tags vars,run_tests,copy_results --extra-vars=@./extra_vars.yaml
+
+gitr-dev-loop:
+ $(Q)$(GITR_KERNEL_CI_LOOP) dev
+
+gitr-dev-kernelci:
+ $(Q)$(GITR_KERNEL_CI_LOOP_KOTD) dev
+
+gitr-dev-reset:
+ $(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
+ -f 30 -i hosts -l dev playbooks/gitr.yml --tags vars,reset --extra-vars=@./extra_vars.yaml
+
+gitr-help-menu:
+ @echo "gitr options:"
+ @echo "gitr - Git clone git, build and install it"
+ @echo "gitr-{baseline,dev} - Run the gitr test on baseline or dev hosts and collect results"
+ @echo "gitr-{baseline,dev}-loop" - Run gitr in a loop until error or steady state
+ @echo "gitr-{baseline,dev}-kernelci - Run gitr kernel-ci loop"
+ @echo ""
+
+HELP_TARGETS += gitr-help-menu
diff --git a/workflows/gitr/btrfs/Kconfig b/workflows/gitr/btrfs/Kconfig
new file mode 100644
index 000000000000..1d5e997cc498
--- /dev/null
+++ b/workflows/gitr/btrfs/Kconfig
@@ -0,0 +1,20 @@
+config GITR_DEVICE
+ string
+ default "/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_NVME
+ default "/dev/disk/by-id/virtio-kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_VIRTIO
+ default "/dev/disk/by-id/ide-kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_IDE
+ default "/dev/nvme2n1" if TERRAFORM_AWS_INSTANCE_M5AD_4XLARGE
+ default "/dev/nvme1n1" if TERRAFORM_GCE
+ default "/dev/sdd" if TERRAFORM_AZURE
+
+config GITR_LABEL
+ string "The file system label to use"
+ default "gitr"
+ help
+ The file system label that marks the file system under test.
+
+config GITR_MOUNT_OPTS
+ string "The file system mount options to use"
+ default "defaults"
+ help
+ The mount options to use when mounting the file system under test.
diff --git a/workflows/gitr/btrfs/Makefile b/workflows/gitr/btrfs/Makefile
new file mode 100644
index 000000000000..1fc0ea050dc8
--- /dev/null
+++ b/workflows/gitr/btrfs/Makefile
@@ -0,0 +1,5 @@
+GITR_ARGS += gitr_fstype=btrfs
+GITR_ARGS += gitr_uses_no_devices='False'
+GITR_ARGS += gitr_device=$(subst ",,$(CONFIG_GITR_DEVICE))
+GITR_ARGS += gitr_label=$(subst ",,$(CONFIG_GITR_LABEL))
+GITR_ARGS += gitr_mount_opts='$(subst ",,$(CONFIG_GITR_MOUNT_OPTS))'
diff --git a/workflows/gitr/ext4/Kconfig b/workflows/gitr/ext4/Kconfig
new file mode 100644
index 000000000000..1d5e997cc498
--- /dev/null
+++ b/workflows/gitr/ext4/Kconfig
@@ -0,0 +1,20 @@
+config GITR_DEVICE
+ string
+ default "/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_NVME
+ default "/dev/disk/by-id/virtio-kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_VIRTIO
+ default "/dev/disk/by-id/ide-kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_IDE
+ default "/dev/nvme2n1" if TERRAFORM_AWS_INSTANCE_M5AD_4XLARGE
+ default "/dev/nvme1n1" if TERRAFORM_GCE
+ default "/dev/sdd" if TERRAFORM_AZURE
+
+config GITR_LABEL
+ string "The file system label to use"
+ default "gitr"
+ help
+ The file system label that marks the file system under test.
+
+config GITR_MOUNT_OPTS
+ string "The file system mount options to use"
+ default "defaults"
+ help
+ The mount options to use when mounting the file system under test.
diff --git a/workflows/gitr/ext4/Makefile b/workflows/gitr/ext4/Makefile
new file mode 100644
index 000000000000..540da14ccf02
--- /dev/null
+++ b/workflows/gitr/ext4/Makefile
@@ -0,0 +1,5 @@
+GITR_ARGS += gitr_fstype=ext4
+GITR_ARGS += gitr_uses_no_devices='False'
+GITR_ARGS += gitr_device=$(subst ",,$(CONFIG_GITR_DEVICE))
+GITR_ARGS += gitr_label=$(subst ",,$(CONFIG_GITR_LABEL))
+GITR_ARGS += gitr_mount_opts='$(subst ",,$(CONFIG_GITR_MOUNT_OPTS))'
diff --git a/workflows/gitr/nfs/Kconfig b/workflows/gitr/nfs/Kconfig
new file mode 100644
index 000000000000..434af79d8fbf
--- /dev/null
+++ b/workflows/gitr/nfs/Kconfig
@@ -0,0 +1,31 @@
+config GITR_USE_KDEVOPS_NFSD
+ bool "Test against the kdevops NFS server"
+ select KDEVOPS_SETUP_NFSD
+ default y
+ help
+ Select this option if you want to test against an NFS
+ server deployed by kdevops. If you want to test against
+ another already-deployed and reachable NFS server, disable
+ this option and specify the server hostname below.
+
+config GITR_NFS_SERVER_HOSTNAME
+ string "Test Server hostname"
+ default ""
+ depends on !GITR_USE_KDEVOPS_NFSD
+ help
+ The NFS server hostname or address that test clients are to
+ mount.
+
+config GITR_NFS_SERVER_EXPORT
+ string "Test Server export path"
+ default "gitr"
+ depends on !GITR_USE_KDEVOPS_NFSD
+ help
+ The NFS server export path that test clients are to mount.
+
+config GITR_MOUNT_OPTS
+ string "The NFS client mount options to use"
+ default "vers=4.2,sec=sys"
+ help
+ The mount options to use when mounting the file system
+ under test.
diff --git a/workflows/gitr/nfs/Makefile b/workflows/gitr/nfs/Makefile
new file mode 100644
index 000000000000..9cebdbf65446
--- /dev/null
+++ b/workflows/gitr/nfs/Makefile
@@ -0,0 +1,14 @@
+GITR_ARGS += gitr_fstype=nfs
+GITR_ARGS += gitr_uses_no_devices='True'
+
+ifeq (y,$(CONFIG_GITR_USE_KDEVOPS_NFSD))
+GITR_ARGS += gitr_nfs_server_host='$(subst ",,$(CONFIG_KDEVOPS_HOSTS_PREFIX))-nfsd'
+GITR_ARGS += gitr_nfs_server_export='gitr'
+GITR_ARGS += gitr_nfs_use_kdevops_nfsd='True'
+else # CONFIG_GITR_USE_KDEVOPS_NFSD
+GITR_ARGS += gitr_nfs_server_host='$(subst ",,$(CONFIG_GITR_NFS_SERVER_HOSTNAME))'
+GITR_ARGS += gitr_nfs_server_export='$(subst ",,$(CONFIG_GITR_NFS_SERVER_EXPORT))'
+GITR_ARGS += gitr_nfs_use_kdevops_nfsd='False'
+endif # CONFIG_GITR_USE_KDEVOPS_NFSD
+
+GITR_ARGS += gitr_mount_opts='$(subst ",,$(CONFIG_GITR_MOUNT_OPTS))'
diff --git a/workflows/gitr/tmpfs/Kconfig b/workflows/gitr/tmpfs/Kconfig
new file mode 100644
index 000000000000..d1ba4dab93ba
--- /dev/null
+++ b/workflows/gitr/tmpfs/Kconfig
@@ -0,0 +1,5 @@
+config GITR_MOUNT_OPTS
+ string "The file system mount options to use"
+ default "size=75%"
+ help
+ The mount options to use when mounting the file system under test.
diff --git a/workflows/gitr/tmpfs/Makefile b/workflows/gitr/tmpfs/Makefile
new file mode 100644
index 000000000000..905a45e62813
--- /dev/null
+++ b/workflows/gitr/tmpfs/Makefile
@@ -0,0 +1,4 @@
+GITR_ARGS += gitr_fstype=tmpfs
+GITR_ARGS += gitr_uses_no_devices='True'
+
+GITR_ARGS += gitr_mount_opts='$(subst ",,$(CONFIG_GITR_MOUNT_OPTS))'
diff --git a/workflows/gitr/xfs/Kconfig b/workflows/gitr/xfs/Kconfig
new file mode 100644
index 000000000000..1d5e997cc498
--- /dev/null
+++ b/workflows/gitr/xfs/Kconfig
@@ -0,0 +1,20 @@
+config GITR_DEVICE
+ string
+ default "/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_NVME
+ default "/dev/disk/by-id/virtio-kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_VIRTIO
+ default "/dev/disk/by-id/ide-kdevops1" if LIBVIRT && LIBVIRT_EXTRA_STORAGE_DRIVE_IDE
+ default "/dev/nvme2n1" if TERRAFORM_AWS_INSTANCE_M5AD_4XLARGE
+ default "/dev/nvme1n1" if TERRAFORM_GCE
+ default "/dev/sdd" if TERRAFORM_AZURE
+
+config GITR_LABEL
+ string "The file system label to use"
+ default "gitr"
+ help
+ The file system label that marks the file system under test.
+
+config GITR_MOUNT_OPTS
+ string "The file system mount options to use"
+ default "defaults"
+ help
+ The mount options to use when mounting the file system under test.
diff --git a/workflows/gitr/xfs/Makefile b/workflows/gitr/xfs/Makefile
new file mode 100644
index 000000000000..7ddb2cf9259e
--- /dev/null
+++ b/workflows/gitr/xfs/Makefile
@@ -0,0 +1,6 @@
+GITR_ARGS += gitr_fstype=xfs
+GITR_ARGS += gitr_uses_no_devices='False'
+
+GITR_ARGS += gitr_device=$(subst ",,$(CONFIG_GITR_DEVICE))
+GITR_ARGS += gitr_label=$(subst ",,$(CONFIG_GITR_LABEL))
+GITR_ARGS += gitr_mount_opts='$(subst ",,$(CONFIG_GITR_MOUNT_OPTS))'
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH v2 7/7] gitr: Adjust the gen_hosts and gen_nodes playbooks
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
` (5 preceding siblings ...)
2024-01-03 15:08 ` [PATCH v2 6/7] Add a workflow for the git regression suite Chuck Lever
@ 2024-01-03 15:09 ` Chuck Lever
2024-01-03 17:10 ` [PATCH v2 0/7] Add a git regression workflow Jeff Layton
7 siblings, 0 replies; 23+ messages in thread
From: Chuck Lever @ 2024-01-03 15:09 UTC (permalink / raw)
To: kdevops
From: Chuck Lever <chuck.lever@oracle.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
playbooks/roles/gen_hosts/defaults/main.yml | 1 +
playbooks/roles/gen_hosts/tasks/main.yml | 13 +++++++++++++
playbooks/roles/gen_hosts/templates/gitr.j2 | 23 +++++++++++++++++++++++
playbooks/roles/gen_nodes/defaults/main.yml | 1 +
playbooks/roles/gen_nodes/tasks/main.yml | 15 +++++++++++++++
5 files changed, 53 insertions(+)
create mode 100644 playbooks/roles/gen_hosts/templates/gitr.j2
diff --git a/playbooks/roles/gen_hosts/defaults/main.yml b/playbooks/roles/gen_hosts/defaults/main.yml
index a62c53c64f53..ff1212a5d1e2 100644
--- a/playbooks/roles/gen_hosts/defaults/main.yml
+++ b/playbooks/roles/gen_hosts/defaults/main.yml
@@ -24,6 +24,7 @@ kdevops_workflow_enable_selftests: False
kdevops_workflow_enable_cxl: False
kdevops_workflow_enable_pynfs: False
+kdevops_workflow_enable_gitr: False
is_fstests: False
fstests_fstyp: "bogus"
diff --git a/playbooks/roles/gen_hosts/tasks/main.yml b/playbooks/roles/gen_hosts/tasks/main.yml
index 0c6029c4c7a7..826a58d31996 100644
--- a/playbooks/roles/gen_hosts/tasks/main.yml
+++ b/playbooks/roles/gen_hosts/tasks/main.yml
@@ -94,6 +94,19 @@
- kdevops_workflow_enable_pynfs
- ansible_hosts_template.stat.exists
+- name: Generate the Ansible hosts file for dedicated gitr workflow
+ tags: [ 'hosts' ]
+ template:
+ src: "{{ kdevops_hosts_template }}"
+ dest: "{{ topdir_path }}/{{ kdevops_hosts }}"
+ force: yes
+ trim_blocks: True
+ lstrip_blocks: True
+ when:
+ - kdevops_workflows_dedicated_workflow
+ - kdevops_workflow_enable_gitr
+ - ansible_hosts_template.stat.exists
+
- name: Set empty enabled test types list for fstests
tags: [ 'hosts' ]
set_fact:
diff --git a/playbooks/roles/gen_hosts/templates/gitr.j2 b/playbooks/roles/gen_hosts/templates/gitr.j2
new file mode 100644
index 000000000000..ba144bfde935
--- /dev/null
+++ b/playbooks/roles/gen_hosts/templates/gitr.j2
@@ -0,0 +1,23 @@
+[all]
+{{ kdevops_hosts_prefix }}
+{% if kdevops_baseline_and_dev == True %}
+{{ kdevops_hosts_prefix }}-dev
+{% endif %}
+[all:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+[baseline]
+{{ kdevops_hosts_prefix }}
+[baseline:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+[dev]
+{% if kdevops_baseline_and_dev %}
+{{ kdevops_hosts_prefix }}-dev
+{% endif %}
+[dev:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
+[nfsd]
+{% if nfsd_threads is defined %}
+{{ kdevops_hosts_prefix }}-nfsd
+{% endif %}
+[nfsd:vars]
+ansible_python_interpreter = "{{ kdevops_python_interpreter }}"
diff --git a/playbooks/roles/gen_nodes/defaults/main.yml b/playbooks/roles/gen_nodes/defaults/main.yml
index 066c6674347c..dc9864003482 100644
--- a/playbooks/roles/gen_nodes/defaults/main.yml
+++ b/playbooks/roles/gen_nodes/defaults/main.yml
@@ -10,6 +10,7 @@ kdevops_nodes: "vagrant/kdevops_nodes.yaml"
kdevops_baseline_and_dev: False
kdevops_workflow_enable_cxl: False
kdevops_workflow_enable_pynfs: False
+kdevops_workflow_enable_gitr: False
kdevops_workflow_enable_selftests: False
virtualbox_provider: False
diff --git a/playbooks/roles/gen_nodes/tasks/main.yml b/playbooks/roles/gen_nodes/tasks/main.yml
index c81350bc8085..2f5c48b6cdcf 100644
--- a/playbooks/roles/gen_nodes/tasks/main.yml
+++ b/playbooks/roles/gen_nodes/tasks/main.yml
@@ -132,6 +132,21 @@
- kdevops_workflow_enable_cxl
- ansible_nodes_template.stat.exists
+- name: Generate the gitr kdevops nodes file using {{ kdevops_nodes_template }} as jinja2 source template
+ tags: [ 'nodes' ]
+ vars:
+ node_template: "{{ kdevops_nodes_template | basename }}"
+ all_generic_nodes: "{{ generic_nodes }}"
+ nodes: "{{ all_generic_nodes }}"
+ template:
+ src: "{{ node_template }}"
+ dest: "{{ topdir_path }}/{{ kdevops_nodes }}"
+ force: yes
+ when:
+ - kdevops_workflows_dedicated_workflow
+ - kdevops_workflow_enable_gitr
+ - ansible_nodes_template.stat.exists
+
- name: Set empty enabled test types list for fstests
tags: [ 'hosts' ]
set_fact:
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/7] Add a git regression workflow
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
` (6 preceding siblings ...)
2024-01-03 15:09 ` [PATCH v2 7/7] gitr: Adjust the gen_hosts and gen_nodes playbooks Chuck Lever
@ 2024-01-03 17:10 ` Jeff Layton
2024-01-03 18:04 ` Chuck Lever III
7 siblings, 1 reply; 23+ messages in thread
From: Jeff Layton @ 2024-01-03 17:10 UTC (permalink / raw)
To: Chuck Lever, kdevops
On Wed, 2024-01-03 at 10:08 -0500, Chuck Lever wrote:
> Since the last series, I've added support for tmpfs and NFS mounts,
> in addition to a few other refinements and loads of testing. I'd
> like a little review before I push these.
>
This all looks pretty great to me. I don't see any need to wait on
merging it, if it seems to be working for you.
> We had talked before about setting up multiple concurrent mounts for
> running tests of different NFS versions in parallel. +1 for running
> tests in parallel, but I think we want to make this a more generic
> mechanism that can test other mount option combinations as well
> multiple NFS versions (and perhaps could be reused with other
> workflows).
>
> I don't have a specific proposal for that yet, but I'm interested in
> discussion.
>
Version selection is just a mount option anyway. Note that even with
fstests, we now can spin up a separate host for TLS testing, so there is
precedent for non-version host variants anyway.
With fstests now, we can test with these options:
- default mount options
- v4.0
- v3
- TLS (with default mount options)
It probably makes sense to test those configurations for the git
regression suite as well, since they offers pretty good coverage for NFS
client and server.
Are there other configurations that might make sense to test regularly?
It's not too hard to add them.
We probably could even add a "Custom" configuration too that takes
arbitrary mount option and export option strings. That'd make it simpler
to test more exotic configurations.
It would be nice to eventually share this sort of option handling across
different workflows, but that's a more difficult problem given that most
of them handle mounting quite differently.
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/7] Add a git regression workflow
2024-01-03 17:10 ` [PATCH v2 0/7] Add a git regression workflow Jeff Layton
@ 2024-01-03 18:04 ` Chuck Lever III
2024-01-04 16:04 ` Chuck Lever III
0 siblings, 1 reply; 23+ messages in thread
From: Chuck Lever III @ 2024-01-03 18:04 UTC (permalink / raw)
To: Jeff Layton; +Cc: Chuck Lever, kdevops@lists.linux.dev
> On Jan 3, 2024, at 12:10 PM, Jeff Layton <jlayton@kernel.org> wrote:
>
> On Wed, 2024-01-03 at 10:08 -0500, Chuck Lever wrote:
>> Since the last series, I've added support for tmpfs and NFS mounts,
>> in addition to a few other refinements and loads of testing. I'd
>> like a little review before I push these.
>>
>
> This all looks pretty great to me. I don't see any need to wait on
> merging it, if it seems to be working for you.
OK, I'll push later this week.
>> We had talked before about setting up multiple concurrent mounts for
>> running tests of different NFS versions in parallel. +1 for running
>> tests in parallel, but I think we want to make this a more generic
>> mechanism that can test other mount option combinations as well
>> multiple NFS versions (and perhaps could be reused with other
>> workflows).
>>
>> I don't have a specific proposal for that yet, but I'm interested in
>> discussion.
>>
>
> Version selection is just a mount option anyway. Note that even with
> fstests, we now can spin up a separate host for TLS testing, so there is
> precedent for non-version host variants anyway.
>
> With fstests now, we can test with these options:
>
> - default mount options
I'd like to see separate runs for 4.1 and 4.2. 4.2 has a bunch
of new mechanisms like READ_PLUS, COPY, and ALLOCATE that the
client is using now and are not used for 4.1 mounts.
> - v4.0
> - v3
> - TLS (with default mount options)
>
> It probably makes sense to test those configurations for the git
> regression suite as well, since they offers pretty good coverage for NFS
> client and server.
>
> Are there other configurations that might make sense to test regularly?
> It's not too hard to add them.
Some of the more interesting options are nocto, readdirplus,
lookupcache and sec=.
I'm testing a playbook that creates siw devices so that both
TCP and RDMA can be covered. RDMA support will also be good
for SMB Direct and NVMeoF.
We might want to add separate runs for pnfs (block) and fscache.
And we could even do some runs with and without delegation
enabled on the server.
> We probably could even add a "Custom" configuration too that takes
> arbitrary mount option and export option strings. That'd make it simpler
> to test more exotic configurations.
>
> It would be nice to eventually share this sort of option handling across
> different workflows, but that's a more difficult problem given that most
> of them handle mounting quite differently.
Well, the difference seems to be whether the mount requires a
block device or not. tmpfs and NFS don't need one. So, maybe
we could split the plays that mount into one that manages
the underlying device and another that handles mounting with a
standard set or a custom set of mount options. <shrug>
--
Chuck Lever
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/7] Add a git regression workflow
2024-01-03 18:04 ` Chuck Lever III
@ 2024-01-04 16:04 ` Chuck Lever III
2024-01-04 16:46 ` Jeff Layton
0 siblings, 1 reply; 23+ messages in thread
From: Chuck Lever III @ 2024-01-04 16:04 UTC (permalink / raw)
To: Jeff Layton; +Cc: Chuck Lever, kdevops@lists.linux.dev
> On Jan 3, 2024, at 1:04 PM, Chuck Lever III <chuck.lever@oracle.com> wrote:
>
>
>
>> On Jan 3, 2024, at 12:10 PM, Jeff Layton <jlayton@kernel.org> wrote:
>>
>> On Wed, 2024-01-03 at 10:08 -0500, Chuck Lever wrote:
>>> Since the last series, I've added support for tmpfs and NFS mounts,
>>> in addition to a few other refinements and loads of testing. I'd
>>> like a little review before I push these.
>>>
>>
>> This all looks pretty great to me. I don't see any need to wait on
>> merging it, if it seems to be working for you.
>
> OK, I'll push later this week.
Blargh.
I pushed the gitr patches this morning, but there are still
some problems.
I've been trying to run down the NFSv4.1 backchannel problem
so I haven't run "make gitr-baseline" recently -- I've been
running the test by hand on the target hosts.
But "make gitr-baseline" is still hitting this error after
the test passes:
TASK [gitr : Save git regression test output to a file] ************************************************************************************************************************************************
fatal: [cel]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout_lines'. 'dict object' has no attribute 'stdout_lines'\n\nThe error appears to be in '/home/cel/src/kdevops/playbooks/roles/gitr/tasks/main.yml': line 162, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Save git regression test output to a file\n ^ here\n"}
I can't seem to nail down why stdout_lines does not exist,
because sometimes it /does/ exist.
My Ansible fu is not strong.
--
Chuck Lever
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/7] Add a git regression workflow
2024-01-04 16:04 ` Chuck Lever III
@ 2024-01-04 16:46 ` Jeff Layton
2024-01-04 18:19 ` Chuck Lever III
0 siblings, 1 reply; 23+ messages in thread
From: Jeff Layton @ 2024-01-04 16:46 UTC (permalink / raw)
To: Chuck Lever III; +Cc: Chuck Lever, kdevops@lists.linux.dev
On Thu, 2024-01-04 at 16:04 +0000, Chuck Lever III wrote:
>
> > On Jan 3, 2024, at 1:04 PM, Chuck Lever III <chuck.lever@oracle.com> wrote:
> >
> >
> >
> > > On Jan 3, 2024, at 12:10 PM, Jeff Layton <jlayton@kernel.org> wrote:
> > >
> > > On Wed, 2024-01-03 at 10:08 -0500, Chuck Lever wrote:
> > > > Since the last series, I've added support for tmpfs and NFS mounts,
> > > > in addition to a few other refinements and loads of testing. I'd
> > > > like a little review before I push these.
> > > >
> > >
> > > This all looks pretty great to me. I don't see any need to wait on
> > > merging it, if it seems to be working for you.
> >
> > OK, I'll push later this week.
>
> Blargh.
>
> I pushed the gitr patches this morning, but there are still
> some problems.
>
> I've been trying to run down the NFSv4.1 backchannel problem
> so I haven't run "make gitr-baseline" recently -- I've been
> running the test by hand on the target hosts.
>
> But "make gitr-baseline" is still hitting this error after
> the test passes:
>
> TASK [gitr : Save git regression test output to a file] ************************************************************************************************************************************************
> fatal: [cel]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout_lines'. 'dict object' has no attribute 'stdout_lines'\n\nThe error appears to be in '/home/cel/src/kdevops/playbooks/roles/gitr/tasks/main.yml': line 162, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Save git regression test output to a file\n ^ here\n"}
>
> I can't seem to nail down why stdout_lines does not exist,
> because sometimes it /does/ exist.
>
> My Ansible fu is not strong.
>
>
---------------8<-------------------
- name: Run the full git regression suite
tags: [ 'run_tests' ]
community.general.make:
chdir: "{{ gitr_mnt }}/git"
target: "test"
jobs: "{{ nproc.stdout }}"
register: gitr_results
failed_when: False
changed_when: False
...
- name: Save git regression test output to a file
tags: [ 'copy_results' ]
ansible.builtin.copy:
content: "{{ gitr_results.stdout_lines|join('\n') }}"
dest: "{{ gitr_mnt }}/gitr.stdout"
---------------8<-------------------
Looking at the first play, you have "failed_when: False". Is it possible
the first play failed and didn't register a result? I'd look closely at
what's happening in that play, in any case...
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 0/7] Add a git regression workflow
2024-01-04 16:46 ` Jeff Layton
@ 2024-01-04 18:19 ` Chuck Lever III
0 siblings, 0 replies; 23+ messages in thread
From: Chuck Lever III @ 2024-01-04 18:19 UTC (permalink / raw)
To: Jeff Layton; +Cc: Chuck Lever, kdevops@lists.linux.dev
> On Jan 4, 2024, at 11:46 AM, Jeff Layton <jlayton@kernel.org> wrote:
>
> On Thu, 2024-01-04 at 16:04 +0000, Chuck Lever III wrote:
>>
>>> On Jan 3, 2024, at 1:04 PM, Chuck Lever III <chuck.lever@oracle.com> wrote:
>>>
>>>
>>>
>>>> On Jan 3, 2024, at 12:10 PM, Jeff Layton <jlayton@kernel.org> wrote:
>>>>
>>>> On Wed, 2024-01-03 at 10:08 -0500, Chuck Lever wrote:
>>>>> Since the last series, I've added support for tmpfs and NFS mounts,
>>>>> in addition to a few other refinements and loads of testing. I'd
>>>>> like a little review before I push these.
>>>>>
>>>>
>>>> This all looks pretty great to me. I don't see any need to wait on
>>>> merging it, if it seems to be working for you.
>>>
>>> OK, I'll push later this week.
>>
>> Blargh.
>>
>> I pushed the gitr patches this morning, but there are still
>> some problems.
>>
>> I've been trying to run down the NFSv4.1 backchannel problem
>> so I haven't run "make gitr-baseline" recently -- I've been
>> running the test by hand on the target hosts.
>>
>> But "make gitr-baseline" is still hitting this error after
>> the test passes:
>>
>> TASK [gitr : Save git regression test output to a file] ************************************************************************************************************************************************
>> fatal: [cel]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout_lines'. 'dict object' has no attribute 'stdout_lines'\n\nThe error appears to be in '/home/cel/src/kdevops/playbooks/roles/gitr/tasks/main.yml': line 162, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Save git regression test output to a file\n ^ here\n"}
>>
>> I can't seem to nail down why stdout_lines does not exist,
>> because sometimes it /does/ exist.
>>
>> My Ansible fu is not strong.
>>
>>
>
>
> ---------------8<-------------------
> - name: Run the full git regression suite
> tags: [ 'run_tests' ]
> community.general.make:
> chdir: "{{ gitr_mnt }}/git"
> target: "test"
> jobs: "{{ nproc.stdout }}"
> register: gitr_results
> failed_when: False
> changed_when: False
>
> ...
>
> - name: Save git regression test output to a file
> tags: [ 'copy_results' ]
> ansible.builtin.copy:
> content: "{{ gitr_results.stdout_lines|join('\n') }}"
> dest: "{{ gitr_mnt }}/gitr.stdout"
> ---------------8<-------------------
>
> Looking at the first play, you have "failed_when: False". Is it possible
> the first play failed and didn't register a result? I'd look closely at
> what's happening in that play, in any case...
One problem might be that the output of community.general.make
is voluminous, and might be overrunning the internals of
Ansible.
I'm going to try switching to running the tests in a "prove"
harness that outputs only the test's passes/fails count.
--
Chuck Lever
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 3/7] create_tmpfs: Enable mounting tmpfs file systems for testing
2024-01-03 15:08 ` [PATCH v2 3/7] create_tmpfs: Enable mounting tmpfs file systems for testing Chuck Lever
@ 2024-01-25 14:26 ` Luis Chamberlain
2024-01-25 14:39 ` Chuck Lever III
0 siblings, 1 reply; 23+ messages in thread
From: Luis Chamberlain @ 2024-01-25 14:26 UTC (permalink / raw)
To: Chuck Lever; +Cc: kdevops
On Wed, Jan 03, 2024 at 10:08:39AM -0500, Chuck Lever wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
>
> To enable tmpfs as a generic test target, add a playbook for
> mounting a tmpfs file system. The new playbook is similar to
> the create_partition playbook: it creates a persistent fstab
> entry, but does not need to reserve a block device for the
> new file system.
>
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
It would seem this is to use tmpfs for some purpose. Note that fstests
in kdevops already does have proper full profiles support to help test
tmpfs with all the possible bells and whistles available as different
profiles. That is, each possibe variation as a different guest / node.
That is being used by Daniel Gomez to first create a public baseline
for tmpfs so that we can then add support for large folios for it.
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 5/7] Add a playbook for the git regression workflow
2024-01-03 15:08 ` [PATCH v2 5/7] Add a playbook for the git regression workflow Chuck Lever
@ 2024-01-25 14:34 ` Luis Chamberlain
2024-01-25 14:41 ` Luis Chamberlain
2024-01-25 15:14 ` Luis Chamberlain
1 sibling, 1 reply; 23+ messages in thread
From: Luis Chamberlain @ 2024-01-25 14:34 UTC (permalink / raw)
To: Chuck Lever; +Cc: kdevops
On Wed, Jan 03, 2024 at 10:08:52AM -0500, Chuck Lever wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
>
> This workflow builds the git tool and runs its built-in regression
> suite on the target file system. The git regression suite acts as an
> extensive file system function test. This workflow runs the suite in
> multiple threads to add timing stress.
>
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
> playbooks/gitr.yml | 4
> playbooks/roles/gitr/defaults/main.yml | 9 +
> .../roles/gitr/tasks/install-deps/debian/main.yml | 28 +++
> playbooks/roles/gitr/tasks/install-deps/main.yml | 12 +
> .../roles/gitr/tasks/install-deps/redhat/main.yml | 38 ++++
> .../roles/gitr/tasks/install-deps/suse/main.yml | 24 ++
> playbooks/roles/gitr/tasks/main.yml | 214 ++++++++++++++++++++
> 7 files changed, 329 insertions(+)
> create mode 100644 playbooks/gitr.yml
> create mode 100644 playbooks/roles/gitr/defaults/main.yml
> create mode 100644 playbooks/roles/gitr/tasks/install-deps/debian/main.yml
> create mode 100644 playbooks/roles/gitr/tasks/install-deps/main.yml
> create mode 100644 playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
> create mode 100644 playbooks/roles/gitr/tasks/install-deps/suse/main.yml
> create mode 100644 playbooks/roles/gitr/tasks/main.yml
>
> diff --git a/playbooks/gitr.yml b/playbooks/gitr.yml
> new file mode 100644
> index 000000000000..4c3e3ba919d7
> --- /dev/null
> +++ b/playbooks/gitr.yml
> @@ -0,0 +1,4 @@
> +---
> +- hosts: all
> + roles:
> + - role: gitr
> diff --git a/playbooks/roles/gitr/defaults/main.yml b/playbooks/roles/gitr/defaults/main.yml
> new file mode 100644
> index 000000000000..fe2763aa8f73
> --- /dev/null
> +++ b/playbooks/roles/gitr/defaults/main.yml
> @@ -0,0 +1,9 @@
> +# SPDX-License-Identifier GPL-2.0+
> +---
> +# Our sensible defaults for the gitr role.
> +#
> +# The default is to not run the gitr tests.
> +kdevops_run_gitr: False
> +
> +gitr_uses_no_devices: False
> +gitr_nfs_use_kdevops_nfsd: False
> diff --git a/playbooks/roles/gitr/tasks/install-deps/debian/main.yml b/playbooks/roles/gitr/tasks/install-deps/debian/main.yml
> new file mode 100644
> index 000000000000..50d4ba1037f3
> --- /dev/null
> +++ b/playbooks/roles/gitr/tasks/install-deps/debian/main.yml
> @@ -0,0 +1,28 @@
> +---
> +- name: Install build dependencies for git
> + become: yes
> + become_method: sudo
> + apt:
> + name:
> + - gcc
> + - git-core
> + - gettext
> + - libcurl4-openssl-dev
> + - libexpat1-dev
> + - libssl-dev
> + - libz-dev
> + state: present
> + update_cache: yes
> + tags: [ 'gitr', 'deps' ]
> +
> +- name: Install run-time dependencies for git regression tests
> + become: yes
> + become_method: sudo
> + apt:
> + name:
> + - cvs
> + - cvsps
> + - sqlite3
> + state: present
> + update_cache: yes
> + tags: [ 'gitr', 'deps' ]
> diff --git a/playbooks/roles/gitr/tasks/install-deps/main.yml b/playbooks/roles/gitr/tasks/install-deps/main.yml
> new file mode 100644
> index 000000000000..c4f0bc1f2192
> --- /dev/null
> +++ b/playbooks/roles/gitr/tasks/install-deps/main.yml
> @@ -0,0 +1,12 @@
> +---
> +- name: Debian-specific set up
> + ansible.builtin.include_tasks: roles/gitr/tasks/install-deps/debian/main.yml
> + when: ansible_os_family == 'Debian'
> +
> +- name: SuSE-specific set up
> + ansible.builtin.include_tasks: roles/gitr/tasks/install-deps/suse/main.yml
> + when: ansible_os_family == 'Suse'
> +
> +- name: Red Hat-specific set up
> + ansible.builtin.include_tasks: roles/gitr/tasks/install-deps/redhat/main.yml
> + when: ansible_os_family == 'RedHat'
> diff --git a/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml b/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
> new file mode 100644
> index 000000000000..30fb0f9b16d5
> --- /dev/null
> +++ b/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
> @@ -0,0 +1,38 @@
> +---
> +- name: Install build dependencies for git
> + become: yes
> + become_method: sudo
> + yum:
> + update_cache: yes
> + name: "{{ packages }}"
> + retries: 3
> + delay: 5
> + register: result
> + until: result.rc == 0
> + vars:
> + packages:
> + - expat-devel
> + - gcc
> + - gettext
> + - git-core
> + - libcurl-devel
> + - openssl-devel
> + - zlib-devel
> +
> +- name: Install run-time dependencies for git regression tests
> + become: yes
> + become_method: sudo
> + yum:
> + update_cache: yes
> + name: "{{ packages }}"
> + retries: 3
> + delay: 5
> + register: result
> + until: result.rc == 0
> + vars:
> + packages:
> + - cvs
> + - cvsps
> + - git-email
> + - subversion
> +# - subversion-perl
> diff --git a/playbooks/roles/gitr/tasks/install-deps/suse/main.yml b/playbooks/roles/gitr/tasks/install-deps/suse/main.yml
> new file mode 100644
> index 000000000000..0db06c5b66d0
> --- /dev/null
> +++ b/playbooks/roles/gitr/tasks/install-deps/suse/main.yml
> @@ -0,0 +1,24 @@
> +---
> +- name: Install build dependencies for git
> + become: yes
> + become_method: sudo
> + zypper:
> + name:
> + - gcc
> + - gettext
> + - git-core
> + - libcurl-devel
> + - libexpat-devel
> + - libopenssl-1_1-devel
> + state: present
> +
> +- name: Install run-time dependencies for git regression tests
> + become: yes
> + become_method: sudo
> + zypper:
> + name:
> + - cvs
> + - cvsps
> + - subversion
> + - subversion-perl
> + state: present
> diff --git a/playbooks/roles/gitr/tasks/main.yml b/playbooks/roles/gitr/tasks/main.yml
> new file mode 100644
> index 000000000000..f225d547dd65
> --- /dev/null
> +++ b/playbooks/roles/gitr/tasks/main.yml
> @@ -0,0 +1,214 @@
> +---
> +- name: Import optional extra_args file
> + include_vars: "{{ item }}"
> + ignore_errors: yes
> + with_first_found:
> + - files:
> + - "../extra_vars.yml"
> + - "../extra_vars.yaml"
> + - "../extra_vars.json"
> + skip: true
> + tags: vars
> +
> +- name: Set the path where we collect gitr test results
> + tags: [ 'vars' ]
> + set_fact:
> + gitr_workflow_dir: "../workflows/gitr"
> +
> +- name: Clean up localhost results directory and files
> + tags: [ 'clean_local_results' ]
> + local_action: file path="{{ gitr_workflow_dir }}/results/" state=absent
> + run_once: true
There is an alternative to local_action which makes the task easier to
read. Something like this:
- name: Clean up localhost results directory and files
tags: [ 'clean_local_results' ]
file
path: "{{ gitr_workflow_dir }}/results/"
state: absent
delegate_to: localhost
run_once: true
It is easier to read because the module splits each line.
> +- name: Create the results directory
> + tags: [ 'first_run' ]
> + local_action: file path="{{ gitr_workflow_dir }}/results/" state=directory
> + run_once: true
Same here.
> +- name: git clone git
> + tags: [ 'gitr' ]
> + ansible.builtin.git:
> + repo: "{{ gitr_repo }}"
> + dest: "{{ gitr_mnt }}/git"
> + update: yes
> + version: "{{ gitr_repo_commit }}"
> + retries: 3
> + delay: 5
> + register: result
> + until: not result.failed
> +
> +- name: Get nproc
> + tags: [ 'gitr', 'build', 'vars' ]
> + ansible.builtin.command: "{{ num_jobs }}"
> + register: nproc
> + changed_when: False
> +
> +- name: Build git
> + tags: [ 'gitr', 'build' ]
> + community.general.make:
> + chdir: "{{ gitr_mnt }}/git"
> + jobs: "{{ nproc.stdout }}"
> +
> +- name: Run the full git regression suite
> + tags: [ 'run_tests' ]
> + community.general.make:
> + chdir: "{{ gitr_mnt }}/git"
> + target: "test"
> + jobs: "{{ nproc.stdout }}"
> + register: gitr_results
> + failed_when: False
> + changed_when: False
> +
> +- name: Capture RPC statistics
> + tags: [ 'copy_results' ]
> + ansible.builtin.command: "mountstats --rpc"
> + register: rpc_results
> + when:
> + - gitr_fstype == "nfs"
> +
> +- name: Capture transport statistics
> + tags: [ 'copy_results' ]
> + ansible.builtin.command: "mountstats --xprt"
> + register: xprt_results
> + when:
> + - gitr_fstype == "nfs"
> +
> +- name: Save git regression test output to a file
> + tags: [ 'copy_results' ]
> + ansible.builtin.copy:
> + content: "{{ gitr_results.stdout_lines|join('\n') }}"
> + dest: "{{ gitr_mnt }}/gitr.stdout"
> +
> +- name: Save RPC statistics to a file
> + tags: [ 'copy_results' ]
> + ansible.builtin.copy:
> + content: "{{ rpc_results.stdout_lines|join('\n') }}"
> + dest: "{{ gitr_mnt }}/gitr.rpc"
> + when:
> + - gitr_fstype == "nfs"
> +
> +- name: Save transport statistics to a file
> + tags: [ 'copy_results' ]
> + ansible.builtin.copy:
> + content: "{{ xprt_results.stdout_lines|join('\n') }}"
> + dest: "{{ gitr_mnt }}/gitr.xprt"
> + when:
> + - gitr_fstype == "nfs"
> +
> +- name: Get used target kernel version
> + tags: [ 'copy_results' ]
> + ansible.builtin.command: "uname -r"
> + register: uname_cmd
> +
> +- name: Store kernel_rev variable
> + tags: [ 'copy_results' ]
> + set_fact:
> + kernel_rev: "{{ uname_cmd.stdout_lines | regex_replace('\\]') | regex_replace('\\[') | replace(\"'\",'') }}"
> + run_once: true
> +
> +- name: Copy regression results over
> + tags: [ 'copy_results' ]
> + ansible.builtin.fetch:
> + src: "{{ gitr_mnt }}/gitr.stdout"
> + dest: "{{ gitr_workflow_dir }}/results/{{ kernel_rev }}.stdout"
> + flat: yes
> +
> +- name: Copy RPC statistics over
> + tags: [ 'copy_results' ]
> + ansible.builtin.fetch:
> + src: "{{ gitr_mnt }}/gitr.rpc"
> + dest: "{{ gitr_workflow_dir }}/results/{{ kernel_rev }}.rpc"
> + flat: yes
> + when:
> + - gitr_fstype == "nfs"
> +
> +- name: Copy transport statistics over
> + tags: [ 'copy_results' ]
> + ansible.builtin.fetch:
> + src: "{{ gitr_mnt }}/gitr.xprt"
> + dest: "{{ gitr_workflow_dir }}/results/{{ kernel_rev }}.xprt"
> + flat: yes
> + when:
> + - gitr_fstype == "nfs"
> +
> +# XXX: This is working now, but it should copy the failures into a log file
> +- name: Report failures
> + tags: [ 'copy_results' ]
> + debug:
> + msg: "{{ gitr_results.stderr_lines }}"
> + when: gitr_results.stderr | length > 0
>
>
>
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 3/7] create_tmpfs: Enable mounting tmpfs file systems for testing
2024-01-25 14:26 ` Luis Chamberlain
@ 2024-01-25 14:39 ` Chuck Lever III
2024-01-25 15:44 ` Luis Chamberlain
0 siblings, 1 reply; 23+ messages in thread
From: Chuck Lever III @ 2024-01-25 14:39 UTC (permalink / raw)
To: Luis Chamberlain; +Cc: Chuck Lever, kdevops@lists.linux.dev
> On Jan 25, 2024, at 9:26 AM, Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Wed, Jan 03, 2024 at 10:08:39AM -0500, Chuck Lever wrote:
>> From: Chuck Lever <chuck.lever@oracle.com>
>>
>> To enable tmpfs as a generic test target, add a playbook for
>> mounting a tmpfs file system. The new playbook is similar to
>> the create_partition playbook: it creates a persistent fstab
>> entry, but does not need to reserve a block device for the
>> new file system.
>>
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>
> It would seem this is to use tmpfs for some purpose. Note that fstests
> in kdevops already does have proper full profiles support to help test
> tmpfs with all the possible bells and whistles available as different
> profiles. That is, each possibe variation as a different guest / node.
> That is being used by Daniel Gomez to first create a public baseline
> for tmpfs so that we can then add support for large folios for it.
The fstests workflow has support for tmpfs, true.
But we need to mount tmpfs shares in the NFS server, and the new
gitr workflow needs to mount tmpfs for testing. That is the
underlying need being addressed here.
--
Chuck Lever
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 5/7] Add a playbook for the git regression workflow
2024-01-25 14:34 ` Luis Chamberlain
@ 2024-01-25 14:41 ` Luis Chamberlain
0 siblings, 0 replies; 23+ messages in thread
From: Luis Chamberlain @ 2024-01-25 14:41 UTC (permalink / raw)
To: Chuck Lever; +Cc: kdevops
On Thu, Jan 25, 2024 at 06:34:28AM -0800, Luis Chamberlain wrote:
> There is an alternative to local_action which makes the task easier to
> read. Something like this:
>
>
> - name: Clean up localhost results directory and files
> tags: [ 'clean_local_results' ]
> file
> path: "{{ gitr_workflow_dir }}/results/"
> state: absent
> delegate_to: localhost
> run_once: true
>
>
> It is easier to read because the module splits each line.
>
> > +- name: Create the results directory
> > + tags: [ 'first_run' ]
> > + local_action: file path="{{ gitr_workflow_dir }}/results/" state=directory
> > + run_once: true
>
> Same here.
Sorry I hit send too soon. I trimmed the reply here as this hopefuly
saves you time in my reply. I'll reply with more review next.
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 5/7] Add a playbook for the git regression workflow
2024-01-03 15:08 ` [PATCH v2 5/7] Add a playbook for the git regression workflow Chuck Lever
2024-01-25 14:34 ` Luis Chamberlain
@ 2024-01-25 15:14 ` Luis Chamberlain
2024-01-25 16:12 ` Chuck Lever III
1 sibling, 1 reply; 23+ messages in thread
From: Luis Chamberlain @ 2024-01-25 15:14 UTC (permalink / raw)
To: Chuck Lever; +Cc: kdevops
On Wed, Jan 03, 2024 at 10:08:52AM -0500, Chuck Lever wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
> diff --git a/playbooks/roles/gitr/defaults/main.yml b/playbooks/roles/gitr/defaults/main.yml
> new file mode 100644
> index 000000000000..fe2763aa8f73
> --- /dev/null
> +++ b/playbooks/roles/gitr/defaults/main.yml
> @@ -0,0 +1,9 @@
> +# SPDX-License-Identifier GPL-2.0+
> +---
> +# Our sensible defaults for the gitr role.
> +#
> +# The default is to not run the gitr tests.
> +kdevops_run_gitr: False
> +
> +gitr_uses_no_devices: False
> +gitr_nfs_use_kdevops_nfsd: False
It would seem this is missing quite a bit of variables such as:
gitr_device
gitr_fstype
gitr_label
gitr_label
gitr_mnt
gitr_mount_opts
gitr_repo
gitr_repo_commit
Note that if it uses some of the variables which other Kconfig magic
adds, the role should still define the defaults too, it is no harm, it
only helps to bring clarity that the role uses them too.
> diff --git a/playbooks/roles/gitr/tasks/install-deps/main.yml b/playbooks/roles/gitr/tasks/install-deps/main.yml
> diff --git a/playbooks/roles/gitr/tasks/install-deps/debian/main.yml b/playbooks/roles/gitr/tasks/install-deps/debian/main.yml
> diff --git a/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml b/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
> diff --git a/playbooks/roles/gitr/tasks/install-deps/suse/main.yml b/playbooks/roles/gitr/tasks/install-deps/suse/main.yml
Thanks for doing all this.
> +- name: Run the full git regression suite
> + tags: [ 'run_tests' ]
> + community.general.make:
> + chdir: "{{ gitr_mnt }}/git"
> + target: "test"
> + jobs: "{{ nproc.stdout }}"
> + register: gitr_results
> + failed_when: False
> + changed_when: False
Ah, neat.
> +# XXX: This is working now, but it should copy the failures into a log file
> +- name: Report failures
> + tags: [ 'copy_results' ]
> + debug:
> + msg: "{{ gitr_results.stderr_lines }}"
> + when: gitr_results.stderr | length > 0
Curious, has this found issues on filesystems before?
I was not aware of the extensive set of tests on git's 'make test', it
certainly is a nice collection, and I can see it being very useful, but
I am just curious if it already has had history of finding some fs bugs.
If so, we could certainly augment our fs testing to run these too after
running fstests.
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 6/7] Add a workflow for the git regression suite
2024-01-03 15:08 ` [PATCH v2 6/7] Add a workflow for the git regression suite Chuck Lever
@ 2024-01-25 15:39 ` Luis Chamberlain
2024-01-25 15:56 ` Luis Chamberlain
0 siblings, 1 reply; 23+ messages in thread
From: Luis Chamberlain @ 2024-01-25 15:39 UTC (permalink / raw)
To: Chuck Lever; +Cc: kdevops
On Wed, Jan 03, 2024 at 10:08:58AM -0500, Chuck Lever wrote:
> diff --git a/docs/gitr.md b/docs/gitr.md
> new file mode 100644
> index 000000000000..73e5c42ec0e4
> --- /dev/null
> +++ b/docs/gitr.md
> @@ -0,0 +1,22 @@
> +# kdevops git regression suite
> +
> +kdevops can run the git tool's internal regression suite against
> +a target file system.
> +
> +Run `make menuconfig` and select:
> +
> + Target workflows -> Dedicated target Linux test workflow ->gitr
So "dedicated workflow" comes from the idea that a workflow could
be enabled solely to run alone, and only that workflow on a kdevops
configuration. Although kdevops started by providing the fstests target
to run xfstests, it later grew to also support blktests, selftests, etc.
After blktests was added it came to a point that one wanted to run both
fstests and blktests targets in one same configuration. So, we needed
a way to allow fstests and blktests to not be mutually exclusive.
The main difference today between dedicated workflows and non-dedicated
ones are that dedicated worfklows suggest a set of target nodes one
could end up creating to test that target workflow. So for testing
XFS with fstests you could end up with about 10 nodes, one per each
target mkfs.xfs profile we support. With LBS this is augmented quite
a bit, as we're also testing 8k, 16k, 32k and 64k block sizes for each
profile. You could opt-in to only allow one profile so you could also
just end up with xfs-reflink for instance to test.
Non-dedicated workflows use the default hosts / target nodes, which is
just one node.
Today for instance the CXL dedicated workflow is rather limited as it
is only instantiating *one* node, and in the future this should be
augmented to allow it to build a node per CXL topology.
The gitr workflow seems like a example new shared workflow. I am
working on a new set of targets which also will be useful for
all the one which allows us to use systemd remote journals, but
that is a bit more involved as we want to set up the server on
the localhost at bringup and then the systemd remote journal on each
node on bringup before devconfig playbook is run in full.
The gitr workflow then seems very generic and so although I think
it can very well be a dedicated workflow, if so, it should support
then bringing up all possible filesystems, one per node, and testing
'make gitr-baseline' for git on each node. But it would also seem very
useful to allow the gitr workflow to be supported in a non-dedicated
workflow, so that the commands are used for another dedicated workflow.
That would allow us then to use 'make fstests-baseline' to test
filesystems with fstests and then after also use 'make gitr-baseline'
so to test git's `make test` regression suite against all filesystem
profiles we are testing.
> +Then configure the test parameters by going to:
> +
> + Target workflows -> Configure and run the git regression suite
> +
> +Choose the file system type to test and the location of the repo
> +that contains the version of git you want to use for the test.
> +
> +Then, run:
> +
> + * `make`
> + * `make bringup`
> + * `make gitr`
I am working on splitting up bringup a bit more, so we will have
provisioning split up in more steps. Today we have bringup do:
a) ssh update so you can ssh to each node
b) run the devconfig playbook, this in turn will only issue the
install-deps if you enabled the kdevops devconfig kernel developer
tools install option (disabled by default).
Since we default to not install the devconfig kernel developer tools
by default today, but we want to install some things on bring up based
on your config, I'm splitting the above two steps with one more in the
midle:
a) ssh update
b) install early deps
c) devconfig playbook
So b) is new, and it is how I am going to fix sytemd-remote-journal.
You could leverage that split by augmenteing a variable I will add
with your 'gitr' target on the early deps. This way, at bringup, the
above commands are reduced from:
* `make`
* `make bringup`
* `make gitr`
To the following two:
* `make`
* `make bringup`
The target `make gitr` will obviously still be avaialble though.
> diff --git a/kconfigs/workflows/Kconfig b/kconfigs/workflows/Kconfig
> index 652f65de7647..1a904258734f 100644
> --- a/kconfigs/workflows/Kconfig
> +++ b/kconfigs/workflows/Kconfig
> @@ -152,6 +152,13 @@ config KDEVOPS_WORKFLOW_DEDICATE_SELFTESTS
> selftests work. Note that you will need at least 8 GiB of RAM
> for your target guests.
>
> +config KDEVOPS_WORKFLOW_DEDICATE_GITR
> + bool "gitr"
> + select KDEVOPS_WORKFLOW_ENABLE_GITR
> + help
> + This will dedicate your configuration to running only the
> + multi-threaded git regression workflow.
> +
> endchoice
>
> endif
> @@ -209,6 +216,14 @@ config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_SELFTESTS
> Select this option if you are doing Linux kernel developent and
> testing with sefltests.
>
> +config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_GITR
> + bool "gitr"
> + select KDEVOPS_WORKFLOW_ENABLE_GITR
> + depends on VAGRANT || TERRAFORM_PRIVATE_NET
> + help
> + Select this option if you want to configure the deployment for
> + testing via the multi-threaded git regression workflow.
> +
> endif # !WORKFLOWS_DEDICATED_WORKFLOW
>
> config KDEVOPS_WORKFLOW_ENABLE_FSTESTS
> @@ -262,6 +277,16 @@ source "workflows/selftests/Kconfig"
> endmenu
> endif # KDEVOPS_WORKFLOW_ENABLE_SELFTESTS
>
> +config KDEVOPS_WORKFLOW_ENABLE_GITR
> + bool
> + default y if KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_GITR || KDEVOPS_WORKFLOW_DEDICATE_GITR
> +
> +if KDEVOPS_WORKFLOW_ENABLE_GITR
> +menu "Configure and run the git regression suite"
> +source "workflows/gitr/Kconfig"
> +endmenu
> +endif # KDEVOPS_WORKFLOW_ENABLE_GITR
> +
> config KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT
> bool
> default y if KDEVOPS_WORKFLOW_ENABLE_FSTESTS || KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
> diff --git a/scripts/workflows/gitr/run_kernel_ci.sh b/scripts/workflows/gitr/run_kernel_ci.sh
> new file mode 120000
> index 000000000000..4fd5dc5e8e13
> --- /dev/null
> +++ b/scripts/workflows/gitr/run_kernel_ci.sh
> @@ -0,0 +1 @@
> +../generic/run_kernel_ci.sh
> \ No newline at end of file
> diff --git a/scripts/workflows/gitr/run_kernel_ci_kotd.sh b/scripts/workflows/gitr/run_kernel_ci_kotd.sh
> new file mode 120000
> index 000000000000..8f94d6ba4f52
> --- /dev/null
> +++ b/scripts/workflows/gitr/run_kernel_ci_kotd.sh
> @@ -0,0 +1 @@
> +../kotd/run_kernel_ci_kotd.sh
> \ No newline at end of file
> diff --git a/scripts/workflows/gitr/run_loop.sh b/scripts/workflows/gitr/run_loop.sh
> new file mode 100755
> index 000000000000..2ba961e2f4cb
> --- /dev/null
> +++ b/scripts/workflows/gitr/run_loop.sh
> @@ -0,0 +1,58 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: copyleft-next-0.3.1
> +
> +# Part of kdevops kernel-ci, this is the script which will run the test workflow
> +# as many times as indicated up to CONFIG_KERNEL_CI_STEADY_STATE_GOAL
> +
> +source ${TOPDIR}/.config
> +source ${TOPDIR}/scripts/lib.sh
> +
> +WORKFLOWDIR=${TOPDIR}/workflows/gitr
> +
> +COUNT=1
> +
> +run_loop()
> +{
> + while true; do
> + echo "== kernel-ci gitr test loop $COUNT start: $(date)" > $KERNEL_CI_FAIL_LOG
> + echo "/usr/bin/time -f %E make gitr-baseline" >> $KERNEL_CI_FAIL_LOG
> + /usr/bin/time -p -o $KERNEL_CI_LOGTIME make gitr-baseline >> $KERNEL_CI_FAIL_LOG
> + ANSIBLE_CALL_RET=$?
> + echo "End $COUNT: $(date)" >> $KERNEL_CI_FAIL_LOG
> + cat $KERNEL_CI_LOGTIME >> $KERNEL_CI_FAIL_LOG
> + echo "git status:" >> $KERNEL_CI_FAIL_LOG
> + git status >> $KERNEL_CI_FAIL_LOG
> + echo "Results:" >> $KERNEL_CI_FAIL_LOG
> +
> + rm -f $KERNEL_CI_DIFF_LOG
> +
> + if [[ "$ANSIBLE_CALL_RET" -ne 0 ]]; then
> + echo "Test $COUNT: FAILED!" >> $KERNEL_CI_DIFF_LOG
> + echo "== Test loop count $COUNT" >> $KERNEL_CI_DIFF_LOG
> + echo "$(git describe)" >> $KERNEL_CI_DIFF_LOG
> + git diff >> $KERNEL_CI_DIFF_LOG
> + cat $KERNEL_CI_DIFF_LOG >> $KERNEL_CI_FAIL_LOG
> + cat $KERNEL_CI_FAIL_LOG >> $KERNEL_CI_FULL_LOG
> + echo $COUNT > $KERNEL_CI_FAIL_FILE
> + exit 1
> + else
> + echo "Test $COUNT: OK!" >> $KERNEL_CI_FAIL_LOG
> + echo "----------------------------------------------------------------" >> $KERNEL_CI_FAIL_LOG
> + cat $KERNEL_CI_FAIL_LOG >> $KERNEL_CI_FULL_LOG
> + fi
> +
> + # This let's us keep track of which loop count was last successful
> + echo $COUNT > $KERNEL_CI_OK_FILE
> +
> + let COUNT=$COUNT+1
> + if [[ "$CONFIG_KERNEL_CI_ENABLE_STEADY_STATE" == "y" &&
> + "$COUNT" -gt "$CONFIG_KERNEL_CI_STEADY_STATE_GOAL" ]]; then
> + exit 0
> + fi
> + sleep 1
> + done
> +}
> +
> +rm -f $KERNEL_CI_FAIL_FILE $KERNEL_CI_OK_FILE
> +echo "= kernel-ci full log" > $KERNEL_CI_FULL_LOG
> +run_loop
> diff --git a/workflows/Makefile b/workflows/Makefile
> index 1c0e13a93f38..122eb98bb66b 100644
> --- a/workflows/Makefile
> +++ b/workflows/Makefile
> @@ -43,6 +43,11 @@ ifeq (y,$(CONFIG_WORKFLOWS_REBOOT_LIMIT))
> include workflows/demos/reboot-limit/Makefile
> endif # CONFIG_WORKFLOWS_REBOOT_LIMIT == y
>
> +ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_GITR))
> +WORKFLOW_ARGS += kdevops_workflow_enable_gitr='True'
> +include workflows/gitr/Makefile
> +endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_GITR == y
> +
> ANSIBLE_EXTRA_ARGS += $(WORKFLOW_ARGS)
> ANSIBLE_EXTRA_ARGS_SEPARATED += $(WORKFLOW_ARGS_SEPARATED)
> ANSIBLE_EXTRA_ARGS_DIRECT += $(WORKFLOW_ARGS_DIRECT)
> diff --git a/workflows/gitr/Kconfig b/workflows/gitr/Kconfig
> new file mode 100644
> index 000000000000..14f69676c5e8
> --- /dev/null
> +++ b/workflows/gitr/Kconfig
> +config GITR_REPO
> + string "The git tool repository to clone"
> + default "https://git.kernel.org/pub/scm/git/git.git"
> + help
> + The git tool repository to clone.
> +
Neat, a respective augment for the mirror service would be nice. But
that can be done later once folks start using this more.
Cool stuff!
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 3/7] create_tmpfs: Enable mounting tmpfs file systems for testing
2024-01-25 14:39 ` Chuck Lever III
@ 2024-01-25 15:44 ` Luis Chamberlain
0 siblings, 0 replies; 23+ messages in thread
From: Luis Chamberlain @ 2024-01-25 15:44 UTC (permalink / raw)
To: Chuck Lever III; +Cc: Chuck Lever, kdevops@lists.linux.dev
On Thu, Jan 25, 2024 at 02:39:51PM +0000, Chuck Lever III wrote:
>
>
> > On Jan 25, 2024, at 9:26 AM, Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > On Wed, Jan 03, 2024 at 10:08:39AM -0500, Chuck Lever wrote:
> >> From: Chuck Lever <chuck.lever@oracle.com>
> >>
> >> To enable tmpfs as a generic test target, add a playbook for
> >> mounting a tmpfs file system. The new playbook is similar to
> >> the create_partition playbook: it creates a persistent fstab
> >> entry, but does not need to reserve a block device for the
> >> new file system.
> >>
> >> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> >
> > It would seem this is to use tmpfs for some purpose. Note that fstests
> > in kdevops already does have proper full profiles support to help test
> > tmpfs with all the possible bells and whistles available as different
> > profiles. That is, each possibe variation as a different guest / node.
> > That is being used by Daniel Gomez to first create a public baseline
> > for tmpfs so that we can then add support for large folios for it.
>
> The fstests workflow has support for tmpfs, true.
>
> But we need to mount tmpfs shares in the NFS server, and the new
> gitr workflow needs to mount tmpfs for testing. That is the
> underlying need being addressed here.
Sure, I get it, now that I read the rest of the patches the goal seems
clear. The notes I made about having gitr support when one is already
enabling a dedicated fstests workflow in the other thread would allow us
to augment existing tmpfs fstests testing with this, so we can do both
in one kdevops setup for tmpfs:
make fstests-baseline
make gitr-baseline
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 6/7] Add a workflow for the git regression suite
2024-01-25 15:39 ` Luis Chamberlain
@ 2024-01-25 15:56 ` Luis Chamberlain
0 siblings, 0 replies; 23+ messages in thread
From: Luis Chamberlain @ 2024-01-25 15:56 UTC (permalink / raw)
To: Chuck Lever; +Cc: kdevops
On Thu, Jan 25, 2024 at 07:39:40AM -0800, Luis Chamberlain wrote:
> Cool stuff!
FWIW stats on the latest git:
fixed 0
success 29591
failed 0
broken 286
total 30173
So 29k tests is pretty nice set of tests, still curious if its found any
issues, so to prioritize including this as part of our filesystem
developer workflows when testing filesystems.
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 5/7] Add a playbook for the git regression workflow
2024-01-25 15:14 ` Luis Chamberlain
@ 2024-01-25 16:12 ` Chuck Lever III
2024-01-25 16:17 ` Luis Chamberlain
0 siblings, 1 reply; 23+ messages in thread
From: Chuck Lever III @ 2024-01-25 16:12 UTC (permalink / raw)
To: Luis Chamberlain; +Cc: Chuck Lever, kdevops@lists.linux.dev
> On Jan 25, 2024, at 10:14 AM, Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Wed, Jan 03, 2024 at 10:08:52AM -0500, Chuck Lever wrote:
>> From: Chuck Lever <chuck.lever@oracle.com>
>> diff --git a/playbooks/roles/gitr/defaults/main.yml b/playbooks/roles/gitr/defaults/main.yml
>> new file mode 100644
>> index 000000000000..fe2763aa8f73
>> --- /dev/null
>> +++ b/playbooks/roles/gitr/defaults/main.yml
>> @@ -0,0 +1,9 @@
>> +# SPDX-License-Identifier GPL-2.0+
>> +---
>> +# Our sensible defaults for the gitr role.
>> +#
>> +# The default is to not run the gitr tests.
>> +kdevops_run_gitr: False
>> +
>> +gitr_uses_no_devices: False
>> +gitr_nfs_use_kdevops_nfsd: False
>
> It would seem this is missing quite a bit of variables such as:
>
> gitr_device
> gitr_fstype
> gitr_label
> gitr_label
> gitr_mnt
> gitr_mount_opts
> gitr_repo
> gitr_repo_commit
>
> Note that if it uses some of the variables which other Kconfig magic
> adds, the role should still define the defaults too, it is no harm, it
> only helps to bring clarity that the role uses them too.
I looked for precedent in some of the other defaults/main.yml
file, and the minimal set seemed to be preferred, though there
are no comments about it anywhere.
I think at this point, the basic gitr workflow as it has been
merged is ready to take a lot of polish and clean up as you
see fit.
>> diff --git a/playbooks/roles/gitr/tasks/install-deps/main.yml b/playbooks/roles/gitr/tasks/install-deps/main.yml
>> diff --git a/playbooks/roles/gitr/tasks/install-deps/debian/main.yml b/playbooks/roles/gitr/tasks/install-deps/debian/main.yml
>> diff --git a/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml b/playbooks/roles/gitr/tasks/install-deps/redhat/main.yml
>> diff --git a/playbooks/roles/gitr/tasks/install-deps/suse/main.yml b/playbooks/roles/gitr/tasks/install-deps/suse/main.yml
>
> Thanks for doing all this.
>
>> +- name: Run the full git regression suite
>> + tags: [ 'run_tests' ]
>> + community.general.make:
>> + chdir: "{{ gitr_mnt }}/git"
>> + target: "test"
>> + jobs: "{{ nproc.stdout }}"
>> + register: gitr_results
>> + failed_when: False
>> + changed_when: False
>
> Ah, neat.
And this has been replaced with "prove" by a later patch.
>> +# XXX: This is working now, but it should copy the failures into a log file
>> +- name: Report failures
>> + tags: [ 'copy_results' ]
>> + debug:
>> + msg: "{{ gitr_results.stderr_lines }}"
>> + when: gitr_results.stderr | length > 0
>
> Curious, has this found issues on filesystems before?
Absolutely. We find a problem using this test every second or
third kernel release. It's the multi-threading that exposes
the most interesting races, but sometimes a test failure can
be reproduced with just one of these subtests.
NFS has benefitted considerably, I think, because the git
community themselves test pretty thoroughly on local
filesystems but not on NFS mounts.
I added tmpfs support because I recently added stable
directory cookie support to tmpfs to fix a problem found
when testing tmpfs NFS exports.
> I was not aware of the extensive set of tests on git's 'make test', it
> certainly is a nice collection, and I can see it being very useful, but
> I am just curious if it already has had history of finding some fs bugs.
>
> If so, we could certainly augment our fs testing to run these too after
> running fstests.
Yes I hope others find it useful. I think it's pretty relevant
to ensure the kernel's preferred version control system is
working reliably on the premier tier of filesystems. :-)
And maybe the git community might find the kdevops framework
useful for their own CI needs, since it gives them quick
access to test results on several filesystem implementations.
--
Chuck Lever
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH v2 5/7] Add a playbook for the git regression workflow
2024-01-25 16:12 ` Chuck Lever III
@ 2024-01-25 16:17 ` Luis Chamberlain
0 siblings, 0 replies; 23+ messages in thread
From: Luis Chamberlain @ 2024-01-25 16:17 UTC (permalink / raw)
To: Chuck Lever III; +Cc: Chuck Lever, kdevops@lists.linux.dev
On Thu, Jan 25, 2024 at 04:12:46PM +0000, Chuck Lever III wrote:
>
> > On Jan 25, 2024, at 10:14 AM, Luis Chamberlain <mcgrof@kernel.org> wrote:
If it doesn't break, don't fix it will be my stance :P
> And this has been replaced with "prove" by a later patch.
I see!
> >> +# XXX: This is working now, but it should copy the failures into a log file
> >> +- name: Report failures
> >> + tags: [ 'copy_results' ]
> >> + debug:
> >> + msg: "{{ gitr_results.stderr_lines }}"
> >> + when: gitr_results.stderr | length > 0
> >
> > Curious, has this found issues on filesystems before?
>
> Absolutely. We find a problem using this test every second or
> third kernel release. It's the multi-threading that exposes
> the most interesting races, but sometimes a test failure can
> be reproduced with just one of these subtests.
>
> NFS has benefitted considerably, I think, because the git
> community themselves test pretty thoroughly on local
> filesystems but not on NFS mounts.
>
Great to know thanks!
> > I was not aware of the extensive set of tests on git's 'make test', it
> > certainly is a nice collection, and I can see it being very useful, but
> > I am just curious if it already has had history of finding some fs bugs.
> >
> > If so, we could certainly augment our fs testing to run these too after
> > running fstests.
>
> Yes I hope others find it useful. I think it's pretty relevant
> to ensure the kernel's preferred version control system is
> working reliably on the premier tier of filesystems. :-)
Fantastic, will do so and also suggest it to other folks!
> And maybe the git community might find the kdevops framework
> useful for their own CI needs, since it gives them quick
> access to test results on several filesystem implementations.
Indeed, hope they find it useful, thanks for doing this!
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2024-01-25 16:17 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-01-03 15:08 [PATCH v2 0/7] Add a git regression workflow Chuck Lever
2024-01-03 15:08 ` [PATCH v2 1/7] nfsd_add_export: Add an export_options variable Chuck Lever
2024-01-03 15:08 ` [PATCH v2 2/7] nfsd_add_export: Set fsid= on tmpfs exports Chuck Lever
2024-01-03 15:08 ` [PATCH v2 3/7] create_tmpfs: Enable mounting tmpfs file systems for testing Chuck Lever
2024-01-25 14:26 ` Luis Chamberlain
2024-01-25 14:39 ` Chuck Lever III
2024-01-25 15:44 ` Luis Chamberlain
2024-01-03 15:08 ` [PATCH v2 4/7] create_nfs_mount: Enable testing arbitrary NFS mounts Chuck Lever
2024-01-03 15:08 ` [PATCH v2 5/7] Add a playbook for the git regression workflow Chuck Lever
2024-01-25 14:34 ` Luis Chamberlain
2024-01-25 14:41 ` Luis Chamberlain
2024-01-25 15:14 ` Luis Chamberlain
2024-01-25 16:12 ` Chuck Lever III
2024-01-25 16:17 ` Luis Chamberlain
2024-01-03 15:08 ` [PATCH v2 6/7] Add a workflow for the git regression suite Chuck Lever
2024-01-25 15:39 ` Luis Chamberlain
2024-01-25 15:56 ` Luis Chamberlain
2024-01-03 15:09 ` [PATCH v2 7/7] gitr: Adjust the gen_hosts and gen_nodes playbooks Chuck Lever
2024-01-03 17:10 ` [PATCH v2 0/7] Add a git regression workflow Jeff Layton
2024-01-03 18:04 ` Chuck Lever III
2024-01-04 16:04 ` Chuck Lever III
2024-01-04 16:46 ` Jeff Layton
2024-01-04 18:19 ` Chuck Lever III
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox