[PATCH] gitr: Make gitr work on RHEL/CentOS

[PATCH] gitr: Make gitr work on RHEL/CentOS

[Scott Mayhew]

RHEL and CentOS do not ship cvsps or perl-TAP-Harness-Archive.  For the
former, we'll have to build it from git.  For the latter, we can install
it from CPAN, but we'll also have to install the perl-App-cpanminus
package to do so.

Also add two other dependencies.  First is 'make', which is currently
only installed if you have KDEVOPS_TRY_INSTALL_KDEV_TOOLS enabled in
your config.  Second is 'tar' which is not installed in the RHEL
virt-builder images by default ('tar' isn't part of the @core dnf
package group).

Signed-off-by: Scott Mayhew <smayhew@redhat.com>
---
 playbooks/roles/gitr/defaults/main.yml |  3 ++
 playbooks/roles/gitr/tasks/main.yml    | 67 ++++++++++++++++++++++++++
 playbooks/roles/gitr/vars/RedHat.yml   |  7 ++-
 3 files changed, 75 insertions(+), 2 deletions(-)

diff --git a/playbooks/roles/gitr/defaults/main.yml b/playbooks/roles/gitr/defaults/main.yml
index bcfbaef..ff737ca 100644
--- a/playbooks/roles/gitr/defaults/main.yml
+++ b/playbooks/roles/gitr/defaults/main.yml
@@ -18,3 +18,6 @@ gitr_thread_single: false
 gitr_thread_fast: false
 gitr_thread_stress: false
 gitr_thread_custom: false
+
+cvsps_git: https://github.com/andreyvit/cvsps.git
+cvsps_data: "{{ data_path }}/cvsps"
diff --git a/playbooks/roles/gitr/tasks/main.yml b/playbooks/roles/gitr/tasks/main.yml
index 330350c..001257a 100644
--- a/playbooks/roles/gitr/tasks/main.yml
+++ b/playbooks/roles/gitr/tasks/main.yml
@@ -11,6 +11,11 @@
   failed_when: false
   tags: vars
 
+- name: Set up the /data mount point
+  tags: ['data_partition']
+  ansible.builtin.include_role:
+    name: create_data_partition
+
 - name: Set the name of the test group
   tags: vars
   ansible.builtin.set_fact:
@@ -42,6 +47,21 @@
       paths:
         - 'vars'
 
+- name: Update gitr depencies for RHEL/Centos
+  tags: gitr
+  ansible.builtin.set_fact:
+    gitr_packages: "{{ gitr_packages + ['perl-App-cpanminus'] }}"
+  when:
+    - ansible_distribution == "RedHat" or
+      ansible_distribution == "CentOS"
+
+- name: Update gitr dependencies for Fedora
+  tags: gitr
+  ansible.builtin.set_fact:
+    gitr_packages: "{{ gitr_packages + ['cvsps', 'perl-TAP-Harness-Archive'] }}"
+  when:
+    - ansible_distribution == "Fedora"
+
 - name: Install dependencies for gitr
   tags: gitr
   become: true
@@ -51,6 +71,53 @@
     name: "{{ gitr_packages }}"
     state: present
 
+- name: Install CPAN modules for gitr
+  tags: gitr
+  become: true
+  become_flags: 'su - -c'
+  become_method: ansible.builtin.sudo
+  community.general.cpanm:
+    name: "{{ item }}"
+  with_items: "{{ gitr_cpan_modules }}"
+  when:
+    - ansible_distribution == "RedHat" or
+      ansible_distribution == "CentOS"
+
+- name: git clone cvsps
+  tags: gitr
+  ansible.builtin.git:
+    repo: "{{ cvsps_git }}"
+    dest: "{{ cvsps_data }}"
+    update: true
+  retries: 3
+  delay: 5
+  register: result
+  until: not result.failed
+  when:
+    - ansible_distribution == "RedHat" or
+      ansible_distribution == "CentOS"
+
+- name: Build cvsps
+  tags: gitr
+  community.general.make:
+    chdir: "{{ cvsps_data }}"
+    jobs: "{{ ansible_processor_nproc }}"
+  when:
+    - ansible_distribution == "RedHat" or
+      ansible_distribution == "CentOS"
+
+- name: Install cvsps
+  tags: gitr
+  become: true
+  become_flags: 'su - -c'
+  become_method: ansible.builtin.sudo
+  community.general.make:
+    target: install
+    chdir: "{{ cvsps_data }}"
+  when:
+    - ansible_distribution == "RedHat" or
+      ansible_distribution == "CentOS"
+
 - name: Set up an iSCSI initiator on the target node
   ansible.builtin.include_role:
     name: iscsi
diff --git a/playbooks/roles/gitr/vars/RedHat.yml b/playbooks/roles/gitr/vars/RedHat.yml
index 57f2f35..caf9840 100644
--- a/playbooks/roles/gitr/vars/RedHat.yml
+++ b/playbooks/roles/gitr/vars/RedHat.yml
@@ -1,7 +1,6 @@
 ---
 gitr_packages:
   - cvs
-  - cvsps
   - expat-devel
   - gcc
   - gettext
@@ -9,11 +8,15 @@ gitr_packages:
   - git-email
   - glibc-utils
   - libcurl-devel
+  - make
   - openssl-devel
   - perl-Memoize
-  - perl-TAP-Harness-Archive
   - perl-Test-Harness
   - perl-Test-Simple
   - subversion
   - subversion-perl
+  - tar
   - zlib-devel
+
+gitr_cpan_modules:
+  - TAP::Harness::Archive
-- 
2.44.0

Re: [PATCH] gitr: Make gitr work on RHEL/CentOS

[Chuck Lever]

On Thu, Aug 15, 2024 at 05:59:41PM -0400, Scott Mayhew wrote:
> RHEL and CentOS do not ship cvsps or perl-TAP-Harness-Archive.  For the
> former, we'll have to build it from git.  For the latter, we can install
> it from CPAN, but we'll also have to install the perl-App-cpanminus
> package to do so.
> 
> Also add two other dependencies.  First is 'make', which is currently
> only installed if you have KDEVOPS_TRY_INSTALL_KDEV_TOOLS enabled in
> your config.

That's a little surprising. I guess Ansible's "make" module does not
install the base make utility?


> Second is 'tar' which is not installed in the RHEL
> virt-builder images by default ('tar' isn't part of the @core dnf
> package group).

Another option might be to disable the CVS-related tests by not
installing CVS on the target nodes. The git test suite should pick
up that omission and then not run those tests.


> Signed-off-by: Scott Mayhew <smayhew@redhat.com>
> ---
>  playbooks/roles/gitr/defaults/main.yml |  3 ++
>  playbooks/roles/gitr/tasks/main.yml    | 67 ++++++++++++++++++++++++++
>  playbooks/roles/gitr/vars/RedHat.yml   |  7 ++-
>  3 files changed, 75 insertions(+), 2 deletions(-)
> 
> diff --git a/playbooks/roles/gitr/defaults/main.yml b/playbooks/roles/gitr/defaults/main.yml
> index bcfbaef..ff737ca 100644
> --- a/playbooks/roles/gitr/defaults/main.yml
> +++ b/playbooks/roles/gitr/defaults/main.yml
> @@ -18,3 +18,6 @@ gitr_thread_single: false
>  gitr_thread_fast: false
>  gitr_thread_stress: false
>  gitr_thread_custom: false
> +
> +cvsps_git: https://github.com/andreyvit/cvsps.git
> +cvsps_data: "{{ data_path }}/cvsps"
> diff --git a/playbooks/roles/gitr/tasks/main.yml b/playbooks/roles/gitr/tasks/main.yml
> index 330350c..001257a 100644
> --- a/playbooks/roles/gitr/tasks/main.yml
> +++ b/playbooks/roles/gitr/tasks/main.yml
> @@ -11,6 +11,11 @@
>    failed_when: false
>    tags: vars
>  
> +- name: Set up the /data mount point
> +  tags: ['data_partition']
> +  ansible.builtin.include_role:
> +    name: create_data_partition
> +
>  - name: Set the name of the test group
>    tags: vars
>    ansible.builtin.set_fact:
> @@ -42,6 +47,21 @@
>        paths:
>          - 'vars'
>  
> +- name: Update gitr depencies for RHEL/Centos
> +  tags: gitr
> +  ansible.builtin.set_fact:
> +    gitr_packages: "{{ gitr_packages + ['perl-App-cpanminus'] }}"
> +  when:
> +    - ansible_distribution == "RedHat" or
> +      ansible_distribution == "CentOS"
> +
> +- name: Update gitr dependencies for Fedora
> +  tags: gitr
> +  ansible.builtin.set_fact:
> +    gitr_packages: "{{ gitr_packages + ['cvsps', 'perl-TAP-Harness-Archive'] }}"
> +  when:
> +    - ansible_distribution == "Fedora"
> +
>  - name: Install dependencies for gitr
>    tags: gitr
>    become: true
> @@ -51,6 +71,53 @@
>      name: "{{ gitr_packages }}"
>      state: present
>  
> +- name: Install CPAN modules for gitr
> +  tags: gitr
> +  become: true
> +  become_flags: 'su - -c'
> +  become_method: ansible.builtin.sudo
> +  community.general.cpanm:
> +    name: "{{ item }}"
> +  with_items: "{{ gitr_cpan_modules }}"
> +  when:
> +    - ansible_distribution == "RedHat" or
> +      ansible_distribution == "CentOS"
> +
> +- name: git clone cvsps
> +  tags: gitr
> +  ansible.builtin.git:
> +    repo: "{{ cvsps_git }}"
> +    dest: "{{ cvsps_data }}"
> +    update: true
> +  retries: 3
> +  delay: 5
> +  register: result
> +  until: not result.failed
> +  when:
> +    - ansible_distribution == "RedHat" or
> +      ansible_distribution == "CentOS"
> +
> +- name: Build cvsps
> +  tags: gitr
> +  community.general.make:
> +    chdir: "{{ cvsps_data }}"
> +    jobs: "{{ ansible_processor_nproc }}"
> +  when:
> +    - ansible_distribution == "RedHat" or
> +      ansible_distribution == "CentOS"
> +
> +- name: Install cvsps
> +  tags: gitr
> +  become: true
> +  become_flags: 'su - -c'
> +  become_method: ansible.builtin.sudo
> +  community.general.make:
> +    target: install
> +    chdir: "{{ cvsps_data }}"
> +  when:
> +    - ansible_distribution == "RedHat" or
> +      ansible_distribution == "CentOS"
> +
>  - name: Set up an iSCSI initiator on the target node
>    ansible.builtin.include_role:
>      name: iscsi
> diff --git a/playbooks/roles/gitr/vars/RedHat.yml b/playbooks/roles/gitr/vars/RedHat.yml
> index 57f2f35..caf9840 100644
> --- a/playbooks/roles/gitr/vars/RedHat.yml
> +++ b/playbooks/roles/gitr/vars/RedHat.yml
> @@ -1,7 +1,6 @@
>  ---
>  gitr_packages:
>    - cvs
> -  - cvsps
>    - expat-devel
>    - gcc
>    - gettext
> @@ -9,11 +8,15 @@ gitr_packages:
>    - git-email
>    - glibc-utils
>    - libcurl-devel
> +  - make
>    - openssl-devel
>    - perl-Memoize
> -  - perl-TAP-Harness-Archive
>    - perl-Test-Harness
>    - perl-Test-Simple
>    - subversion
>    - subversion-perl
> +  - tar
>    - zlib-devel
> +
> +gitr_cpan_modules:
> +  - TAP::Harness::Archive
> -- 
> 2.44.0
> 
> 

-- 
Chuck Lever

Re: [PATCH] gitr: Make gitr work on RHEL/CentOS

[Scott Mayhew]

On Fri, 16 Aug 2024, Chuck Lever wrote:

> On Thu, Aug 15, 2024 at 05:59:41PM -0400, Scott Mayhew wrote:
> > RHEL and CentOS do not ship cvsps or perl-TAP-Harness-Archive.  For the
> > former, we'll have to build it from git.  For the latter, we can install
> > it from CPAN, but we'll also have to install the perl-App-cpanminus
> > package to do so.
> > 
> > Also add two other dependencies.  First is 'make', which is currently
> > only installed if you have KDEVOPS_TRY_INSTALL_KDEV_TOOLS enabled in
> > your config.
> 
> That's a little surprising. I guess Ansible's "make" module does not
> install the base make utility?

AFAIK ansible modules just run the commands.  It's up to the user to
make sure the software that provide those commands is already installed
on the target systems.
> 
> 
> > Second is 'tar' which is not installed in the RHEL
> > virt-builder images by default ('tar' isn't part of the @core dnf
> > package group).
> 
> Another option might be to disable the CVS-related tests by not
> installing CVS on the target nodes. The git test suite should pick
> up that omission and then not run those tests.

You're saying that only the CVS-related tests are using tar?  Sounds
weird, but I can try it if you want.  Personally I'd rather have the
same set of tests running on my RHEL targets that are running on my
Fedora targets.

<snip/>

BTW, I think this will probably break the test on Oracle Linux.
> > +  when:
> > +    - ansible_distribution == "RedHat" or
> > +      ansible_distribution == "CentOS"

I think 'when: ansible_distribution != "Fedora"' will work better.

-Scott

Re: [PATCH] gitr: Make gitr work on RHEL/CentOS

[Chuck Lever III]

> On Aug 19, 2024, at 8:53 AM, Scott Mayhew <smayhew@redhat.com> wrote:
> 
> On Fri, 16 Aug 2024, Chuck Lever wrote:
> 
>> On Thu, Aug 15, 2024 at 05:59:41PM -0400, Scott Mayhew wrote:
>>> RHEL and CentOS do not ship cvsps or perl-TAP-Harness-Archive.  For the
>>> former, we'll have to build it from git.  For the latter, we can install
>>> it from CPAN, but we'll also have to install the perl-App-cpanminus
>>> package to do so.
>>> 
>>> Also add two other dependencies.  First is 'make', which is currently
>>> only installed if you have KDEVOPS_TRY_INSTALL_KDEV_TOOLS enabled in
>>> your config.
>> 
>> That's a little surprising. I guess Ansible's "make" module does not
>> install the base make utility?
> 
> AFAIK ansible modules just run the commands.  It's up to the user to
> make sure the software that provide those commands is already installed
> on the target systems.

My Buildbot configurations do not have INSTALL_KDEV_TOOLS enabled,
but they seem to have no problem running the gitr workflow. I'm
thinking there might be some other reason why "make" is not
available -- ie, my quibble is with the patch description, not
with adding the dependency.


>>> Second is 'tar' which is not installed in the RHEL
>>> virt-builder images by default ('tar' isn't part of the @core dnf
>>> package group).
>> 
>> Another option might be to disable the CVS-related tests by not
>> installing CVS on the target nodes. The git test suite should pick
>> up that omission and then not run those tests.
> 
> You're saying that only the CVS-related tests are using tar?  Sounds
> weird, but I can try it if you want.  Personally I'd rather have the
> same set of tests running on my RHEL targets that are running on my
> Fedora targets.

That's sensible -- let's keep them the same.


> <snip/>
> 
> BTW, I think this will probably break the test on Oracle Linux.
>>> +  when:
>>> +    - ansible_distribution == "RedHat" or
>>> +      ansible_distribution == "CentOS"
> 
> I think 'when: ansible_distribution != "Fedora"' will work better.

No argument from me. I probably copied that from somewhere else.


--
Chuck Lever

Re: [PATCH] gitr: Make gitr work on RHEL/CentOS

[Scott Mayhew]

On Mon, 19 Aug 2024, Chuck Lever III wrote:

> 
> 
> > On Aug 19, 2024, at 8:53 AM, Scott Mayhew <smayhew@redhat.com> wrote:
> > 
> > On Fri, 16 Aug 2024, Chuck Lever wrote:
> > 
> >> On Thu, Aug 15, 2024 at 05:59:41PM -0400, Scott Mayhew wrote:
> >>> RHEL and CentOS do not ship cvsps or perl-TAP-Harness-Archive.  For the
> >>> former, we'll have to build it from git.  For the latter, we can install
> >>> it from CPAN, but we'll also have to install the perl-App-cpanminus
> >>> package to do so.
> >>> 
> >>> Also add two other dependencies.  First is 'make', which is currently
> >>> only installed if you have KDEVOPS_TRY_INSTALL_KDEV_TOOLS enabled in
> >>> your config.
> >> 
> >> That's a little surprising. I guess Ansible's "make" module does not
> >> install the base make utility?
> > 
> > AFAIK ansible modules just run the commands.  It's up to the user to
> > make sure the software that provide those commands is already installed
> > on the target systems.
> 
> My Buildbot configurations do not have INSTALL_KDEV_TOOLS enabled,
> but they seem to have no problem running the gitr workflow. I'm
> thinking there might be some other reason why "make" is not
> available -- ie, my quibble is with the patch description, not
> with adding the dependency.

Ok, let me double-check the Fedora images to see where they're getting
'make' from.
> 
> 
> >>> Second is 'tar' which is not installed in the RHEL
> >>> virt-builder images by default ('tar' isn't part of the @core dnf
> >>> package group).
> >> 
> >> Another option might be to disable the CVS-related tests by not
> >> installing CVS on the target nodes. The git test suite should pick
> >> up that omission and then not run those tests.
> > 
> > You're saying that only the CVS-related tests are using tar?  Sounds
> > weird, but I can try it if you want.  Personally I'd rather have the
> > same set of tests running on my RHEL targets that are running on my
> > Fedora targets.
> 
> That's sensible -- let's keep them the same.

Sounds good.
> 
> 
> > <snip/>
> > 
> > BTW, I think this will probably break the test on Oracle Linux.
> >>> +  when:
> >>> +    - ansible_distribution == "RedHat" or
> >>> +      ansible_distribution == "CentOS"
> > 
> > I think 'when: ansible_distribution != "Fedora"' will work better.
> 
> No argument from me. I probably copied that from somewhere else.

Those new 'when' clauses were added by my patch and I was just pointing
out my mistake.

-Scott
> 
> 
> --
> Chuck Lever
> 
> 

Re: [PATCH] gitr: Make gitr work on RHEL/CentOS

[Scott Mayhew]

On Mon, 19 Aug 2024, Scott Mayhew wrote:

> On Mon, 19 Aug 2024, Chuck Lever III wrote:
> 
> > 
> > 
> > > On Aug 19, 2024, at 8:53 AM, Scott Mayhew <smayhew@redhat.com> wrote:
> > > 
> > > On Fri, 16 Aug 2024, Chuck Lever wrote:
> > > 
> > >> On Thu, Aug 15, 2024 at 05:59:41PM -0400, Scott Mayhew wrote:
> > >>> RHEL and CentOS do not ship cvsps or perl-TAP-Harness-Archive.  For the
> > >>> former, we'll have to build it from git.  For the latter, we can install
> > >>> it from CPAN, but we'll also have to install the perl-App-cpanminus
> > >>> package to do so.
> > >>> 
> > >>> Also add two other dependencies.  First is 'make', which is currently
> > >>> only installed if you have KDEVOPS_TRY_INSTALL_KDEV_TOOLS enabled in
> > >>> your config.
> > >> 
> > >> That's a little surprising. I guess Ansible's "make" module does not
> > >> install the base make utility?
> > > 
> > > AFAIK ansible modules just run the commands.  It's up to the user to
> > > make sure the software that provide those commands is already installed
> > > on the target systems.
> > 
> > My Buildbot configurations do not have INSTALL_KDEV_TOOLS enabled,
> > but they seem to have no problem running the gitr workflow. I'm
> > thinking there might be some other reason why "make" is not
> > available -- ie, my quibble is with the patch description, not
> > with adding the dependency.
> 
> Ok, let me double-check the Fedora images to see where they're getting
> 'make' from.

It looks like the real culprit is that the "Install btrfs-progs" play in
playbooks/roles/devconfig/tasks/install-deps/redhat/main.yml doesn't
really do what it claims.  It's actually installing the entire set of
packages from the "Build install package list" play right before it.
That's how we're magically getting both the make and tar packages on
Fedora... make included in that list explicitly, and tar is getting
pulled in as a dependency of libtool.

> > 
> > 
> > >>> Second is 'tar' which is not installed in the RHEL
> > >>> virt-builder images by default ('tar' isn't part of the @core dnf
> > >>> package group).
> > >> 
> > >> Another option might be to disable the CVS-related tests by not
> > >> installing CVS on the target nodes. The git test suite should pick
> > >> up that omission and then not run those tests.
> > > 
> > > You're saying that only the CVS-related tests are using tar?  Sounds
> > > weird, but I can try it if you want.  Personally I'd rather have the
> > > same set of tests running on my RHEL targets that are running on my
> > > Fedora targets.
> > 
> > That's sensible -- let's keep them the same.
> 
> Sounds good.
> > 
> > 
> > > <snip/>
> > > 
> > > BTW, I think this will probably break the test on Oracle Linux.
> > >>> +  when:
> > >>> +    - ansible_distribution == "RedHat" or
> > >>> +      ansible_distribution == "CentOS"
> > > 
> > > I think 'when: ansible_distribution != "Fedora"' will work better.
> > 
> > No argument from me. I probably copied that from somewhere else.
> 
> Those new 'when' clauses were added by my patch and I was just pointing
> out my mistake.
> 
> -Scott
> > 
> > 
> > --
> > Chuck Lever
> > 
> > 

Re: [PATCH] gitr: Make gitr work on RHEL/CentOS

[Chuck Lever III]

> On Aug 19, 2024, at 5:28 PM, Scott Mayhew <smayhew@redhat.com> wrote:
> 
> On Mon, 19 Aug 2024, Scott Mayhew wrote:
> 
>> On Mon, 19 Aug 2024, Chuck Lever III wrote:
>> 
>>>> On Aug 19, 2024, at 8:53 AM, Scott Mayhew <smayhew@redhat.com> wrote:
>>>> 
>>>> On Fri, 16 Aug 2024, Chuck Lever wrote:
>>>> 
>>>>> On Thu, Aug 15, 2024 at 05:59:41PM -0400, Scott Mayhew wrote:
>>>>>> RHEL and CentOS do not ship cvsps or perl-TAP-Harness-Archive.  For the
>>>>>> former, we'll have to build it from git.  For the latter, we can install
>>>>>> it from CPAN, but we'll also have to install the perl-App-cpanminus
>>>>>> package to do so.
>>>>>> 
>>>>>> Also add two other dependencies.  First is 'make', which is currently
>>>>>> only installed if you have KDEVOPS_TRY_INSTALL_KDEV_TOOLS enabled in
>>>>>> your config.
>>>>> 
>>>>> That's a little surprising. I guess Ansible's "make" module does not
>>>>> install the base make utility?
>>>> 
>>>> AFAIK ansible modules just run the commands.  It's up to the user to
>>>> make sure the software that provide those commands is already installed
>>>> on the target systems.
>>> 
>>> My Buildbot configurations do not have INSTALL_KDEV_TOOLS enabled,
>>> but they seem to have no problem running the gitr workflow. I'm
>>> thinking there might be some other reason why "make" is not
>>> available -- ie, my quibble is with the patch description, not
>>> with adding the dependency.
>> 
>> Ok, let me double-check the Fedora images to see where they're getting
>> 'make' from.
> 
> It looks like the real culprit is that the "Install btrfs-progs" play in
> playbooks/roles/devconfig/tasks/install-deps/redhat/main.yml doesn't
> really do what it claims.  It's actually installing the entire set of
> packages from the "Build install package list" play right before it.
> That's how we're magically getting both the make and tar packages on
> Fedora... make included in that list explicitly, and tar is getting
> pulled in as a dependency of libtool.

That might explain why that step was hanging for so long when
the public package repositories are slow.

No objection from me to adding make/tar explicitly for the gitr
workflow. Root cause now looks plausible to me.


--
Chuck Lever