[bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[Marciniszyn, Mike]

Seen on Rocky Linux 8.8 with rdma-core v51.0. 

I suspect 8.9 shows the same issue and I know the RC for RHEL 8.10 does as well.

The cmake version is 3.20.2.

The linux kernel version and hardware is not relevant since this is a cmake/build issue.

The hardware is also not relevant.

To reproduce:
1. download an untar the v51.0 tar ball to ~
cd ~
wget  https://github.com/linux-rdma/rdma-core/releases/download/v51.0/rdma-core-51.0.tar.gz
tar -zxvf tar -zxvf rdma-core-51.0.tar.gz
2. create the following directories
 mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS,OTHER}
3. copy the spec file 
cp cp rdma-core-51.0/redhat/rdma-core.spec ~/rpmbuild/SPECS
4. Create SOURCES
tar -zcvf ~/rpmbuild/SOURCES/rdma-core-51.0.tar.gz rdma-core-51.0
5. load build dependencies
ensure AppStream, BaseOS, PowerTools repos are accessible
sudo dnf install -y dnf-plugins-core rpm-build
cd ~/rpmbuild/SPECS
sudo dnf builddep -y rdma-core.spec
6. Load an additional pythion interpreter
sudo dnf install python39
6. do the build
rpmbuild  -ba rdma-core.spec

The build gets the following errors:
error: File not found: /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/lib64/python3.6/site-packages/pyverbs
error: File not found: /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/share/doc/rdma-core/tests/*.py


RPM build errors:
    File not found: /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/lib64/python3.6/site-packages/pyverbs
    File not found: /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/share/doc/rdma-core/tests/*.py

Earlier in the build this is seen:
-- Found Python: /usr/bin/python3.9 (found version "3.9.16") found components: Interpreter
-- Could NOT find cython (missing: CYTHON_EXECUTABLE CYTHON_VERSION_STRING)

The issue appears to have been introduced by:
1462a8737 build: Fix cmake warning

cmake appears to find the 3.9 python despite having:
-DPYTHON_EXECUTABLE:PATH=%{__python3}

The issue will impact 8.10 because valgrind-deve/valgrind will bring in python3.11 100% of the time.

Mike

Re: [bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[Jason Gunthorpe]

On Thu, May 16, 2024 at 07:48:06PM +0000, Marciniszyn, Mike wrote:
> Seen on Rocky Linux 8.8 with rdma-core v51.0. 
> 
> I suspect 8.9 shows the same issue and I know the RC for RHEL 8.10 does as well.
> 
> The cmake version is 3.20.2.
> 
> The linux kernel version and hardware is not relevant since this is a cmake/build issue.
> 
> The hardware is also not relevant.
> 
> To reproduce:
> 1. download an untar the v51.0 tar ball to ~
> cd ~
> wget  https://github.com/linux-rdma/rdma-core/releases/download/v51.0/rdma-core-51.0.tar.gz
> tar -zxvf tar -zxvf rdma-core-51.0.tar.gz
> 2. create the following directories
>  mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS,OTHER}
> 3. copy the spec file 
> cp cp rdma-core-51.0/redhat/rdma-core.spec ~/rpmbuild/SPECS
> 4. Create SOURCES
> tar -zcvf ~/rpmbuild/SOURCES/rdma-core-51.0.tar.gz rdma-core-51.0
> 5. load build dependencies
> ensure AppStream, BaseOS, PowerTools repos are accessible
> sudo dnf install -y dnf-plugins-core rpm-build
> cd ~/rpmbuild/SPECS
> sudo dnf builddep -y rdma-core.spec
> 6. Load an additional pythion interpreter
> sudo dnf install python39
> 6. do the build
> rpmbuild  -ba rdma-core.spec
> 
> The build gets the following errors:
> error: File not found: /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/lib64/python3.6/site-packages/pyverbs
> error: File not found: /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/share/doc/rdma-core/tests/*.py
> 
> 
> RPM build errors:
>     File not found: /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/lib64/python3.6/site-packages/pyverbs
>     File not found: /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/share/doc/rdma-core/tests/*.py
> 
> Earlier in the build this is seen:
> -- Found Python: /usr/bin/python3.9 (found version "3.9.16") found components: Interpreter
> -- Could NOT find cython (missing: CYTHON_EXECUTABLE CYTHON_VERSION_STRING)
> 
> The issue appears to have been introduced by:
> 1462a8737 build: Fix cmake warning
> 
> cmake appears to find the 3.9 python despite having:
> -DPYTHON_EXECUTABLE:PATH=%{__python3}

I looked at this briefly but didn't guess what the issue really was?
Do you know more?

Jason

RE: [bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[Marciniszyn, Mike]

> > /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/shar
> > e/doc/rdma-core/tests/*.py
> >
> > Earlier in the build this is seen:
> > -- Found Python: /usr/bin/python3.9 (found version "3.9.16") found
> > components: Interpreter
> > -- Could NOT find cython (missing: CYTHON_EXECUTABLE
> > CYTHON_VERSION_STRING)
> >
> > The issue appears to have been introduced by:
> > 1462a8737 build: Fix cmake warning
> >
> > cmake appears to find the 3.9 python despite having:
> > -DPYTHON_EXECUTABLE:PATH=%{__python3}
> 
> I looked at this briefly but didn't guess what the issue really was?
> Do you know more?
> 
> Jason

I looks like the above commit changes the cmake to use a new >= 3.12 code path:

  # FindPython looks preferably for Python3. If not found, version 2 is searched
  FIND_PACKAGE(Python COMPONENTS Interpreter REQUIRED)
  set(PYTHON_EXECUTABLE ${Python_EXECUTABLE})
  if (NOT NO_PYVERBS AND Python_VERSION_MAJOR EQUAL 3)
    FIND_PACKAGE(cython)
  endif()

There is an additional refinement from:
5dcc1f402 Improve python searching logic in buildscripts.

That patch does NOT fix the issue.

The rhel 8.10 cython requires the 3.6.8 Python, but the FIND_PACKAGE() for python
returns the python3.11 required by valgrind dependency resolution.

The issue happens on any system where the cython required interpreter is older than the most recent python.

This can be reproduced by loading a newer python and rebuilding the 51.0 rdma-core from a git clone using either 51.0 or master.

An RPM based RHEL build can replace the existing PYTHON_EXECUABLE with Python_EXECUTABLE.

The issue can also be fixed by adding a variable that can be passed on the command line with a compensating
fix in the CMakeLists.txt to pass use the EXACT python version required by cython.

Thanks to looking into this!
Mike

Re: [bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[Jason Gunthorpe]

On Fri, May 24, 2024 at 04:40:20PM +0000, Marciniszyn, Mike wrote:
> > > /home/mmarcini/rpmbuild/BUILDROOT/rdma-core-51.0-1.el8.x86_64/usr/shar
> > > e/doc/rdma-core/tests/*.py
> > >
> > > Earlier in the build this is seen:
> > > -- Found Python: /usr/bin/python3.9 (found version "3.9.16") found
> > > components: Interpreter
> > > -- Could NOT find cython (missing: CYTHON_EXECUTABLE
> > > CYTHON_VERSION_STRING)
> > >
> > > The issue appears to have been introduced by:
> > > 1462a8737 build: Fix cmake warning
> > >
> > > cmake appears to find the 3.9 python despite having:
> > > -DPYTHON_EXECUTABLE:PATH=%{__python3}
> > 
> > I looked at this briefly but didn't guess what the issue really was?
> > Do you know more?
> > 
> > Jason
> 
> I looks like the above commit changes the cmake to use a new >= 3.12 code path:
> 
>   # FindPython looks preferably for Python3. If not found, version 2 is searched
>   FIND_PACKAGE(Python COMPONENTS Interpreter REQUIRED)
>   set(PYTHON_EXECUTABLE ${Python_EXECUTABLE})
>   if (NOT NO_PYVERBS AND Python_VERSION_MAJOR EQUAL 3)
>     FIND_PACKAGE(cython)
>   endif()
> 
> There is an additional refinement from:
> 5dcc1f402 Improve python searching logic in buildscripts.
> 
> That patch does NOT fix the issue.
> 
> The rhel 8.10 cython requires the 3.6.8 Python, but the FIND_PACKAGE() for python
> returns the python3.11 required by valgrind dependency resolution.
> 
> The issue happens on any system where the cython required interpreter is older than the most recent python.
> 
> This can be reproduced by loading a newer python and rebuilding the 51.0 rdma-core from a git clone using either 51.0 or master.
> 
> An RPM based RHEL build can replace the existing PYTHON_EXECUABLE
> with Python_EXECUTABLE.

You mean at some point we lost that PYTHON_EXECUTABLE works and it got
renamed to Python_EXECUTABLE ?

Jason

RE: [bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[Marciniszyn, Mike]

> >
> > This can be reproduced by loading a newer python and rebuilding the 51.0
> rdma-core from a git clone using either 51.0 or master.
> >
> > An RPM based RHEL build can replace the existing PYTHON_EXECUABLE with
> > Python_EXECUTABLE.
> 
> You mean at some point we lost that PYTHON_EXECUTABLE works and it got
> renamed to Python_EXECUTABLE ?
> 
> Jason

The Python_EXECUTABLE is workaround that can be applied to a spec file.

What happened is we added a new code path in our cmake list that behaved differently on handling cython.

I'm kind of new to cmake. Sorry.
Mike

Re: [bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[Jason Gunthorpe]

On Fri, May 24, 2024 at 06:01:51PM +0000, Marciniszyn, Mike wrote:
> > >
> > > This can be reproduced by loading a newer python and rebuilding the 51.0
> > rdma-core from a git clone using either 51.0 or master.
> > >
> > > An RPM based RHEL build can replace the existing PYTHON_EXECUABLE with
> > > Python_EXECUTABLE.
> > 
> > You mean at some point we lost that PYTHON_EXECUTABLE works and it got
> > renamed to Python_EXECUTABLE ?
> > 
> > Jason
> 
> The Python_EXECUTABLE is workaround that can be applied to a spec file.

OK, that is clear. If PYTHON_EXECUTABLE no longer works that is a well
understandable issue.

Something like this maybe?

--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -216,6 +216,9 @@ endif()
 
 # Use Python modules based on CMake version for backward compatibility
 set(CYTHON_EXECUTABLE "")
+if (${PYTHON_EXCUTABLE})
+  set(Python_EXECUTABLE ${PYTHON_EXCUTABLE})
+endif()
 if (${CMAKE_VERSION} VERSION_LESS "3.12")
   # Look for Python. We prefer some variant of python 3 if the system has it
   FIND_PACKAGE(PythonInterp 3 QUIET)

Jason

RE: [bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[Marciniszyn, Mike]

> >
> > The Python_EXECUTABLE is workaround that can be applied to a spec file.
> 
> OK, that is clear. If PYTHON_EXECUTABLE no longer works that is a well
> understandable issue.
> 
> Something like this maybe?
> 
> --- a/CMakeLists.txt
> +++ b/CMakeLists.txt
> @@ -216,6 +216,9 @@ endif()
> 
>  # Use Python modules based on CMake version for backward compatibility
> set(CYTHON_EXECUTABLE "")
> +if (${PYTHON_EXCUTABLE})
> +  set(Python_EXECUTABLE ${PYTHON_EXCUTABLE})
> +endif()
>  if (${CMAKE_VERSION} VERSION_LESS "3.12")
>    # Look for Python. We prefer some variant of python 3 if the system has it
>    FIND_PACKAGE(PythonInterp 3 QUIET)
> 
> Jason
Ok.  I will give that a try.

Mike

RE: [bug-report]rdma-core v51.0 build error with Rocky Linux 8.8

[Marciniszyn, Mike]

> > The Python_EXECUTABLE is workaround that can be applied to a spec file.
> 
> OK, that is clear. If PYTHON_EXECUTABLE no longer works that is a well
> understandable issue.
> 
> Something like this maybe?
> 
> --- a/CMakeLists.txt
> +++ b/CMakeLists.txt
> @@ -216,6 +216,9 @@ endif()
> 
>  # Use Python modules based on CMake version for backward compatibility
> set(CYTHON_EXECUTABLE "")
> +if (${PYTHON_EXCUTABLE})
> +  set(Python_EXECUTABLE ${PYTHON_EXCUTABLE})
> +endif()
>  if (${CMAKE_VERSION} VERSION_LESS "3.12")
>    # Look for Python. We prefer some variant of python 3 if the system has it
>    FIND_PACKAGE(PythonInterp 3 QUIET)
> 
> Jason

Jason,

After a lot of more testing, I have found that the tip of the master branch is ok!

These patches resolve the issue:
5dcc1f402 Improve python searching logic in buildscripts.
462a8737 build: Fix cmake warning

Technically, the earlier patch is the one that breaks the -D, and the second fixes that.

On a system with two python 3 interpreters, when PYTHON_EXECUTABLE is NOT passed in from the command line,
the pyverbs is not built as expected, but the build otherwise works.

When it is, the pyverbs is built.

We can pull these two patches back to fix the issue to earlier versions as we need.

Thanks for your help!
Mike