OFED-4.8, rdma-core, and library paths

OFED-4.8, rdma-core, and library paths

[Steve Wise]

Hey,

I think we have an issue with the new rdma-core packaging and OFED-4.8.  I think
OFED-4.8 installs provider libraries in a different location than previous
releases of the provider libs.  The result of this, I think, is that OFED-4.8
installed over a destro with its own provider libs installed will result in two
versions of the libs installed, and further, the system might end up using the
distro provider libs with newer OFED drivers, which could be problematic.  I
believe the OFED installer uninstalls previous OFED rpms, but not distro rpms.
>From what I can tell, it uses the ofed_info  command, if it exists, to determine
which rpms to uninstall.  So if there is no previous OFED installed, then
ofed_info will not exist so the distro rdma rpms will not be uninstalled.  Prior
to OFED-4.8, I think this was somewhat benign, because OFED would install the
provider libs over the currently installed distro libs, and thus nobody noticed.
But now the rdma-core package puts the provider libs in a different location,
thus exposing this issue.

What do folks think about this?  Should OFED-4.8 try and uninstall rdma
cmds/libs regardless of where they came from?  Perhaps optionally.  Or should
this just be documented so the admin is required to deal with it?  I think if we
leave OFED as-is, we'll end up with lots of support issues where old libs are
being loaded causing problems.

Thoughts?  Am I missing something in the OFED-4.8 installer that avoids this
issue?


Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED-4.8, rdma-core, and library paths

[Vladimir Sokolovsky]

On 02/07/2017 05:07 PM, Steve Wise wrote:
> Hey,
>
> I think we have an issue with the new rdma-core packaging and OFED-4.8.  I think
> OFED-4.8 installs provider libraries in a different location than previous
> releases of the provider libs.  The result of this, I think, is that OFED-4.8
> installed over a destro with its own provider libs installed will result in two
> versions of the libs installed, and further, the system might end up using the
> distro provider libs with newer OFED drivers, which could be problematic.  I
> believe the OFED installer uninstalls previous OFED rpms, but not distro rpms.
>  From what I can tell, it uses the ofed_info  command, if it exists, to determine
> which rpms to uninstall.  So if there is no previous OFED installed, then
> ofed_info will not exist so the distro rdma rpms will not be uninstalled.  Prior
> to OFED-4.8, I think this was somewhat benign, because OFED would install the
> provider libs over the currently installed distro libs, and thus nobody noticed.
> But now the rdma-core package puts the provider libs in a different location,
> thus exposing this issue.
>
> What do folks think about this?  Should OFED-4.8 try and uninstall rdma
> cmds/libs regardless of where they came from?  Perhaps optionally.  Or should
> this just be documented so the admin is required to deal with it?  I think if we
> leave OFED as-is, we'll end up with lots of support issues where old libs are
> being loaded causing problems.
>
> Thoughts?  Am I missing something in the OFED-4.8 installer that avoids this
> issue?
>
>
> Steve.

Hi Steve,
OFED's install script uninstalls previous OFED versions and 
corresponding in-box RPMs as well.
The list of in-box RPMs that should be uninstalled is maintained in the 
install.pl script itself. ofed_info is not used by the install.pl script.
So, there should be no issue here.

Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: OFED-4.8, rdma-core, and library paths

[Steve Wise]

> 
> > Thoughts?  Am I missing something in the OFED-4.8 installer that avoids this
> > issue?
> >
> >
> > Steve.
> 
> Hi Steve,
> OFED's install script uninstalls previous OFED versions and
> corresponding in-box RPMs as well.
> The list of in-box RPMs that should be uninstalled is maintained in the
> install.pl script itself. ofed_info is not used by the install.pl script.
> So, there should be no issue here.
> 

That's good!  I missed that.  I guess I was looking at uninstall.sh, and assumed
install.pl called that.  

Thanks,

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED-4.8, rdma-core, and library paths

[Jason Gunthorpe]

On Tue, Feb 07, 2017 at 09:07:28AM -0600, Steve Wise wrote:

> I think we have an issue with the new rdma-core packaging and OFED-4.8.  I think
> OFED-4.8 installs provider libraries in a different location than previous
> releases of the provider libs.

On Redhat, yes. OFED could build using the

-DVERBS_PROVIDER_DIR=''

To disable this new location if absolutely necessary.

> The result of this, I think, is that OFED-4.8 installed over a
> destro with its own provider libs installed will result in two
> versions of the libs installed,

Yes...

> and further, the system might end up using the distro provider libs
> with newer OFED drivers, which could be problematic.

Hm, possibly yes. ibverbs first checks the new location, if the
provider is not there then it will fall back to a naked dlopen which
could find providers in the system library path if there was a .driver
file for it.

However, starting in rdma-core 13 the distro providers will not be
link compatible with new libibverbs and will silently fail to load.

> What do folks think about this?  Should OFED-4.8 try and uninstall rdma
> cmds/libs regardless of where they came from?  Perhaps optionally.  Or should
> this just be documented so the admin is required to deal with it?  I think if we
> leave OFED as-is, we'll end up with lots of support issues where old libs are
> being loaded causing problems.

I'm not sure it is likely someone will hit this. The user would need a
distro packaged provider that is not included in rdma-core. AFAIK no
such thing exists...

Do you know of another way to trigger wrong loading?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: OFED-4.8, rdma-core, and library paths

[Steve Wise]

> 
> Hm, possibly yes. ibverbs first checks the new location, if the
> provider is not there then it will fall back to a naked dlopen which
> could find providers in the system library path if there was a .driver
> file for it.
> 

Hmm, so it will load the provider libraries directly specifying the full path?
IE 'ldconfig -p' doesn't matter?

> However, starting in rdma-core 13 the distro providers will not be
> link compatible with new libibverbs and will silently fail to load.
> 
> > What do folks think about this?  Should OFED-4.8 try and uninstall rdma
> > cmds/libs regardless of where they came from?  Perhaps optionally.  Or
should
> > this just be documented so the admin is required to deal with it?  I think
if we
> > leave OFED as-is, we'll end up with lots of support issues where old libs
are
> > being loaded causing problems.
> 
> I'm not sure it is likely someone will hit this. The user would need a
> distro packaged provider that is not included in rdma-core. AFAIK no
> such thing exists...
>

Vlad already clarified that the OFED installer uninstalls distro rpms, so I
think OFED is good.

> 
> Do you know of another way to trigger wrong loading?
> 

Actually, I was assuming the load path used by libibverbs would be done based on
'ldconfig -p'.  If that is not the case, then everything is ok...I think. 

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED-4.8, rdma-core, and library paths

[Jason Gunthorpe]

On Tue, Feb 07, 2017 at 11:19:59AM -0600, Steve Wise wrote:
> > 
> > Hm, possibly yes. ibverbs first checks the new location, if the
> > provider is not there then it will fall back to a naked dlopen which
> > could find providers in the system library path if there was a .driver
> > file for it.
> 
> Hmm, so it will load the provider libraries directly specifying the
> full path?  IE 'ldconfig -p' doesn't matter?

As the first try, yes. That is the usual way to locate
plugins. Typically you don't want the system linker searching plugin
directories.

Jason
_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/mailman/listinfo/ewg

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]

On Tue, Feb 07, 2017 at 10:27:52AM -0700, Jason Gunthorpe wrote:
> On Tue, Feb 07, 2017 at 11:19:59AM -0600, Steve Wise wrote:
> > >
> > > Hm, possibly yes. ibverbs first checks the new location, if the
> > > provider is not there then it will fall back to a naked dlopen which
> > > could find providers in the system library path if there was a .driver
> > > file for it.
> >
> > Hmm, so it will load the provider libraries directly specifying the
> > full path?  IE 'ldconfig -p' doesn't matter?
>
> As the first try, yes. That is the usual way to locate
> plugins. Typically you don't want the system linker searching plugin
> directories.

Jason,
I have a slightly different question, but it is still in context of
plugins and linkers.

In v0 of DV, you asked from us to create normal shared library for mlx5,
e.g. libmlx5.so. In order to be visible to "gcc -lmlx5"i command, it should be
placed in the same directory as libibverbs.so, while plugins should be placed
in libibverbs folder.

I did it by using symlinks
$l /usr/lib64/libibverbs
lrwxrwxrwx    1 root root   10 Feb  7 18:09 libmlx5-rdmav2.so -> ../libmlx5.so
$l /usr/lib64 | grep mlx5
lrwxrwxrwx    1 root root   12 Feb  7 18:09 libmlx5.so -> libmlx5.so.1
lrwxrwxrwx    1 root root   17 Feb  7 18:09 libmlx5.so.1 -> libmlx5.so.1.0.13

It works, but I don't know if it is right approach from distro/packaging
perspective. Is it ok?

Thanks

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Jason Gunthorpe]

On Tue, Feb 07, 2017 at 08:06:42PM +0200, Leon Romanovsky wrote:
> On Tue, Feb 07, 2017 at 10:27:52AM -0700, Jason Gunthorpe wrote:
> > On Tue, Feb 07, 2017 at 11:19:59AM -0600, Steve Wise wrote:
> > > >
> > > > Hm, possibly yes. ibverbs first checks the new location, if the
> > > > provider is not there then it will fall back to a naked dlopen which
> > > > could find providers in the system library path if there was a .driver
> > > > file for it.
> > >
> > > Hmm, so it will load the provider libraries directly specifying the
> > > full path?  IE 'ldconfig -p' doesn't matter?
> >
> > As the first try, yes. That is the usual way to locate
> > plugins. Typically you don't want the system linker searching plugin
> > directories.
> 
> Jason,
> I have a slightly different question, but it is still in context of
> plugins and linkers.
> 
> In v0 of DV, you asked from us to create normal shared library for mlx5,
> e.g. libmlx5.so. In order to be visible to "gcc -lmlx5"i command, it should be
> placed in the same directory as libibverbs.so, while plugins should be placed
> in libibverbs folder.
> 
> I did it by using symlinks
> $l /usr/lib64/libibverbs
> lrwxrwxrwx    1 root root   10 Feb  7 18:09 libmlx5-rdmav2.so -> ../libmlx5.so
> $l /usr/lib64 | grep mlx5
> lrwxrwxrwx    1 root root   12 Feb  7 18:09 libmlx5.so -> libmlx5.so.1
> lrwxrwxrwx    1 root root   17 Feb  7 18:09 libmlx5.so.1 -> libmlx5.so.1.0.13
> 
> It works, but I don't know if it is right approach from distro/packaging
> perspective. Is it ok?

I think this is basically OK. The symlink should point to the real
file (libmlx5.so.1.0.13) though.

Also, make sure that cmake computes the .. properly, something like

execute_process("realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)

You'll want to create a new rdma_dv_provider function to handle all of
this.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2497 bytes --]

On Tue, Feb 07, 2017 at 11:18:14AM -0700, Jason Gunthorpe wrote:
> On Tue, Feb 07, 2017 at 08:06:42PM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 07, 2017 at 10:27:52AM -0700, Jason Gunthorpe wrote:
> > > On Tue, Feb 07, 2017 at 11:19:59AM -0600, Steve Wise wrote:
> > > > >
> > > > > Hm, possibly yes. ibverbs first checks the new location, if the
> > > > > provider is not there then it will fall back to a naked dlopen which
> > > > > could find providers in the system library path if there was a .driver
> > > > > file for it.
> > > >
> > > > Hmm, so it will load the provider libraries directly specifying the
> > > > full path?  IE 'ldconfig -p' doesn't matter?
> > >
> > > As the first try, yes. That is the usual way to locate
> > > plugins. Typically you don't want the system linker searching plugin
> > > directories.
> >
> > Jason,
> > I have a slightly different question, but it is still in context of
> > plugins and linkers.
> >
> > In v0 of DV, you asked from us to create normal shared library for mlx5,
> > e.g. libmlx5.so. In order to be visible to "gcc -lmlx5"i command, it should be
> > placed in the same directory as libibverbs.so, while plugins should be placed
> > in libibverbs folder.
> >
> > I did it by using symlinks
> > $l /usr/lib64/libibverbs
> > lrwxrwxrwx    1 root root   10 Feb  7 18:09 libmlx5-rdmav2.so -> ../libmlx5.so
> > $l /usr/lib64 | grep mlx5
> > lrwxrwxrwx    1 root root   12 Feb  7 18:09 libmlx5.so -> libmlx5.so.1
> > lrwxrwxrwx    1 root root   17 Feb  7 18:09 libmlx5.so.1 -> libmlx5.so.1.0.13
> >
> > It works, but I don't know if it is right approach from distro/packaging
> > perspective. Is it ok?
>
> I think this is basically OK. The symlink should point to the real
> file (libmlx5.so.1.0.13) though.

Right, to be sure that provider and libibverbs are coming from same version.

>
> Also, make sure that cmake computes the .. properly, something like
>
> execute_process("realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)
>
> You'll want to create a new rdma_dv_provider function to handle all of
> this.

I made it (rdma_shared_provider function), but have a very hard time
to properly create ".." symlink, because during the build (in place too)
the output is placed in build/lib in flat structure and symlinks need to
be without "..". But during installation phase, these symlinks should
be changed to ".." and it doesn't work for me in automatic way :(

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Jason Gunthorpe]

On Tue, Feb 07, 2017 at 08:35:38PM +0200, Leon Romanovsky wrote:
> I made it (rdma_shared_provider function), but have a very hard time
> to properly create ".." symlink, because during the build (in place too)
> the output is placed in build/lib in flat structure and symlinks need to
> be without "..". But during installation phase, these symlinks should
> be changed to ".." and it doesn't work for me in automatic way :(

For symlinks build/ and installed are two different flows, they don't intermix

Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13

For install, use realpath like this:

 execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)
 rdma_install_symlink("${LINK_PATH}/libibverbs-dv-mlx5.so.1.0.13" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")

The rdma_install_symlink helper takes care of the install step.

Jason
_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/mailman/listinfo/ewg

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2143 bytes --]

On Tue, Feb 07, 2017 at 11:42:06AM -0700, Jason Gunthorpe wrote:
> On Tue, Feb 07, 2017 at 08:35:38PM +0200, Leon Romanovsky wrote:
> > I made it (rdma_shared_provider function), but have a very hard time
> > to properly create ".." symlink, because during the build (in place too)
> > the output is placed in build/lib in flat structure and symlinks need to
> > be without "..". But during installation phase, these symlinks should
> > be changed to ".." and it doesn't work for me in automatic way :(
>
> For symlinks build/ and installed are two different flows, they don't intermix
>
> Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
> and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13

Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.

>
> For install, use realpath like this:
>
>  execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)

It doesn't work for me :(
-- verbs_provder_dir = /usr/lib64/libibverbs
-- install_libdir = /usr/lib64
-- link_path =

lrwxrwxrwx 1 root root     18 Feb  7 21:43 /lib64/libibverbs/libmlx5-rdmav2.so -> /libmlx5.so.1.0.13
lrwxrwxrwx 1 root root     10 Feb  7 21:43 /lib64/libmlx5-rdmav2.so -> libmlx5.so
lrwxrwxrwx 1 root root     12 Feb  7 21:43 /lib64/libmlx5.so -> libmlx5.so.1
lrwxrwxrwx 1 root root     17 Feb  7 21:43 /lib64/libmlx5.so.1 -> libmlx5.so.1.0.13
-rwxr-xr-x 1 root root 666637 Feb  7 21:43 /lib64/libmlx5.so.1.0.13
lrwxrwxrwx 1 root root     18 Feb  7 21:43 /usr/lib64/libibverbs/libmlx5-rdmav2.so -> /libmlx5.so.1.0.13
lrwxrwxrwx 1 root root     10 Feb  7 21:43 /usr/lib64/libmlx5-rdmav2.so -> libmlx5.so
lrwxrwxrwx 1 root root     12 Feb  7 21:43 /usr/lib64/libmlx5.so -> libmlx5.so.1
lrwxrwxrwx 1 root root     17 Feb  7 21:43 /usr/lib64/libmlx5.so.1 -> libmlx5.so.1.0.13
-rwxr-xr-x 1 root root 666637 Feb  7 21:43 /usr/lib64/libmlx5.so.1.0.13


>  rdma_install_symlink("${LINK_PATH}/libibverbs-dv-mlx5.so.1.0.13" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")
>
> The rdma_install_symlink helper takes care of the install step.
>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 4311 bytes --]

On Tue, Feb 07, 2017 at 09:47:59PM +0200, Leon Romanovsky wrote:
> On Tue, Feb 07, 2017 at 11:42:06AM -0700, Jason Gunthorpe wrote:
> > On Tue, Feb 07, 2017 at 08:35:38PM +0200, Leon Romanovsky wrote:
> > > I made it (rdma_shared_provider function), but have a very hard time
> > > to properly create ".." symlink, because during the build (in place too)
> > > the output is placed in build/lib in flat structure and symlinks need to
> > > be without "..". But during installation phase, these symlinks should
> > > be changed to ".." and it doesn't work for me in automatic way :(
> >
> > For symlinks build/ and installed are two different flows, they don't intermix
> >
> > Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
> > and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13
>
> Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.
>
> >
> > For install, use realpath like this:
> >
> >  execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)
>
> It doesn't work for me :(
> -- verbs_provder_dir = /usr/lib64/libibverbs
> -- install_libdir = /usr/lib64
> -- link_path =
>
> lrwxrwxrwx 1 root root     18 Feb  7 21:43 /lib64/libibverbs/libmlx5-rdmav2.so -> /libmlx5.so.1.0.13
> lrwxrwxrwx 1 root root     10 Feb  7 21:43 /lib64/libmlx5-rdmav2.so -> libmlx5.so
> lrwxrwxrwx 1 root root     12 Feb  7 21:43 /lib64/libmlx5.so -> libmlx5.so.1
> lrwxrwxrwx 1 root root     17 Feb  7 21:43 /lib64/libmlx5.so.1 -> libmlx5.so.1.0.13
> -rwxr-xr-x 1 root root 666637 Feb  7 21:43 /lib64/libmlx5.so.1.0.13
> lrwxrwxrwx 1 root root     18 Feb  7 21:43 /usr/lib64/libibverbs/libmlx5-rdmav2.so -> /libmlx5.so.1.0.13
> lrwxrwxrwx 1 root root     10 Feb  7 21:43 /usr/lib64/libmlx5-rdmav2.so -> libmlx5.so
> lrwxrwxrwx 1 root root     12 Feb  7 21:43 /usr/lib64/libmlx5.so -> libmlx5.so.1
> lrwxrwxrwx 1 root root     17 Feb  7 21:43 /usr/lib64/libmlx5.so.1 -> libmlx5.so.1.0.13
> -rwxr-xr-x 1 root root 666637 Feb  7 21:43 /usr/lib64/libmlx5.so.1.0.13
>
>
> >  rdma_install_symlink("${LINK_PATH}/libibverbs-dv-mlx5.so.1.0.13" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")
> >
> > The rdma_install_symlink helper takes care of the install step.
> >

At the end, my function looks as follow:

# Create a special provider with exported symbols in it
function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
  # Installed driver file
  file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n")
  install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")

  # Uninstalled driver file
  file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
  file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")

  # Create a static provider library
  if (ENABLE_STATIC)
    add_library(${DEST} STATIC ${ARGN})
    set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
    install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")

    list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
    set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")
  endif()

  # Create the plugin shared library
  add_library(${DEST} SHARED ${ARGN})
  # Even though these are modules we still want to use Wl,--no-undefined
  set_target_properties(${DEST} PROPERTIES LINK_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
  rdma_set_library_map(${DEST} ${VERSION_SCRIPT})

  target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
  target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
  target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
  set_target_properties(${DEST} PROPERTIES
  	SOVERSION ${SOVERSION}
  	VERSION ${VERSION}
  	LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
  add_custom_target(share_link ALL DEPENDS "${DEST}"  COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"
	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")
  add_dependencies(share_link ${DEST})

  install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
  install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")
  rdma_install_symlink("${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION}" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")
endfunction()

> > Jason



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Jason Gunthorpe]

On Tue, Feb 07, 2017 at 10:14:28PM +0200, Leon Romanovsky wrote:

> > > Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
> > > and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13
> >
> > Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.

That will clash with the legacy providers, don't recommend it.

> > > For install, use realpath like this:
> > >
> > >  execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)
> >
> > It doesn't work for me :(

Ah realpath needs the paths to exist.. Sigh

Another alternative is this:

python -c "import os; print(os.path.relpath('/usr/lib64/libmlx4.so','/usr/lib64/libibverbs'));"

>   add_custom_target(share_link ALL DEPENDS "${DEST}"  COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"
> 	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")
>   add_dependencies(share_link ${DEST})

The doesn't need a build time, just do:

    execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink
                 "lib${DEST}.so.${VERSION}"
		 "${BUILD_LIB}/lib${DEST}-rdmav2.so")

It creates a dangling link until compilation, which is fine..
 
>   install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")

This line isn't needed, it is part of the next function

>   rdma_install_symlink("${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION}" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")

Should be

   rdma_install_symlink("${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")

And most distros will not allow an absolute link under /usr which is
why you need to make a relpath work and use that instead of
${CMAKE_INSTALL_LIBDIR}

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2770 bytes --]

On Tue, Feb 07, 2017 at 01:59:30PM -0700, Jason Gunthorpe wrote:
> On Tue, Feb 07, 2017 at 10:14:28PM +0200, Leon Romanovsky wrote:
>
> > > > Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
> > > > and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13
> > >
> > > Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.
>
> That will clash with the legacy providers, don't recommend it.

Is it real scenario? Will these legacy providers co-exists with
rdma-core library? I really don't like libibverbs-dv-mlx5 name, it is
too long and have feature name in the name.

>
> > > > For install, use realpath like this:
> > > >
> > > >  execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)
> > >
> > > It doesn't work for me :(
>
> Ah realpath needs the paths to exist.. Sigha
>
> Another alternative is this:
>
> python -c "import os; print(os.path.relpath('/usr/lib64/libmlx4.so','/usr/lib64/libibverbs'));"

It prints to the screen, but doesn't update OUTPUT_VARAIBLE :(
115  execute_process(COMMAND python -c "import os; print(os.path.relpath('/usr/lib64/libmlx5.so','/usr/lib64/libibverbs'));"
116                 OUTPUT_VARIBALE LINK_PATH)
117   message(STATUS "link_path = ${LINK_PATH}")

..
-- Performing Test HAVE_C_WREDUNDANT_DECLS - Failed
../libmlx5.so
-- link_path =
-- Missing Optional Items:

>
> >   add_custom_target(share_link ALL DEPENDS "${DEST}"  COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"
> > 	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")
> >   add_dependencies(share_link ${DEST})
>
> The doesn't need a build time, just do:
>
>     execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink
>                  "lib${DEST}.so.${VERSION}"
> 		 "${BUILD_LIB}/lib${DEST}-rdmav2.so")
>
> It creates a dangling link until compilation, which is fine..

In non-ninja builds, it doesn't create libmlx5-rdmav2.so symlink in
build/lib, because libmlx5.so is not created yet. This is why I ended
with custom target.

>
> >   install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")
>
> This line isn't needed, it is part of the next function

Not for build/lib

>
> >   rdma_install_symlink("${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION}" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")
>
> Should be
>
>    rdma_install_symlink("${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")a

Thanks for catching this.

>
> And most distros will not allow an absolute link under /usr which is
> why you need to make a relpath work and use that instead of
> ${CMAKE_INSTALL_LIBDIR}

It was my desperate attempt to make it work.

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

On Wed, Feb 08, 2017 at 08:37:58AM +0200, Leon Romanovsky wrote:
> On Tue, Feb 07, 2017 at 01:59:30PM -0700, Jason Gunthorpe wrote:
> > Ah realpath needs the paths to exist.. Sigha
> >
> > Another alternative is this:
> >
> > python -c "import os; print(os.path.relpath('/usr/lib64/libmlx4.so','/usr/lib64/libibverbs'));"
>
> It prints to the screen, but doesn't update OUTPUT_VARAIBLE :(
> 115  execute_process(COMMAND python -c "import os; print(os.path.relpath('/usr/lib64/libmlx5.so','/usr/lib64/libibverbs'));"
> 116                 OUTPUT_VARIBALE LINK_PATH)
> 117   message(STATUS "link_path = ${LINK_PATH}")
>
> ..
> -- Performing Test HAVE_C_WREDUNDANT_DECLS - Failed
> ../libmlx5.so
> -- link_path =
> -- Missing Optional Items:

Awesome,
It was misspelled "OUTPUT_VARIBALE" instead of "OUTPUT_VARIABLE"
together with beautiful world of cmake.
https://cmake.org/Wiki/CMake_FAQ#How_can_I_get_quoting_and_escapes_to_work_properly.3F

At the end, I put it into standalone python script to overcome it.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Jason Gunthorpe]

On Wed, Feb 08, 2017 at 08:37:58AM +0200, Leon Romanovsky wrote:
> On Tue, Feb 07, 2017 at 01:59:30PM -0700, Jason Gunthorpe wrote:
> > On Tue, Feb 07, 2017 at 10:14:28PM +0200, Leon Romanovsky wrote:
> >
> > > > > Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
> > > > > and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13
> > > >
> > > > Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.
> >
> > That will clash with the legacy providers, don't recommend it.
> 
> Is it real scenario? Will these legacy providers co-exists with
> rdma-core library? I really don't like libibverbs-dv-mlx5 name, it is
> too long and have feature name in the name.

I'd save having the 'dv' in the name is sort of the point...

The possible issue would be if an app links to this new library and is
run on an old system with the old type of library - then you'd get a
really ugly linker error message. It is best to avoid re using library
names.

That said, the old name is libmlx4-rdmav2.so so it should be OK..

> >     execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink
> >                  "lib${DEST}.so.${VERSION}"
> > 		 "${BUILD_LIB}/lib${DEST}-rdmav2.so")
> >
> > It creates a dangling link until compilation, which is fine..
> 
> In non-ninja builds, it doesn't create libmlx5-rdmav2.so symlink in
> build/lib, because libmlx5.so is not created yet. This is why I ended
> with custom target.

Eh? That doesn't sound right.

> > >   install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")
> >
> > This line isn't needed, it is part of the next function
> 
> Not for build/lib
> 

? Do not install something calleld lib*-rdmav2.so to /usr/lib/

Jason
_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/mailman/listinfo/ewg

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 5973 bytes --]

On Wed, Feb 08, 2017 at 10:33:35AM -0700, Jason Gunthorpe wrote:
> On Wed, Feb 08, 2017 at 08:37:58AM +0200, Leon Romanovsky wrote:
> > On Tue, Feb 07, 2017 at 01:59:30PM -0700, Jason Gunthorpe wrote:
> > > On Tue, Feb 07, 2017 at 10:14:28PM +0200, Leon Romanovsky wrote:
> > >
> > > > > > Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
> > > > > > and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13
> > > > >
> > > > > Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.
> > >
> > > That will clash with the legacy providers, don't recommend it.
> >
> > Is it real scenario? Will these legacy providers co-exists with
> > rdma-core library? I really don't like libibverbs-dv-mlx5 name, it is
> > too long and have feature name in the name.
>
> I'd save having the 'dv' in the name is sort of the point...
>
> The possible issue would be if an app links to this new library and is
> run on an old system with the old type of library - then you'd get a
> really ugly linker error message. It is best to avoid re using library
> names.
>
> That said, the old name is libmlx4-rdmav2.so so it should be OK..
>
> > >     execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink
> > >                  "lib${DEST}.so.${VERSION}"
> > > 		 "${BUILD_LIB}/lib${DEST}-rdmav2.so")
> > >
> > > It creates a dangling link until compilation, which is fine..
> >
> > In non-ninja builds, it doesn't create libmlx5-rdmav2.so symlink in
> > build/lib, because libmlx5.so is not created yet. This is why I ended
> > with custom target.
>
> Eh? That doesn't sound right.
>
> > > >   install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")
> > >
> > > This line isn't needed, it is part of the next function
> >
> > Not for build/lib
> >
>
> ? Do not install something calleld lib*-rdmav2.so to /usr/lib/

Thanks for the help, my final version which works correctly for build in place, install from
sources and packages for centos6/centos6 is below:

----
# Create a special provider with exported symbols in it
function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
  # Installed driver file
  file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n")
  install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")

  # Uninstalled driver file
  file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
  file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")

  # Create a static provider library
  if (ENABLE_STATIC)
    add_library(${DEST} STATIC ${ARGN})
    set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
    install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")

    list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
    set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")
  endif()

  # Create the plugin shared library
  add_library(${DEST} SHARED ${ARGN})
  # Even though these are modules we still want to use Wl,--no-undefined
  set_target_properties(${DEST} PROPERTIES LINK_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
  rdma_set_library_map(${DEST} ${VERSION_SCRIPT})

  target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
  target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
  target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
  set_target_properties(${DEST} PROPERTIES
  	SOVERSION ${SOVERSION}
  	VERSION ${VERSION}
  	LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
  add_custom_target(share_link ALL DEPENDS "${DEST}"  COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"
	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")
  add_dependencies(share_link ${DEST})

  install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
  execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
    ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
    OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
  rdma_install_symlink("${DEST_LINK_PATH}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")
endfunction()

----
and buildlib/relpath
---
import os
import sys

print(os.path.relpath(sys.argv[1], sys.argv[2]))
---

But I still have issues with DEB package.
It creates absolute (and wrong)  symlink instead of relative one.

build/cbuild pkg jessie
...
DESTDIR=/home/leonro/src/debian/tmp ninja -C build-deb install
ninja: Entering directory `build-deb'
[1/2] cd /home/leonro/src/build-deb/providers/mlx5 && /usr/bin/cmake -E create_symlink libmlx5.so.1.0.13 /home/leonro/src/build-deb/lib/libmlx5-rdmav2.so
[2/2] Install the project...
....

➜  ls -l  usr/lib/x86_64-linux-gnu/libibverbs
total 364K
drwxr-xr-x 2 leonro leonro 320 Feb  8 17:17 .
drwxr-xr-x 3 leonro leonro 120 Feb  8 17:17 ..
-rw-r--r-- 1 leonro leonro 22K Feb  8 17:13 libcxgb3-rdmav2.so
-rw-r--r-- 1 leonro leonro 30K Feb  8 17:13 libcxgb4-rdmav2.so
-rw-r--r-- 1 leonro leonro 18K Feb  8 17:13 libhfi1verbs-rdmav2.so
-rw-r--r-- 1 leonro leonro 22K Feb  8 17:13 libhns-rdmav2.so
-rw-r--r-- 1 leonro leonro 30K Feb  8 17:13 libi40iw-rdmav2.so
-rw-r--r-- 1 leonro leonro 18K Feb  8 17:13 libipathverbs-rdmav2.so
-rw-r--r-- 1 leonro leonro 46K Feb  8 17:13 libmlx4-rdmav2.so
lrwxrwxrwx 1 leonro leonro  65 Feb  8 17:13 libmlx5-rdmav2.so -> /home/leonro/src/build-deb/lib/x86_64-linux-gnu/libmlx5.so.1.0.13
-rw-r--r-- 1 leonro leonro 34K Feb  8 17:13 libmthca-rdmav2.so
-rw-r--r-- 1 leonro leonro 23K Feb  8 17:13 libnes-rdmav2.so
-rw-r--r-- 1 leonro leonro 26K Feb  8 17:13 libocrdma-rdmav2.so
-rw-r--r-- 1 leonro leonro 34K Feb  8 17:13 libqedr-rdmav2.so
-rw-r--r-- 1 leonro leonro 18K Feb  8 17:13 librxe-rdmav2.so
-rw-r--r-- 1 leonro leonro 18K Feb  8 17:13 libvmw_pvrdma-rdmav2.so

Any pointer what should I need to fix in debian/ to make it work?

Thanks

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Jason Gunthorpe]

On Wed, Feb 08, 2017 at 08:01:04PM +0200, Leon Romanovsky wrote:
> On Wed, Feb 08, 2017 at 10:33:35AM -0700, Jason Gunthorpe wrote:

> Thanks for the help, my final version which works correctly for build in place, install from
> sources and packages for centos6/centos6 is below:

Looks good

> # Create a special provider with exported symbols in it
> function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
>   # Installed driver file
>   file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n")
>   install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")
> 
>   # Uninstalled driver file
>   file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
>   file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")
> 
>   # Create a static provider library
>   if (ENABLE_STATIC)
>     add_library(${DEST} STATIC ${ARGN})
>     set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
>     install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
> 
>     list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
>     set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")
>   endif()
> 
>   # Create the plugin shared library
>   add_library(${DEST} SHARED ${ARGN})
>   # Even though these are modules we still want to use Wl,--no-undefined
>   set_target_properties(${DEST} PROPERTIES LINK_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
>   rdma_set_library_map(${DEST} ${VERSION_SCRIPT})
> 
>   target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
>   target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
>   target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
>   set_target_properties(${DEST} PROPERTIES
>   	SOVERSION ${SOVERSION}
>   	VERSION ${VERSION}
>   	LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
>   add_custom_target(share_link ALL DEPENDS "${DEST}"  COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"
> 	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")
>   add_dependencies(share_link ${DEST})

Except this really shouldn't be a rule.  The non-rule method is used
everywhere else (eg man pages), so it must work here, it doesn't make
sense that ninja vs make would be any different. When you get
everything working put it back to execute_process..

>   install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
>   execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
>     ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
>     OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
>   rdma_install_symlink("${DEST_LINK_PATH}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")

> and buildlib/relpath
> import os
> import sys
> 
> print(os.path.relpath(sys.argv[1], sys.argv[2]))

Sure, yes, doing the escaping right would probably be quite hard, that is the
usual nightmare with shell stuff unfortunately.

> But I still have issues with DEB package.
> It creates absolute (and wrong)  symlink instead of relative one.

Hmm. Can you update your github? I'll look for you.

> -rw-r--r-- 1 leonro leonro 46K Feb  8 17:13 libmlx4-rdmav2.so
> lrwxrwxrwx 1 leonro leonro  65 Feb  8 17:13 libmlx5-rdmav2.so -> /home/leonro/src/build-deb/lib/x86_64-linux-gnu/libmlx5.so.1.0.13

Most likely this means that DEST_LINK_PATH passed to
rdma_install_symlink is not correct. Which suggests python relpath is
not working..

relpath in python is done textually, from these args:

>     ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}

But

 set(VERBS_PROVIDER_DIR "${CMAKE_INSTALL_FULL_LIBDIR}/libibverbs"

So if CMAKE_INSTALL_FULL_LIBDIR != CMAKE_INSTALL_LIBDIR (which depends
on how the distro build script configures cmake) then it will fail.

Switch to:

     ${CMAKE_INSTALL_FULL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}

Jason
_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/mailman/listinfo/ewg

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 5771 bytes --]

On Wed, Feb 08, 2017 at 11:18:20AM -0700, Jason Gunthorpe wrote:
> On Wed, Feb 08, 2017 at 08:01:04PM +0200, Leon Romanovsky wrote:
> > On Wed, Feb 08, 2017 at 10:33:35AM -0700, Jason Gunthorpe wrote:
>
> > Thanks for the help, my final version which works correctly for build in place, install from
> > sources and packages for centos6/centos6 is below:
>
> Looks good
>
> > # Create a special provider with exported symbols in it
> > function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
> >   # Installed driver file
> >   file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n")
> >   install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")
> >
> >   # Uninstalled driver file
> >   file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
> >   file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")
> >
> >   # Create a static provider library
> >   if (ENABLE_STATIC)
> >     add_library(${DEST} STATIC ${ARGN})
> >     set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
> >     install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
> >
> >     list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
> >     set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")
> >   endif()
> >
> >   # Create the plugin shared library
> >   add_library(${DEST} SHARED ${ARGN})
> >   # Even though these are modules we still want to use Wl,--no-undefined
> >   set_target_properties(${DEST} PROPERTIES LINK_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
> >   rdma_set_library_map(${DEST} ${VERSION_SCRIPT})
> >
> >   target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
> >   target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
> >   target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
> >   set_target_properties(${DEST} PROPERTIES
> >   	SOVERSION ${SOVERSION}
> >   	VERSION ${VERSION}
> >   	LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
> >   add_custom_target(share_link ALL DEPENDS "${DEST}"  COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"
> > 	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")
> >   add_dependencies(share_link ${DEST})
>
> Except this really shouldn't be a rule.  The non-rule method is used
> everywhere else (eg man pages), so it must work here, it doesn't make
> sense that ninja vs make would be any different. When you get
> everything working put it back to execute_process..
>
> >   install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
> >   execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
> >     ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
> >     OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
> >   rdma_install_symlink("${DEST_LINK_PATH}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")
>
> > and buildlib/relpath
> > import os
> > import sys
> >
> > print(os.path.relpath(sys.argv[1], sys.argv[2]))
>
> Sure, yes, doing the escaping right would probably be quite hard, that is the
> usual nightmare with shell stuff unfortunately.
>
> > But I still have issues with DEB package.
> > It creates absolute (and wrong)  symlink instead of relative one.
>
> Hmm. Can you update your github? I'll look for you.

I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
https://github.com/rleon/rdma-core/commits/m/dv-v1
Updated cover letter with changelog:
https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977fbf95aa7ee47a69e
And all cmake/packages stuff in the patch:
https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d568476314c1f65ddc

>
> > -rw-r--r-- 1 leonro leonro 46K Feb  8 17:13 libmlx4-rdmav2.so
> > lrwxrwxrwx 1 leonro leonro  65 Feb  8 17:13 libmlx5-rdmav2.so -> /home/leonro/src/build-deb/lib/x86_64-linux-gnu/libmlx5.so.1.0.13
>
> Most likely this means that DEST_LINK_PATH passed to
> rdma_install_symlink is not correct. Which suggests python relpath is
> not working..
>
> relpath in python is done textually, from these args:
>
> >     ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
>
> But
>
>  set(VERBS_PROVIDER_DIR "${CMAKE_INSTALL_FULL_LIBDIR}/libibverbs"
>
> So if CMAKE_INSTALL_FULL_LIBDIR != CMAKE_INSTALL_LIBDIR (which depends
> on how the distro build script configures cmake) then it will fail.
>
> Switch to:
>
>      ${CMAKE_INSTALL_FULL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}

You are right, it fixed debian, I'm testing other packages now.
diff --git a/buildlib/rdma_functions.cmake b/buildlib/rdma_functions.cmake
index 3d0683d0..3a1758f8 100644
--- a/buildlib/rdma_functions.cmake
+++ b/buildlib/rdma_functions.cmake
@@ -110,9 +110,15 @@ function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
   add_dependencies(share_link ${DEST})

   install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
-  execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
-    ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
-    OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
+  if (NOT ${CMAKE_INSTALL_FULL_LIBDIR} STREQUAL ${CMAKE_INSTALL_LIBDIR})
+    execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
+	    ${CMAKE_INSTALL_FULL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
+      OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
+  else()
+    execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
+      ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
+      OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
+  endif()
   rdma_install_symlink("${DEST_LINK_PATH}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")
 endfunction()


>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Jason Gunthorpe]

On Wed, Feb 08, 2017 at 08:54:46PM +0200, Leon Romanovsky wrote:

> > Hmm. Can you update your github? I'll look for you.
> 
> I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
> https://github.com/rleon/rdma-core/commits/m/dv-v1
> Updated cover letter with changelog:
> https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977fbf95aa7ee47a69e
> And all cmake/packages stuff in the patch:
> https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d568476314c1f65ddc

This should take care of most things

https://github.com/jgunthorpe/rdma-plumbing/commit/490b7fb184c414ab16b7056f1effa9136e8afe46

The debian packaging still needs some adjusting.

I suspect it is against Debian policy to include libmlx.so in the
providers package, it probably needs to have its own package and its
own -dev package.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1031 bytes --]

On Wed, Feb 08, 2017 at 01:56:22PM -0700, Jason Gunthorpe wrote:
> On Wed, Feb 08, 2017 at 08:54:46PM +0200, Leon Romanovsky wrote:
>
> > > Hmm. Can you update your github? I'll look for you.
> >
> > I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
> > https://github.com/rleon/rdma-core/commits/m/dv-v1
> > Updated cover letter with changelog:
> > https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977fbf95aa7ee47a69e
> > And all cmake/packages stuff in the patch:
> > https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d568476314c1f65ddc
>
> This should take care of most things
>
> https://github.com/jgunthorpe/rdma-plumbing/commit/490b7fb184c414ab16b7056f1effa9136e8afe46

Thanks a lot, we will add into our v1.

>
> The debian packaging still needs some adjusting.
>
> I suspect it is against Debian policy to include libmlx.so in the
> providers package, it probably needs to have its own package and its
> own -dev package.

I don't know, at the end, it is provider.

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OFED-4.8, rdma-core, and library paths

[Benjamin Drung]

Am Mittwoch, den 08.02.2017, 13:56 -0700 schrieb Jason Gunthorpe:
> On Wed, Feb 08, 2017 at 08:54:46PM +0200, Leon Romanovsky wrote:
> 
> > > Hmm. Can you update your github? I'll look for you.
> > 
> > I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
> > https://github.com/rleon/rdma-core/commits/m/dv-v1
> > Updated cover letter with changelog:
> > https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977f
> > bf95aa7ee47a69e
> > And all cmake/packages stuff in the patch:
> > https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d56
> > 8476314c1f65ddc
> 
> This should take care of most things
> 
> https://github.com/jgunthorpe/rdma-plumbing/commit/490b7fb184c414ab16
> b7056f1effa9136e8afe46
> 
> The debian packaging still needs some adjusting.
> 
> I suspect it is against Debian policy to include libmlx.so in the
> providers package, it probably needs to have its own package and its
> own -dev package.

When other packages should be able to link against libmlx.so and need
header files for that, it is better to have separate binary packages
(libmlx and libmlx-dev) for it.

-- 
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL:  http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
Geschäftsführer: Andreas Gauger, Achim Weiss.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1.1: Type: text/plain, Size: 1959 bytes --]

On Thu, Feb 09, 2017 at 10:53:26AM +0100, Benjamin Drung wrote:
> Am Mittwoch, den 08.02.2017, 13:56 -0700 schrieb Jason Gunthorpe:
> > On Wed, Feb 08, 2017 at 08:54:46PM +0200, Leon Romanovsky wrote:
> >
> > > > Hmm. Can you update your github? I'll look for you.
> > >
> > > I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
> > > https://github.com/rleon/rdma-core/commits/m/dv-v1
> > > Updated cover letter with changelog:
> > > https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977f
> > > bf95aa7ee47a69e
> > > And all cmake/packages stuff in the patch:
> > > https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d56
> > > 8476314c1f65ddc
> >
> > This should take care of most things
> >
> > https://github.com/jgunthorpe/rdma-plumbing/commit/490b7fb184c414ab16
> > b7056f1effa9136e8afe46
> >
> > The debian packaging still needs some adjusting.
> >
> > I suspect it is against Debian policy to include libmlx.so in the
> > providers package, it probably needs to have its own package and its
> > own -dev package.
>
> When other packages should be able to link against libmlx.so and need
> header files for that, it is better to have separate binary packages
> (libmlx and libmlx-dev) for it.

This libmlx is dependent on ibverbs-provider and libibverbs. It is useless
without them and will be always installed together, so why do we need to
complicate users life by adding new library?

And we can always spin-off it as a separate package once someone will need it.

Thanks

>
> --
> Benjamin Drung
> System Developer
> Debian & Ubuntu Developer
>
> ProfitBricks GmbH
> Greifswalder Str. 207
> D - 10405 Berlin
>
> Email: benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
> URL:  http://www.profitbricks.com
>
> Sitz der Gesellschaft: Berlin.
> Registergericht: Amtsgericht Charlottenburg, HRB 125506B.
> Geschäftsführer: Andreas Gauger, Achim Weiss.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/mailman/listinfo/ewg

Re: OFED-4.8, rdma-core, and library paths

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2737 bytes --]

On Wed, Feb 08, 2017 at 11:18:20AM -0700, Jason Gunthorpe wrote:
> On Wed, Feb 08, 2017 at 08:01:04PM +0200, Leon Romanovsky wrote:
> > On Wed, Feb 08, 2017 at 10:33:35AM -0700, Jason Gunthorpe wrote:
>
> > Thanks for the help, my final version which works correctly for build in place, install from
> > sources and packages for centos6/centos6 is below:
>
> Looks good
>
> > # Create a special provider with exported symbols in it
> > function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
> >   # Installed driver file
> >   file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n")
> >   install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")
> >
> >   # Uninstalled driver file
> >   file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
> >   file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")
> >
> >   # Create a static provider library
> >   if (ENABLE_STATIC)
> >     add_library(${DEST} STATIC ${ARGN})
> >     set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
> >     install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
> >
> >     list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
> >     set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")
> >   endif()
> >
> >   # Create the plugin shared library
> >   add_library(${DEST} SHARED ${ARGN})
> >   # Even though these are modules we still want to use Wl,--no-undefined
> >   set_target_properties(${DEST} PROPERTIES LINK_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
> >   rdma_set_library_map(${DEST} ${VERSION_SCRIPT})
> >
> >   target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
> >   target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
> >   target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
> >   set_target_properties(${DEST} PROPERTIES
> >   	SOVERSION ${SOVERSION}
> >   	VERSION ${VERSION}
> >   	LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
> >   add_custom_target(share_link ALL DEPENDS "${DEST}"  COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"
> > 	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")
> >   add_dependencies(share_link ${DEST})
>
> Except this really shouldn't be a rule.  The non-rule method is used
> everywhere else (eg man pages), so it must work here, it doesn't make
> sense that ninja vs make would be any different. When you get
> everything working put it back to execute_process..

I tried it now and with ninja run, I'm getting the following error:
-- Performing Test HAVE_C_WREDUNDANT_DECLS - Success
failed to create symbolic link '/home/leonro/src/rdma-core/build/lib/libmlx5-rdmav2.so': No such file or directory
-- Missing Optional Items:

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]