rdma-core, cmake/ninja question

rdma-core, cmake/ninja question

[Steve Wise]

Hey Jason, newb question:  

How do I force the rdma-core to build with /usr as the install prefix, /etc as
the config dir, and /usr/lib64 as the lib dir?  It seems to default to
/usr/local..

Thanks!

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core, cmake/ninja question

[Jason Gunthorpe]

On Thu, Sep 22, 2016 at 10:35:06AM -0500, Steve Wise wrote:
> Hey Jason, newb question:  
> 
> How do I force the rdma-core to build with /usr as the install prefix, /etc as
> the config dir, and /usr/lib64 as the lib dir?  It seems to default to
> /usr/local..

Yes, that is the GNU standard - it is what configure did.

The equivilant to configure --prefix is:
     -DCMAKE_INSTALL_PREFIX=/usr

You often also need.
     -DCMAKE_INSTALL_SYSCONFDIR=/etc

There are many paths that are configurable, each distro sets them as
appropriate via the packaging scripts. For instance this is what RPM
tells cmake to do on centos6:

/usr/bin/cmake -DCMAKE_VERBOSE_MAKEFILE=ON
-DCMAKE_INSTALL_PREFIX:PATH=/usr
-DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64
-DINCLUDE_INSTALL_DIR:PATH=/usr/include
-DLIB_INSTALL_DIR:PATH=/usr/lib64
-DSYSCONF_INSTALL_DIR:PATH=/etc
-DSHARE_INSTALL_PREFIX:PATH=/usr/share
-DLIB_SUFFIX=64
-DCMAKE_SKIP_RPATH:BOOL=ON
-DBUILD_SHARED_LIBS:BOOL=ON
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_INSTALL_BINDIR:PATH=/usr/bin
-DCMAKE_INSTALL_SBINDIR:PATH=/usr/sbin
-DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64
-DCMAKE_INSTALL_LIBEXECDIR:PATH=/usr/libexec
-DCMAKE_INSTALL_LOCALSTATEDIR:PATH=/var
-DCMAKE_INSTALL_SHAREDSTATEDIR:PATH=/var/lib
-DCMAKE_INSTALL_INCLUDEDIR:PATH=/usr/include
-DCMAKE_INSTALL_INFODIR:PATH=/usr/share/info
-DCMAKE_INSTALL_MANDIR:PATH=/usr/share/man
-DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc
-DCMAKE_INSTALL_SYSTEMD_SERVICEDIR:PATH=/tmp/
-DCMAKE_INSTALL_INITDDIR:PATH=/etc/rc.d/init.d

Each distro is a little different. This is all the same stuff as
configure, just different names..

I recommend *against* running 'make install' to /usr. This is a big
package now, it is hard to unwind it once it installs over your
system. I really should finish the run in place patch :|

If you do this:

$ echo /usr/local/lib64/ > /etc/ld.so.conf.d/usrlocal.conf && ldconfig

Then you can install to /usr/local/ and things will work as you
expect. Note, this depends on a verbs patch that is still only in my
preview tree.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core, cmake/ninja question

[Steve Wise]

> 
> I recommend *against* running 'make install' to /usr. This is a big
> package now, it is hard to unwind it once it installs over your
> system. I really should finish the run in place patch :|
> 
> If you do this:
> 
> $ echo /usr/local/lib64/ > /etc/ld.so.conf.d/usrlocal.conf && ldconfig
> 
> Then you can install to /usr/local/ and things will work as you
> expect. Note, this depends on a verbs patch that is still only in my
> preview tree.

So I have a dev system setup with everything installed from the distro in /usr.
I have my rdma-core git tree.  I change libcxgb4 to fix a bug.  I want to test
it.  I install it to /usr/local and add /etc/ld.so.conf.d/usrlocal.conf.  But
that still doesn't work because ldconfig still finds libcxgb4 in /usr/lib64
first...

How can I force ldconfig to look in /usr/local/lib64 first?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core, cmake/ninja question

[Jason Gunthorpe]

On Thu, Sep 22, 2016 at 11:21:53AM -0500, Steve Wise wrote:
> > 
> > I recommend *against* running 'make install' to /usr. This is a big
> > package now, it is hard to unwind it once it installs over your
> > system. I really should finish the run in place patch :|
> > 
> > If you do this:
> > 
> > $ echo /usr/local/lib64/ > /etc/ld.so.conf.d/usrlocal.conf && ldconfig
> > 
> > Then you can install to /usr/local/ and things will work as you
> > expect. Note, this depends on a verbs patch that is still only in my
> > preview tree.
> 
> So I have a dev system setup with everything installed from the distro in /usr.
> I have my rdma-core git tree.  I change libcxgb4 to fix a bug.  I want to test
> it.  I install it to /usr/local and add /etc/ld.so.conf.d/usrlocal.conf.  But
> that still doesn't work because ldconfig still finds libcxgb4 in /usr/lib64
> first...

Apply this patch:

https://github.com/jgunthorpe/rdma-plumbing/commit/64ed4fd30a2157bdaf0a8f7f1b04d2dad02c1b8f

It hardwires the search path into libverbs and libverbs will look into
the place it was installed before searching the system library
path. That will eliminate the problem you mentioned above.

Further, after that patch is applied you can do a rough imitation of
'run-in-place' like this:

$ cmake .. -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib -DSYSCONF_INSTALL_DIR:PATH=/etc

(untested) That hardwires the build directory into libibverbs, so it
will try to load all drivers from there first.

Then use

$ export LD_LIBRARY_PATH=`pwd`/lib

And all verbs using programs started from that shell will use the full
new library set and the new providers without having to do make
install at all.

I plan to automate this basic approach for 'run in place', please let
me know if it works for you and makes sense.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core, cmake/ninja question

[Steve Wise]

> > So I have a dev system setup with everything installed from the distro in
/usr.
> > I have my rdma-core git tree.  I change libcxgb4 to fix a bug.  I want to
test
> > it.  I install it to /usr/local and add /etc/ld.so.conf.d/usrlocal.conf.
But
> > that still doesn't work because ldconfig still finds libcxgb4 in /usr/lib64
> > first...
> 
> Apply this patch:
> 
> https://github.com/jgunthorpe/rdma-
> plumbing/commit/64ed4fd30a2157bdaf0a8f7f1b04d2dad02c1b8f
> 
> It hardwires the search path into libverbs and libverbs will look into
> the place it was installed before searching the system library
> path. That will eliminate the problem you mentioned above.
> 
> Further, after that patch is applied you can do a rough imitation of
> 'run-in-place' like this:
> 
> $ cmake .. -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib -
> DSYSCONF_INSTALL_DIR:PATH=/etc
>

And how do I build this after the cmake?

 
> (untested) That hardwires the build directory into libibverbs, so it
> will try to load all drivers from there first.
> 
> Then use
> 
> $ export LD_LIBRARY_PATH=`pwd`/lib
> 
> And all verbs using programs started from that shell will use the full
> new library set and the new providers without having to do make
> install at all.
> 
> I plan to automate this basic approach for 'run in place', please let
> me know if it works for you and makes sense.
> 
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core, cmake/ninja question

[Jason Gunthorpe]

On Thu, Sep 22, 2016 at 01:43:36PM -0500, Steve Wise wrote:

> > Apply this patch:
> > 
> > https://github.com/jgunthorpe/rdma-
> > plumbing/commit/64ed4fd30a2157bdaf0a8f7f1b04d2dad02c1b8f
> > 
> > It hardwires the search path into libverbs and libverbs will look into
> > the place it was installed before searching the system library
> > path. That will eliminate the problem you mentioned above.
> > 
> > Further, after that patch is applied you can do a rough imitation of
> > 'run-in-place' like this:
> > 
> > $ cmake .. -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib -
> > DSYSCONF_INSTALL_DIR:PATH=/etc
> >
> 
> And how do I build this after the cmake?

Sorry, you always need the -GNinja, I was just being brief

I also mistyped the ETC name.

I tested this:

$ mkdir build ; cd build
$ cmake .. -GNinja -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib -DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc
$ ninja
$ strace bin/ibv_devinfo  2>&1 | grep -i rdmav2
open("/tmp/rdma-core/build/lib/libmlx4-rdmav2.so", O_RDONLY|O_CLOEXEC) = 3

Note LD_LIBRARY_PATH is needed to run things from outside build/bin/

I have some thoughts to make this simpler, call it a proof of concept for
run-from-build ..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core, cmake/ninja question

[Steve Wise]

> On Thu, Sep 22, 2016 at 01:43:36PM -0500, Steve Wise wrote:
> 
> > > Apply this patch:
> > >
> > > https://github.com/jgunthorpe/rdma-
> > > plumbing/commit/64ed4fd30a2157bdaf0a8f7f1b04d2dad02c1b8f
> > >
> > > It hardwires the search path into libverbs and libverbs will look into
> > > the place it was installed before searching the system library
> > > path. That will eliminate the problem you mentioned above.
> > >
> > > Further, after that patch is applied you can do a rough imitation of
> > > 'run-in-place' like this:
> > >
> > > $ cmake .. -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib -
> > > DSYSCONF_INSTALL_DIR:PATH=/etc
> > >
> >
> > And how do I build this after the cmake?
> 
> Sorry, you always need the -GNinja, I was just being brief
> 
> I also mistyped the ETC name.
> 
> I tested this:
> 
> $ mkdir build ; cd build
> $ cmake .. -GNinja -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib -
> DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc
> $ ninja
> $ strace bin/ibv_devinfo  2>&1 | grep -i rdmav2
> open("/tmp/rdma-core/build/lib/libmlx4-rdmav2.so", O_RDONLY|O_CLOEXEC) = 3
> 
> Note LD_LIBRARY_PATH is needed to run things from outside build/bin/
> 
> I have some thoughts to make this simpler, call it a proof of concept for
> run-from-build ..
> 

No worries.  I'm just thick headed. :)  Its working for me.  I've made a change
to libcxgb4 and can test by running in the build tree. I compile with and
without my change and see that the correct libcxgb4 is being used.  Ship it!  :)

So if I have a libcxgb4 change to submit, I should submit it against rdma-core,
ja?

Thanks Jason!

Steve.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma-core, cmake/ninja question

[Jason Gunthorpe]

On Thu, Sep 22, 2016 at 02:34:28PM -0500, Steve Wise wrote:

> So if I have a libcxgb4 change to submit, I should submit it against rdma-core,
> ja?

Yes, Doug or Leon will apply it.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma-core, cmake/ninja question

[Steve Wise]

> > I recommend *against* running 'make install' to /usr. This is a big
> > package now, it is hard to unwind it once it installs over your
> > system. I really should finish the run in place patch :|
> >
> > If you do this:
> >
> > $ echo /usr/local/lib64/ > /etc/ld.so.conf.d/usrlocal.conf && ldconfig
> >
> > Then you can install to /usr/local/ and things will work as you
> > expect. Note, this depends on a verbs patch that is still only in my
> > preview tree.
> 
> So I have a dev system setup with everything installed from the distro in
> /usr.  I have my rdma-core git tree.  I change libcxgb4 to fix a bug.  I
> want to test it.  I install it to /usr/local and add
> /etc/ld.so.conf.d/usrlocal.conf.  But that still doesn't work because
> ldconfig still finds libcxgb4 in /usr/lib64 first...
> 
> How can I force ldconfig to look in /usr/local/lib64 first?

Actually, I just adjusted the search order with LD_LIBRARY_PATH and got what I
needed.

Thanks!

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html