xio messenger is not building by default

xio messenger is not building by default

[Somnath Roy]

Hi Sage,
Why we are not building xio messenger by default like other experimental features ?
Let me know if you want to, I can put a patch in.

Thanks & Regards
Somnath

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: xio messenger is not building by default

[Sage Weil]

On Fri, 13 Mar 2015, Somnath Roy wrote:
> Hi Sage,
> Why we are not building xio messenger by default like other experimental features ?
> Let me know if you want to, I can put a patch in.

This is a build/packaging issue.  I think right now libxio needs to be 
present at build time and it's not available in any of the distros we 
build for.  Usually what we do in this situation is embed it as a git 
submodule (like we do with rocksdb).  Happy to take patches that do that 
if that's the best route (I suspect it is?).  There'll be a few final 
tweaks in the build environment then to set up the git repo mirror but 
nothing major.

I'd look at how things are handled with rocksdb now as a model?

sage

RE: xio messenger is not building by default

[Somnath Roy]

Hmm...I missed that part..I thought it would be a quick config fix :-(
Yes, git submodule should be the approach.

Thanks & Regards
Somnath

-----Original Message-----
From: Sage Weil [mailto:sage@newdream.net]
Sent: Friday, March 13, 2015 4:45 PM
To: Somnath Roy
Cc: ceph-devel@vger.kernel.org
Subject: Re: xio messenger is not building by default

On Fri, 13 Mar 2015, Somnath Roy wrote:
> Hi Sage,
> Why we are not building xio messenger by default like other experimental features ?
> Let me know if you want to, I can put a patch in.

This is a build/packaging issue.  I think right now libxio needs to be present at build time and it's not available in any of the distros we build for.  Usually what we do in this situation is embed it as a git submodule (like we do with rocksdb).  Happy to take patches that do that if that's the best route (I suspect it is?).  There'll be a few final tweaks in the build environment then to set up the git repo mirror but nothing major.

I'd look at how things are handled with rocksdb now as a model?

sage

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: xio messenger is not building by default

[Matt W. Benjamin]

I'll send a submodule change.  Apologies for the delay.

Matt

----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

> Hmm...I missed that part..I thought it would be a quick config fix
> :-(
> Yes, git submodule should be the approach.
> 
> Thanks & Regards
> Somnath
> 
> -----Original Message-----
> From: Sage Weil [mailto:sage@newdream.net]
> Sent: Friday, March 13, 2015 4:45 PM
> To: Somnath Roy
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: xio messenger is not building by default
> 
> On Fri, 13 Mar 2015, Somnath Roy wrote:
> > Hi Sage,
> > Why we are not building xio messenger by default like other
> experimental features ?
> > Let me know if you want to, I can put a patch in.
> 
> This is a build/packaging issue.  I think right now libxio needs to be
> present at build time and it's not available in any of the distros we
> build for.  Usually what we do in this situation is embed it as a git
> submodule (like we do with rocksdb).  Happy to take patches that do
> that if that's the best route (I suspect it is?).  There'll be a few
> final tweaks in the build environment then to set up the git repo
> mirror but nothing major.
> 
> I'd look at how things are handled with rocksdb now as a model?
> 
> sage
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message
> is intended only for the use of the designated recipient(s) named
> above. If the reader of this message is not the intended recipient,
> you are hereby notified that you have received this message in error
> and that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify the sender by telephone or
> e-mail (as shown above) immediately and destroy any and all copies of
> this message in your possession (whether hard copies or electronically
> stored copies).
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: xio messenger is not building by default

[Ken Dreyer]

On 03/13/2015 05:44 PM, Sage Weil wrote:
> On Fri, 13 Mar 2015, Somnath Roy wrote:
>> Hi Sage,
>> Why we are not building xio messenger by default like other experimental features ?
>> Let me know if you want to, I can put a patch in.
> 
> This is a build/packaging issue.  I think right now libxio needs to be 
> present at build time and it's not available in any of the distros we 
> build for.  Usually what we do in this situation is embed it as a git 
> submodule (like we do with rocksdb).  Happy to take patches that do that 
> if that's the best route (I suspect it is?).  There'll be a few final 
> tweaks in the build environment then to set up the git repo mirror but 
> nothing major.
> 
> I'd look at how things are handled with rocksdb now as a model?
> 

What are the packages that we would need in the distros? From
README.xio, it looks like we'd need the following to be packaged:

https://www.openfabrics.org/downloads/verbs/
https://github.com/accelio/accelio

Is there anything else that's missing?

- Ken

Re: xio messenger is not building by default

[Matt W. Benjamin]

Hi,

Arent OFED dependencies already packaged?  I assumed that only Accelio would
be a submodule.

Matt

----- "Ken Dreyer" <kdreyer@redhat.com> wrote:

> On 03/13/2015 05:44 PM, Sage Weil wrote:
> > On Fri, 13 Mar 2015, Somnath Roy wrote:
> >> Hi Sage,
> >> Why we are not building xio messenger by default like other
> experimental features ?
> >> Let me know if you want to, I can put a patch in.
> > 
> > This is a build/packaging issue.  I think right now libxio needs to
> be 
> > present at build time and it's not available in any of the distros
> we 
> > build for.  Usually what we do in this situation is embed it as a
> git 
> > submodule (like we do with rocksdb).  Happy to take patches that do
> that 
> > if that's the best route (I suspect it is?).  There'll be a few
> final 
> > tweaks in the build environment then to set up the git repo mirror
> but 
> > nothing major.
> > 
> > I'd look at how things are handled with rocksdb now as a model?
> > 
> 
> What are the packages that we would need in the distros? From
> README.xio, it looks like we'd need the following to be packaged:
> 
> https://www.openfabrics.org/downloads/verbs/
> https://github.com/accelio/accelio
> 
> Is there anything else that's missing?
> 
> - Ken
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: xio messenger is not building by default

[Ken Dreyer]

On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
> Hi,
> 
> Arent OFED dependencies already packaged?  I assumed that only Accelio would
> be a submodule.
> 

Thanks for pointing that out. It's in Fedora as libibverbs-devel, and
it's in Debian as libibverbs-dev. So you're right, Accelio is the only
one to be packaged.

Looking at configure.ac, does README.xio need to be updated to mention
librdmacm?

- Ken

Re: xio messenger is not building by default

[Matt W. Benjamin]

Hi,

Yes, agree.  We do depend on librdmacm!

Matt

----- "Ken Dreyer" <kdreyer@redhat.com> wrote:

> On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
> > Hi,
> > 
> > Arent OFED dependencies already packaged?  I assumed that only
> Accelio would
> > be a submodule.
> > 
> 
> Thanks for pointing that out. It's in Fedora as libibverbs-devel, and
> it's in Debian as libibverbs-dev. So you're right, Accelio is the
> only
> one to be packaged.
> 
> Looking at configure.ac, does README.xio need to be updated to
> mention
> librdmacm?
> 
> - Ken

-- 
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

RE: xio messenger is not building by default

[Somnath Roy]

I think installing the MLNX_OFED driver package will install most of the stuff(including libibverbs-dev). We need the following extra packages for building accelio.
numactl libnuma-dev

But, what if user is building it without OFED driver package installed ?
BTW, I saw accelio was giving assert if I manually install libibverbs-dev from Ubuntu repo and upgrade this OFED driver later. Cleaning the repo and rebuilding solved the issue. Seems it was linking wrong libraries.

Thanks & Regards
Somnath


-----Original Message-----
From: Matt W. Benjamin [mailto:matt@cohortfs.com]
Sent: Monday, March 16, 2015 9:34 AM
To: Ken Dreyer
Cc: ceph-devel@vger.kernel.org; Sage Weil; Somnath Roy
Subject: Re: xio messenger is not building by default

Hi,

Yes, agree.  We do depend on librdmacm!

Matt

----- "Ken Dreyer" <kdreyer@redhat.com> wrote:

> On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
> > Hi,
> >
> > Arent OFED dependencies already packaged?  I assumed that only
> Accelio would
> > be a submodule.
> >
>
> Thanks for pointing that out. It's in Fedora as libibverbs-devel, and
> it's in Debian as libibverbs-dev. So you're right, Accelio is the only
> one to be packaged.
>
> Looking at configure.ac, does README.xio need to be updated to mention
> librdmacm?
>
> - Ken

--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689
fax.  734-769-8938
cel.  734-216-5309

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: xio messenger is not building by default

[Matt W. Benjamin]

Hi,

----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

> I think installing the MLNX_OFED driver package will install most of
> the stuff(including libibverbs-dev). We need the following extra
> packages for building accelio.
> numactl libnuma-dev

Ah, that's also true.

> 
> But, what if user is building it without OFED driver package installed
> ?

Should it be possible?  I mean, to both build Xio and not have the header interface?

> BTW, I saw accelio was giving assert if I manually install
> libibverbs-dev from Ubuntu repo and upgrade this OFED driver later.
> Cleaning the repo and rebuilding solved the issue. Seems it was
> linking wrong libraries.

I think I've seen this.

> 
> Thanks & Regards
> Somnath
> 
> 
> -----Original Message-----
> From: Matt W. Benjamin [mailto:matt@cohortfs.com]
> Sent: Monday, March 16, 2015 9:34 AM
> To: Ken Dreyer
> Cc: ceph-devel@vger.kernel.org; Sage Weil; Somnath Roy
> Subject: Re: xio messenger is not building by default
> 
> Hi,
> 
> Yes, agree.  We do depend on librdmacm!
> 
> Matt
> 
> ----- "Ken Dreyer" <kdreyer@redhat.com> wrote:
> 
> > On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
> > > Hi,
> > >
> > > Arent OFED dependencies already packaged?  I assumed that only
> > Accelio would
> > > be a submodule.
> > >
> >
> > Thanks for pointing that out. It's in Fedora as libibverbs-devel,
> and
> > it's in Debian as libibverbs-dev. So you're right, Accelio is the
> only
> > one to be packaged.
> >
> > Looking at configure.ac, does README.xio need to be updated to
> mention
> > librdmacm?
> >
> > - Ken
> 
> --
> Matt Benjamin
> CohortFS, LLC.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
> 
> http://cohortfs.com
> 
> tel.  734-761-4689
> fax.  734-769-8938
> cel.  734-216-5309
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message
> is intended only for the use of the designated recipient(s) named
> above. If the reader of this message is not the intended recipient,
> you are hereby notified that you have received this message in error
> and that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify the sender by telephone or
> e-mail (as shown above) immediately and destroy any and all copies of
> this message in your possession (whether hard copies or electronically
> stored copies).

-- 
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

RE: xio messenger is not building by default

[Somnath Roy]

Matt,
It is possible to build accelio without OFED packages installed.
But, my suspicion is , it may not work properly as I have mentioned below.

Thanks & Regards
Somnath

-----Original Message-----
From: Matt W. Benjamin [mailto:matt@cohortfs.com]
Sent: Monday, March 16, 2015 11:22 AM
To: Somnath Roy
Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
Subject: Re: xio messenger is not building by default

Hi,

----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

> I think installing the MLNX_OFED driver package will install most of
> the stuff(including libibverbs-dev). We need the following extra
> packages for building accelio.
> numactl libnuma-dev

Ah, that's also true.

>
> But, what if user is building it without OFED driver package installed
> ?

Should it be possible?  I mean, to both build Xio and not have the header interface?

> BTW, I saw accelio was giving assert if I manually install
> libibverbs-dev from Ubuntu repo and upgrade this OFED driver later.
> Cleaning the repo and rebuilding solved the issue. Seems it was
> linking wrong libraries.

I think I've seen this.

>
> Thanks & Regards
> Somnath
>
>
> -----Original Message-----
> From: Matt W. Benjamin [mailto:matt@cohortfs.com]
> Sent: Monday, March 16, 2015 9:34 AM
> To: Ken Dreyer
> Cc: ceph-devel@vger.kernel.org; Sage Weil; Somnath Roy
> Subject: Re: xio messenger is not building by default
>
> Hi,
>
> Yes, agree.  We do depend on librdmacm!
>
> Matt
>
> ----- "Ken Dreyer" <kdreyer@redhat.com> wrote:
>
> > On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
> > > Hi,
> > >
> > > Arent OFED dependencies already packaged?  I assumed that only
> > Accelio would
> > > be a submodule.
> > >
> >
> > Thanks for pointing that out. It's in Fedora as libibverbs-devel,
> and
> > it's in Debian as libibverbs-dev. So you're right, Accelio is the
> only
> > one to be packaged.
> >
> > Looking at configure.ac, does README.xio need to be updated to
> mention
> > librdmacm?
> >
> > - Ken
>
> --
> Matt Benjamin
> CohortFS, LLC.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://cohortfs.com
>
> tel.  734-761-4689
> fax.  734-769-8938
> cel.  734-216-5309
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message
> is intended only for the use of the designated recipient(s) named
> above. If the reader of this message is not the intended recipient,
> you are hereby notified that you have received this message in error
> and that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify the sender by telephone or
> e-mail (as shown above) immediately and destroy any and all copies of
> this message in your possession (whether hard copies or electronically
> stored copies).

--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689
fax.  734-769-8938
cel.  734-216-5309

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: xio messenger is not building by default

[Matt W. Benjamin]

It might be possible to us Accelio over TCP, I guess.

Matt

----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

> Matt,
> It is possible to build accelio without OFED packages installed.
> But, my suspicion is , it may not work properly as I have mentioned
> below.
> 
> Thanks & Regards
> Somnath
> 
> -----Original Message-----
> From: Matt W. Benjamin [mailto:matt@cohortfs.com]
> Sent: Monday, March 16, 2015 11:22 AM
> To: Somnath Roy
> Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
> Subject: Re: xio messenger is not building by default
> 
> Hi,
> 
> ----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:
> 
> > I think installing the MLNX_OFED driver package will install most
> of
> > the stuff(including libibverbs-dev). We need the following extra
> > packages for building accelio.
> > numactl libnuma-dev
> 
> Ah, that's also true.
> 
> >
> > But, what if user is building it without OFED driver package
> installed
> > ?
> 
> Should it be possible?  I mean, to both build Xio and not have the
> header interface?
> 
> > BTW, I saw accelio was giving assert if I manually install
> > libibverbs-dev from Ubuntu repo and upgrade this OFED driver later.
> > Cleaning the repo and rebuilding solved the issue. Seems it was
> > linking wrong libraries.
> 
> I think I've seen this.
> 
> >
> > Thanks & Regards
> > Somnath
> >
> >
> > -----Original Message-----
> > From: Matt W. Benjamin [mailto:matt@cohortfs.com]
> > Sent: Monday, March 16, 2015 9:34 AM
> > To: Ken Dreyer
> > Cc: ceph-devel@vger.kernel.org; Sage Weil; Somnath Roy
> > Subject: Re: xio messenger is not building by default
> >
> > Hi,
> >
> > Yes, agree.  We do depend on librdmacm!
> >
> > Matt
> >
> > ----- "Ken Dreyer" <kdreyer@redhat.com> wrote:
> >
> > > On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
> > > > Hi,
> > > >
> > > > Arent OFED dependencies already packaged?  I assumed that only
> > > Accelio would
> > > > be a submodule.
> > > >
> > >
> > > Thanks for pointing that out. It's in Fedora as libibverbs-devel,
> > and
> > > it's in Debian as libibverbs-dev. So you're right, Accelio is the
> > only
> > > one to be packaged.
> > >
> > > Looking at configure.ac, does README.xio need to be updated to
> > mention
> > > librdmacm?
> > >
> > > - Ken
> >
> > --
> > Matt Benjamin
> > CohortFS, LLC.
> > 315 West Huron Street, Suite 140A
> > Ann Arbor, Michigan 48103
> >
> > http://cohortfs.com
> >
> > tel.  734-761-4689
> > fax.  734-769-8938
> > cel.  734-216-5309
> >
> > ________________________________
> >
> > PLEASE NOTE: The information contained in this electronic mail
> message
> > is intended only for the use of the designated recipient(s) named
> > above. If the reader of this message is not the intended recipient,
> > you are hereby notified that you have received this message in
> error
> > and that any review, dissemination, distribution, or copying of
> this
> > message is strictly prohibited. If you have received this
> > communication in error, please notify the sender by telephone or
> > e-mail (as shown above) immediately and destroy any and all copies
> of
> > this message in your possession (whether hard copies or
> electronically
> > stored copies).
> 
> --
> Matt Benjamin
> CohortFS, LLC.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
> 
> http://cohortfs.com
> 
> tel.  734-761-4689
> fax.  734-769-8938
> cel.  734-216-5309
> 
> ________________________________
> 
> PLEASE NOTE: The information contained in this electronic mail message
> is intended only for the use of the designated recipient(s) named
> above. If the reader of this message is not the intended recipient,
> you are hereby notified that you have received this message in error
> and that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify the sender by telephone or
> e-mail (as shown above) immediately and destroy any and all copies of
> this message in your possession (whether hard copies or electronically
> stored copies).

-- 
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

RE: xio messenger is not building by default

[Somnath Roy]

Yes, but what I am saying is, building accelio library itself is having some problem if I build it before I install OFED packages and do apt-get libibvirbs-dev manually.
Now, if I follow these steps it is building fine.

1. Install OFED packages (this will install libibvirbs)

2. apt-get numactl libnuma-dev

3. Build accelio

The accelio library generated is working fine.

So, it seems OFED driver installing some correct libraries (probably libibvirbs or something ?) and accelio is building fine by linking with those. We need to find out what are those  as dependency with driver packages (installing correct libraries) might not be a good idea.
BTW, Mellanox is recommending the latest OFED packages both for TCP/RDMA. I didn't check the accelio + tcp connection is going on fine with faulty accelio library of mine though.

Thanks & Regards
Somnath

-----Original Message-----
From: Matt W. Benjamin [mailto:matt@cohortfs.com]
Sent: Monday, March 16, 2015 11:29 AM
To: Somnath Roy
Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
Subject: Re: xio messenger is not building by default

It might be possible to us Accelio over TCP, I guess.

Matt

----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

> Matt,
> It is possible to build accelio without OFED packages installed.
> But, my suspicion is , it may not work properly as I have mentioned
> below.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Matt W. Benjamin [mailto:matt@cohortfs.com]
> Sent: Monday, March 16, 2015 11:22 AM
> To: Somnath Roy
> Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
> Subject: Re: xio messenger is not building by default
>
> Hi,
>
> ----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:
>
> > I think installing the MLNX_OFED driver package will install most
> of
> > the stuff(including libibverbs-dev). We need the following extra
> > packages for building accelio.
> > numactl libnuma-dev
>
> Ah, that's also true.
>
> >
> > But, what if user is building it without OFED driver package
> installed
> > ?
>
> Should it be possible?  I mean, to both build Xio and not have the
> header interface?
>
> > BTW, I saw accelio was giving assert if I manually install
> > libibverbs-dev from Ubuntu repo and upgrade this OFED driver later.
> > Cleaning the repo and rebuilding solved the issue. Seems it was
> > linking wrong libraries.
>
> I think I've seen this.
>
> >
> > Thanks & Regards
> > Somnath
> >
> >
> > -----Original Message-----
> > From: Matt W. Benjamin [mailto:matt@cohortfs.com]
> > Sent: Monday, March 16, 2015 9:34 AM
> > To: Ken Dreyer
> > Cc: ceph-devel@vger.kernel.org; Sage Weil; Somnath Roy
> > Subject: Re: xio messenger is not building by default
> >
> > Hi,
> >
> > Yes, agree.  We do depend on librdmacm!
> >
> > Matt
> >
> > ----- "Ken Dreyer" <kdreyer@redhat.com> wrote:
> >
> > > On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
> > > > Hi,
> > > >
> > > > Arent OFED dependencies already packaged?  I assumed that only
> > > Accelio would
> > > > be a submodule.
> > > >
> > >
> > > Thanks for pointing that out. It's in Fedora as libibverbs-devel,
> > and
> > > it's in Debian as libibverbs-dev. So you're right, Accelio is the
> > only
> > > one to be packaged.
> > >
> > > Looking at configure.ac, does README.xio need to be updated to
> > mention
> > > librdmacm?
> > >
> > > - Ken
> >
> > --
> > Matt Benjamin
> > CohortFS, LLC.
> > 315 West Huron Street, Suite 140A
> > Ann Arbor, Michigan 48103
> >
> > http://cohortfs.com
> >
> > tel.  734-761-4689
> > fax.  734-769-8938
> > cel.  734-216-5309
> >
> > ________________________________
> >
> > PLEASE NOTE: The information contained in this electronic mail
> message
> > is intended only for the use of the designated recipient(s) named
> > above. If the reader of this message is not the intended recipient,
> > you are hereby notified that you have received this message in
> error
> > and that any review, dissemination, distribution, or copying of
> this
> > message is strictly prohibited. If you have received this
> > communication in error, please notify the sender by telephone or
> > e-mail (as shown above) immediately and destroy any and all copies
> of
> > this message in your possession (whether hard copies or
> electronically
> > stored copies).
>
> --
> Matt Benjamin
> CohortFS, LLC.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://cohortfs.com
>
> tel.  734-761-4689
> fax.  734-769-8938
> cel.  734-216-5309
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message
> is intended only for the use of the designated recipient(s) named
> above. If the reader of this message is not the intended recipient,
> you are hereby notified that you have received this message in error
> and that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify the sender by telephone or
> e-mail (as shown above) immediately and destroy any and all copies of
> this message in your possession (whether hard copies or electronically
> stored copies).

--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689
fax.  734-769-8938
cel.  734-216-5309

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: xio messenger is not building by default

[Vu Pham]

Hi Somnath,

It's possible to build Accelio without OFED packages (download from open 
fabric website or MLNX_OFED is Mellanox version of the opensource 
package) installed; however, you need to install the libibverbs-dev, 
librdmacm-dev from the distribution you are running.

If you compile against original libibverbs-dev, librdmacm-dev from 
distributions, you must run with RDMA kernel stack and RDMA libraries 
from the distribution as well. You must not mix these with RDMA kernel 
stack and RDMA libraries provided by OFED packages

-vu



On 3/16/2015 11:25:44 AM, "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

>Matt,
>It is possible to build accelio without OFED packages installed.
>But, my suspicion is , it may not work properly as I have mentioned 
>below.
>
>Thanks & Regards
>Somnath
>
>-----Original Message-----
>From: Matt W. Benjamin [mailto:matt@cohortfs.com]
>Sent: Monday, March 16, 2015 11:22 AM
>To: Somnath Roy
>Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
>Subject: Re: xio messenger is not building by default
>
>Hi,
>
>----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:
>
>>  I think installing the MLNX_OFED driver package will install most of
>>  the stuff(including libibverbs-dev). We need the following extra
>>  packages for building accelio.
>>  numactl libnuma-dev
>
>Ah, that's also true.
>
>>
>>  But, what if user is building it without OFED driver package 
>>installed
>>  ?
>
>Should it be possible? I mean, to both build Xio and not have the 
>header interface?
>
>>  BTW, I saw accelio was giving assert if I manually install
>>  libibverbs-dev from Ubuntu repo and upgrade this OFED driver later.
>>  Cleaning the repo and rebuilding solved the issue. Seems it was
>>  linking wrong libraries.
>
>I think I've seen this.
>
>>
>>  Thanks & Regards
>>  Somnath
>>
>>
>>  -----Original Message-----
>>  From: Matt W. Benjamin [mailto:matt@cohortfs.com]
>>  Sent: Monday, March 16, 2015 9:34 AM
>>  To: Ken Dreyer
>>  Cc: ceph-devel@vger.kernel.org; Sage Weil; Somnath Roy
>>  Subject: Re: xio messenger is not building by default
>>
>>  Hi,
>>
>>  Yes, agree. We do depend on librdmacm!
>>
>>  Matt
>>
>>  ----- "Ken Dreyer" <kdreyer@redhat.com> wrote:
>>
>>  > On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
>>  > > Hi,
>>  > >
>>  > > Arent OFED dependencies already packaged? I assumed that only
>>  > Accelio would
>>  > > be a submodule.
>>  > >
>>  >
>>  > Thanks for pointing that out. It's in Fedora as libibverbs-devel,
>>  and
>>  > it's in Debian as libibverbs-dev. So you're right, Accelio is the
>>  only
>>  > one to be packaged.
>>  >
>>  > Looking at configure.ac, does README.xio need to be updated to
>>  mention
>>  > librdmacm?
>>  >
>>  > - Ken
>>
>>  --
>>  Matt Benjamin
>>  CohortFS, LLC.
>>  315 West Huron Street, Suite 140A
>>  Ann Arbor, Michigan 48103
>>
>>  http://cohortfs.com
>>
>>  tel. 734-761-4689
>>  fax. 734-769-8938
>>  cel. 734-216-5309
>>
>>  ________________________________
>>
>>  PLEASE NOTE: The information contained in this electronic mail 
>>message
>>  is intended only for the use of the designated recipient(s) named
>>  above. If the reader of this message is not the intended recipient,
>>  you are hereby notified that you have received this message in error
>>  and that any review, dissemination, distribution, or copying of this
>>  message is strictly prohibited. If you have received this
>>  communication in error, please notify the sender by telephone or
>>  e-mail (as shown above) immediately and destroy any and all copies of
>>  this message in your possession (whether hard copies or 
>>electronically
>>  stored copies).
>
>--
>Matt Benjamin
>CohortFS, LLC.
>315 West Huron Street, Suite 140A
>Ann Arbor, Michigan 48103
>
>http://cohortfs.com
>
>tel. 734-761-4689
>fax. 734-769-8938
>cel. 734-216-5309
>
>________________________________
>
>PLEASE NOTE: The information contained in this electronic mail message 
>is intended only for the use of the designated recipient(s) named 
>above. If the reader of this message is not the intended recipient, you 
>are hereby notified that you have received this message in error and 
>that any review, dissemination, distribution, or copying of this 
>message is strictly prohibited. If you have received this communication 
>in error, please notify the sender by telephone or e-mail (as shown 
>above) immediately and destroy any and all copies of this message in 
>your possession (whether hard copies or electronically stored copies).
>
>\x13��칻\x1c�&�~�&�\x18��+-��ݶ\x17��w��˛���m�\x1e�\x17^��b��^n�r���z�\x1a��h����&��\x1e�G���h�\x03(�階�ݢj"��\x1a�^[m�����z�ޖ���f���h���~�m�

RE: xio messenger is not building by default

[Somnath Roy]

Vu,
Yes, that means we can't use the apt-get version of libibverbs etc as installing the OFED packages later will overwrite those. I am talking about installing to system path not to any customized location as that will not be a solution to the user.
So, it seems we need to fix the Ubuntu repo to get the proper library version that OFED packages are installing (?) or always install OFED packages before building accelio, isn't it ?

Thanks & Regards
Somnath

-----Original Message-----
From: Vu Pham [mailto:vuhuong@mellanox.com]
Sent: Monday, March 16, 2015 11:51 AM
To: Somnath Roy; Matt W. Benjamin
Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
Subject: Re: xio messenger is not building by default

Hi Somnath,

It's possible to build Accelio without OFED packages (download from open fabric website or MLNX_OFED is Mellanox version of the opensource
package) installed; however, you need to install the libibverbs-dev, librdmacm-dev from the distribution you are running.

If you compile against original libibverbs-dev, librdmacm-dev from distributions, you must run with RDMA kernel stack and RDMA libraries from the distribution as well. You must not mix these with RDMA kernel stack and RDMA libraries provided by OFED packages

-vu



On 3/16/2015 11:25:44 AM, "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

>Matt,
>It is possible to build accelio without OFED packages installed.
>But, my suspicion is , it may not work properly as I have mentioned
>below.
>
>Thanks & Regards
>Somnath
>
>-----Original Message-----
>From: Matt W. Benjamin [mailto:matt@cohortfs.com]
>Sent: Monday, March 16, 2015 11:22 AM
>To: Somnath Roy
>Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
>Subject: Re: xio messenger is not building by default
>
>Hi,
>
>----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:
>
>>  I think installing the MLNX_OFED driver package will install most of
>> the stuff(including libibverbs-dev). We need the following extra
>> packages for building accelio.
>>  numactl libnuma-dev
>
>Ah, that's also true.
>
>>
>>  But, what if user is building it without OFED driver package
>>installed
>>  ?
>
>Should it be possible? I mean, to both build Xio and not have the
>header interface?
>
>>  BTW, I saw accelio was giving assert if I manually install
>> libibverbs-dev from Ubuntu repo and upgrade this OFED driver later.
>>  Cleaning the repo and rebuilding solved the issue. Seems it was
>> linking wrong libraries.
>
>I think I've seen this.
>
>>
>>  Thanks & Regards
>>  Somnath
>>
>>
>>  -----Original Message-----
>>  From: Matt W. Benjamin [mailto:matt@cohortfs.com]
>>  Sent: Monday, March 16, 2015 9:34 AM
>>  To: Ken Dreyer
>>  Cc: ceph-devel@vger.kernel.org; Sage Weil; Somnath Roy
>>  Subject: Re: xio messenger is not building by default
>>
>>  Hi,
>>
>>  Yes, agree. We do depend on librdmacm!
>>
>>  Matt
>>
>>  ----- "Ken Dreyer" <kdreyer@redhat.com> wrote:
>>
>>  > On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
>>  > > Hi,
>>  > >
>>  > > Arent OFED dependencies already packaged? I assumed that only  >
>> Accelio would  > > be a submodule.
>>  > >
>>  >
>>  > Thanks for pointing that out. It's in Fedora as libibverbs-devel,
>> and  > it's in Debian as libibverbs-dev. So you're right, Accelio is
>> the  only  > one to be packaged.
>>  >
>>  > Looking at configure.ac, does README.xio need to be updated to
>> mention  > librdmacm?
>>  >
>>  > - Ken
>>
>>  --
>>  Matt Benjamin
>>  CohortFS, LLC.
>>  315 West Huron Street, Suite 140A
>>  Ann Arbor, Michigan 48103
>>
>>  http://cohortfs.com
>>
>>  tel. 734-761-4689
>>  fax. 734-769-8938
>>  cel. 734-216-5309
>>
>>  ________________________________
>>
>>  PLEASE NOTE: The information contained in this electronic mail
>>message
>>  is intended only for the use of the designated recipient(s) named
>>  above. If the reader of this message is not the intended recipient,
>>  you are hereby notified that you have received this message in error
>>  and that any review, dissemination, distribution, or copying of this
>>  message is strictly prohibited. If you have received this
>>  communication in error, please notify the sender by telephone or
>>  e-mail (as shown above) immediately and destroy any and all copies
>>of
>>  this message in your possession (whether hard copies or
>>electronically
>>  stored copies).
>
>--
>Matt Benjamin
>CohortFS, LLC.
>315 West Huron Street, Suite 140A
>Ann Arbor, Michigan 48103
>
>http://cohortfs.com
>
>tel. 734-761-4689
>fax. 734-769-8938
>cel. 734-216-5309
>
>________________________________
>
>PLEASE NOTE: The information contained in this electronic mail message
>is intended only for the use of the designated recipient(s) named
>above. If the reader of this message is not the intended recipient, you
>are hereby notified that you have received this message in error and
>that any review, dissemination, distribution, or copying of this
>message is strictly prohibited. If you have received this communication
>in error, please notify the sender by telephone or e-mail (as shown
>above) immediately and destroy any and all copies of this message in
>your possession (whether hard copies or electronically stored copies).
>
>\x13  칻\x1c & ~ & \x18  +-  ݶ\x17  w  ˛   m \x1e \x17^  b  ^n r   z \x1a  h    &  \x1e G   h \x03(
> 階 ݢj"  \x1a ^[m     z ޖ   f   h   ~ m

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: xio messenger is not building by default

[Vu Pham]

Somnath,

Yes, that's correct.

You must compile and run with rdma kernel stack & libraries provided by distributions (Ubuntu, RH...)
OR 
compile and run with rdma kernel stack + libraries provided by OFED packages.
You CANNOT mix and match them.

Thanks,
-vu

-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@sandisk.com] 
Sent: Monday, March 16, 2015 11:59 AM
To: Vu Pham; Matt W. Benjamin
Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
Subject: RE: xio messenger is not building by default

Vu,
Yes, that means we can't use the apt-get version of libibverbs etc as installing the OFED packages later will overwrite those. I am talking about installing to system path not to any customized location as that will not be a solution to the user.
So, it seems we need to fix the Ubuntu repo to get the proper library version that OFED packages are installing (?) or always install OFED packages before building accelio, isn't it ?

Thanks & Regards
Somnath

-----Original Message-----
From: Vu Pham [mailto:vuhuong@mellanox.com]
Sent: Monday, March 16, 2015 11:51 AM
To: Somnath Roy; Matt W. Benjamin
Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
Subject: Re: xio messenger is not building by default

Hi Somnath,

It's possible to build Accelio without OFED packages (download from open fabric website or MLNX_OFED is Mellanox version of the opensource
package) installed; however, you need to install the libibverbs-dev, librdmacm-dev from the distribution you are running.

If you compile against original libibverbs-dev, librdmacm-dev from distributions, you must run with RDMA kernel stack and RDMA libraries from the distribution as well. You must not mix these with RDMA kernel stack and RDMA libraries provided by OFED packages

-vu



On 3/16/2015 11:25:44 AM, "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:

>Matt,
>It is possible to build accelio without OFED packages installed.
>But, my suspicion is , it may not work properly as I have mentioned 
>below.
>
>Thanks & Regards
>Somnath
>
>-----Original Message-----
>From: Matt W. Benjamin [mailto:matt@cohortfs.com]
>Sent: Monday, March 16, 2015 11:22 AM
>To: Somnath Roy
>Cc: ceph-devel@vger.kernel.org; Sage Weil; Ken Dreyer
>Subject: Re: xio messenger is not building by default
>
>Hi,
>
>----- "Somnath Roy" <Somnath.Roy@sandisk.com> wrote:
>
>>  I think installing the MLNX_OFED driver package will install most of 
>> the stuff(including libibverbs-dev). We need the following extra 
>> packages for building accelio.
>>  numactl libnuma-dev
>
>Ah, that's also true.
>
>>
>>  But, what if user is building it without OFED driver package 
>>installed
>>  ?
>
>Should it be possible? I mean, to both build Xio and not have the 
>header interface?
>
>>  BTW, I saw accelio was giving assert if I manually install 
>> libibverbs-dev from Ubuntu repo and upgrade this OFED driver later.
>>  Cleaning the repo and rebuilding solved the issue. Seems it was 
>> linking wrong libraries.
>
>I think I've seen this.
>
>>
>>  Thanks & Regards
>>  Somnath
>>
>>
>>  -----Original Message-----
>>  From: Matt W. Benjamin [mailto:matt@cohortfs.com]
>>  Sent: Monday, March 16, 2015 9:34 AM
>>  To: Ken Dreyer
>>  Cc: ceph-devel@vger.kernel.org; Sage Weil; Somnath Roy
>>  Subject: Re: xio messenger is not building by default
>>
>>  Hi,
>>
>>  Yes, agree. We do depend on librdmacm!
>>
>>  Matt
>>
>>  ----- "Ken Dreyer" <kdreyer@redhat.com> wrote:
>>
>>  > On 03/16/2015 10:20 AM, Matt W. Benjamin wrote:
>>  > > Hi,
>>  > >
>>  > > Arent OFED dependencies already packaged? I assumed that only  > 
>> Accelio would  > > be a submodule.
>>  > >
>>  >
>>  > Thanks for pointing that out. It's in Fedora as libibverbs-devel, 
>> and  > it's in Debian as libibverbs-dev. So you're right, Accelio is 
>> the  only  > one to be packaged.
>>  >
>>  > Looking at configure.ac, does README.xio need to be updated to 
>> mention  > librdmacm?
>>  >
>>  > - Ken
>>
>>  --
>>  Matt Benjamin
>>  CohortFS, LLC.
>>  315 West Huron Street, Suite 140A
>>  Ann Arbor, Michigan 48103
>>
>>  http://cohortfs.com
>>
>>  tel. 734-761-4689
>>  fax. 734-769-8938
>>  cel. 734-216-5309
>>
>>  ________________________________
>>
>>  PLEASE NOTE: The information contained in this electronic mail 
>>message
>>  is intended only for the use of the designated recipient(s) named
>>  above. If the reader of this message is not the intended recipient,
>>  you are hereby notified that you have received this message in error
>>  and that any review, dissemination, distribution, or copying of this
>>  message is strictly prohibited. If you have received this
>>  communication in error, please notify the sender by telephone or
>>  e-mail (as shown above) immediately and destroy any and all copies 
>>of
>>  this message in your possession (whether hard copies or 
>>electronically
>>  stored copies).
>
>--
>Matt Benjamin
>CohortFS, LLC.
>315 West Huron Street, Suite 140A
>Ann Arbor, Michigan 48103
>
>http://cohortfs.com
>
>tel. 734-761-4689
>fax. 734-769-8938
>cel. 734-216-5309
>
>________________________________
>
>PLEASE NOTE: The information contained in this electronic mail message 
>is intended only for the use of the designated recipient(s) named 
>above. If the reader of this message is not the intended recipient, you 
>are hereby notified that you have received this message in error and 
>that any review, dissemination, distribution, or copying of this 
>message is strictly prohibited. If you have received this communication 
>in error, please notify the sender by telephone or e-mail (as shown
>above) immediately and destroy any and all copies of this message in 
>your possession (whether hard copies or electronically stored copies).
>
>\x13  칻\x1c & ~ & \x18  +-  ݶ\x17  w  ˛   m \x1e \x17^  b  ^n r   z \x1a  h    &  \x1e G   h \x03(
> 階 ݢj"  \x1a ^[m     z ޖ   f   h   ~ m

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).