[lustre-devel] CentOS 6 - Build problems with kmod

[lustre-devel] CentOS 6 - Build problems with kmod

[Patrick Farrell]

Good afternoon,


I'm trying to build and install updated Lustre master on CentOS 6 for the first time in a month or two, and I'm having trouble.


I use the build procedure documented here:

https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821


I've got some new kmod-* RPMs, and I need to install those to install Lustre, which is fine, except I get a huge string of messages like this when I try:

error: Failed dependencies:
        ksym(__init_waitqueue_head) = 0xffc7c184 is needed by kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
        ksym(__mutex_init) = 0x4bf79039 is needed by kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64

I've rebuilt and reinstalled my kernel with this latest version of Lustre.  Error messages remain the same, and I can't install.


Any thoughts or advice?


- Patrick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160818/9eda7c04/attachment.htm>

[lustre-devel] CentOS 6 - Build problems with kmod

[Christopher J. Morrone]

Yes, those instructions should be taken with a huge grain of salt.  For
instance, instructions for compiling lustre should really employ a
custom user and talk about "useradd" and such.  Also, most users can
skip the whole custom-patched-kernel section.  Hopefully all developers
will be able to stop that too by the time 2.9.0 comes out.

The error you are seeing almost certainly means that you don't have a
kernel installed that offers symbols compatible with the kernel that
lustre was compiled against.

It sounds like you built your own kernel.  Did you install all the
resulting kernel packages before building lustre (including any
devel-related packages)?  Is your custom kernel the newest kernel
installed on your system?

Chris

On 08/18/2016 01:00 PM, Patrick Farrell wrote:
> Good afternoon,
> 
> 
> I'm trying to build and install updated Lustre master on CentOS 6 for
> the first time in a month or two, and I'm having trouble.
> 
> 
> I use the build procedure documented here:
> 
> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
> 
> 
> I've got some new kmod-* RPMs, and I need to install those to install
> Lustre, which is fine, except I get a huge string of messages like this
> when I try:
> 
> error: Failed dependencies:
>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>         ksym(__mutex_init) = 0x4bf79039 is needed by
> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
> 
> I've rebuilt and reinstalled my kernel with this latest version of
> Lustre.  Error messages remain the same, and I can't install.
> 
> 
> Any thoughts or advice?
> 
> 
> - Patrick
> 
> 
> 
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
> 

[lustre-devel] CentOS 6 - Build problems with kmod

[Christopher J. Morrone]

On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
> Yes, those instructions should be taken with a huge grain of salt.  For
> instance, instructions for compiling lustre should really employ a

s/should/should NOT/

> custom user and talk about "useradd" and such.  Also, most users can
> skip the whole custom-patched-kernel section.  Hopefully all developers
> will be able to stop that too by the time 2.9.0 comes out.
> 
> The error you are seeing almost certainly means that you don't have a
> kernel installed that offers symbols compatible with the kernel that
> lustre was compiled against.
> 
> It sounds like you built your own kernel.  Did you install all the
> resulting kernel packages before building lustre (including any
> devel-related packages)?  Is your custom kernel the newest kernel
> installed on your system?
> 
> Chris
> 
> On 08/18/2016 01:00 PM, Patrick Farrell wrote:
>> Good afternoon,
>>
>>
>> I'm trying to build and install updated Lustre master on CentOS 6 for
>> the first time in a month or two, and I'm having trouble.
>>
>>
>> I use the build procedure documented here:
>>
>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>>
>>
>> I've got some new kmod-* RPMs, and I need to install those to install
>> Lustre, which is fine, except I get a huge string of messages like this
>> when I try:
>>
>> error: Failed dependencies:
>>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>         ksym(__mutex_init) = 0x4bf79039 is needed by
>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>
>> I've rebuilt and reinstalled my kernel with this latest version of
>> Lustre.  Error messages remain the same, and I can't install.
>>
>>
>> Any thoughts or advice?
>>
>>
>> - Patrick
>>
>>
>>
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>>
> 
> .
> 

[lustre-devel] CentOS 6 - Build problems with kmod

[Patrick Farrell]

Chris,


I agree with your contention about the kernel symbols, that's why I rebuilt from scratch and reinstalled.  Just did it again.  Still getting the error.


> It sounds like you built your own kernel.  Did you install all the
> resulting kernel packages before building lustre (including any
> devel-related packages)?

Yes, but this process doesn't produce anything other than the kernel RPM.


> Is your custom kernel the newest kernel
> installed on your system?

Yes.  It's the newest and it's what's booted.


So, does anyone have any insight in to what needs to change in the documented build process so I can build and install Lustre on CentOS 6?  (ldisksfs, not ZFS, so I must build the kernel)  It seems likely that I'm missing some symbol RPMs or similar, but kernel-syms is a SuSE only thing, I believe.  I believe Intel is still building and installing ldiskfs Lustre on CentOS 6, so there must be something...?


- Patrick

________________________________
From: lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of Christopher J. Morrone <morrone2@llnl.gov>
Sent: Thursday, August 18, 2016 3:44:52 PM
To: lustre-devel at lists.lustre.org
Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod

On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
> Yes, those instructions should be taken with a huge grain of salt.  For
> instance, instructions for compiling lustre should really employ a

s/should/should NOT/

> custom user and talk about "useradd" and such.  Also, most users can
> skip the whole custom-patched-kernel section.  Hopefully all developers
> will be able to stop that too by the time 2.9.0 comes out.
>
> The error you are seeing almost certainly means that you don't have a
> kernel installed that offers symbols compatible with the kernel that
> lustre was compiled against.
>
> It sounds like you built your own kernel.  Did you install all the
> resulting kernel packages before building lustre (including any
> devel-related packages)?  Is your custom kernel the newest kernel
> installed on your system?
>
> Chris
>
> On 08/18/2016 01:00 PM, Patrick Farrell wrote:
>> Good afternoon,
>>
>>
>> I'm trying to build and install updated Lustre master on CentOS 6 for
>> the first time in a month or two, and I'm having trouble.
>>
>>
>> I use the build procedure documented here:
>>
>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>>
>>
>> I've got some new kmod-* RPMs, and I need to install those to install
>> Lustre, which is fine, except I get a huge string of messages like this
>> when I try:
>>
>> error: Failed dependencies:
>>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>         ksym(__mutex_init) = 0x4bf79039 is needed by
>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>
>> I've rebuilt and reinstalled my kernel with this latest version of
>> Lustre.  Error messages remain the same, and I can't install.
>>
>>
>> Any thoughts or advice?
>>
>>
>> - Patrick
>>
>>
>>
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>>
>
> .
>

_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160818/8f9908ea/attachment.htm>

[lustre-devel] CentOS 6 - Build problems with kmod

[Ben Evans]

When I ran into the problem, I did an rpm -I --nodeps and it worked fine.  The errors you are seeing are dependencies listed in the RPMs, not ones used by the actual .ko's.

I spent some time trying to dig up where the 'kmod-' prefix for all the kernel symbols was getting inserted, but couldn't track it down.

-Ben Evans

From: lustre-devel <lustre-devel-bounces at lists.lustre.org<mailto:lustre-devel-bounces@lists.lustre.org>> on behalf of Patrick Farrell <paf at cray.com<mailto:paf@cray.com>>
Date: Thursday, August 18, 2016 at 6:11 PM
To: "Christopher J. Morrone" <morrone2 at llnl.gov<mailto:morrone2@llnl.gov>>, "lustre-devel at lists.lustre.org<mailto:lustre-devel@lists.lustre.org>" <lustre-devel at lists.lustre.org<mailto:lustre-devel@lists.lustre.org>>
Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod



Chris,


I agree with your contention about the kernel symbols, that's why I rebuilt from scratch and reinstalled.  Just did it again.  Still getting the error.


> It sounds like you built your own kernel.  Did you install all the
> resulting kernel packages before building lustre (including any
> devel-related packages)?

Yes, but this process doesn't produce anything other than the kernel RPM.


> Is your custom kernel the newest kernel
> installed on your system?

Yes.  It's the newest and it's what's booted.


So, does anyone have any insight in to what needs to change in the documented build process so I can build and install Lustre on CentOS 6?  (ldisksfs, not ZFS, so I must build the kernel)  It seems likely that I'm missing some symbol RPMs or similar, but kernel-syms is a SuSE only thing, I believe.  I believe Intel is still building and installing ldiskfs Lustre on CentOS 6, so there must be something...?


- Patrick

________________________________
From: lustre-devel <lustre-devel-bounces at lists.lustre.org<mailto:lustre-devel-bounces@lists.lustre.org>> on behalf of Christopher J. Morrone <morrone2 at llnl.gov<mailto:morrone2@llnl.gov>>
Sent: Thursday, August 18, 2016 3:44:52 PM
To: lustre-devel at lists.lustre.org<mailto:lustre-devel@lists.lustre.org>
Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod

On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
> Yes, those instructions should be taken with a huge grain of salt.  For
> instance, instructions for compiling lustre should really employ a

s/should/should NOT/

> custom user and talk about "useradd" and such.  Also, most users can
> skip the whole custom-patched-kernel section.  Hopefully all developers
> will be able to stop that too by the time 2.9.0 comes out.
>
> The error you are seeing almost certainly means that you don't have a
> kernel installed that offers symbols compatible with the kernel that
> lustre was compiled against.
>
> It sounds like you built your own kernel.  Did you install all the
> resulting kernel packages before building lustre (including any
> devel-related packages)?  Is your custom kernel the newest kernel
> installed on your system?
>
> Chris
>
> On 08/18/2016 01:00 PM, Patrick Farrell wrote:
>> Good afternoon,
>>
>>
>> I'm trying to build and install updated Lustre master on CentOS 6 for
>> the first time in a month or two, and I'm having trouble.
>>
>>
>> I use the build procedure documented here:
>>
>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>>
>>
>> I've got some new kmod-* RPMs, and I need to install those to install
>> Lustre, which is fine, except I get a huge string of messages like this
>> when I try:
>>
>> error: Failed dependencies:
>>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>         ksym(__mutex_init) = 0x4bf79039 is needed by
>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>
>> I've rebuilt and reinstalled my kernel with this latest version of
>> Lustre.  Error messages remain the same, and I can't install.
>>
>>
>> Any thoughts or advice?
>>
>>
>> - Patrick
>>
>>
>>
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org<mailto:lustre-devel@lists.lustre.org>
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>>
>
> .
>

_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org<mailto:lustre-devel@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160819/9b00260c/attachment-0001.htm>

[lustre-devel] CentOS 6 - Build problems with kmod

[Patrick Farrell]

Thanks, Ben - That looks likely to work.  Still more than a bit weird.

Looking at why things are still working internally at Cray for CentOS 6, 
I think it's because we're not using lustre.spec (for boring and 
complicated reasons).

It seems very likely the problem was introduced by LU-5614 - 
http://review.whamcloud.com/#/c/12063/

Is Intel still building and installing on CentOS 6 automatically? Or has 
all automated testing moved to CentOS 7?

- Patrick
On 08/19/2016 08:57 AM, Ben Evans wrote:
> When I ran into the problem, I did an rpm -I --nodeps and it worked 
> fine.  The errors you are seeing are dependencies listed in the RPMs, 
> not ones used by the actual .ko's.
>
> I spent some time trying to dig up where the 'kmod-' prefix for all 
> the kernel symbols was getting inserted, but couldn't track it down.
>
> -Ben Evans
>
> From: lustre-devel <lustre-devel-bounces@lists.lustre.org 
> <mailto:lustre-devel-bounces@lists.lustre.org>> on behalf of Patrick 
> Farrell <paf at cray.com <mailto:paf@cray.com>>
> Date: Thursday, August 18, 2016 at 6:11 PM
> To: "Christopher J. Morrone" <morrone2@llnl.gov 
> <mailto:morrone2@llnl.gov>>, "lustre-devel at lists.lustre.org 
> <mailto:lustre-devel@lists.lustre.org>" <lustre-devel@lists.lustre.org 
> <mailto:lustre-devel@lists.lustre.org>>
> Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod
>
>
> Chris,
>
>
> I agree with your contention about the kernel symbols, that's why I 
> rebuilt from scratch and reinstalled.  Just did it again.  Still 
> getting the error.
>
>
> > It sounds like you built your own kernel.  Did you install all the
> > resulting kernel packages before building lustre (including any
> > devel-related packages)?
>
> Yes, but this process doesn't produce anything other than the kernel RPM.
>
>
> > Is your custom kernel the newest kernel
> > installed on your system?
>
> Yes.  It's the newest and it's what's booted.
>
>
> So, does anyone have any insight in to what needs to change in the 
> documented build process so I can build and install Lustre on CentOS 
> 6?  (ldisksfs, not ZFS, so I must build the kernel)  It seems likely 
> that I'm missing some symbol RPMs or similar, but kernel-syms is a 
> SuSE only thing, I believe.  I believe Intel is still building and 
> installing ldiskfs Lustre on CentOS 6, so there must be something...?
>
>
> - Patrick
>
> ------------------------------------------------------------------------
> *From:* lustre-devel <lustre-devel-bounces@lists.lustre.org 
> <mailto:lustre-devel-bounces@lists.lustre.org>> on behalf of 
> Christopher J. Morrone <morrone2 at llnl.gov <mailto:morrone2@llnl.gov>>
> *Sent:* Thursday, August 18, 2016 3:44:52 PM
> *To:* lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>
> *Subject:* Re: [lustre-devel] CentOS 6 - Build problems with kmod
> On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
> > Yes, those instructions should be taken with a huge grain of salt.  For
> > instance, instructions for compiling lustre should really employ a
>
> s/should/should NOT/
>
> > custom user and talk about "useradd" and such. Also, most users can
> > skip the whole custom-patched-kernel section. Hopefully all developers
> > will be able to stop that too by the time 2.9.0 comes out.
> >
> > The error you are seeing almost certainly means that you don't have a
> > kernel installed that offers symbols compatible with the kernel that
> > lustre was compiled against.
> >
> > It sounds like you built your own kernel.  Did you install all the
> > resulting kernel packages before building lustre (including any
> > devel-related packages)?  Is your custom kernel the newest kernel
> > installed on your system?
> >
> > Chris
> >
> > On 08/18/2016 01:00 PM, Patrick Farrell wrote:
> >> Good afternoon,
> >>
> >>
> >> I'm trying to build and install updated Lustre master on CentOS 6 for
> >> the first time in a month or two, and I'm having trouble.
> >>
> >>
> >> I use the build procedure documented here:
> >>
> >> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
> >>
> >>
> >> I've got some new kmod-* RPMs, and I need to install those to install
> >> Lustre, which is fine, except I get a huge string of messages like this
> >> when I try:
> >>
> >> error: Failed dependencies:
> >>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
> >> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
> >>         ksym(__mutex_init) = 0x4bf79039 is needed by
> >> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
> >>
> >> I've rebuilt and reinstalled my kernel with this latest version of
> >> Lustre.  Error messages remain the same, and I can't install.
> >>
> >>
> >> Any thoughts or advice?
> >>
> >>
> >> - Patrick
> >>
> >>
> >>
> >> _______________________________________________
> >> lustre-devel mailing list
> >> lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>
> >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
> >>
> >
> > .
> >
>
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160819/b5a6f3ce/attachment.htm>

[lustre-devel] CentOS 6 - Build problems with kmod

[Patrick Farrell]

Unfortunately, when I do that, I am apparently still missing something - 
lustre-osd is not found, and the lustre modules are not found when I try 
to load them.  (running depmod didn't help there)

Reverting LU-5614 solves this for me, for now.  I'm hoping someone who's 
still building and installing for CentOS 6 has some insight.

- Patrick

On 08/19/2016 10:27 AM, Patrick Farrell wrote:
> Thanks, Ben - That looks likely to work.  Still more than a bit weird.
>
> Looking at why things are still working internally at Cray for CentOS 
> 6, I think it's because we're not using lustre.spec (for boring and 
> complicated reasons).
>
> It seems very likely the problem was introduced by LU-5614 - 
> http://review.whamcloud.com/#/c/12063/
>
> Is Intel still building and installing on CentOS 6 automatically? Or 
> has all automated testing moved to CentOS 7?
>
> - Patrick
> On 08/19/2016 08:57 AM, Ben Evans wrote:
>> When I ran into the problem, I did an rpm -I --nodeps and it worked 
>> fine.  The errors you are seeing are dependencies listed in the RPMs, 
>> not ones used by the actual .ko's.
>>
>> I spent some time trying to dig up where the 'kmod-' prefix for all 
>> the kernel symbols was getting inserted, but couldn't track it down.
>>
>> -Ben Evans
>>
>> From: lustre-devel <lustre-devel-bounces@lists.lustre.org 
>> <mailto:lustre-devel-bounces@lists.lustre.org>> on behalf of Patrick 
>> Farrell <paf at cray.com <mailto:paf@cray.com>>
>> Date: Thursday, August 18, 2016 at 6:11 PM
>> To: "Christopher J. Morrone" <morrone2@llnl.gov 
>> <mailto:morrone2@llnl.gov>>, "lustre-devel at lists.lustre.org 
>> <mailto:lustre-devel@lists.lustre.org>" 
>> <lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>>
>> Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod
>>
>>
>> Chris,
>>
>>
>> I agree with your contention about the kernel symbols, that's why I 
>> rebuilt from scratch and reinstalled.  Just did it again.  Still 
>> getting the error.
>>
>>
>> > It sounds like you built your own kernel.  Did you install all the
>> > resulting kernel packages before building lustre (including any
>> > devel-related packages)?
>>
>> Yes, but this process doesn't produce anything other than the kernel RPM.
>>
>>
>> > Is your custom kernel the newest kernel
>> > installed on your system?
>>
>> Yes.  It's the newest and it's what's booted.
>>
>>
>> So, does anyone have any insight in to what needs to change in the 
>> documented build process so I can build and install Lustre on CentOS 
>> 6?  (ldisksfs, not ZFS, so I must build the kernel)  It seems likely 
>> that I'm missing some symbol RPMs or similar, but kernel-syms is a 
>> SuSE only thing, I believe.  I believe Intel is still building and 
>> installing ldiskfs Lustre on CentOS 6, so there must be something...?
>>
>>
>> - Patrick
>>
>> ------------------------------------------------------------------------
>> *From:* lustre-devel <lustre-devel-bounces@lists.lustre.org 
>> <mailto:lustre-devel-bounces@lists.lustre.org>> on behalf of 
>> Christopher J. Morrone <morrone2 at llnl.gov <mailto:morrone2@llnl.gov>>
>> *Sent:* Thursday, August 18, 2016 3:44:52 PM
>> *To:* lustre-devel at lists.lustre.org 
>> <mailto:lustre-devel@lists.lustre.org>
>> *Subject:* Re: [lustre-devel] CentOS 6 - Build problems with kmod
>> On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
>> > Yes, those instructions should be taken with a huge grain of salt.  For
>> > instance, instructions for compiling lustre should really employ a
>>
>> s/should/should NOT/
>>
>> > custom user and talk about "useradd" and such.  Also, most users can
>> > skip the whole custom-patched-kernel section.  Hopefully all developers
>> > will be able to stop that too by the time 2.9.0 comes out.
>> >
>> > The error you are seeing almost certainly means that you don't have a
>> > kernel installed that offers symbols compatible with the kernel that
>> > lustre was compiled against.
>> >
>> > It sounds like you built your own kernel. Did you install all the
>> > resulting kernel packages before building lustre (including any
>> > devel-related packages)?  Is your custom kernel the newest kernel
>> > installed on your system?
>> >
>> > Chris
>> >
>> > On 08/18/2016 01:00 PM, Patrick Farrell wrote:
>> >> Good afternoon,
>> >>
>> >>
>> >> I'm trying to build and install updated Lustre master on CentOS 6 for
>> >> the first time in a month or two, and I'm having trouble.
>> >>
>> >>
>> >> I use the build procedure documented here:
>> >>
>> >> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>> >>
>> >>
>> >> I've got some new kmod-* RPMs, and I need to install those to install
>> >> Lustre, which is fine, except I get a huge string of messages like 
>> this
>> >> when I try:
>> >>
>> >> error: Failed dependencies:
>> >>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
>> >> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>> >>         ksym(__mutex_init) = 0x4bf79039 is needed by
>> >> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>> >>
>> >> I've rebuilt and reinstalled my kernel with this latest version of
>> >> Lustre.  Error messages remain the same, and I can't install.
>> >>
>> >>
>> >> Any thoughts or advice?
>> >>
>> >>
>> >> - Patrick
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> lustre-devel mailing list
>> >> lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>
>> >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>> >>
>> >
>> > .
>> >
>>
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>
>
>
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20160819/234a8183/attachment-0001.htm>

[lustre-devel] CentOS 6 - Build problems with kmod

[James Simmons]

> Unfortunately, when I do that, I am apparently still missing something - lustre-osd is not found, and the lustre modules are
> not found when I try to load them.? (running depmod didn't help there)
> 
> Reverting LU-5614 solves this for me, for now.? I'm hoping someone who's still building and installing for CentOS 6 has some
> insight.

Do you have kabi-whitelist installed? I had a similar problem and if I 
remember right after I installed kabi-whitelist the issues went away.
Be aware you need the kabi-whitelist for the kernel you are building
against installed.

> On 08/19/2016 10:27 AM, Patrick Farrell wrote:
>       Thanks, Ben - That looks likely to work.? Still more than a bit weird.
> 
>       Looking at why things are still working internally at Cray for CentOS 6, I think it's because we're not using
>       lustre.spec (for boring and complicated reasons).
> 
>       It seems very likely the problem was introduced by LU-5614 - http://review.whamcloud.com/#/c/12063/
> 
>       Is Intel still building and installing on CentOS 6 automatically?? Or has all automated testing moved to CentOS 7?
> 
>       - Patrick
>       On 08/19/2016 08:57 AM, Ben Evans wrote:
>       When I ran into the problem, I did an rpm -I --nodeps and it worked fine. ?The errors you are seeing are
>       dependencies listed in the RPMs, not ones used by the actual .ko's.
> 
> I spent some time trying to dig up where the 'kmod-' prefix for all the kernel symbols was getting inserted, but
> couldn't track it down.
> 
> -Ben Evans
> 
> From: lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of Patrick Farrell <paf@cray.com>
> Date: Thursday, August 18, 2016 at 6:11 PM
> To: "Christopher J. Morrone" <morrone2@llnl.gov>, "lustre-devel at lists.lustre.org" <lustre-devel@lists.lustre.org>
> Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod
> 
> 
> Chris,
> 
> 
> I agree with your contention about the kernel symbols, that's why I rebuilt from scratch and reinstalled. ?Just
> did it again.? Still getting the error.
> 
> 
> > It sounds like you built your own kernel.? Did you install all the
> > resulting kernel packages before building lustre (including any
> > devel-related packages)??
> 
> Yes, but this process doesn't produce anything other than the kernel?RPM.
> 
> 
> >?Is your custom kernel the newest kernel
> > installed on your system?
> 
> Yes. ?It's the newest and it's what's booted.
> 
> 
> So, does anyone have any insight in to what needs to change in the documented build process so I can build and
> install Lustre on CentOS 6? ?(ldisksfs, not ZFS, so I must build the kernel)? It seems likely that I'm missing
> some symbol RPMs or similar, but kernel-syms is a SuSE only thing, I believe. ?I believe Intel is still building
> and installing ldiskfs Lustre on CentOS 6, so there must be something...?
> 
> 
> - Patrick
> 
> ______________________________________________________________________________________________________________________________
> From: lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of Christopher J. Morrone <morrone2@llnl.gov>
> Sent: Thursday, August 18, 2016 3:44:52 PM
> To: lustre-devel at lists.lustre.org
> Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod ?
> On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
> > Yes, those instructions should be taken with a huge grain of salt.? For
> > instance, instructions for compiling lustre should really employ a
> 
> s/should/should NOT/
> 
> > custom user and talk about "useradd" and such.? Also, most users can
> > skip the whole custom-patched-kernel section.? Hopefully all developers
> > will be able to stop that too by the time 2.9.0 comes out.
> >
> > The error you are seeing almost certainly means that you don't have a
> > kernel installed that offers symbols compatible with the kernel that
> > lustre was compiled against.
> >
> > It sounds like you built your own kernel.? Did you install all the
> > resulting kernel packages before building lustre (including any
> > devel-related packages)?? Is your custom kernel the newest kernel
> > installed on your system?
> >
> > Chris
> >
> > On 08/18/2016 01:00 PM, Patrick Farrell wrote:
> >> Good afternoon,
> >>
> >>
> >> I'm trying to build and install updated Lustre master on CentOS 6 for
> >> the first time in a month or two, and I'm having trouble.
> >>
> >>
> >> I use the build procedure documented here:
> >>
> >> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
> >>
> >>
> >> I've got some new kmod-* RPMs, and I need to install those to install
> >> Lustre, which is fine, except I get a huge string of messages like this
> >> when I try:
> >>
> >> error: Failed dependencies:
> >>???????? ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
> >> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
> >>???????? ksym(__mutex_init) = 0x4bf79039 is needed by
> >> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
> >>
> >> I've rebuilt and reinstalled my kernel with this latest version of
> >> Lustre.? Error messages remain the same, and I can't install.
> >>
> >>
> >> Any thoughts or advice?
> >>
> >>
> >> - Patrick
> >>
> >>
> >>
> >> _______________________________________________
> >> lustre-devel mailing list
> >> lustre-devel at lists.lustre.org
> >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
> >>
> >
> > .
> >
> 
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
> 
> 
> 
> 
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
> 
> 
> 
> 

[lustre-devel] CentOS 6 - Build problems with kmod

[Ben Evans]

kabi-whitelist didn't seem to help for me when I ran into this.

-Ben

On 8/19/16, 1:03 PM, "lustre-devel on behalf of James Simmons"
<lustre-devel-bounces at lists.lustre.org on behalf of
jsimmons@infradead.org> wrote:

>
>> Unfortunately, when I do that, I am apparently still missing something
>>- lustre-osd is not found, and the lustre modules are
>> not found when I try to load them.  (running depmod didn't help there)
>> 
>> Reverting LU-5614 solves this for me, for now.  I'm hoping someone
>>who's still building and installing for CentOS 6 has some
>> insight.
>
>Do you have kabi-whitelist installed? I had a similar problem and if I
>remember right after I installed kabi-whitelist the issues went away.
>Be aware you need the kabi-whitelist for the kernel you are building
>against installed.
>
>> On 08/19/2016 10:27 AM, Patrick Farrell wrote:
>>       Thanks, Ben - That looks likely to work.  Still more than a bit
>>weird.
>> 
>>       Looking at why things are still working internally at Cray for
>>CentOS 6, I think it's because we're not using
>>       lustre.spec (for boring and complicated reasons).
>> 
>>       It seems very likely the problem was introduced by LU-5614 -
>>http://review.whamcloud.com/#/c/12063/
>> 
>>       Is Intel still building and installing on CentOS 6 automatically?
>> Or has all automated testing moved to CentOS 7?
>> 
>>       - Patrick
>>       On 08/19/2016 08:57 AM, Ben Evans wrote:
>>       When I ran into the problem, I did an rpm -I --nodeps and it
>>worked fine.  The errors you are seeing are
>>       dependencies listed in the RPMs, not ones used by the actual
>>.ko's.
>> 
>> I spent some time trying to dig up where the 'kmod-' prefix for all the
>>kernel symbols was getting inserted, but
>> couldn't track it down.
>> 
>> -Ben Evans
>> 
>> From: lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of
>>Patrick Farrell <paf@cray.com>
>> Date: Thursday, August 18, 2016 at 6:11 PM
>> To: "Christopher J. Morrone" <morrone2@llnl.gov>,
>>"lustre-devel at lists.lustre.org" <lustre-devel@lists.lustre.org>
>> Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod
>> 
>> 
>> Chris,
>> 
>> 
>> I agree with your contention about the kernel symbols, that's why I
>>rebuilt from scratch and reinstalled.  Just
>> did it again.  Still getting the error.
>> 
>> 
>> > It sounds like you built your own kernel.  Did you install all the
>> > resulting kernel packages before building lustre (including any
>> > devel-related packages)?
>> 
>> Yes, but this process doesn't produce anything other than the kernel
>>RPM.
>> 
>> 
>> > Is your custom kernel the newest kernel
>> > installed on your system?
>> 
>> Yes.  It's the newest and it's what's booted.
>> 
>> 
>> So, does anyone have any insight in to what needs to change in the
>>documented build process so I can build and
>> install Lustre on CentOS 6?  (ldisksfs, not ZFS, so I must build the
>>kernel)  It seems likely that I'm missing
>> some symbol RPMs or similar, but kernel-syms is a SuSE only thing, I
>>believe.  I believe Intel is still building
>> and installing ldiskfs Lustre on CentOS 6, so there must be
>>something...?
>> 
>> 
>> - Patrick
>> 
>> 
>>_________________________________________________________________________
>>_____________________________________________________
>> From: lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf of
>>Christopher J. Morrone <morrone2@llnl.gov>
>> Sent: Thursday, August 18, 2016 3:44:52 PM
>> To: lustre-devel at lists.lustre.org
>> Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod
>> On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
>> > Yes, those instructions should be taken with a huge grain of salt.
>>For
>> > instance, instructions for compiling lustre should really employ a
>> 
>> s/should/should NOT/
>> 
>> > custom user and talk about "useradd" and such.  Also, most users can
>> > skip the whole custom-patched-kernel section.  Hopefully all
>>developers
>> > will be able to stop that too by the time 2.9.0 comes out.
>> >
>> > The error you are seeing almost certainly means that you don't have a
>> > kernel installed that offers symbols compatible with the kernel that
>> > lustre was compiled against.
>> >
>> > It sounds like you built your own kernel.  Did you install all the
>> > resulting kernel packages before building lustre (including any
>> > devel-related packages)?  Is your custom kernel the newest kernel
>> > installed on your system?
>> >
>> > Chris
>> >
>> > On 08/18/2016 01:00 PM, Patrick Farrell wrote:
>> >> Good afternoon,
>> >>
>> >>
>> >> I'm trying to build and install updated Lustre master on CentOS 6 for
>> >> the first time in a month or two, and I'm having trouble.
>> >>
>> >>
>> >> I use the build procedure documented here:
>> >>
>> >> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>> >>
>> >>
>> >> I've got some new kmod-* RPMs, and I need to install those to install
>> >> Lustre, which is fine, except I get a huge string of messages like
>>this
>> >> when I try:
>> >>
>> >> error: Failed dependencies:
>> >>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
>> >> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>> >>         ksym(__mutex_init) = 0x4bf79039 is needed by
>> >> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>> >>
>> >> I've rebuilt and reinstalled my kernel with this latest version of
>> >> Lustre.  Error messages remain the same, and I can't install.
>> >>
>> >>
>> >> Any thoughts or advice?
>> >>
>> >>
>> >> - Patrick
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> lustre-devel mailing list
>> >> lustre-devel at lists.lustre.org
>> >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>> >>
>> >
>> > .
>> >
>> 
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>> 
>> 
>> 
>> 
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>> 
>> 
>> 
>> 

[lustre-devel] CentOS 6 - Build problems with kmod

[Christopher J. Morrone]

On 08/19/2016 06:57 AM, Ben Evans wrote:
> When I ran into the problem, I did an rpm -I --nodeps and it worked
> fine.  The errors you are seeing are dependencies listed in the RPMs,
> not ones used by the actual .ko's.

Please do not use --nodeps unless you are an rpm expert and entirely
understand why the problem is happening, and what the impacts of using
--nodeps will be.

In this case, the warnings are correct, and the underlying problem
really needs to be resolved.

> I spent some time trying to dig up where the 'kmod-' prefix for all the
> kernel symbols was getting inserted, but couldn't track it down.

I'm not sure what you mean.  The kernel symbol names don't have a kmod-
prefix added...

Are you asking where RHEL kmod-lustre-* packages put their kernel
modules?  They are installed in /lib/module/<kernel version>/extra,
where <kernel version> is the version against which Lustre was compiled.
 The modules are then symlinked into the /lib/module/<kernel
version>/weak-updates of the other kernels that offer symbols compatible
to the particular module being symlinked.

Or do you mean where does the kernel's provided symbols get listed?
Those do not involve kmod- named packages.  Most (all?) of the compiled
modules are simply in the base kernel-<kernel version> package.  There
is a file listing all symbols offered by the kernel in
/boot/symvers-<kernel-version>.gz.

Chris

> -Ben Evans
> 
> From: lustre-devel <lustre-devel-bounces@lists.lustre.org
> <mailto:lustre-devel-bounces@lists.lustre.org>> on behalf of Patrick
> Farrell <paf at cray.com <mailto:paf@cray.com>>
> Date: Thursday, August 18, 2016 at 6:11 PM
> To: "Christopher J. Morrone" <morrone2@llnl.gov
> <mailto:morrone2@llnl.gov>>, "lustre-devel at lists.lustre.org
> <mailto:lustre-devel@lists.lustre.org>" <lustre-devel@lists.lustre.org
> <mailto:lustre-devel@lists.lustre.org>>
> Subject: Re: [lustre-devel] CentOS 6 - Build problems with kmod
> 
> 
> Chris,
> 
> 
> I agree with your contention about the kernel symbols, that's why I
> rebuilt from scratch and reinstalled.  Just did it again.  Still getting
> the error.
> 
> 
>> It sounds like you built your own kernel.  Did you install all the
>> resulting kernel packages before building lustre (including any
>> devel-related packages)? 
> 
> Yes, but this process doesn't produce anything other than the kernel RPM.
> 
> 
>> Is your custom kernel the newest kernel
>> installed on your system?
> 
> Yes.  It's the newest and it's what's booted.
> 
> 
> So, does anyone have any insight in to what needs to change in the
> documented build process so I can build and install Lustre on CentOS 6?
>  (ldisksfs, not ZFS, so I must build the kernel)  It seems likely that
> I'm missing some symbol RPMs or similar, but kernel-syms is a SuSE only
> thing, I believe.  I believe Intel is still building and installing
> ldiskfs Lustre on CentOS 6, so there must be something...?
> 
> 
> - Patrick
> 
> ------------------------------------------------------------------------
> *From:* lustre-devel <lustre-devel-bounces@lists.lustre.org
> <mailto:lustre-devel-bounces@lists.lustre.org>> on behalf of Christopher
> J. Morrone <morrone2 at llnl.gov <mailto:morrone2@llnl.gov>>
> *Sent:* Thursday, August 18, 2016 3:44:52 PM
> *To:* lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>
> *Subject:* Re: [lustre-devel] CentOS 6 - Build problems with kmod
>  
> On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
>> Yes, those instructions should be taken with a huge grain of salt.  For
>> instance, instructions for compiling lustre should really employ a
> 
> s/should/should NOT/
> 
>> custom user and talk about "useradd" and such.  Also, most users can
>> skip the whole custom-patched-kernel section.  Hopefully all developers
>> will be able to stop that too by the time 2.9.0 comes out.
>> 
>> The error you are seeing almost certainly means that you don't have a
>> kernel installed that offers symbols compatible with the kernel that
>> lustre was compiled against.
>> 
>> It sounds like you built your own kernel.  Did you install all the
>> resulting kernel packages before building lustre (including any
>> devel-related packages)?  Is your custom kernel the newest kernel
>> installed on your system?
>> 
>> Chris
>> 
>> On 08/18/2016 01:00 PM, Patrick Farrell wrote:
>>> Good afternoon,
>>>
>>>
>>> I'm trying to build and install updated Lustre master on CentOS 6 for
>>> the first time in a month or two, and I'm having trouble.
>>>
>>>
>>> I use the build procedure documented here:
>>>
>>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>>>
>>>
>>> I've got some new kmod-* RPMs, and I need to install those to install
>>> Lustre, which is fine, except I get a huge string of messages like this
>>> when I try:
>>>
>>> error: Failed dependencies:
>>>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
>>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>>         ksym(__mutex_init) = 0x4bf79039 is needed by
>>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>>
>>> I've rebuilt and reinstalled my kernel with this latest version of
>>> Lustre.  Error messages remain the same, and I can't install.
>>>
>>>
>>> Any thoughts or advice?
>>>
>>>
>>> - Patrick
>>>
>>>
>>>
>>> _______________________________________________
>>> lustre-devel mailing list
>>> lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>
>>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>>>
>> 
>> .
>> 
> 
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org <mailto:lustre-devel@lists.lustre.org>
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

[lustre-devel] CentOS 6 - Build problems with kmod

[Christopher J. Morrone]

On 08/18/2016 03:11 PM, Patrick Farrell wrote:
> 
> Chris,
> 
> 
> I agree with your contention about the kernel symbols, that's why I
> rebuilt from scratch and reinstalled.  Just did it again.  Still getting
> the error.
> 
> 
>> It sounds like you built your own kernel.  Did you install all the
>> resulting kernel packages before building lustre (including any
>> devel-related packages)? 
> 
> Yes, but this process doesn't produce anything other than the kernel RPM.

You are saying that literally only one rpm is produced?  If that isn't
what your are saying, please list all of the produced rpms, and also
list which ones you are installing.

If you only have a single kernel rpm, then you almost certainly don't
have the correct packages installed to allow Lustre to compile against
that kernel.  Lustre is probably compiling against some other installed
kernel.

>> Is your custom kernel the newest kernel
>> installed on your system?
> 
> Yes.  It's the newest and it's what's booted.

> So, does anyone have any insight in to what needs to change in the
> documented build process so I can build and install Lustre on CentOS 6?
>  (ldisksfs, not ZFS, so I must build the kernel)  It seems likely that
> I'm missing some symbol RPMs or similar, but kernel-syms is a SuSE only
> thing, I believe.  I believe Intel is still building and installing
> ldiskfs Lustre on CentOS 6, so there must be something...?

There aren't separate "symbol" rpms for RHEL.

I still think that it is likely that you compiled against a kernel on
your build node that either does not exist on your lustre node, or the
kernel that lustre compiled about on your build node was old enough that
its symbols are incompatible with the booted kernel on your lustre node.

Here are somethings that you can try to eliminate problems:

1) Remove the lustre-patched kernel altogether.  Purge it from your
system.  Build Lustre against the stock kernel.

2) Figure out which kernel you are actually compiling against.  Or at
the very least, which ones your lustre packages are compatible with.

I would probably do one or both of these:

- Run "rpm -qp --requires kmod-lustre-<whatever>.rpm".  Pick out a few
of the required kernel symbols for which you saw complaints when trying
to install your kernel.  This will eliminate the issue of your kernel
build that seems to be going wrong.  You can come back to this later
when you verify that the lustre build is working correctly.

- Run "rpm -q --provides" on each installed kernel package (the packages
that have the name of the form 'kernel-<version>'.  Rum that output
through grep a few times for each of the specific symbol names that you
picked in the previous step.  For example:

# rpm -q --provides kernel-3.10.0-327.28.2.1chaos.ch6.x86_64 |grep
__mutex_init
kernel(__mutex_init) = 0x9a025cd5

Now, compare the hex symbol version required by the kmod-lustre package
with the hex symbol versions provided by the various kernels that you
currently have installed.  Which are offering compatible symbols?

For instance:

# rpm -qp --requires
kmod-lustre-2.8.0_0.0.llnlpreview.33-1.ch6.x86_64.rpm | grep __mutex_init
kernel(__mutex_init) = 0x9a025cd5

Look, they match!  I can install this lustre modules with this kernel
installed, and have no rpm requirement complaints.

You are going to find that you do _not_ have a kernel package installed
that offers the symbols that the kmod-lustre- package(s) require.

Chris


> - Patrick
> 
> ------------------------------------------------------------------------
> *From:* lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf
> of Christopher J. Morrone <morrone2@llnl.gov>
> *Sent:* Thursday, August 18, 2016 3:44:52 PM
> *To:* lustre-devel at lists.lustre.org
> *Subject:* Re: [lustre-devel] CentOS 6 - Build problems with kmod
>  
> On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
>> Yes, those instructions should be taken with a huge grain of salt.  For
>> instance, instructions for compiling lustre should really employ a
> 
> s/should/should NOT/
> 
>> custom user and talk about "useradd" and such.  Also, most users can
>> skip the whole custom-patched-kernel section.  Hopefully all developers
>> will be able to stop that too by the time 2.9.0 comes out.
>> 
>> The error you are seeing almost certainly means that you don't have a
>> kernel installed that offers symbols compatible with the kernel that
>> lustre was compiled against.
>> 
>> It sounds like you built your own kernel.  Did you install all the
>> resulting kernel packages before building lustre (including any
>> devel-related packages)?  Is your custom kernel the newest kernel
>> installed on your system?
>> 
>> Chris
>> 
>> On 08/18/2016 01:00 PM, Patrick Farrell wrote:
>>> Good afternoon,
>>>
>>>
>>> I'm trying to build and install updated Lustre master on CentOS 6 for
>>> the first time in a month or two, and I'm having trouble.
>>>
>>>
>>> I use the build procedure documented here:
>>>
>>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>>>
>>>
>>> I've got some new kmod-* RPMs, and I need to install those to install
>>> Lustre, which is fine, except I get a huge string of messages like this
>>> when I try:
>>>
>>> error: Failed dependencies:
>>>         ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
>>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>>         ksym(__mutex_init) = 0x4bf79039 is needed by
>>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>>
>>> I've rebuilt and reinstalled my kernel with this latest version of
>>> Lustre.  Error messages remain the same, and I can't install.
>>>
>>>
>>> Any thoughts or advice?
>>>
>>>
>>> - Patrick
>>>
>>>
>>>
>>> _______________________________________________
>>> lustre-devel mailing list
>>> lustre-devel at lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>>>
>> 
>> .
>> 
> 
> _______________________________________________
> lustre-devel mailing list
> lustre-devel at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

[lustre-devel] CentOS 6 - Build problems with kmod

[Patrick Farrell]

On 08/19/2016 03:44 PM, Christopher J. Morrone wrote:
> On 08/18/2016 03:11 PM, Patrick Farrell wrote:
>> Chris,
>>
>>
>> I agree with your contention about the kernel symbols, that's why I
>> rebuilt from scratch and reinstalled.  Just did it again.  Still getting
>> the error.
>>
>>
>>> It sounds like you built your own kernel.  Did you install all the
>>> resulting kernel packages before building lustre (including any
>>> devel-related packages)?
>> Yes, but this process doesn't produce anything other than the kernel RPM.
> You are saying that literally only one rpm is produced?  If that isn't
> what your are saying, please list all of the produced rpms, and also
> list which ones you are installing.
Yes, one non-source RPM.  I don't install any RPMs as part of the build 
process itself.
>
> If you only have a single kernel rpm, then you almost certainly don't
> have the correct packages installed to allow Lustre to compile against
> that kernel.  Lustre is probably compiling against some other installed
> kernel.
It's compiling the whole kernel from source, so I don't need any other 
packages.  I build Lustre against the kernel bits directly, in the 
directories where they were built, not by installing any kernel RPMs.  I 
just extract the kernel source, patch it, and then build it, then build 
Lustre against the results.  The path for Lustre ./configure 
--with-linux[or whatever that option is]= is down in the build 
directories for the kernel.  (It's definitely not building against 
another installed kernel - I can make modifications  in this source and 
have them show up on the nodes where I install Lustre and this kernel.)
>>> Is your custom kernel the newest kernel
>>> installed on your system?
>> Yes.  It's the newest and it's what's booted.
>> So, does anyone have any insight in to what needs to change in the
>> documented build process so I can build and install Lustre on CentOS 6?
>>   (ldisksfs, not ZFS, so I must build the kernel)  It seems likely that
>> I'm missing some symbol RPMs or similar, but kernel-syms is a SuSE only
>> thing, I believe.  I believe Intel is still building and installing
>> ldiskfs Lustre on CentOS 6, so there must be something...?
> There aren't separate "symbol" rpms for RHEL.
>
> I still think that it is likely that you compiled against a kernel on
> your build node that either does not exist on your lustre node, or the
> kernel that lustre compiled about on your build node was old enough that
> its symbols are incompatible with the booted kernel on your lustre node.
>
> Here are somethings that you can try to eliminate problems:
>
> 1) Remove the lustre-patched kernel altogether.  Purge it from your
> system.  Build Lustre against the stock kernel.
>
> 2) Figure out which kernel you are actually compiling against.  Or at
> the very least, which ones your lustre packages are compatible with.
>
> I would probably do one or both of these:
>
> - Run "rpm -qp --requires kmod-lustre-<whatever>.rpm".  Pick out a few
> of the required kernel symbols for which you saw complaints when trying
> to install your kernel.  This will eliminate the issue of your kernel
> build that seems to be going wrong.  You can come back to this later
> when you verify that the lustre build is working correctly.
>
> - Run "rpm -q --provides" on each installed kernel package (the packages
> that have the name of the form 'kernel-<version>'.  Rum that output
> through grep a few times for each of the specific symbol names that you
> picked in the previous step.  For example:
>
> # rpm -q --provides kernel-3.10.0-327.28.2.1chaos.ch6.x86_64 |grep
> __mutex_init
> kernel(__mutex_init) = 0x9a025cd5
>
> Now, compare the hex symbol version required by the kmod-lustre package
> with the hex symbol versions provided by the various kernels that you
> currently have installed.  Which are offering compatible symbols?
>
> For instance:
>
> # rpm -qp --requires
> kmod-lustre-2.8.0_0.0.llnlpreview.33-1.ch6.x86_64.rpm | grep __mutex_init
> kernel(__mutex_init) = 0x9a025cd5
>
> Look, they match!  I can install this lustre modules with this kernel
> installed, and have no rpm requirement complaints.
>
> You are going to find that you do _not_ have a kernel package installed
> that offers the symbols that the kmod-lustre- package(s) require.
>
> Chris
>
>
>> - Patrick
>>
>> ------------------------------------------------------------------------
>> *From:* lustre-devel <lustre-devel-bounces@lists.lustre.org> on behalf
>> of Christopher J. Morrone <morrone2@llnl.gov>
>> *Sent:* Thursday, August 18, 2016 3:44:52 PM
>> *To:* lustre-devel at lists.lustre.org
>> *Subject:* Re: [lustre-devel] CentOS 6 - Build problems with kmod
>>   
>> On 08/18/2016 01:43 PM, Christopher J. Morrone wrote:
>>> Yes, those instructions should be taken with a huge grain of salt.  For
>>> instance, instructions for compiling lustre should really employ a
>> s/should/should NOT/
>>
>>> custom user and talk about "useradd" and such.  Also, most users can
>>> skip the whole custom-patched-kernel section.  Hopefully all developers
>>> will be able to stop that too by the time 2.9.0 comes out.
>>>
>>> The error you are seeing almost certainly means that you don't have a
>>> kernel installed that offers symbols compatible with the kernel that
>>> lustre was compiled against.
>>>
>>> It sounds like you built your own kernel.  Did you install all the
>>> resulting kernel packages before building lustre (including any
>>> devel-related packages)?  Is your custom kernel the newest kernel
>>> installed on your system?
>>>
>>> Chris
>>>
>>> On 08/18/2016 01:00 PM, Patrick Farrell wrote:
>>>> Good afternoon,
>>>>
>>>>
>>>> I'm trying to build and install updated Lustre master on CentOS 6 for
>>>> the first time in a month or two, and I'm having trouble.
>>>>
>>>>
>>>> I use the build procedure documented here:
>>>>
>>>> https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821
>>>>
>>>>
>>>> I've got some new kmod-* RPMs, and I need to install those to install
>>>> Lustre, which is fine, except I get a huge string of messages like this
>>>> when I try:
>>>>
>>>> error: Failed dependencies:
>>>>          ksym(__init_waitqueue_head) = 0xffc7c184 is needed by
>>>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>>>          ksym(__mutex_init) = 0x4bf79039 is needed by
>>>> kmod-lustre-2.8.56_44_g288e55b_dirty-1.el6.x86_64
>>>>
>>>> I've rebuilt and reinstalled my kernel with this latest version of
>>>> Lustre.  Error messages remain the same, and I can't install.
>>>>
>>>>
>>>> Any thoughts or advice?
>>>>
>>>>
>>>> - Patrick
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> lustre-devel mailing list
>>>> lustre-devel at lists.lustre.org
>>>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org
>>>>
>>> .
>>>
>> _______________________________________________
>> lustre-devel mailing list
>> lustre-devel at lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

[lustre-devel] CentOS 6 - Build problems with kmod

[Christopher J. Morrone]

On 08/19/2016 01:51 PM, Patrick Farrell wrote:
> 
> On 08/19/2016 03:44 PM, Christopher J. Morrone wrote:
>> On 08/18/2016 03:11 PM, Patrick Farrell wrote:
>>> Chris,
>>>
>>>
>>> I agree with your contention about the kernel symbols, that's why I
>>> rebuilt from scratch and reinstalled.  Just did it again.  Still getting
>>> the error.
>>>
>>>
>>>> It sounds like you built your own kernel.  Did you install all the
>>>> resulting kernel packages before building lustre (including any
>>>> devel-related packages)?
>>> Yes, but this process doesn't produce anything other than the kernel
>>> RPM.
>> You are saying that literally only one rpm is produced?  If that isn't
>> what your are saying, please list all of the produced rpms, and also
>> list which ones you are installing.
> Yes, one non-source RPM.  I don't install any RPMs as part of the build
> process itself.

You should be installing RPMs as part of the build process.  You should
install Lustre's BuildRequires before building Lustre.

>> If you only have a single kernel rpm, then you almost certainly don't
>> have the correct packages installed to allow Lustre to compile against
>> that kernel.  Lustre is probably compiling against some other installed
>> kernel.
> It's compiling the whole kernel from source, so I don't need any other
> packages.  I build Lustre against the kernel bits directly, in the
> directories where they were built, not by installing any kernel RPMs.

That is only going to work if you don't want to make lustre RPM
packages.  If you want to build RPM packages, you need to follow
standard RPM practices.  In other words, you need to have Lustre's
prerequite packages installed before you build Lustre's RPMs.

So you are going to need to those other kernel packages, and you need to
install them before building Lustre.

Chris

[lustre-devel] CentOS 6 - Build problems with kmod

[Patrick Farrell]

On 08/19/2016 03:58 PM, Christopher J. Morrone wrote:
> On 08/19/2016 01:51 PM, Patrick Farrell wrote:
>> On 08/19/2016 03:44 PM, Christopher J. Morrone wrote:
>>> On 08/18/2016 03:11 PM, Patrick Farrell wrote:
>>>> Chris,
>>>>
>>>>
>>>> I agree with your contention about the kernel symbols, that's why I
>>>> rebuilt from scratch and reinstalled.  Just did it again.  Still getting
>>>> the error.
>>>>
>>>>
>>>>> It sounds like you built your own kernel.  Did you install all the
>>>>> resulting kernel packages before building lustre (including any
>>>>> devel-related packages)?
>>>> Yes, but this process doesn't produce anything other than the kernel
>>>> RPM.
>>> You are saying that literally only one rpm is produced?  If that isn't
>>> what your are saying, please list all of the produced rpms, and also
>>> list which ones you are installing.
>> Yes, one non-source RPM.  I don't install any RPMs as part of the build
>> process itself.
> You should be installing RPMs as part of the build process.  You should
> install Lustre's BuildRequires before building Lustre.
Huh.  OK.  I have never needed to do so before when building everything 
from scratch.

Is this build process documented somewhere, so I can switch to it in my 
testing environment?  I don't know how to invoke a CentOS kernel build 
to create those RPMs I now need.
>
>>> If you only have a single kernel rpm, then you almost certainly don't
>>> have the correct packages installed to allow Lustre to compile against
>>> that kernel.  Lustre is probably compiling against some other installed
>>> kernel.
>> It's compiling the whole kernel from source, so I don't need any other
>> packages.  I build Lustre against the kernel bits directly, in the
>> directories where they were built, not by installing any kernel RPMs.
> That is only going to work if you don't want to make lustre RPM
> packages.  If you want to build RPM packages, you need to follow
> standard RPM practices.  In other words, you need to have Lustre's
> prerequite packages installed before you build Lustre's RPMs.
>
> So you are going to need to those other kernel packages, and you need to
> install them before building Lustre.
Funnily enough, I've been building Lustre RPM packages this way for 
years, and they've always worked fine.  I get that it's not standard 
practice and I need to adjust - But it did work up until now, and I 
believe it's the only documented process for building Lustre and the 
kernel from source.

- Patrick

>
> Chris
>

[lustre-devel] CentOS 6 - Build problems with kmod

[Christopher J. Morrone]

On 08/19/2016 02:10 PM, Patrick Farrell wrote:
> 
> On 08/19/2016 03:58 PM, Christopher J. Morrone wrote:
>> On 08/19/2016 01:51 PM, Patrick Farrell wrote:
>>> On 08/19/2016 03:44 PM, Christopher J. Morrone wrote:
>>>> On 08/18/2016 03:11 PM, Patrick Farrell wrote:
>>>>> Chris,
>>>>>
>>>>>
>>>>> I agree with your contention about the kernel symbols, that's why I
>>>>> rebuilt from scratch and reinstalled.  Just did it again.  Still
>>>>> getting
>>>>> the error.
>>>>>
>>>>>
>>>>>> It sounds like you built your own kernel.  Did you install all the
>>>>>> resulting kernel packages before building lustre (including any
>>>>>> devel-related packages)?
>>>>> Yes, but this process doesn't produce anything other than the kernel
>>>>> RPM.
>>>> You are saying that literally only one rpm is produced?  If that isn't
>>>> what your are saying, please list all of the produced rpms, and also
>>>> list which ones you are installing.
>>> Yes, one non-source RPM.  I don't install any RPMs as part of the build
>>> process itself.
>> You should be installing RPMs as part of the build process.  You should
>> install Lustre's BuildRequires before building Lustre.
> Huh.  OK.  I have never needed to do so before when building everything
> from scratch.
> 
> Is this build process documented somewhere, so I can switch to it in my
> testing environment?  I don't know how to invoke a CentOS kernel build
> to create those RPMs I now need.

I don't know.  That sounds like something you should look up in
CentOS/RHEL/Fedora documentation.

Or you maybe you can just skip the custom kernel altogether.  That is
largly what I do.  I haven't built a kernel myself in years, actually.
When LU-684's patch lands, Lustre-owned custom kernels can be a thing of
the past for everyone!

Get on the bandwagon early and stop using Lustre-patched kernels now. :)

Chris