question on how to (correctly) build an initrd for a root disk on a RAID1

question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

Hi folks:

  We have been trying to build an operational initrd for a root drive
on  software RAID1.

  Basic specs:   2.6.23.14 kernel (though same issues with 2.6.27.2
kernel), on Centos 5.2, pair of SATA drives using libata and other
drivers, x86_64 architecture.

  Using mkinitrd --with=raid1 --with=libata --with=sd_mod ...

  I am getting a kernel panic on switchroot, it can't find the root
file system.  I did some googling, but could not find a similar
example.  Do I need to create a specialized init script?  Is there an
example of this somewhere?  I did look at the Documentation/md.txt and
tried the

     md=0,sda1,sdb1 md=1,sda2,sdb2  md=2,sda3,sdb3

  kernel boot time option, to no avail.  Persistent superblocks are
on, and the devices have the right partition types.

  Any suggestions?  More things to read?

  Thanks in advance.

Joe

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Steve Cousins]

Joe Landman wrote:
> Hi folks:
>
>   We have been trying to build an operational initrd for a root drive
> on  software RAID1.
>
>   Basic specs:   2.6.23.14 kernel (though same issues with 2.6.27.2
> kernel), on Centos 5.2, pair of SATA drives using libata and other
> drivers, x86_64 architecture.
>
>   Using mkinitrd --with=raid1 --with=libata --with=sd_mod ...
>
>   I am getting a kernel panic on switchroot, it can't find the root
> file system.  

Hi Joe,

What does the "root=" kernel parameter look like in your grub.conf (are 
you using grub?)? Mine looks like root=/dev/md2. My md0 is the boot 
partition and md1 is swap. I know this is simplistic but I didn't find 
it in your message.

Steve

> I did some googling, but could not find a similar
> example.  Do I need to create a specialized init script?  Is there an
> example of this somewhere?  I did look at the Documentation/md.txt and
> tried the
>
>      md=0,sda1,sdb1 md=1,sda2,sdb2  md=2,sda3,sdb3
>
>   kernel boot time option, to no avail.  Persistent superblocks are
> on, and the devices have the right partition types.
>
>   Any suggestions?  More things to read?
>
>   Thanks in advance.
>
> Joe
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

On Sun, Oct 26, 2008 at 11:22 PM, Steve Cousins <steve.cousins@maine.edu> wrote:
> Joe Landman wrote:
>>
>> Hi folks:
>>
>>  We have been trying to build an operational initrd for a root drive
>> on  software RAID1.
>>
>>  Basic specs:   2.6.23.14 kernel (though same issues with 2.6.27.2
>> kernel), on Centos 5.2, pair of SATA drives using libata and other
>> drivers, x86_64 architecture.
>>
>>  Using mkinitrd --with=raid1 --with=libata --with=sd_mod ...
>>
>>  I am getting a kernel panic on switchroot, it can't find the root
>> file system.
>
> Hi Joe,
>
> What does the "root=" kernel parameter look like in your grub.conf (are you
> using grub?)? Mine looks like root=/dev/md2. My md0 is the boot partition

Hi Steve:

  root=/dev/md2

and the boot device is (hd0,0).



Joe

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Luca Berra]

On Sun, Oct 26, 2008 at 10:37:17PM -0400, Joe Landman wrote:
>Hi folks:
>
>  We have been trying to build an operational initrd for a root drive
>on  software RAID1.
>
>  Basic specs:   2.6.23.14 kernel (though same issues with 2.6.27.2
>kernel), on Centos 5.2, pair of SATA drives using libata and other
>drivers, x86_64 architecture.
Custom built kernel or centos provided kernel?


>  Using mkinitrd --with=raid1 --with=libata --with=sd_mod ...
>
>  I am getting a kernel panic on switchroot, it can't find the root
>file system.  I did some googling, but could not find a similar
>example.
unfortunately the kernel panic is a consequence of a previous error,
someone willing to help you would need to know everithing that was
spitted between the 'starting red-hat nash' message and the kernel
panic.

>Do I need to create a specialized init script?  Is there an
usually no, redhat/centos initrd should be able to boot from a raid1

>example of this somewhere?  I did look at the Documentation/md.txt and
>tried the
>
>     md=0,sda1,sdb1 md=1,sda2,sdb2  md=2,sda3,sdb3
if you are using the centos kernel, raid is modular and this parameter
has no effect.
mkinitrd should use your /etc/mdadm.conf to discover how the array
should be laid out.

>  kernel boot time option, to no avail.  Persistent superblocks are
>on, and the devices have the right partition types.
partition types do not matter anymore.

>
>  Any suggestions?  More things to read?
is your fstab correct?
is your mdadm.conf correct?
are you loading the correct drivers for your hd controller?

mkinitrd is a shell script, if all else fails, run it with sh -x and
read the output.

L.
-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

On Mon, Oct 27, 2008 at 3:16 AM, Luca Berra <bluca@comedia.it> wrote:
> On Sun, Oct 26, 2008 at 10:37:17PM -0400, Joe Landman wrote:
>>
>> Hi folks:
>>
>>  We have been trying to build an operational initrd for a root drive
>> on  software RAID1.
>>
>>  Basic specs:   2.6.23.14 kernel (though same issues with 2.6.27.2
>> kernel), on Centos 5.2, pair of SATA drives using libata and other
>> drivers, x86_64 architecture.
>
> Custom built kernel or centos provided kernel?

Custom.  The Centos provided does in fact work with this.  I did
(previously) look at the initrd (uncompress and un-cpio it), and
didn't see anything in particular different from what I had done.

>
>
>>  Using mkinitrd --with=raid1 --with=libata --with=sd_mod ...
>>
>>  I am getting a kernel panic on switchroot, it can't find the root
>> file system.  I did some googling, but could not find a similar
>> example.
>
> unfortunately the kernel panic is a consequence of a previous error,
> someone willing to help you would need to know everithing that was
> spitted between the 'starting red-hat nash' message and the kernel
> panic.

I looked and this was actually the earliest error.  I will try to
capture the rest of this and post it for download (it is long).

>
>> Do I need to create a specialized init script?  Is there an
>
> usually no, redhat/centos initrd should be able to boot from a raid1

The baseline provided kernel can.  This is a custom kernel, and I am
attempting to find the right mkinitrd parameters.  One of the nicer
things in the Ubuntu/Debian process is that they have the ability to
add most/all modules in to an initrd.  Somehow this is my current
thought, that we are missing something critical.

>
>> example of this somewhere?  I did look at the Documentation/md.txt and
>> tried the
>>
>>    md=0,sda1,sdb1 md=1,sda2,sdb2  md=2,sda3,sdb3
>
> if you are using the centos kernel, raid is modular and this parameter
> has no effect.
> mkinitrd should use your /etc/mdadm.conf to discover how the array
> should be laid out.

Ok.  This is where it gets interesting.  If the /etc/mdadm.conf is on
the RAID, you have a bit of a "chicken and egg" problem to deal with
here.  How does the system find /etc/mdadm.conf to make the RAID when
/etc/mdadm.conf is on the RAID?

This implies that /etc/mdadm.conf is in the initrd.  Peeking inside
the default centos one, I can see that this is not true.  Which
implies that it doesn't see /etc/mdadm.conf before it assembles the
array, as the /etc/mdadm.conf is on the array, and not in the initrd
image.

Is there a way to include the /etc/mdadm.conf into the initrd?

>
>>  kernel boot time option, to no avail.  Persistent superblocks are
>> on, and the devices have the right partition types.
>
> partition types do not matter anymore.

Autoassembly is depracated?


>>  Any suggestions?  More things to read?
>
> is your fstab correct?

Yes.  Boots fine with older kernel.  fstab doesn't change between the two.

> is your mdadm.conf correct?


Yes.  Boots fine with older kernel.  mdadm.conf doesn't change between the two.

> are you loading the correct drivers for your hd controller?

I believe so:  looked at the list from lsmod and replicated this in
the mkinitrd.  However, I have seen drivers change markedly release to
release.  Hence my interest in building an initrd with the maximum
number of modules (specific to SATA/SCSI)

>
> mkinitrd is a shell script, if all else fails, run it with sh -x and
> read the output.

Already done.  Even instrumented the code so I could see what the
handleraid function was in fact doing.  It looks correct, figures out
the right modules, and makes sure they are in there.

So I am wondering if it is a module load order, or something else like
that., apart from just a missing module.  Which could be a depmod
issue.

Thanks.

Joe

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Brendan Conoboy]

Joe Landman wrote:
> Ok.  This is where it gets interesting.  If the /etc/mdadm.conf is on
> the RAID, you have a bit of a "chicken and egg" problem to deal with
> here.  How does the system find /etc/mdadm.conf to make the RAID when
> /etc/mdadm.conf is on the RAID?
> 
> This implies that /etc/mdadm.conf is in the initrd.  Peeking inside
> the default centos one, I can see that this is not true.  Which
> implies that it doesn't see /etc/mdadm.conf before it assembles the
> array, as the /etc/mdadm.conf is on the array, and not in the initrd
> image.
> 
> Is there a way to include the /etc/mdadm.conf into the initrd?

This is almost certainly your problem.  If / is a RAID volume, the 
initrd itself needs to have an /etc/mdadm.conf that reflects how it is 
constructed.  This is something that mkinitrd should be doing for you. 
I'll second the suggestion of running mkinitrd with sh's -x option to 
see why it's not doing this for you.  Good luck.

-- 
Brendan Conoboy / Red Hat, Inc. / blc@redhat.com

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

On Mon, 2008-10-27 at 10:36 -0600, Brendan Conoboy wrote:
> Joe Landman wrote:
> > Ok.  This is where it gets interesting.  If the /etc/mdadm.conf is on
> > the RAID, you have a bit of a "chicken and egg" problem to deal with
> > here.  How does the system find /etc/mdadm.conf to make the RAID when
> > /etc/mdadm.conf is on the RAID?
> > 
> > This implies that /etc/mdadm.conf is in the initrd.  Peeking inside
> > the default centos one, I can see that this is not true.  Which
> > implies that it doesn't see /etc/mdadm.conf before it assembles the
> > array, as the /etc/mdadm.conf is on the array, and not in the initrd
> > image.
> > 
> > Is there a way to include the /etc/mdadm.conf into the initrd?
> 
> This is almost certainly your problem.  If / is a RAID volume, the 
> initrd itself needs to have an /etc/mdadm.conf that reflects how it is 
> constructed.  This is something that mkinitrd should be doing for you. 
> I'll second the suggestion of running mkinitrd with sh's -x option to 
> see why it's not doing this for you.  Good luck.

Keep in mind that mkinitrd senses what it needs to boot the machine from
the currently running setup.  So, if you run mkinitrd with a root raid
array, it should detect that it needs to start the md raid device, copy
mdadm.static and mdadm.conf to the initrd, and add the appropriate lines
to the initrd script.  If you aren't running the md raid array (maybe
it's a raid1 and you just mounted a bare member long enough to create an
initrd for instance), then it won't.  But, since the system works with a
centos provided kernel, and not with the custom kernel, I'm guessing
that this isn't the issue, that mkinitrd is just fine, and instead the
custom kernel's config is somehow fubar'ed and that's what's causing all
the problems.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

On Mon, Oct 27, 2008 at 12:36 PM, Brendan Conoboy <blc@redhat.com> wrote:
> Joe Landman wrote:
>>
>> Ok.  This is where it gets interesting.  If the /etc/mdadm.conf is on
>> the RAID, you have a bit of a "chicken and egg" problem to deal with
>> here.  How does the system find /etc/mdadm.conf to make the RAID when
>> /etc/mdadm.conf is on the RAID?
>>
>> This implies that /etc/mdadm.conf is in the initrd.  Peeking inside
>> the default centos one, I can see that this is not true.  Which
>> implies that it doesn't see /etc/mdadm.conf before it assembles the
>> array, as the /etc/mdadm.conf is on the array, and not in the initrd
>> image.
>>
>> Is there a way to include the /etc/mdadm.conf into the initrd?
>
> This is almost certainly your problem.  If / is a RAID volume, the initrd
> itself needs to have an /etc/mdadm.conf that reflects how it is constructed.

Hmmm... there was no mdadm.conf even in the centos initrd.  I'll see
if I can force this to be the case.

Since I didn't see the mdadm.conf in the initrd, I assumed it was
autoassembling.  Though it was
my understanding that this is a "bad" thing.

>  This is something that mkinitrd should be doing for you. I'll second the
> suggestion of running mkinitrd with sh's -x option to see why it's not doing
> this for you.  Good luck.

I think I need to dig into the resulting initrd more.   Compare it to
the initrd from the default install.
I did this before, but now I think we need to do this with greater diligence.

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

On Mon, Oct 27, 2008 at 1:56 PM, Doug Ledford <dledford@redhat.com> wrote:
> On Mon, 2008-10-27 at 10:36 -0600, Brendan Conoboy wrote:
>> Joe Landman wrote:

[...]

> Keep in mind that mkinitrd senses what it needs to boot the machine from
> the currently running setup.  So, if you run mkinitrd with a root raid
> array, it should detect that it needs to start the md raid device, copy
> mdadm.static and mdadm.conf to the initrd, and add the appropriate lines
> to the initrd script.  If you aren't running the md raid array (maybe

It did not copy mdadm.static or mdadm.conf.  Interesting.  I added the
"-x" argument, and
instrumented the handleraid function so I could see what it was doing.
 It looks like I
have to dig into this deeper.

> it's a raid1 and you just mounted a bare member long enough to create an
> initrd for instance), then it won't.  But, since the system works with a
> centos provided kernel, and not with the custom kernel, I'm guessing
> that this isn't the issue, that mkinitrd is just fine, and instead the
> custom kernel's config is somehow fubar'ed and that's what's causing all
> the problems.

Hmmm... not likely.  My rationale for that is that the same kernel
(identical .config) runs perfectly
in this mode (software RAID1) on a Ubuntu/Debian machine (the exact
same machine, e.g. same physical hardware, different OS load, as it
turns out).  Moreover, this kernel has been in production for months
at many  sites with nary a peep as to problems (it actually solves
other problems for users with stock kernels).

Will dig into this more, and let people know what I find.  It seems
the consensus is that there should be an mdadm.conf and mdadm.static
in the initrd, and I'll see if the mkinitrd generates this correctly
or not.  Note that I didn't see this in the default initrd, I will
look again.



>
> --
> Doug Ledford <dledford@redhat.com>
>              GPG KeyID: CFBFF194
>              http://people.redhat.com/dledford
>
> Infiniband specific RPMs available at
>              http://people.redhat.com/dledford/Infiniband
>
>

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Luca Berra]

On Mon, Oct 27, 2008 at 08:34:01AM -0400, Joe Landman wrote:
>> unfortunately the kernel panic is a consequence of a previous error,
>> someone willing to help you would need to know everithing that was
>> spitted between the 'starting red-hat nash' message and the kernel
>> panic.
>
>I looked and this was actually the earliest error.  I will try to
>capture the rest of this and post it for download (it is long).
it should at least spit out the modules it is loading and some of the
action it is doing. comparing this with a running one might help in
understanding what the difference is.

>>> Do I need to create a specialized init script?  Is there an
>>
>> usually no, redhat/centos initrd should be able to boot from a raid1
>
>The baseline provided kernel can.  This is a custom kernel, and I am
>attempting to find the right mkinitrd parameters.  One of the nicer
>things in the Ubuntu/Debian process is that they have the ability to
>add most/all modules in to an initrd.  Somehow this is my current
>thought, that we are missing something critical.
is md modular (CONFIG_MD)?

>>
>>> example of this somewhere?  I did look at the Documentation/md.txt and
>>> tried the
>>>
>>>    md=0,sda1,sdb1 md=1,sda2,sdb2  md=2,sda3,sdb3
>>
>> if you are using the centos kernel, raid is modular and this parameter
>> has no effect.
>> mkinitrd should use your /etc/mdadm.conf to discover how the array
>> should be laid out.
>
>Ok.  This is where it gets interesting.  If the /etc/mdadm.conf is on
>the RAID, you have a bit of a "chicken and egg" problem to deal with
>here.  How does the system find /etc/mdadm.conf to make the RAID when
>/etc/mdadm.conf is on the RAID?
>
>This implies that /etc/mdadm.conf is in the initrd.  Peeking inside
>the default centos one, I can see that this is not true.  Which
>implies that it doesn't see /etc/mdadm.conf before it assembles the
>array, as the /etc/mdadm.conf is on the array, and not in the initrd
>image.
>
>Is there a way to include the /etc/mdadm.conf into the initrd?
mkinitrd should include mdadm and /etc/mdadm.conf if it decides your
root is on md

>>
>>>  kernel boot time option, to no avail.  Persistent superblocks are
>>> on, and the devices have the right partition types.
>>
>> partition types do not matter anymore.
>
>Autoassembly is depracated?
yes
>
>>>  Any suggestions?  More things to read?
>>
>> is your fstab correct?
>
>Yes.  Boots fine with older kernel.  fstab doesn't change between the two.

i mean, mkinitrd reads /etc/fstab to decide what your root device is,
sometimes it fails and creates a wrong initrd

>> is your mdadm.conf correct?
>
>
>Yes.  Boots fine with older kernel.  mdadm.conf doesn't change between the two.
>
>> are you loading the correct drivers for your hd controller?
>
>I believe so:  looked at the list from lsmod and replicated this in
>the mkinitrd.  However, I have seen drivers change markedly release to
>release.  Hence my interest in building an initrd with the maximum
>number of modules (specific to SATA/SCSI)
>
>>
>> mkinitrd is a shell script, if all else fails, run it with sh -x and
>> read the output.
>
>Already done.  Even instrumented the code so I could see what the
>handleraid function was in fact doing.  It looks correct, figures out
>the right modules, and makes sure they are in there.

post it and the content of both initrds on pastebin or such,
you seem to have drawn enough interest that someone will look at it.

>So I am wondering if it is a module load order, or something else like
>that., apart from just a missing module.  Which could be a depmod
>issue.
>

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

On Tue, Oct 28, 2008 at 2:43 AM, Luca Berra <bluca@comedia.it> wrote:
> On Mon, Oct 27, 2008 at 08:34:01AM -0400, Joe Landman wrote:
>>>
>>> unfortunately the kernel panic is a consequence of a previous error,
>>> someone willing to help you would need to know everithing that was
>>> spitted between the 'starting red-hat nash' message and the kernel
>>> panic.
>>
>> I looked and this was actually the earliest error.  I will try to
>> capture the rest of this and post it for download (it is long).
>
> it should at least spit out the modules it is loading and some of the
> action it is doing. comparing this with a running one might help in
> understanding what the difference is.

Ok, this makes sense.  Will bring this to a slightly different machine
today and see if we can continue work (put on hold for a few days)

>
>>>> Do I need to create a specialized init script?  Is there an
>>>
>>> usually no, redhat/centos initrd should be able to boot from a raid1
>>
>> The baseline provided kernel can.  This is a custom kernel, and I am
>> attempting to find the right mkinitrd parameters.  One of the nicer
>> things in the Ubuntu/Debian process is that they have the ability to
>> add most/all modules in to an initrd.  Somehow this is my current
>> thought, that we are missing something critical.
>
> is md modular (CONFIG_MD)?

Yes.  Same .config for a different kernel exhibiting the same problem.

grep CONFIG_MD .config
CONFIG_MD=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MD_RAID5_RESHAPE=y
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
# CONFIG_MDIO_BITBANG is not set

[...]

>> This implies that /etc/mdadm.conf is in the initrd.  Peeking inside
>> the default centos one, I can see that this is not true.  Which
>> implies that it doesn't see /etc/mdadm.conf before it assembles the
>> array, as the /etc/mdadm.conf is on the array, and not in the initrd
>> image.
>>
>> Is there a way to include the /etc/mdadm.conf into the initrd?
>
> mkinitrd should include mdadm and /etc/mdadm.conf if it decides your
> root is on md

Ok.  This is what I think (e.g. am guessing) is the likely culprit at
the moment.

[...]

>>>>  Any suggestions?  More things to read?
>>>
>>> is your fstab correct?
>>
>> Yes.  Boots fine with older kernel.  fstab doesn't change between the two.
>
> i mean, mkinitrd reads /etc/fstab to decide what your root device is,
> sometimes it fails and creates a wrong initrd

Ok.  I was unaware that it would build an incorrect initrd, I thought
we could just miss the modules if they weren't explicitly included
using --with=  or similar.  I have seen that with a number of RAID
cards, that mkinitrd will not generate a workable initrd (no software
RAID) as it leaves out the right drivers.  I have been (defensively)
including RAID card drivers in our initrd builds.

>
>>> is your mdadm.conf correct?
>>
>>
>> Yes.  Boots fine with older kernel.  mdadm.conf doesn't change between the
>> two.
>>
>>> are you loading the correct drivers for your hd controller?
>>
>> I believe so:  looked at the list from lsmod and replicated this in
>> the mkinitrd.  However, I have seen drivers change markedly release to
>> release.  Hence my interest in building an initrd with the maximum
>> number of modules (specific to SATA/SCSI)
>>
>>>
>>> mkinitrd is a shell script, if all else fails, run it with sh -x and
>>> read the output.
>>
>> Already done.  Even instrumented the code so I could see what the
>> handleraid function was in fact doing.  It looks correct, figures out
>> the right modules, and makes sure they are in there.
>
> post it and the content of both initrds on pastebin or such,
> you seem to have drawn enough interest that someone will look at it.

Ok, will bring the disks to a slightly different machine so I can do
the serial port captures.  Will post when I can (later
today/tomorrow).

Thanks for all the suggestions and guidance!

Joe

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

Update:

On Tue, Oct 28, 2008 at 8:20 AM, Joe Landman <joe.landman@gmail.com> wrote:
> On Tue, Oct 28, 2008 at 2:43 AM, Luca Berra <bluca@comedia.it> wrote:
>> On Mon, Oct 27, 2008 at 08:34:01AM -0400, Joe Landman wrote:

[..]

>>> This implies that /etc/mdadm.conf is in the initrd.  Peeking inside
>>> the default centos one, I can see that this is not true.  Which
>>> implies that it doesn't see /etc/mdadm.conf before it assembles the
>>> array, as the /etc/mdadm.conf is on the array, and not in the initrd
>>> image.
>>>
>>> Is there a way to include the /etc/mdadm.conf into the initrd?
>>
>> mkinitrd should include mdadm and /etc/mdadm.conf if it decides your
>> root is on md
>
> Ok.  This is what I think (e.g. am guessing) is the likely culprit at
> the moment.

In fact, now looking at the generated initrd and the base distro
initrd, neither one has mdadm.conf nor mdadm.

>
> [...]
>
>>>>>  Any suggestions?  More things to read?
>>>>
>>>> is your fstab correct?
>>>
>>> Yes.  Boots fine with older kernel.  fstab doesn't change between the two.
>>
>> i mean, mkinitrd reads /etc/fstab to decide what your root device is,
>> sometimes it fails and creates a wrong initrd
>
> Ok.  I was unaware that it would build an incorrect initrd, I thought
> we could just miss the modules if they weren't explicitly included
> using --with=  or similar.  I have seen that with a number of RAID
> cards, that mkinitrd will not generate a workable initrd (no software
> RAID) as it leaves out the right drivers.  I have been (defensively)
> including RAID card drivers in our initrd builds.

It looks like nash is the thing responsible for raidautorun, and it
doesn't look like this is doing the right thing in the generated
initrd.  Will post the captured trace from nash on downwards soon.

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

On Tue, Oct 28, 2008 at 9:42 PM, Joe Landman <joe.landman@gmail.com> wrote:
> Update:

capture is at http://pastebin.com/f57b6ba19  .  I think I see part of
the problem

From the trace:

Scanning and configuring dmraid supported devices
raidautorun: RAID_AUTORUN failed: No such device
raidautorun: RAID_AUTORUN failed: No such device

Seeing some of the other notes from here
[http://www.centos.org/modules/newbb/viewtopic.php?topic_id=15305]
someone noted that to make this work correctly, they had to rebuild
with the raid drivers built in.  I am ok with this if we have to do
it, but it seems strange ...  is RAID built as a module or as a built
in for the default distro kernel?

It is also interesting to note that in Fedora, the mkinitrd was
patched to use mdadm
(http://cvs.fedoraproject.org/viewvc/rpms/mkinitrd/F-7/mkinitrd.spec}.
 This is for mkinitrd-6.x.  This machine (Centos 5.2) uses
mkinitrd-5.1.19.6-28

[root@jackrabbitm ~]# rpm -qf /sbin/mkinitrd
mkinitrd-5.1.19.6-28
mkinitrd-5.1.19.6-28

It is starting to look like we need to switch to a later model
mkinitrd.  I'll try this, but are there any patches running around for
this for this mkinitrd?  Conversely, does anyone else with MD raid on
their root/boot drive see mdadm or mdadm.conf in their initrd with a
similar mkinitrd on their system?

Thanks!

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Joe Landman]

On Tue, Oct 28, 2008 at 10:03 PM, Joe Landman <joe.landman@gmail.com> wrote:
> On Tue, Oct 28, 2008 at 9:42 PM, Joe Landman <joe.landman@gmail.com> wrote:
>> Update:
>
> capture is at http://pastebin.com/f57b6ba19  .  I think I see part of
> the problem

Update 2: It appears that the issue is a buggy mkinitrd.  v5.1.19.6-28
does not put the mdadm.conf into the initrd, and may rely upon
something being built into the kernel that our kernel did not have
built in.

Replacing this mkinitrd with 6.0.19-4 (by doing lots of rpmbuild 's
against FC8/9 packages and installing), we did in fact see the
mdadm.conf get explicitly added in during the mkinitrd.

/sbin/mdadm -> /tmp/initrd.Jz3686/sbin/mdadm
/etc/mdadm.conf -> /tmp/initrd.Jz3686/etc/mdadm.conf

Upon reboot, it works.

[root@jackrabbitm ~]# uname -r
2.6.23.14
[root@jackrabbitm ~]# mount
/dev/md0 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

[root@jackrabbitm ~]# rpm -qa | grep -i mkinitrd
mkinitrd-devel-6.0.19-4
mkinitrd-6.0.19-4

While this works, this is not a solution I want to hand to our
users/customers.  Could someone toss me a pointer as to who owns
mkinitrd at RHEL/Centos so I can report this?  Basically this is not a
linux-raid bug, but a bug in the mkinitrd assumptions (or possibly
something that needs to be built into the kernel which is not
explicitly indicated in mkinitrd).

Thanks for all the pointers/suggestions.

Re: question on how to (correctly) build an initrd for a root disk on a RAID1

[Tru Huynh]

On Tue, Oct 28, 2008 at 10:49:27PM -0400, Joe Landman wrote:
...
> Replacing this mkinitrd with 6.0.19-4 (by doing lots of rpmbuild 's
> against FC8/9 packages and installing), we did in fact see the
> mdadm.conf get explicitly added in during the mkinitrd.
> 
> /sbin/mdadm -> /tmp/initrd.Jz3686/sbin/mdadm
> /etc/mdadm.conf -> /tmp/initrd.Jz3686/etc/mdadm.conf
> 
> Upon reboot, it works.
> 
...
> 
> While this works, this is not a solution I want to hand to our
> users/customers.  Could someone toss me a pointer as to who owns
> mkinitrd at RHEL/Centos so I can report this?  Basically this is not a
> linux-raid bug, but a bug in the mkinitrd assumptions (or possibly
> something that needs to be built into the kernel which is not
> explicitly indicated in mkinitrd).
http://bugzilla.redhat.com would be the first step since CentOS
is rebuilding from upstream (bugs included), although 
it could provided it's own features in the extras
repository.
http://bugs.centos.org could also be used for RFE in CentOS extras
http://wiki.centos.org/AdditionalResources/Repositories has a list
of "recommended" repositories.
One caveat, I am not sure that this bug will ever be fixed
since rebuilding a custom kernel is not really supported 
on both distributions.

Cheers,

Tru
-- 
Dr Tru Huynh          | http://www.pasteur.fr/recherche/unites/Binfs/
mailto:tru@pasteur.fr | tel/fax +33 1 45 68 87 37/19
Institut Pasteur, 25-28 rue du Docteur Roux, 75724 Paris CEDEX 15 France