2.6.15: mdrun, udev -- who creates nodes?

2.6.15: mdrun, udev -- who creates nodes?

[Moshe Yudkowsky]

Here's a snippet from /sbin/mdrun:

 >       mdadm -A -a $NUMBER -f `arr MDS $MD` && setarr MDS $MD "started"
 >       # just to be sure
 >       ln /dev/md/$NUMBER /dev/md$NUMBER 2>/dev/null


The first line is the problem I'm having, namely, that the arrays are 
being assembled with the wrong numbers. I'm about to try a replacement 
strategy that ignores /sbin/mdrun entirely.

The last line of the snippet shows /sbin/mdrun attempting to create a 
device that udev creates.

This won't crash anything, to the best of my knowledge; but which 
package is supposed to have "control" of creating /dev/md[0-9]*? udev 
has a rule (under compat-full) that creates this same symlink.

-- 
  Moshe Yudkowsky
  work: http://www.Disaggregate.com
  book: http://www.PebbleAndAvalanche.com


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Marco d'Itri]

On Jan 29, Moshe Yudkowsky <msha5_17@bl.com> wrote:

> This won't crash anything, to the best of my knowledge; but which 
> package is supposed to have "control" of creating /dev/md[0-9]*? udev 
> has a rule (under compat-full) that creates this same symlink.
Indeed I do not understand why mdrun does this.

Anyway, the problem is that while udev is supposed to create the devices
it cannot do it until the RAID array has been assembled, and the array
cannot be assembled unless the device node is available.
This needs to be fixed in the kernel, but apparently nobody cares enough.

-- 
ciao,
Marco


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Moshe Yudkowsky]

Marco d'Itri wrote:
> On Jan 29, Moshe Yudkowsky <msha5_17@bl.com> wrote:
> 
> 
>>This won't crash anything, to the best of my knowledge; but which 
>>package is supposed to have "control" of creating /dev/md[0-9]*? udev 
>>has a rule (under compat-full) that creates this same symlink.
> 
> Indeed I do not understand why mdrun does this.
> 
> Anyway, the problem is that while udev is supposed to create the devices
> it cannot do it until the RAID array has been assembled, and the array
> cannot be assembled unless the device node is available.
> This needs to be fixed in the kernel, but apparently nobody cares enough.

There's an option on mdadm, --auto, that creates the device nodes if 
needed.

That's in theory. In practice, I find that "mdadm --assemble --scan 
--config=/etc/mdadm/mdadm.conf --auto" doesn't work. All three drives 
listed in the config file fail to start. I'm wondering just what the 
problem might be (config file requires UUID, not just ARRAY?); but it's 
clearly an issue for mdadm. And the inability to startup 2.6.15 using 
RAID drives that don't start at /dev/md/0 is quite disquieting.

-- 
Moshe Yudkowsky * moshe@pobox.com * www.pobox.com/~moshe
"I leave my soul to G-d, my service to my Prince, my good wishes to my 
friends,
and my love and charity to you all."  - Montrose, on the gallows, 21 May 
1650


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[linas]

On Sun, Jan 29, 2006 at 08:49:56AM -0600, Moshe Yudkowsky was heard to remark:
> That's in theory. In practice, I find that "mdadm --assemble --scan 
> --config=/etc/mdadm/mdadm.conf --auto" doesn't work. All three drives 

I'm surprised by this. Disk drives which have partitiions marked 
as being of type "Linux md" will have an md superblock on them.
This superblock identifies what md device the disk partition belongs to.
Manually specifying /etc/mdadm/mdadm.conf is dangerous, because it is
likely to introduce erroneous information about the actual structure 
of the system. 

The correct mode of operation should be this:
-- udev creates /dev/hda1, hda2, etc. 
-- kernel scans /dev/hda1, looking for md superblock
-- kernel assembles devices according to info found in the superblocks
-- udev creates /dev/md0, etc. 

Disclaimer: this is a knee-jerk email; I have not actually
thought about the issues.

--linas



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Marco d'Itri]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]

On Jan 30, linas <linas@austin.ibm.com> wrote:

> -- kernel scans /dev/hda1, looking for md superblock
> -- kernel assembles devices according to info found in the superblocks
> -- udev creates /dev/md0, etc. 
The problem is that some users and distributions build the drivers as
modules and/or disable in-kernel auto-assembly.

-- 
ciao,
Marco

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Jason Lunz]

md@Linux.IT said:
>> -- kernel scans /dev/hda1, looking for md superblock
>> -- kernel assembles devices according to info found in the superblocks
>> -- udev creates /dev/md0, etc. 
> The problem is that some users and distributions build the drivers as
> modules and/or disable in-kernel auto-assembly.

Not only that, the raid developers themselves consider autoassembly
deprecated.

http://article.gmane.org/gmane.linux.kernel/373620

With some platforms (partition types, really) it's never been supported
in the first place.

Jason



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[linas]

On Tue, Jan 31, 2006 at 04:40:46PM +0000, Jason Lunz was heard to remark:
> md@Linux.IT said:
> >> -- kernel scans /dev/hda1, looking for md superblock
> >> -- kernel assembles devices according to info found in the superblocks
> >> -- udev creates /dev/md0, etc. 
> > The problem is that some users and distributions build the drivers as
> > modules and/or disable in-kernel auto-assembly.
> 
> Not only that, the raid developers themselves consider autoassembly
> deprecated.
> 
> http://article.gmane.org/gmane.linux.kernel/373620

Hmm. My knee-jerk, didn't-stop-to-think-about-it reaction is that 
this is one of the finest features of linux raid, so why remove it?

Speaking as a real-life sysadmin, with actual servers and actual failed
disks, disk cables and disk controllers, this is a life-saving feature. 
Persistant naming of devices in Linux has long been a problem, and in
this case, it seemed to work.

<story>
I once had an ide controller fail on an x86 board. I bought a new 
controller at the local store, recabled the disks, and booted. 
I was alarmed to find that the system was trying to mount /home 
as /usr, and /usr as /lib, etc. Turned out that /dev/hdc had  
gotten renamed as /dev/hde, etc. and  had to go through a long,
painful, rocket-science (yes, I *do* have a PhD) boot-floppy rescue
to restore the system to working order.
 
I shudder to think what would have happened if RAID reconstruction 
had started based on faulty device names. Worse, as part of my rescue
ops, I had to make multle copies of /etc/fstab, which resided on
different disks (my root volume was raided), as well as the boot 
floppy, and each contained inconsistent info (needed to bootstrap 
my way back). Along the way, I made multiple errors in editing 
the /etc/fstab since I could not keep them straight; twiddling 
BIOS settings added to the confusion.  If this had been /etc/raid.conf 
instead, with reconstruction triggered off of it, this could have 
been an absolute disaster.
</story>

Based on the above, real-life experience, my gut reaction is 
raid assembly based on config files is a bad idea. I don't 
understand how innocent, "minor" errors made by the sysadmin 
won't result in catastrophic data loss.

--linas


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Molle Bestefich]

linas@austin.ibm.com wrote:
> > Not only that, the raid developers themselves
> > consider autoassembly deprecated.
> >
> > http://article.gmane.org/gmane.linux.kernel/373620
>
> Hmm. My knee-jerk, didn't-stop-to-think-about-it reaction is that
> this is one of the finest features of linux raid, so why remove it?

I *think* that "the raid developers" may be, for once, choosing words
not-so-wisely when talking about "deprecating autoassembly".

Last time I heard that I choked as well, only to find out later that
Neil's notion of what auto-assembly is differed substantially from my
own.

Isn't there a faq/wiki somewhere where the official opinion on
"autoassembly deprecation" and exactly what that means can go?


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Jason Lunz]

On Tue, Jan 31, 2006 at 09:19:16PM +0100, Molle Bestefich wrote:
> I *think* that "the raid developers" may be, for once, choosing words
> not-so-wisely when talking about "deprecating autoassembly".

You're right, I should be careful not to imply anything Neil didn't
actually say. What he said was:

> From: Neil Brown <neilb <at> suse.de>
...
> And if other partition styles wanted to add support for raid auto
> detect, tell them "no". It is perfectly possible and even preferable
> to live without autodetect.   We should support legacy usage (those
> above) but should discourage any new usage.

which is a far cry from officially deprecating anything. My point was
that the outlined solution for having udev create /dev/md* nodes could
not work on all platforms because it required raid autodetect, and it
looks unlikely that raid autodetect will be universally supported.

Jason


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[linas]

I was reacting to the following:

> > From: Neil Brown <neilb <at> suse.de>
> > It is perfectly possible and even preferable
> > to live without autodetect.   

I didn't see how it could be "preferable", and illustrated this
with the sysadmin story.

--linas


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Molle Bestefich]

linas wrote:
> I was reacting to the following:
> > > From: Neil Brown <neilb <at> suse.de>
> > > It is perfectly possible and even preferable
> > > to live without autodetect.
>
> I didn't see how it could be "preferable", and illustrated this
> with the sysadmin story.

Certainly it isn't, and now I remember the point.

Quoting Luca Berra:
> the in kernel auto assembly should be removed for good
> it should be replaced by auto assembly in user space (mdadm),
> which does not suffer from the problems that in-kernel has.

If people could start saying
 "we need to get rid of in-kernel autodetection"
instead of
 "we need to get rid of autodetection",
I think there would be much less confusion.


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Luca Berra]

On Tue, Jan 31, 2006 at 01:26:43PM -0600, linas wrote:
>On Tue, Jan 31, 2006 at 04:40:46PM +0000, Jason Lunz was heard to remark:
>> md@Linux.IT said:
>> >> -- kernel scans /dev/hda1, looking for md superblock
>> >> -- kernel assembles devices according to info found in the superblocks
>> >> -- udev creates /dev/md0, etc. 
>> > The problem is that some users and distributions build the drivers as
>> > modules and/or disable in-kernel auto-assembly.
>> 
>> Not only that, the raid developers themselves consider autoassembly
>> deprecated.
>> 
>> http://article.gmane.org/gmane.linux.kernel/373620
>
>Hmm. My knee-jerk, didn't-stop-to-think-about-it reaction is that 
>this is one of the finest features of linux raid, so why remove it?

please, no, not again:
this particular horse has been beaten to death many times before.

the in kernel auto assembly should be removed for good (read the
linux-raid list archive to understand why)

it should be replaced by auto assembly in user space (mdadm), which does
not suffer from the problems that in-kernel has. neither it does suffer
from a poor configuration file like the (unmaintained) raidtools had.
and is much more cleaner and maintainable.

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: 2.6.15: mdrun, udev -- who creates nodes?

[Bill Davidsen]

linas wrote:

>On Tue, Jan 31, 2006 at 04:40:46PM +0000, Jason Lunz was heard to remark:
>  
>
>>md@Linux.IT said:
>>    
>>
>>>>-- kernel scans /dev/hda1, looking for md superblock
>>>>-- kernel assembles devices according to info found in the superblocks
>>>>-- udev creates /dev/md0, etc. 
>>>>        
>>>>
>>>The problem is that some users and distributions build the drivers as
>>>modules and/or disable in-kernel auto-assembly.
>>>      
>>>
>>Not only that, the raid developers themselves consider autoassembly
>>deprecated.
>>
>>http://article.gmane.org/gmane.linux.kernel/373620
>>    
>>
>
>Hmm. My knee-jerk, didn't-stop-to-think-about-it reaction is that 
>this is one of the finest features of linux raid, so why remove it?
>
>Speaking as a real-life sysadmin, with actual servers and actual failed
>disks, disk cables and disk controllers, this is a life-saving feature. 
>Persistant naming of devices in Linux has long been a problem, and in
>this case, it seemed to work.
>
><story>
>I once had an ide controller fail on an x86 board. I bought a new 
>controller at the local store, recabled the disks, and booted. 
>I was alarmed to find that the system was trying to mount /home 
>as /usr, and /usr as /lib, etc. Turned out that /dev/hdc had  
>gotten renamed as /dev/hde, etc. and  had to go through a long,
>painful, rocket-science (yes, I *do* have a PhD) boot-floppy rescue
>to restore the system to working order.
> 
>I shudder to think what would have happened if RAID reconstruction 
>had started based on faulty device names. Worse, as part of my rescue
>ops, I had to make multle copies of /etc/fstab, which resided on
>different disks (my root volume was raided), as well as the boot 
>floppy, and each contained inconsistent info (needed to bootstrap 
>my way back). Along the way, I made multiple errors in editing 
>the /etc/fstab since I could not keep them straight; twiddling 
>BIOS settings added to the confusion.  If this had been /etc/raid.conf 
>instead, with reconstruction triggered off of it, this could have 
>been an absolute disaster.
></story>
>
>Based on the above, real-life experience, my gut reaction is 
>raid assembly based on config files is a bad idea. I don't 
>understand how innocent, "minor" errors made by the sysadmin 
>won't result in catastrophic data loss.
>

I fear you don't understand how the auto detect and assemble works. Or 
more to the point what it does, since how it works is somewhat more 
complex. If you use partitions and UUID, you can just plug in the drives 
any old place and they will be found and recognised in spite of that. As 
long as you have a boot drive where the BIOS will use it, mdadm with 
find your stuff and put it together correctly. Neil does more magic than 
harry Potter!

I know someone who gave this a real life test, although I'd not say who.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x103432&bid#0486&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel