(unknown)

(unknown)

[bhess]

linux-raid@vger.kernel.org



I originally sent this to Neil Brown who suggested I sent it to you.







Any help would be appreciated.







Has anyone put an effort into building a raid 1 based on USB connected

drives under Redhat 3/4 not as the root/boot

drive. A year ago I don't think this made any sense but with the

price of drives being far less that then the equivalent tape media

and the simple USB to IDE smart cable I am evaluating an expandable

USB disk farm for two uses. A reasonable robust place to store

data until I can select what I want to put on tape. The second

is secondary storage for all of the family video tapes that I am

capturing in preparation for editing to DVD. The system does not

have to be fast just large, robust, expandable and cheep.



I currently run a Redhat sandbox with a hardware raid 5 and 4 120G

SATA drives. I have added USB drives and have them mount with

the LABEL=/PNAME option in fstab. In this manner they end up

in the right place after reboot. I do not know enough about

the Linux drive interface to know if USB attached devices will

get properly mounted into the raid at reboot and after changes

or additions of drives to the USB.



I am a retired Bell Labs Research supervisor. I was in Murray Hill

when UNIX was born and still use Intel based UNIX in the current

form of SCO Unixware both professionally and personally. Unixware

is no longer a viable product since I see no future in it and

Oracle is not supported. I know way to much about how the guts

of Unixware works thanks to a friend who was one of SCO's kernel

and storage designers. I know way to little how Linux works to

get a USB based raid up without a lot of research and tinkering.

I don't mind research and tinkering but I don't like reinventing

the wheel.



I have read The Software-RAID HOWTO by Jakob 0stergaard and

Emilio Bueso and downloaded mdadm. I have not tried it yet.





The system I have in mind uses a Intel server motherboard,

hardware raid 1 SATA root/boot/swap drive, SCSI tape drive

and a 4 port USB card. In a 2U chasses. A second 2U chassis

will contain a supply, up to 14 drives and lots of fans.

I have everything except the drives. The sole use of this system

will be a disk farm with a NFS and Samba server. It will run

under Redhat 3 or 4. I am leaning toward Redhat 4 since I

understand SCSI tape support is more stable under 4. Any

comment in this area would also be appreciated.



Can you point me in the direction of newer articles that cover

Linux raid using USB connected drives or do you have any

suggestions on the configuration of a system. My main concern

is how to get USB drives correctly put back in the raid after

boot and/or a USB change since I do not know how they are assigned

to /dev/sdxy in the first place and how USB hubs interact with

the assignments. I realize I should have other concerns and

just don't know enough. Ignorance is bliss, up to an init 6.



Thank You for your time.



Bill Hess



bhess@patmedia.net 

Re: your mail

[Ross Vandegrift]

On Wed, Jan 11, 2006 at 09:47:03AM -0500, bhess@patmedia.net wrote:
> Has anyone put an effort into building a raid 1 based on USB connected
> drives under Redhat 3/4 not as the root/boot
> drive.

No, I haven't actually tried this, but I do know that it'll work
without any problem.

The main issue that you'll confront - USB devices do not have fixed
device numbers.  They can change every time they are plugged/unplugged.
As a result, you may not be able to depend on your disks to always be
(for example) /dev/sde and /dev/sdf.

There's a user-space daemon called hotplug that manages this stuff.
It load appropriate drivers for the devices it sees plugged into the
USB bus.  I don't know enough about to tell you specifically how it
will handle your setup.

My initial suggestion:
	1) Start with no driver plugged
	2) cat /proc/partitions > ~/nousb
	3) Plug one drive in
	4) cat /proc/partitions > ~/oneusb
	5) Plug second drive in
	6) cat /proc/parititons > ~/twousb
	7) diff -u ~/nousb ~/oneusb
	8) diff -u ~/oneusb ~/twousb

This should give you a good idea of how the kernel treats the
situation from the device perspective.

As far as the RAID part - that's easy.  Once you know what the block
devices look like, make your RAID.  I promise the md driver will work.
I've run md across all sorts of weird device setups and so long as
it's a working block device, md is happy!

Finally, if you use mdadm's --uuid option to assemble your array after
creating it, it should just find the USB disks, regardless of where
they came up.  That way, you only need to worry about what device node
they are using when you first create the array.

Good luck! 


-- 
Ross Vandegrift
ross@lug.udel.edu

"The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell."
	--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37

Re:

[David Greaves]

bhess@patmedia.net wrote:

><snip - a lot!!>
>  
>
can I summarise (!) as:
I want to create a non-system data-storage raid array (ie can be
initialised well after boot)
I want to use a mix of SCSI, sata + USB devices
Will this work if the USB devices change their 'order'

Short answer: yes, with the right kernel/mdadm versions

Longer:

I use debian so don't know what version of the kernel you use? also mdadm?
You need to look at UUIDs
Use the --assemble, --uid and --scan options *after* you know usb
devices are online (and make sure they're listed in the conf (or use
--config=partitions)). It's safe, mdadm won't assemble an array that's
not complete.

Maybe you'd like to script a test for the usb devices and only attempt
an assemble when all devices are available.

AFAIK there are no issues with the hardware/bus type. A block device is
a block device is a block device is a.... :)

Now, advice...

ok, first off: a 14 device raid1 is 14 times more likely to lose *all*
your data than a single device. I have a scsi device running in one of
my machines since 1995. I am about to RMA a 1yr old maxtor sata drive
(I've RMA'd about 30% of my consumer grade ides/sata drives over the
last few years - ie failed *in* warranty). You appear to be jumping into
consumer grade (usb) devices and it may bite you!

Why not just have a lot of individual filesystems and manage the data by
hand?

Also, if you go the one-device way, why not consider lvm2 instead?
The reason is that it seems *very* likely that you'll be looking to swap
devices in and maybe out (when smart tells you a drive is about to fail)
of this ad-hoc storage.
LVM allows you to see each device as a large number of 'chunks'. You
then gather all those chunks from many devices into a 'pool'. You can
then allocate chunks from the pool to create virtual devices and then
make filesystems on them.

This is good because you can then add another device to the 'pool' and
use those chunks to either:
* swap out a failing (SMART) drive if you're lucky
* grow the virtual drive
* take 'snapshots' (OK you don't need this but it's cool!)

finally, watch the filesystem - eg xfs is excellent for big files but
can't shrink

HTH

David

-- 

Re:

[Ross Vandegrift]

On Thu, Jan 12, 2006 at 11:16:36AM +0000, David Greaves wrote:
> ok, first off: a 14 device raid1 is 14 times more likely to lose *all*
> your data than a single device.

No, this is completely incorrect.  Let A denote the event that a single
disk has failed, A_i denote the event that i disks have failed.
Suppose P(A) = x.  Then by Bayes's Law the probability that an n disk RAID
will lose all of your data is:

n_1 = P(A) = x
n_2 = P(A_2) = P(A) * P(A_1 | A) = x^2
n_3 = P(A_3) = P(A) * P(A_2 | A) = x^3
...
n_i = P(A_i) = P(A) * P(A_{i-1} | A) = x^i

ie, RAID1 is expoentially more reliable as you add extra disks!

This assumes that disk failures are independant - ie, that you
correctly configure disks (don't use master and slave on an IDE
channel!), and replace failed disks as soon as they fail.

This is why adding more disks to a RAID1 is rare - x^2 is going to be
a really low probability!  It will be far, far more common for
operator error to break a RAID than for both devices to honestly fail.

-- 
Ross Vandegrift
ross@lug.udel.edu

"The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell."
	--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37

Re:

[David Greaves]

Ross Vandegrift wrote:

>On Thu, Jan 12, 2006 at 11:16:36AM +0000, David Greaves wrote:
>  
>
>>ok, first off: a 14 device raid1 is 14 times more likely to lose *all*
>>your data than a single device.
>>    
>>
>
>No, this is completely incorrect.  Let A denote the event that a single
>disk has failed, A_i denote the event that i disks have failed.
>Suppose P(A) = x.  Then by Bayes's Law the probability that an n disk RAID
>will lose all of your data is:
>
>n_1 = P(A) = x
>n_2 = P(A_2) = P(A) * P(A_1 | A) = x^2
>n_3 = P(A_3) = P(A) * P(A_2 | A) = x^3
>...
>n_i = P(A_i) = P(A) * P(A_{i-1} | A) = x^i
>
>ie, RAID1 is expoentially more reliable as you add extra disks!
>
>This assumes that disk failures are independant - ie, that you
>correctly configure disks (don't use master and slave on an IDE
>channel!), and replace failed disks as soon as they fail.
>
>This is why adding more disks to a RAID1 is rare - x^2 is going to be
>a really low probability!  It will be far, far more common for
>operator error to break a RAID than for both devices to honestly fail.
>
>  
>
sorry, read it all as 'linear', not mirrored which is why I was writing
drivel ;)

David


-- 

(unknown),

[Lelsie Rhorer]

I'm having a severe problem whose root cause I cannot determine.  I have a
RAID 6 array managed by mdadm running on Debian "Lenny" with a 3.2GHz AMD
Athlon 64 x 2 processor and 8G of RAM.  There are ten 1 Terabyte SATA
drives, unpartitioned, fully allocated to the /dev/md0 device. The drive
are served by 3 Silicon Image SATA port multipliers and a Silicon Image 4
port eSATA controller.  The /dev/md0 device is also unpartitioned, and all
8T of active space is formatted as a single Reiserfs file system.  The
entire volume is mounted to /RAID.  Various directories on the volume are
shared using both NFS and SAMBA.

Performance of the RAID system is very good.  The array can read and write
at over 450 Mbps, and I don't know if the limit is the array itself or the
network, but since the performance is more than adequate I really am not
concerned which is the case.

The issue is the entire array will occasionally pause completely for about
40 seconds when a file is created.  This does not always happen, but the
situation is easily reproducible.  The frequency at which the symptom
occurs seems to be related to the transfer load on the array.  If no other
transfers are in process, then the failure seems somewhat more rare,
perhaps accompanying less than 1 file creation in 10..  During heavy file
transfer activity, sometimes the system halts with every other file
creation.  Although I have observed many dozens of these events, I have
never once observed it to happen except when a file creation occurs. 
Reading and writing existing files never triggers the event, although any
read or write occurring during the event is halted for the duration. 
(There is one cron jog which runs every half-hour that creates a tiny file;
this is the most common failure vector.)  There are other drives formatted
with other file systems on the machine, but the issue has never been seen
on any of the other drives.  When the array runs its regularly scheduled
health check, the problem is much worse.  Not only does it lock up with
almost every single file creation, but the lock-up time is much longer -
sometimes in excess of 2 minutes.

Transfers via Linux based utilities (ftp, NFS, cp, mv, rsync, etc) all
recover after the event, but SAMBA based transfers frequently fail, both
reads and writes.

How can I troubleshoot and more importantly resolve this issue?

Re: your mail

[Luca Berra]

On Wed, Apr 01, 2009 at 11:16:06PM -0500, Lelsie Rhorer wrote:
>8T of active space is formatted as a single Reiserfs file system.  The
....
>The issue is the entire array will occasionally pause completely for about
>40 seconds when a file is created.  This does not always happen, but the
>situation is easily reproducible.  The frequency at which the symptom
i wonder how costly b-tree operaton are for a 8Tb filesystem...

L.

-- 
Luca Berra -- bluca@comedia.it
         Communication Media & Services S.r.l.
  /"\
  \ /     ASCII RIBBON CAMPAIGN
   X        AGAINST HTML MAIL
  / \

(unknown),

[Frédéric Mantegazza]

subscribe linux-raid
-- 
   Frédéric

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: your mail

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 352 bytes --]

You need to e-mail majordomo@vger.kernel.org.

On Tue, 9 Oct 2007, Frédéric Mantegazza wrote:

> subscribe linux-raid
> --
>   Frédéric
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

(unknown)

[Michael Robinton]

>I've taken a look at the ML archives, and found an old thread (06/2002)
>on this subject, but found no solution.
>
>I've a working setup with a two disks RAID1 root, which boots
>flawlessly. Troubles arise when simulating hw failure. RAID setup is as
>follows:
>
>raiddev                 /dev/md0
>raid-level              1
>nr-raid-disks           2
>nr-spare-disks          0
>chunk-size              4
>
>device                  /dev/hda1
>raid-disk               0
>
>device                  /dev/hdc1
>raid-disk               1

>If I disconnect /dev/hda before booting, the kernel tries to initialize
>the array, can't access /dev/hda1 (no wonder), marks it as faulty, then
>refuses to initialize the array, dieing with a kernel panic, unable to
>mount root.
>
>If I disconnect /dev/hdc before booting, the array gets started in
>degraded mode, and the startup goes on without a glitch.
>
>If I disconnect /dev/hda and move /dev/hdc to its place (so it's now
>/dev/hda), the array gets started in degraded mode and the startup goes
>on.
>
>Actually, this is already a workable solution (if the first disk dies, I
>just "promote" the second to hda and go looking for a replacement of the
>broken disk), but I think this is not _elegant_. 8)
>
>Could anyone help me shedding some light on the subject?
>
>Tnx in advance.
>--
>Massimiliano Masserelli

There is no standard for the behavior of the motherboard bios when the
first device 0x80 is not available at boot time. Some motherboards will
automove 0x81 -> 0x80, some can do it as a bios change, some you're stuck.

Most scsii controllers will do this and a few IDE controllers will as
well.

Generally for best flexibility, use an independent lilo file for each hard
disk and set the boot disk pointer individually for each drive to 0x80 or
0x81 as needed for your environment rather than using the "raid" feature
of lilo.

See the Boot-Root-Raid-LILO howto for examples. This doc is a bit out of
date, but the examples and setups are all applicable for the 2.4 series
kernels.

Michael

Re: your mail

[Massimiliano Masserelli]

On Tue, Oct 29, 2002 at 05:26:23PM -0800, Michael Robinton wrote:

> There is no standard for the behavior of the motherboard bios when the
> first device 0x80 is not available at boot time. Some motherboards
> will automove 0x81 -> 0x80, some can do it as a bios change, some
> you're stuck.

Yes, I know. Anyway the problem doesn't seem connected with LILO, since
the kernel uncompresses smoothly, loads up the initrd image and THEN
hangs. The point is that the raidstart seems unable to initialize the
array if the /dev/hda disk is not present, and I can't understand why.
Is there some sort of priority between disks in a RAID-1 array?

Bye.
-- 
Massimiliano Masserelli
-------------------------------------------------------------------------------
He who hesitates is sometimes saved.