Re:

Re:

[Jure Pecar]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

On Tue,  4 Jun 2002 08:47:12 -0700 (PDT)
klink@clouddancer.com (Colonel) wrote:

> 
> True, I think that the point is that of the 5 possible 2 disk
> failures, 2 of them (in striped mirrors, not mirrored stripes) kill
> the array.  For RAID5, all of them kill the array.  But the fancy RAID
> setups are for _large_ arrays, not 4 disks, unless you are after the
> small write speed improvement (as I am).

going offtopic here ...
what kind of raid setup is the best for write intensive load like mail
queues & co?
 
> Plus any raid metadevice made of metadevices cannot autostart, which
> means tinkering during startup, which is only worth it for those large
> drive arrays.

hm? it does for me. probalby the redhat's rc.sysinit does the right
thing ... 

-- 


Jure Pecar

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re:

[bouche]

Hey man, here's that site I was telling you about. They are offering huge discounts now on Penis Enhancement Patches

http://www.poqz.com/md/

A top team of British scientists and medical doctors have worked to develop the state-of-the-art Penis Enlargement Patch delivery system which automatically increases penis size up to 3-4 full inches. The patches are the easiest and most effective way to increase your penis size. You won't have to take pills, get under the knife to perform expensive and very painful surgery, use any pumps or other devices. No one will ever find out that you are using our product. Just apply one patch on your body and wear it for 3 days and you will start noticing dramatic results.

Millions of men are taking advantage of this revolutionary new product - Don't be left behind!

As an added incentive, they are offering huge discount specials right now, check out the site to see for yourself !

http://www.poqz.com/md/









u n s u b s c r i b e  
http://www.yzewa.com/un.php 
 

Re:

[jfire]

Meet the newest and most aggressive addition to Internet commerce - http://5139.2005clearance.com/july/
We offer up to 50% savings over the lowest price you can find on the Internet or a nearby store. 
Our prices are low during our Special July 4 Sale, but not for long, as items run out before you get a chance to buy them. 
We offer quick order processing, online account reports, and live customer support for existing customers. 
What are you waiting for??? Click and shop!!!

http://3470.2005clearance.com/july/

Re:

[David Greaves]

bhess@patmedia.net wrote:

><snip - a lot!!>
>  
>
can I summarise (!) as:
I want to create a non-system data-storage raid array (ie can be
initialised well after boot)
I want to use a mix of SCSI, sata + USB devices
Will this work if the USB devices change their 'order'

Short answer: yes, with the right kernel/mdadm versions

Longer:

I use debian so don't know what version of the kernel you use? also mdadm?
You need to look at UUIDs
Use the --assemble, --uid and --scan options *after* you know usb
devices are online (and make sure they're listed in the conf (or use
--config=partitions)). It's safe, mdadm won't assemble an array that's
not complete.

Maybe you'd like to script a test for the usb devices and only attempt
an assemble when all devices are available.

AFAIK there are no issues with the hardware/bus type. A block device is
a block device is a block device is a.... :)

Now, advice...

ok, first off: a 14 device raid1 is 14 times more likely to lose *all*
your data than a single device. I have a scsi device running in one of
my machines since 1995. I am about to RMA a 1yr old maxtor sata drive
(I've RMA'd about 30% of my consumer grade ides/sata drives over the
last few years - ie failed *in* warranty). You appear to be jumping into
consumer grade (usb) devices and it may bite you!

Why not just have a lot of individual filesystems and manage the data by
hand?

Also, if you go the one-device way, why not consider lvm2 instead?
The reason is that it seems *very* likely that you'll be looking to swap
devices in and maybe out (when smart tells you a drive is about to fail)
of this ad-hoc storage.
LVM allows you to see each device as a large number of 'chunks'. You
then gather all those chunks from many devices into a 'pool'. You can
then allocate chunks from the pool to create virtual devices and then
make filesystems on them.

This is good because you can then add another device to the 'pool' and
use those chunks to either:
* swap out a failing (SMART) drive if you're lucky
* grow the virtual drive
* take 'snapshots' (OK you don't need this but it's cool!)

finally, watch the filesystem - eg xfs is excellent for big files but
can't shrink

HTH

David

-- 

Re:

[Ross Vandegrift]

On Thu, Jan 12, 2006 at 11:16:36AM +0000, David Greaves wrote:
> ok, first off: a 14 device raid1 is 14 times more likely to lose *all*
> your data than a single device.

No, this is completely incorrect.  Let A denote the event that a single
disk has failed, A_i denote the event that i disks have failed.
Suppose P(A) = x.  Then by Bayes's Law the probability that an n disk RAID
will lose all of your data is:

n_1 = P(A) = x
n_2 = P(A_2) = P(A) * P(A_1 | A) = x^2
n_3 = P(A_3) = P(A) * P(A_2 | A) = x^3
...
n_i = P(A_i) = P(A) * P(A_{i-1} | A) = x^i

ie, RAID1 is expoentially more reliable as you add extra disks!

This assumes that disk failures are independant - ie, that you
correctly configure disks (don't use master and slave on an IDE
channel!), and replace failed disks as soon as they fail.

This is why adding more disks to a RAID1 is rare - x^2 is going to be
a really low probability!  It will be far, far more common for
operator error to break a RAID than for both devices to honestly fail.

-- 
Ross Vandegrift
ross@lug.udel.edu

"The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell."
	--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37

Re:

[David Greaves]

Ross Vandegrift wrote:

>On Thu, Jan 12, 2006 at 11:16:36AM +0000, David Greaves wrote:
>  
>
>>ok, first off: a 14 device raid1 is 14 times more likely to lose *all*
>>your data than a single device.
>>    
>>
>
>No, this is completely incorrect.  Let A denote the event that a single
>disk has failed, A_i denote the event that i disks have failed.
>Suppose P(A) = x.  Then by Bayes's Law the probability that an n disk RAID
>will lose all of your data is:
>
>n_1 = P(A) = x
>n_2 = P(A_2) = P(A) * P(A_1 | A) = x^2
>n_3 = P(A_3) = P(A) * P(A_2 | A) = x^3
>...
>n_i = P(A_i) = P(A) * P(A_{i-1} | A) = x^i
>
>ie, RAID1 is expoentially more reliable as you add extra disks!
>
>This assumes that disk failures are independant - ie, that you
>correctly configure disks (don't use master and slave on an IDE
>channel!), and replace failed disks as soon as they fail.
>
>This is why adding more disks to a RAID1 is rare - x^2 is going to be
>a really low probability!  It will be far, far more common for
>operator error to break a RAID than for both devices to honestly fail.
>
>  
>
sorry, read it all as 'linear', not mirrored which is why I was writing
drivel ;)

David


-- 

Re:

[Hillary]

Hello linux-raid@vger.kernel.org,

We sell brand-name and exact gen_ericcc equivalents.

Our prompt, courteous and disc#reet ser_vice will make you smile.

---------------------------------

copy the address below and paste in your web browser:

anorthite.techsmartjobs.com/?zz=3Dlowcost

----------------------------------



push the "Perform Currency Conversion" button..
Three hundred years in the deepest South:=20.
Gnarled, twisted, and curled, the nails yellow and claw-like, it is as if,=

Re: Re:

[Norberto X. Milton]

Hey,

We have Special    0ff3rss      and some  New       Pr0ducctss.

WiDE variety of       pre_scr!p_t!0n       med!_c@-t!0ns     to choose fro=
m.

------------------------------------------------------------------------

copy the address below and paste in o your web browser:

afterdays.grindscull.net

------------------------------------------------------------------------


His workbook is wedged in the window,.
Make of my pass a road to the light=20.
It's autumn, 1991, and I'm sitting on the edge of the bed.
Of a tear that runs down an angel's face..
I had nothing further to do with them,.

Get back to you later,

Olen Labovitz

RE:.

[Jake White]

Hello!
Our company Barcelo Travel Inc. seek enthusiastic, organised and alert individual to 
support our busy sales offices. If you live in germany our offer its good chance change your liife.
You must have excellent customer relations, communication and administration skills 
Successful candidates will be required to work in Main our Office for approximately 
one month.
To apply, please email CV to barcelotravinc@aol.com
regards,
Dominico Barcelo

Re:

[David Greaves]

Henry, Andrew wrote:
> I'm new to software RAID and this list.  I read a few months of archives to see if I found answers but only partly...
OK - good idea to start with a simple setup then... oh, wait...

> 1. badblocks -c 10240 -s -w -t random -v /dev/sd[ab]
fine
> 2. parted /dev/sdX mklabel msdos ##on both drives
> 3a. parted /dev/sdX mkpart primary 0 500.1GB ##on both drives
> 3b. parted /dev/sdX set 1 raid on ##on both drives
no point setting raid type since autodetect is not needed
> 4. mdadm --create --verbose /dev/md0 --metadata=1.0 --raid-devices=2 --level=raid1 --name=backupArray /dev/sd[ab]1
a mirror - so the same data/partitions should go to /dev/sda1 /dev/sdb1
> 5. mdadm --examine --scan | tee /etc/mdadm.conf and set 'DEVICES partitions' so that I don't hard code any devide names that may change on reboot.
hmm - on my Debian box I'd get /dev/md/backupArray as the device name I think -
I override this though

> 6. mdadm --assemble --name=mdBackup /dev/md0 ##assemble is run during --create it seems and this was not needed.
> 7. cryptsetup --verbose --verify-passphrase luksFormat /dev/md0
> 8. cryptsetup luksOpen /dev/md0 raid500
good luck with that
> 9. pvcreate /dev/mapper/raid500
> 10. vgcreate vgbackup /dev/mapper/raid500
> 11. lvcreate --name lvbackup --size 450G vgbackup ## check PEs first with vgdisplay
and that...


Seriously, they should work fine - but not a lot of people do this kind of thing
and there may be issues layering this many device layers (eg ISTR a suggestion
that 4K stacks may not be good). Be prepared to submit bug reports and have good
backups.

> 12. mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/vgbackup/lvbackup
Well, I suppose you could have partitioned the lvm volume and used XFS and a
separate journal for maximum complexity <grin>

> 13. mkdir /mnt/raid500; mount /dev/vgbackup/lvbackup /mnt/raid500"
> This worked perfectly.  Did not test but everything lokked fine and I could use the mount.  Thought: lets see if everything comes up at boot (yes, I had edited fstab to mount /dev/vgbackup/lvbackup and set crypttab to start luks on raid500.
> Reboot failed.
I suspect you mean that the filesystem wasn't mounted.
Do you really mean that the machine wouldn't boot - that's bad - you may have
blatted some bootsector somewhere.
Raid admin does not need you to use dd or hack at disk partitions any more than
mkfs does.

> Fsck could not check raid device and would not boot.  Kernel had not
autodetected md0.  I now know this is because superblock format 1.0 puts
metadata at end of device and therefore kernel cannot autodetect.
Technically it's not the sb location that prevents the kernel autodetecting -
it's a design decision that only supports autodetect for v0.9
You don't need autodetect - if you wanted an encrypted lvm root fs then you'd
need an initrd anyhow.
Just make sure you're using a distro that 'does the right thing' and assembles
arrays according to your mdadm.conf at rc?.d time
(nb what distro/kernel are you using)

> I started a LiveCD, mounted my root lvm, removed entries from fstab/crypttab and rebooted.  Reboot was now OK.
> Now I tried to wipe the array so I can re-create with 0.9 metadata superblock.
mdadm --zero-superblock
> I ran dd on sd[ab] for a few hundred megs, which wiped partitions.  I removed /etc/mdadm.conf.  I then repartitioned and rebooted.  I then tried to recreate the array with:
which failed since the sb is at the end of the device
http://linux-raid.osdl.org/index.php/Superblock

> mdadm --create --verbose /dev/md0 --raid-devices=2 --level=raid1 /dev/sd[ab]1
> 
> but it reports that the devices are already part of an array and do I want to continue??  I say yes and it then immedialtely  says "out of sync, resyncing existing array" (not exact words but I suppose you get the idea)
> I reboot to kill sync and then dd again, repartition, etc ect then reboot.
> Now when server comes up, fdisk reports (it's the two 500GB discs that are in the array):
This is all probably down to randomly dd'ing the disks/partitions...
> 
> [root@k2 ~]# fdisk -l
> 
> Disk /dev/hda: 80.0 GB, 80026361856 bytes
> 255 heads, 63 sectors/track, 9729 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> 
>    Device Boot      Start         End      Blocks   Id  System
> /dev/hda1   *           1          19      152586   83  Linux
> /dev/hda2              20        9729    77995575   8e  Linux LVM
> 
> Disk /dev/sda: 500.1 GB, 500107862016 bytes
> 255 heads, 63 sectors/track, 60801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> 
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sda1               1       60801   488384001   fd  Linux raid autodetect
> 
> Disk /dev/sdb: 320.0 GB, 320072933376 bytes
> 255 heads, 63 sectors/track, 38913 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> 
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sdb1               1       38913   312568641   83  Linux


Err, this ^^^ is a 320GB drive. You said 2 500Gb drives...
Mirroring them will work but it will (silently-ish) only use the first 320Gb


> 
> Disk /dev/md0: 500.1 GB, 500105150464 bytes
> 2 heads, 4 sectors/track, 122095984 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
and somehow md0 is sized at 500Gb

what does /proc/mdstat say?

> Disk /dev/md0 doesn't contain a valid partition table
> 
> Where previously, I had /dev/sdc that was the same as /dev/sda above (ignore the 320GB, that is separate and on boot, they sometimes come up in different order).
So what kernel/distro did you use for the liveCD/main OS?

> Now, I cannot write to sda above (500GB disc) with commands: dd, mdadm -zero-superblock etc etc.  I can write to md0 with dd but what the heck happened to sdc??  Why did it become /dev/md0??
> Now I read the forum thread and ran dd on beginning and end of sda and md0 with /dev/zero using seek to skip first 490GB and deleted /dev/md0 then rebooted and now I see sda but there is no sdc or md0.
What's /dev/sdc?

> I cannot see any copy of mdadm.conf in /boot and initramfs-update does not work on CentOS, but I am more used to Debian and do not know the CentOS equivalent.  I do know that I have now completely dd'ed the first 10MB and last 2MB of sda and md0 and have deleted (with rm -f) /dev/md0, and now *only* /dev/sda (plus internal had and extra 320GB sdb) shows up in fdisk -l:  There is no md0 or sdc.
> 
> So after all that rambling, my question is:
> 
> Why did /dev/md0 appear in fdisk -l when it had previously been sda/sdb even after successfully creating my array before reboot?
fdisk -l looks at all the devices for partitions.
sdc isn't there (hardware failure?)

> How do I remove the array?  Have I now done everything to remove it?
mdadm --stop
> I suppose (hope) that if I go to the server and power cycle it and the esata discs, my sdc probably will appear again ( I have not done this yet-no chance today) but why does it not appear after a soft reboot after having dd'd /dev/md0?


Got to admit - I'm confused....


Go and try to make a simple ext3 on a mirror of your 2 500Gb drives. No 'dd'
required.
Once you have that working try playing with mdadm.
Then encrypt it and layer ext3 on that.
I have no idea what you're trying to achieve with lvm - do you need it?

Have a good luck here too : http://linux-raid.osdl.org/

David

RE:

[David Lethe]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Lelsie Rhorer
> Sent: Wednesday, April 01, 2009 11:16 PM
> To: linux-raid@vger.kernel.org
> Subject:
> 
> I'm having a severe problem whose root cause I cannot determine.  I
> have a
> RAID 6 array managed by mdadm running on Debian "Lenny" with a 3.2GHz
> AMD
> Athlon 64 x 2 processor and 8G of RAM.  There are ten 1 Terabyte SATA
> drives, unpartitioned, fully allocated to the /dev/md0 device. The
> drive
> are served by 3 Silicon Image SATA port multipliers and a Silicon
Image
> 4
> port eSATA controller.  The /dev/md0 device is also unpartitioned, and
> all
> 8T of active space is formatted as a single Reiserfs file system.  The
> entire volume is mounted to /RAID.  Various directories on the volume
> are
> shared using both NFS and SAMBA.
> 
> Performance of the RAID system is very good.  The array can read and
> write
> at over 450 Mbps, and I don't know if the limit is the array itself or
> the
> network, but since the performance is more than adequate I really am
> not
> concerned which is the case.
> 
> The issue is the entire array will occasionally pause completely for
> about
> 40 seconds when a file is created.  This does not always happen, but
> the
> situation is easily reproducible.  The frequency at which the symptom
> occurs seems to be related to the transfer load on the array.  If no
> other
> transfers are in process, then the failure seems somewhat more rare,
> perhaps accompanying less than 1 file creation in 10..  During heavy
> file
> transfer activity, sometimes the system halts with every other file
> creation.  Although I have observed many dozens of these events, I
have
> never once observed it to happen except when a file creation occurs.
> Reading and writing existing files never triggers the event, although
> any
> read or write occurring during the event is halted for the duration.
> (There is one cron jog which runs every half-hour that creates a tiny
> file;
> this is the most common failure vector.)  There are other drives
> formatted
> with other file systems on the machine, but the issue has never been
> seen
> on any of the other drives.  When the array runs its regularly
> scheduled
> health check, the problem is much worse.  Not only does it lock up
with
> almost every single file creation, but the lock-up time is much longer
> -
> sometimes in excess of 2 minutes.
> 
> Transfers via Linux based utilities (ftp, NFS, cp, mv, rsync, etc) all
> recover after the event, but SAMBA based transfers frequently fail,
> both
> reads and writes.
> 
> How can I troubleshoot and more importantly resolve this issue?
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

I would try to first run hardware diagnostics.  Maybe you will get
"lucky" and one or more disks will fail diagnostics, which at least
means it will be easy to repair the problem.

This could very well be situation where you have a lot of bad blocks
that have to get restriped, and parity has to be regenerated.   Are
these the cheap consumer SATA disk drives, or enterprise class disks? 

David

Re:

[Peter Grandi]

> The issue is the entire array will occasionally pause completely
> for about 40 seconds when a file is created. [ ... ] During heavy
> file transfer activity, sometimes the system halts with every
> other file creation. [ ... ] There are other drives formatted
> with other file systems on the machine, but the issue has never
> been seen on any of the other drives.  When the array runs its
> regularly scheduled health check, the problem is much worse. [
> ... ]

Looks like that either you have hw issues (transfer errors, bad
blocks) or more likely the cache flusher and elevator settings have
not been tuned for a steady flow.

> How can I troubleshoot and more importantly resolve this issue?

Well, troubleshooting would require a good understanding of file
system design and storage subsystem design, and quite a bit of time.

However, for hardware errors check the kernel logs, and for cache
flusher and elevator settings check the 'bi'/'bo' numbers of
'vmstat 1' while the pause happens.

For a deeper profile of per-drive IO run 'watch iostat 1 2' while
this is happening. This might also help indicate drive errors (no
IO is happening) or flusher/elevator tuning issues (lots of IO is
happening suddenfly).

Re:

[Andrew Burgess]

On Wed, 2009-04-01 at 23:16 -0500, Lelsie Rhorer wrote:

> The issue is the entire array will occasionally pause completely for about
> 40 seconds when a file is created. 

I had symptoms like this once. It turned out to be a defective disk. The
disk would never return a read or write error but just intermittently
took a really long time to respond.

I found it by running atop. All the other drives would be running at low
utilization and this one drive would be at 100% when the symptoms
occurred (which in atop gets colored red so it jumps out at you)

RE:

[Lelsie Rhorer]

> I would try to first run hardware diagnostics.  Maybe you will get
> "lucky" and one or more disks will fail diagnostics, which at least
> means it will be easy to repair the problem.
> 
> This could very well be situation where you have a lot of bad blocks
> that have to get restriped, and parity has to be regenerated.   Are
> these the cheap consumer SATA disk drives, or enterprise class disks?


I don't buy that for a second.  First of all, restriping parity can and does
occur in the background.  Secondly, how is it the system writes many
terrabytes of data post file creation, then chokes on a 0 byte file?

Re:

[Greg Freemyer]

On Sat, Apr 4, 2009 at 8:12 PM, Lelsie Rhorer <lrhorer@satx.rr.com> wrote:
>> I would try to first run hardware diagnostics.  Maybe you will get
>> "lucky" and one or more disks will fail diagnostics, which at least
>> means it will be easy to repair the problem.
>>
>> This could very well be situation where you have a lot of bad blocks
>> that have to get restriped, and parity has to be regenerated.   Are
>> these the cheap consumer SATA disk drives, or enterprise class disks?
>
>
> I don't buy that for a second.  First of all, restriping parity can and does
> occur in the background.  Secondly, how is it the system writes many
> terrabytes of data post file creation, then chokes on a 0 byte file?
>

Alternate theory - serious fsync performance issue

I don't know if it's related, but there is a lot of recent discussion
related to fsync causing large delays in ext3.  Linus is saying his
highspeed SDD is seeing multisecond delays.  He is very frustrated
because the SDD should be more or less instantaneous.

The current thread is http://markmail.org/message/adiyz3lri6tlueaf

In one of the other threads I saw someone saying that in one test they
had a fsync() call take minutes to return.  Apparently no one yet
fully understands what is going on.  Seems like something that could
in some way be related to what you are seeing.

Greg
-- 
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re:

[Roger Heflin]

Lelsie Rhorer wrote:
>> I would try to first run hardware diagnostics.  Maybe you will get
>> "lucky" and one or more disks will fail diagnostics, which at least
>> means it will be easy to repair the problem.
>>
>> This could very well be situation where you have a lot of bad blocks
>> that have to get restriped, and parity has to be regenerated.   Are
>> these the cheap consumer SATA disk drives, or enterprise class disks?
> 
> 
> I don't buy that for a second.  First of all, restriping parity can and does
> occur in the background.  Secondly, how is it the system writes many
> terrabytes of data post file creation, then chokes on a 0 byte file?
> 


You should note that the drive won't know a sector it just wrote is 
bad until it reads it....are you sure you actually successfully wrote 
all of that data and that it is still there?

And it is not the writes that kill when you have a drive going bad, it 
is the reads of the bad sectors.    And to create a file, a number of 
things will likely need to be read to finish the file creation, and if 
one of those is a bad sector things get ugly.

RE:

[Lelsie Rhorer]

> Alternate theory - serious fsync performance issue
> 
> I don't know if it's related, but there is a lot of recent discussion
> related to fsync causing large delays in ext3.  Linus is saying his
> highspeed SDD is seeing multisecond delays.  He is very frustrated
> because the SDD should be more or less instantaneous.
> 
> The current thread is http://markmail.org/message/adiyz3lri6tlueaf
> 
> In one of the other threads I saw someone saying that in one test they
> had a fsync() call take minutes to return.  Apparently no one yet
> fully understands what is going on.  Seems like something that could
> in some way be related to what you are seeing.

Well, it could be.  I tried flushing the cashes numerous times while
testing, but I never could see it made a difference one way or the other.

RE:

[Lelsie Rhorer]

> You should note that the drive won't know a sector it just wrote is
> bad until it reads it

Yes, but it also won't halt the write for 40 seconds because it was bad.
More to  the point, there is no difference at the drive level between a bad
sector written for a 30Gb file and a 30 byte file.

> ....are you sure you actually successfully wrote all of that data and that
>it is still there?  Pretty sure, yeah.  There are no errors in the
filesystem, and every file I have written works.  Again, however, the point
is there is never a problem once the file is created, no matter how long it
takes to write it out to disk.  The moment the file is created, however,
there may be up to a 2 minute delay in writing its data to the drive.

> And it is not the writes that kill when you have a drive going bad, it
> is the reads of the bad sectors.    And to create a file, a number of
> things will likely need to be read to finish the file creation, and if
> one of those is a bad sector things get ugly.

Well, I agree to some extent, except that why would it be loosely related to
the volume of drive activity, and why is it 5 drives stop reading altogether
and 5 do not?  Furthermore, every single video file gets read, re-written,
edited, re-written again, and finally read again at least once, sometimes
several times, before being finally archived.  Why does the kernel log never
report any errors of any sort?

RE:

[David Lethe]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Lelsie Rhorer
> Sent: Sunday, April 05, 2009 12:21 AM
> To: linux-raid@vger.kernel.org
> Subject: RE:
> 
> > You should note that the drive won't know a sector it just wrote is
> > bad until it reads it
> 
> Yes, but it also won't halt the write for 40 seconds because it was
> bad.
> More to  the point, there is no difference at the drive level between
a
> bad
> sector written for a 30Gb file and a 30 byte file.
> 
> > ....are you sure you actually successfully wrote all of that data
and
> that
> >it is still there?  Pretty sure, yeah.  There are no errors in the
> filesystem, and every file I have written works.  Again, however, the
> point
> is there is never a problem once the file is created, no matter how
> long it
> takes to write it out to disk.  The moment the file is created,
> however,
> there may be up to a 2 minute delay in writing its data to the drive.
> 
> > And it is not the writes that kill when you have a drive going bad,
> it
> > is the reads of the bad sectors.    And to create a file, a number
of
> > things will likely need to be read to finish the file creation, and
> if
> > one of those is a bad sector things get ugly.
> 
> Well, I agree to some extent, except that why would it be loosely
> related to
> the volume of drive activity, and why is it 5 drives stop reading
> altogether
> and 5 do not?  Furthermore, every single video file gets read, re-
> written,
> edited, re-written again, and finally read again at least once,
> sometimes
> several times, before being finally archived.  Why does the kernel log
> never
> report any errors of any sort?
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

All of what you report is still consistent with delays caused by having
to remap bad blocks
The O/S will not report recovered errors, as this gets done internally
by the disk drive, and the O/S never learns about it. (Queue depth
settings can account for some of the other "weirdness" you reported.

Really, if this was my system I would run non-destructive read tests on
all blocks; along with the embedded self-test on the disk.  It is often
a lot easier and more productive to eliminate what ISN'T the problem
rather than chase all of the potential reasons for the problem.  

Re:

[Greg Freemyer]

On Sun, Apr 5, 2009 at 1:05 AM, Lelsie Rhorer <lrhorer@satx.rr.com> wrote:
>> Alternate theory - serious fsync performance issue
>>
>> I don't know if it's related, but there is a lot of recent discussion
>> related to fsync causing large delays in ext3.  Linus is saying his
>> highspeed SDD is seeing multisecond delays.  He is very frustrated
>> because the SDD should be more or less instantaneous.
>>
>> The current thread is http://markmail.org/message/adiyz3lri6tlueaf
>>
>> In one of the other threads I saw someone saying that in one test they
>> had a fsync() call take minutes to return.  Apparently no one yet
>> fully understands what is going on.  Seems like something that could
>> in some way be related to what you are seeing.
>
> Well, it could be.  I tried flushing the cashes numerous times while
> testing, but I never could see it made a difference one way or the other.
>

In a separate thread you said it was reiser and what I have seen
discussed is ext3, so you may be safe from that bug.

As to flushing caches, I don't think that is the same thing.    This
bug specifically impacts fsyncs on a small file while a heavy i/o load
is underway via other processes.  The elevators were being discussed
as part of the problem and fsync triggers different elevator logic
than sync or drop_caches does.

Greg
-- 
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re:

[Roger Heflin]

Jack Etherington wrote:
> Hello,
> I am not sure whether troubleshooting messages are allowed on the mdadm
> mailing list (or it is for development and bugs only) so please point me in
> the right direction if this is not the right place.
> 
> Before posting here I have tried using the following resources for
> information:
>> Google
>> Distribution IRC channel (Ubuntu)
>> Linuxquestions.org
> 
> My knowledge of Linux is beginner/moderate.
> 
> My setup is:
> 9x1tb Hard Drives (2xhitachi and 7x Samsung HD103UJ)
> Supermicro AOC-SAT2-MV8 8 Port SATA Card
> 1xMotherboard SATA port
> Single RAID5 array created with mdadm, printout of /proc/mdstat:
> 
> root@server3:~# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]
> [raid10]
> md0 : active raid5 sdj1[7] sdc1[0] sda1[8] sdg1[6] sdi1[9](F) sdd1[4]
> sde1[3] sdh1[2] sdf1[10](F)
>       7814079488 blocks level 5, 64k chunk, algorithm 2 [9/7] [U_UUU_UUU]
> 
> 
> A printout of /var/messages is available here: http://pastebin.com/m6499846
> as not to make this post any longer...
> (The array has been down for about a month now. It is my home storage
> server, non-critical, but I do not have a backup)
> 
> Also a printout of ‘mdadm --detail /dev/md0’ is available here:
> http://pastebin.com/f44b6e069
> 
> I have used ‘mdadm -v -A -f /dev/md0’ to get the array online again, and can
> read data (intact without errors) from the array, but it soon becomes
> degraded again.
> 
> Any help on where to start would be greatly appreciated :)
> 
> Jack


What kind of MB do you have on this setup?

And that card is in a PCI-X slot isn't it?

And how big of power supply do you have?   What is the rating of the 
Amps on the +12V line?

And is it always the same disks or disks that fault, or is any disk 
just as likely to have issues?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re:

[Michael Evans]

On Wed, Jan 6, 2010 at 6:19 AM, Lapohos Tibor <tibor.lapohos@rogers.com> wrote:
> Hello,
>
> I successfully set up an Intel Matrix Raid device with a RAID1 and a RAID0 volume, each having a couple of partitions, but then I could not install GRUB2 on the RAID1 volume, which I wanted to use to boot from and mount as root. It turned out that the "IMSM" metadata is not supported in GRUB2 (v1.97.1) just yet, so I had to turn away from my original plan.
>
> To "imitate" the setup I originally wanded, I turned both of my drives into AHCI controlled devices in the BIOS (instead of RAID), and I partitioned them to obtain /dev/sda[12] and /dev/sdb[12].
>
> Then I used /dev/sd[ab]1 to build a RAID1 set, and /dev/sd[ab]2 to create a RAID0 set using mdadm v 3.0.3:
>
>> mdadm -C /dev/md0 -v -e 0 -l 1 -n 2 /dev/sda1 /dev/sdb1
>> mdadm -C /dev/md1 -v -e 0 -l 0 -n 2 /dev/sda2 /dev/sdb2
>
> I set the metadata type to 0.90 because I would like to boot from it and allow the kernel to auto-detect the RAID devices while it's booting, in order to can get away from using an intitrd (I am building my own distribution based on CLFS x86_64 multilib).
>
> I used cfdisk to partition both of the /dev/md[01] devices, and I obtained /dev/md0p[123] and /dev/md1p[12]. The plan is to use /dev/md0p1 as a RAID1 root partition, and have the system boot from /dev/md0. I formatted /dev/md0p1 as
>
>> mk2efs -t ext4 -L OS /dev/md0p1
>
> To this point, things went smoothly. mdadm -D... and mdadm -E... did report back working devices as intended. Then mounted /dev/md0p1 on a directory called /root/os, and I did
>
>> grub-install --root-directory=/root/os /dev/md0
>
> or
>
>> grub-install --root-directory=/root/os "md0"
>
> and I got a warning and an error message: "Your embedding area is unusually small.  core.img won't fit in it." and "Embedding is not possible, but this is required when the root device is on a RAID array or LVM volume."
>
> What did I do wrong, and how do I fix it? Thanks ahead,
> Tibor
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Grub wants to embed (copy it's executable) in to the area between the
MBR style layout (sector 0 of a drive) and the first partition on that
drive (typically starts as early as sector 63).  Try starting the
first partition at something like a 1 or 2 mb offset from the start of
the drive.  That should likely be enough space.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re:

[Antonio Perez]

Michael Evans wrote:
[...] 
> Grub wants to embed (copy it's executable) in to the area between the
> MBR style layout (sector 0 of a drive) and the first partition on that
> drive (typically starts as early as sector 63).  Try starting the
> first partition at something like a 1 or 2 mb offset from the start of
> the drive.  That should likely be enough space.

	...something like a 1 or 2 mb offset...

Grub-core is about 26k, anything bigger than 30k should be enough, right?

-- 
Antonio Perez

Re:

[Neil Brown]

On Sun, 7 Mar 2010 19:37:15 -0600
"Leslie Rhorer" <lrhorer@satx.rr.com> wrote:

> I am running mdadm 2.6.7.2-1, and 2.6.7.2-3 is available under my distro.
> Do either of these versions support reshaping an array from RAID5 to RAID6?

No

> Does any later version?

Yes.


You need mdadm-3.1.1 plus linux 2.6.32.

NeilBrown

> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE:

[Leslie Rhorer]

	Thanks, Neil.  I guess I'll just tear it down and rebuild.  Debian
"Squeeze" is definitely not ready for prime time, and I don't think even it
supplies kernel 2.6.32 or mdadm 3.1.1.  Oh, well.

> -----Original Message-----
> From: Neil Brown [mailto:neilb@suse.de]
> Sent: Sunday, March 07, 2010 7:53 PM
> To: Leslie Rhorer
> Cc: linux-raid@vger.kernel.org
> Subject: Re:
> 
> On Sun, 7 Mar 2010 19:37:15 -0600
> "Leslie Rhorer" <lrhorer@satx.rr.com> wrote:
> 
> > I am running mdadm 2.6.7.2-1, and 2.6.7.2-3 is available under my
> distro.
> > Do either of these versions support reshaping an array from RAID5 to
> RAID6?
> 
> No
> 
> > Does any later version?
> 
> Yes.
> 
> 
> You need mdadm-3.1.1 plus linux 2.6.32.
> 
> NeilBrown
> 
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re:

[Michael Evans]

On Sun, Mar 7, 2010 at 6:01 PM, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
>        Thanks, Neil.  I guess I'll just tear it down and rebuild.  Debian
> "Squeeze" is definitely not ready for prime time, and I don't think even it
> supplies kernel 2.6.32 or mdadm 3.1.1.  Oh, well.
>
>> -----Original Message-----
>> From: Neil Brown [mailto:neilb@suse.de]
>> Sent: Sunday, March 07, 2010 7:53 PM
>> To: Leslie Rhorer
>> Cc: linux-raid@vger.kernel.org
>> Subject: Re:
>>
>> On Sun, 7 Mar 2010 19:37:15 -0600
>> "Leslie Rhorer" <lrhorer@satx.rr.com> wrote:
>>
>> > I am running mdadm 2.6.7.2-1, and 2.6.7.2-3 is available under my
>> distro.
>> > Do either of these versions support reshaping an array from RAID5 to
>> RAID6?
>>
>> No
>>
>> > Does any later version?
>>
>> Yes.
>>
>>
>> You need mdadm-3.1.1 plus linux 2.6.32.
>>
>> NeilBrown
>>
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

What are you talking about?  Have you not synced to the online
repository?  I grabbed this out of the Package.bz2 file.

(Yes, BTW, they should probably have a bug filed to get mdadm 3.1
included before the freeze...)

Package: mdadm
Priority: optional
Section: admin
Installed-Size: 1064
Maintainer: Debian mdadm maintainers <pkg-mdadm-devel@lists.alioth.debian.org>
Architecture: i386
Version: 3.0.3-2
Replaces: mdctl
Depends: libc6 (>= 2.3.3), udev | makedev, debconf (>= 1.4.72),
lsb-base (>= 3.1-6)
Recommends: default-mta | mail-transport-agent, module-init-tools
Conflicts: initramfs-tools (<< 0.65), mdctl (<< 0.7.2), raidtools2 (<<
1.00.3-12.1)
Filename: pool/main/m/mdadm/mdadm_3.0.3-2_i386.deb
Size: 418426
MD5sum: ab27fb8bfde438bc76dcb42ce8717626
SHA1: a89f1b90ac08bcf29c6c49cc310db0823edf8562
SHA256: 5f9fae56cba6aa6dbb747ddf86402615319d32f3d692da64a818f0f1fab399af
Description: tool to administer Linux MD arrays (software RAID)
 The mdadm utility can be used to create, manage, and monitor MD
 (multi-disk) arrays for software RAID or multipath I/O.
 .
 This package automatically configures mdadm to assemble arrays during the
 system startup process. If not needed, this functionality can be disabled.
Homepage: http://neil.brown.name/blog/mdadm
Tag: admin::boot, admin::configuring, hardware::storage,
implemented-in::c, implemented-in::shell, interface::commandline,
interface::daemon, role::program, scope::utility, use::configuring,
use::monitor


Package: linux-image-2.6.32-trunk-686
Priority: optional
Section: kernel
Installed-Size: 74220
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: i386
Source: linux-2.6
Version: 2.6.32-5
Provides: linux-image, linux-image-2.6, linux-modules-2.6.32-trunk-686
Depends: module-init-tools, initramfs-tools (>= 0.55) | linux-initramfs-tool
Pre-Depends: debconf | debconf-2.0
Recommends: firmware-linux-free (>= 2.6.32), libc6-i686
Suggests: linux-doc-2.6.32, grub | lilo
Conflicts: initramfs-tools (<< 0.55)
Filename: pool/main/l/linux-2.6/linux-image-2.6.32-trunk-686_2.6.32-5_i386.deb
Size: 26282748
MD5sum: 776c9e57322b9e154bcc0518a9224df3
SHA1: 210cc48a91120ebd36cb20d9516e6e34e4c52da4
SHA256: bf8e8936cacf902b09dde1950bf10dcf50b1cceeeb4d1c351250a5c13da42e83
Description: Linux 2.6.32 for modern PCs
 The Linux kernel 2.6.32 and modules for use on PCs with Intel Pentium
 Pro/II/III/4/4M/D/M, Xeon, Celeron, Core or Atom; AMD K6, Geode LX/NX,
 Athlon (K7), Duron, Opteron, Sempron, Turion or Phenom; Transmeta
 Efficeon; VIA C3 "Nehemiah" or C7 processors.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE:

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Michael Evans
> Sent: Sunday, March 07, 2010 8:22 PM
> To: Leslie Rhorer
> Cc: Neil Brown; linux-raid@vger.kernel.org
> Subject: Re:
> 
> On Sun, Mar 7, 2010 at 6:01 PM, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
> >        Thanks, Neil.  I guess I'll just tear it down and rebuild.
>  Debian
> > "Squeeze" is definitely not ready for prime time, and I don't think even
> it
> > supplies kernel 2.6.32 or mdadm 3.1.1.  Oh, well.
> >
> >> -----Original Message-----
> >> From: Neil Brown [mailto:neilb@suse.de]
> >> Sent: Sunday, March 07, 2010 7:53 PM
> >> To: Leslie Rhorer
> >> Cc: linux-raid@vger.kernel.org
> >> Subject: Re:
> >>
> >> On Sun, 7 Mar 2010 19:37:15 -0600
> >> "Leslie Rhorer" <lrhorer@satx.rr.com> wrote:
> >>
> >> > I am running mdadm 2.6.7.2-1, and 2.6.7.2-3 is available under my
> >> distro.
> >> > Do either of these versions support reshaping an array from RAID5 to
> >> RAID6?
> >>
> >> No
> >>
> >> > Does any later version?
> >>
> >> Yes.
> >>
> >>
> >> You need mdadm-3.1.1 plus linux 2.6.32.
> >>
> >> NeilBrown
> >>
> >> >
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> What are you talking about?

	Referring to what, specifically?

> Have you not synced to the online
> repository?  I grabbed this out of the Package.bz2 file.

	Of course.  I'm running "Lenny" for an AMD-64.  The output below is
definitely not for AMD-64, and it looks to me like it might be "Squeeze" or
"Sid", not "Lenny".

From the "Lenny" AMD-64 Package.bz2 file:

Package: mdadm
Priority: optional
Section: admin
Installed-Size: 776
Maintainer: Debian mdadm maintainers
<pkg-mdadm-devel@lists.alioth.debian.org>
Architecture: amd64
Version: 2.6.7.2-3
Replaces: mdctl
Depends: libc6 (>= 2.7-1), udev | makedev, debconf (>= 1.4.72), lsb-base (>=
3.1-6)
Recommends: exim4 | mail-transport-agent, module-init-tools
Conflicts: initramfs-tools (<< 0.65), mdctl (<< 0.7.2), raidtools2 (<<
1.00.3-12.1)
Filename: pool/main/m/mdadm/mdadm_2.6.7.2-3_amd64.deb
Size: 273876
MD5sum: a3755364dcc80be5d940a3d423eb55a9
SHA1: a0ee5083f213f70bbb96cb8fd5b2f89a5cb8ddd3
SHA256: 10add842a74034592b8647c8e70a69502e3d915413a7da640c0172d02ef9ee7d
Description: tool to administer Linux MD arrays (software RAID)
 The mdadm utility can be used to create, manage, and monitor MD
 (multi-disk) arrays for software RAID or multipath I/O.
 .
 This package automatically configures mdadm to assemble arrays during the
 system startup process. If not needed, this functionality can be disabled.
Homepage: http://neil.brown.name/blog/mdadm
Tag: admin::boot, admin::configuring, hardware::storage, implemented-in::c,
implemented-in::shell, interface::commandline, interface::daemon,
role::program, scope::utility, use::configuring, use::monitor


	"Lenny" i386 lists the same thing.
 
> (Yes, BTW, they should probably have a bug filed to get mdadm 3.1
> included before the freeze...)
> 
> Package: mdadm
> Priority: optional
> Section: admin
> Installed-Size: 1064
> Maintainer: Debian mdadm maintainers <pkg-mdadm-
> devel@lists.alioth.debian.org>
> Architecture: i386
> Version: 3.0.3-2
> Replaces: mdctl
> Depends: libc6 (>= 2.3.3), udev | makedev, debconf (>= 1.4.72),
> lsb-base (>= 3.1-6)
> Recommends: default-mta | mail-transport-agent, module-init-tools
> Conflicts: initramfs-tools (<< 0.65), mdctl (<< 0.7.2), raidtools2 (<<
> 1.00.3-12.1)
> Filename: pool/main/m/mdadm/mdadm_3.0.3-2_i386.deb
> Size: 418426
> MD5sum: ab27fb8bfde438bc76dcb42ce8717626
> SHA1: a89f1b90ac08bcf29c6c49cc310db0823edf8562
> SHA256: 5f9fae56cba6aa6dbb747ddf86402615319d32f3d692da64a818f0f1fab399af
> Description: tool to administer Linux MD arrays (software RAID)
>  The mdadm utility can be used to create, manage, and monitor MD
>  (multi-disk) arrays for software RAID or multipath I/O.
>  .
>  This package automatically configures mdadm to assemble arrays during the
>  system startup process. If not needed, this functionality can be
> disabled.
> Homepage: http://neil.brown.name/blog/mdadm
> Tag: admin::boot, admin::configuring, hardware::storage,
> implemented-in::c, implemented-in::shell, interface::commandline,
> interface::daemon, role::program, scope::utility, use::configuring,
> use::monitor
> 
> 
> Package: linux-image-2.6.32-trunk-686
> Priority: optional
> Section: kernel
> Installed-Size: 74220
> Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
> Architecture: i386
> Source: linux-2.6
> Version: 2.6.32-5
> Provides: linux-image, linux-image-2.6, linux-modules-2.6.32-trunk-686
> Depends: module-init-tools, initramfs-tools (>= 0.55) | linux-initramfs-
> tool
> Pre-Depends: debconf | debconf-2.0
> Recommends: firmware-linux-free (>= 2.6.32), libc6-i686
> Suggests: linux-doc-2.6.32, grub | lilo
> Conflicts: initramfs-tools (<< 0.55)
> Filename: pool/main/l/linux-2.6/linux-image-2.6.32-trunk-686_2.6.32-
> 5_i386.deb
> Size: 26282748
> MD5sum: 776c9e57322b9e154bcc0518a9224df3
> SHA1: 210cc48a91120ebd36cb20d9516e6e34e4c52da4
> SHA256: bf8e8936cacf902b09dde1950bf10dcf50b1cceeeb4d1c351250a5c13da42e83
> Description: Linux 2.6.32 for modern PCs
>  The Linux kernel 2.6.32 and modules for use on PCs with Intel Pentium
>  Pro/II/III/4/4M/D/M, Xeon, Celeron, Core or Atom; AMD K6, Geode LX/NX,
>  Athlon (K7), Duron, Opteron, Sempron, Turion or Phenom; Transmeta
>  Efficeon; VIA C3 "Nehemiah" or C7 processors.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re:

[Michael Evans]

On Sun, Mar 7, 2010 at 7:20 PM, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
>
>
>> -----Original Message-----
>> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
>> owner@vger.kernel.org] On Behalf Of Michael Evans
>> Sent: Sunday, March 07, 2010 8:22 PM
>> To: Leslie Rhorer
>> Cc: Neil Brown; linux-raid@vger.kernel.org
>> Subject: Re:
>>
>> On Sun, Mar 7, 2010 at 6:01 PM, Leslie Rhorer <lrhorer@satx.rr.com> wrote:
>> >        Thanks, Neil.  I guess I'll just tear it down and rebuild.
>>  Debian
>> > "Squeeze" is definitely not ready for prime time, and I don't think even
>> it
>> > supplies kernel 2.6.32 or mdadm 3.1.1.  Oh, well.
>> >
>> >> -----Original Message-----
>> >> From: Neil Brown [mailto:neilb@suse.de]
>> >> Sent: Sunday, March 07, 2010 7:53 PM
>> >> To: Leslie Rhorer
>> >> Cc: linux-raid@vger.kernel.org
>> >> Subject: Re:
>> >>
>> >> On Sun, 7 Mar 2010 19:37:15 -0600
>> >> "Leslie Rhorer" <lrhorer@satx.rr.com> wrote:
>> >>
>> >> > I am running mdadm 2.6.7.2-1, and 2.6.7.2-3 is available under my
>> >> distro.
>> >> > Do either of these versions support reshaping an array from RAID5 to
>> >> RAID6?
>> >>
>> >> No
>> >>
>> >> > Does any later version?
>> >>
>> >> Yes.
>> >>
>> >>
>> >> You need mdadm-3.1.1 plus linux 2.6.32.
>> >>
>> >> NeilBrown
>> >>
>> >> >
>> >> > --
>> >> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
>> in
>> >> > the body of a message to majordomo@vger.kernel.org
>> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>>
>> What are you talking about?
>
>        Referring to what, specifically?
>
>> Have you not synced to the online
>> repository?  I grabbed this out of the Package.bz2 file.
>
>        Of course.  I'm running "Lenny" for an AMD-64.  The output below is
> definitely not for AMD-64, and it looks to me like it might be "Squeeze" or
> "Sid", not "Lenny".
>
> From the "Lenny" AMD-64 Package.bz2 file:
>
> Package: mdadm
> Priority: optional
> Section: admin
> Installed-Size: 776
> Maintainer: Debian mdadm maintainers
> <pkg-mdadm-devel@lists.alioth.debian.org>
> Architecture: amd64
> Version: 2.6.7.2-3
> Replaces: mdctl
> Depends: libc6 (>= 2.7-1), udev | makedev, debconf (>= 1.4.72), lsb-base (>=
> 3.1-6)
> Recommends: exim4 | mail-transport-agent, module-init-tools
> Conflicts: initramfs-tools (<< 0.65), mdctl (<< 0.7.2), raidtools2 (<<
> 1.00.3-12.1)
> Filename: pool/main/m/mdadm/mdadm_2.6.7.2-3_amd64.deb
> Size: 273876
> MD5sum: a3755364dcc80be5d940a3d423eb55a9
> SHA1: a0ee5083f213f70bbb96cb8fd5b2f89a5cb8ddd3
> SHA256: 10add842a74034592b8647c8e70a69502e3d915413a7da640c0172d02ef9ee7d
> Description: tool to administer Linux MD arrays (software RAID)
>  The mdadm utility can be used to create, manage, and monitor MD
>  (multi-disk) arrays for software RAID or multipath I/O.
>  .
>  This package automatically configures mdadm to assemble arrays during the
>  system startup process. If not needed, this functionality can be disabled.
> Homepage: http://neil.brown.name/blog/mdadm
> Tag: admin::boot, admin::configuring, hardware::storage, implemented-in::c,
> implemented-in::shell, interface::commandline, interface::daemon,
> role::program, scope::utility, use::configuring, use::monitor
>
>
>        "Lenny" i386 lists the same thing.
>
>> (Yes, BTW, they should probably have a bug filed to get mdadm 3.1
>> included before the freeze...)
>>
>> Package: mdadm
>> Priority: optional
>> Section: admin
>> Installed-Size: 1064
>> Maintainer: Debian mdadm maintainers <pkg-mdadm-
>> devel@lists.alioth.debian.org>
>> Architecture: i386
>> Version: 3.0.3-2
>> Replaces: mdctl
>> Depends: libc6 (>= 2.3.3), udev | makedev, debconf (>= 1.4.72),
>> lsb-base (>= 3.1-6)
>> Recommends: default-mta | mail-transport-agent, module-init-tools
>> Conflicts: initramfs-tools (<< 0.65), mdctl (<< 0.7.2), raidtools2 (<<
>> 1.00.3-12.1)
>> Filename: pool/main/m/mdadm/mdadm_3.0.3-2_i386.deb
>> Size: 418426
>> MD5sum: ab27fb8bfde438bc76dcb42ce8717626
>> SHA1: a89f1b90ac08bcf29c6c49cc310db0823edf8562
>> SHA256: 5f9fae56cba6aa6dbb747ddf86402615319d32f3d692da64a818f0f1fab399af
>> Description: tool to administer Linux MD arrays (software RAID)
>>  The mdadm utility can be used to create, manage, and monitor MD
>>  (multi-disk) arrays for software RAID or multipath I/O.
>>  .
>>  This package automatically configures mdadm to assemble arrays during the
>>  system startup process. If not needed, this functionality can be
>> disabled.
>> Homepage: http://neil.brown.name/blog/mdadm
>> Tag: admin::boot, admin::configuring, hardware::storage,
>> implemented-in::c, implemented-in::shell, interface::commandline,
>> interface::daemon, role::program, scope::utility, use::configuring,
>> use::monitor
>>
>>
>> Package: linux-image-2.6.32-trunk-686
>> Priority: optional
>> Section: kernel
>> Installed-Size: 74220
>> Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
>> Architecture: i386
>> Source: linux-2.6
>> Version: 2.6.32-5
>> Provides: linux-image, linux-image-2.6, linux-modules-2.6.32-trunk-686
>> Depends: module-init-tools, initramfs-tools (>= 0.55) | linux-initramfs-
>> tool
>> Pre-Depends: debconf | debconf-2.0
>> Recommends: firmware-linux-free (>= 2.6.32), libc6-i686
>> Suggests: linux-doc-2.6.32, grub | lilo
>> Conflicts: initramfs-tools (<< 0.55)
>> Filename: pool/main/l/linux-2.6/linux-image-2.6.32-trunk-686_2.6.32-
>> 5_i386.deb
>> Size: 26282748
>> MD5sum: 776c9e57322b9e154bcc0518a9224df3
>> SHA1: 210cc48a91120ebd36cb20d9516e6e34e4c52da4
>> SHA256: bf8e8936cacf902b09dde1950bf10dcf50b1cceeeb4d1c351250a5c13da42e83
>> Description: Linux 2.6.32 for modern PCs
>>  The Linux kernel 2.6.32 and modules for use on PCs with Intel Pentium
>>  Pro/II/III/4/4M/D/M, Xeon, Celeron, Core or Atom; AMD K6, Geode LX/NX,
>>  Athlon (K7), Duron, Opteron, Sempron, Turion or Phenom; Transmeta
>>  Efficeon; VIA C3 "Nehemiah" or C7 processors.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Yes, it is for Squeeze, if you want the latest bugfixes and security
updates you should seriously consider running debian-testing instead
of stable.  Stable is reserved for 'mature' features.  Testing, as far
as I'm aware, will almost never (and should never if you are paying
attention) cause data-loss, but might occasionally get in to a
situation where something breaks; mostly just during upgrades (but
then that's true of any upgrade).
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re:

[Neil Brown]

On Sat, 13 Nov 2010 01:01:47 -0500
Mike Viau <viaum@sheridanc.on.ca> wrote:

> 
> Hello,
> 
> I am trying to re-setup my fake-raid (RAID1) volume with LVM2 like setup previously. I had been using dmraid on a Lenny installation which gave me (from memory) a block device like /dev/mapper/isw_xxxxxxxxxxx_ and also a /dev/One1TB, but have discovered that the mdadm has replaced the older and believed to be obsolete dmraid for multiple disk/raid support.
> 
> Automatically the fake-raid LVM physical volume does not seem to be set up. I believe my data is safe as I can insert a knoppix live-cd in the system and mount the fake-raid volume (and browse the files). I am planning on perhaps purchasing another at least 1TB drive to backup the data before trying to much fancy stuff with mdadm in fear of loosing the data.
> 
> A few commands that might shed more light on the situation:
> 
> 
> pvdisplay (showing the /dev/md/[device] not recognized yet by LVM2, note sdc another single drive with LVM)
> 
>   --- Physical volume ---
>   PV Name               /dev/sdc7
>   VG Name               XENSTORE-VG
>   PV Size               46.56 GiB / not usable 2.00 MiB
>   Allocatable           yes (but full)
>   PE Size               4.00 MiB
>   Total PE              11920
>   Free PE               0
>   Allocated PE          11920
>   PV UUID               wRa8xM-lcGZ-GwLX-F6bA-YiCj-c9e1-eMpPdL
> 
> 
> cat /proc/mdstat (showing what mdadm shows/discovers)
> 
> Personalities :
> md127 : inactive sda[1](S) sdb[0](S)
>       4514 blocks super external:imsm
> 
> unused devices: 

As imsm can have several arrays described by one set of metadata, mdadm
creates an inactive arrive just like this which just holds the set of
devices, and then should create other arrays made of from different regions
of those devices.
It looks like mdadm hasn't done that you.  You can ask it to with:

  mdadm -I /dev/md/imsm0

That should created the real raid1 array in /dev/md/something.

NeilBrown


> 
> 
> ls -l /dev/md/imsm0 (showing contents of /dev/md/* [currently only one file/link ])
> 
> lrwxrwxrwx 1 root root 8 Nov  7 08:07 /dev/md/imsm0 -> ../md127
> 
> 
> ls -l /dev/md127 (showing the block device)
> 
> brw-rw---- 1 root disk 9, 127 Nov  7 08:07 /dev/md127
> 
> 
> 
> 
> It looks like I can not even access the md device the system created on boot. 
> 
> Does anyone have a guide or tips to migrating from the older dmraid to mdadm for fake-raid?
> 
> 
> fdisk -uc /dev/md127  (showing the block device is inaccessible)
> 
> Unable to read /dev/md127
> 
> 
> dmesg (pieces of dmesg/booting)
> 
> [    4.214092] device-mapper: uevent: version 1.0.3
> [    4.214495] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised: dm-devel@redhat.com
> [    5.509386] udev[446]: starting version 163
> [    7.181418] md: md127 stopped.
> [    7.183088] md: bind<sdb>
> [    7.183179] md: bind<sda>
> 
> 
> 
> update-initramfs -u (Perhaps the most interesting error of them all, I can confirm this occurs with a few different kernels)
> 
> update-initramfs: Generating /boot/initrd.img-2.6.32-5-xen-amd64
> mdadm: cannot open /dev/md/OneTB-RAID1-PV: No such file or directory
> 
> 
> Revised my information, inital thread on Debian-users thread at:
> http://lists.debian.org/debian-user/2010/11/msg01015.html
> 
> Thanks for any ones help :)
> 
> -M
>  		 	   		  --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re:....

[Young Chang]

May I ask if you would be eligible to pursue a Business Proposal of $19.7m with me if you dont mind? Let me know if you are interested?

Re:

[Phil Turmel]

Hi Dragon,

[Fixed subject line]

On 06/09/2011 02:50 AM, Dragon wrote:
> Hi Phil,
> i know that there is something odd with the raid, thats why i need help.
> No i didnt scamble the report. thats what the system output. Sorry for confusing with sdo, this is my usb disk and doesnt belong to the raid. because of the size i didnt have any backup ;(

Well, we don't know yet if your data is intact.  You might get lucky.  For what its worth now, you should know that raid5 isn't considered safe for arrays this size.  When the array is running 12 disks again, you might want to consider using the 13th to change your array to raid6.

> I do not let the system run 24/7 and as i started at in the morning the sequence has changed.

The SCSI driver stack in linux doesn't guarantee the order the drives get named.  And custom udev scripts could massage the names further.

>  fdisk -l |grep sd
> Disk /dev/sda: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdc: 20.4 GB, 20409532416 bytes
> /dev/sdc1   *           1        2372    19053058+  83  Linux
> /dev/sdc2            2373        2481      875542+   5  Extended
> /dev/sdc5            2373        2481      875511   82  Linux swap / Solaris
> Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sde: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdg: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdf: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdh: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdi: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdj: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdk: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdl: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdm: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdn: 1500.3 GB, 1500301910016 bytes
> Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
> Yesterday was the system on disk sdk. now its on sdc?! the system is now and up to the evening online.
> here the actual data of the drives again:
[...]
> 
> as far as i can see, now there is no error with a missing superblock of one disk.

Well, the superblocks indicate that the array is still configured for 13 drives, two of which are missing.  One of the missing drives has been misidentified as a spare, and the other missing drive ialso thinks it is a spare, but has not been attached.  With your most recent listing, they are /dev/sdd and /dev/sdn.

> 
> how can i download lsdrv with "wget"? Yes the way backwards by shrinking lead to the actual problem.

wget https://github.com/pturmel/lsdrv/raw/master/lsdrv
chmod +x lsdrv
./lsdrv

Phil

Re:

[Phil Turmel]

On 06/09/2011 08:16 AM, Dragon wrote:
> Yes if all things get back to normal i will change to raid6. that was my idea for the future too.
> here the result of the script:
> 
> ./lsdrv
> **Warning** The following utility(ies) failed to execute:
>   pvs
>   lvs
> Some information may be missing.
> 
> PCI [pata_atiixp] 00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller
>  ââscsi 0:0:0:0 ATA SAMSUNG HD154UI {S1XWJ1WZ401747}
>  â  ââsda: [8:0] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  â     ââmd0: [9:0] Empty/Unknown 0.00k
>  ââscsi 0:0:1:0 ATA SAMSUNG HD154UI {S1XWJ1WZ405098}
>  â  ââsdb: [8:16] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 1:0:0:0 ATA SAMSUNG SV2044D {0244J1BN626842}
>     ââsdc: [8:32] Partitioned (dos) 19.01g
>        ââsdc1: [8:33] (ext3) 18.17g {6858fc38-9fee-4ab5-8135-029f305b9198}
>        â  ââMounted as /dev/disk/by-uuid/6858fc38-9fee-4ab5-8135-029f305b9198 @ /
>        ââsdc2: [8:34] Partitioned (dos) 1.00k
>        ââsdc5: [8:37] (swap) 854.99m {f67c7f23-e5ac-4c05-992c-a9a494687026}
> PCI [sata_mv] 02:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02)
>  ââscsi 2:0:0:0 ATA SAMSUNG HD154UI {S1XWJD2Z907626}
>  â  ââsdd: [8:48] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 4:0:0:0 ATA SAMSUNG HD154UI {S1XWJ90ZA03442}
>  â  ââsde: [8:64] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 6:0:0:0 ATA SAMSUNG HD154UI {S1XWJ9AB200390}
>  â  ââsdf: [8:80] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 8:0:0:0 ATA SAMSUNG HD154UI {61833B761A63RP}
>     ââsdg: [8:96] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
> PCI [sata_promise] 04:02.0 Mass storage controller: Promise Technology, Inc. PDC40718 (SATA 300 TX4) (rev 02)
>  ââscsi 3:0:0:0 ATA SAMSUNG HD154UI {S1XWJD5B201174}
>  â  ââsdh: [8:112] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 5:0:0:0 ATA SAMSUNG HD154UI {S1XWJ9CB201815}
>  â  ââsdi: [8:128] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 7:x:x:x [Empty]
>  ââscsi 9:0:0:0 ATA SAMSUNG HD154UI {A6311B761A3XPB}
>     ââsdj: [8:144] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
> PCI [ahci] 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [IDE mode]
>  ââscsi 10:0:0:0 ATA SAMSUNG HD154UI {S1XWJ1KS915803}
>  â  ââsdk: [8:160] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 11:0:0:0 ATA SAMSUNG HD154UI {S1XWJ1KS915802}
>  â  ââsdl: [8:176] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 12:0:0:0 ATA SAMSUNG HD154UI {S1XWJ1KSC08024}
>  â  ââsdm: [8:192] MD raid5 (none/13) 1.36t md0 inactive spare {975d6eb2-285e-ed11-021d-f236c2d05073}
>  ââscsi 13:0:0:0 ATA SAMSUNG HD154UI {S1XWJ1KS915804}
>     ââsdn: [8:208] MD raid5 (13) 1.36t inactive {975d6eb2-285e-ed11-021d-f236c2d05073}
> 

Very interesting.  You've exposed a limitation of my script.  I'll have to reconsider how I extract information from members of a partially started array.

Its also clear that you are using a fast-boot kernel with parallel probing of your scsi hosts.  That's why your device names sometimes change.

/dev/sdn is definitely the holdout, though.  Notice the "(13)" where the others are "(none/13)".

Before continuing, I've made the assumption that "mdadm --grow -n 12" was the last major operation attempted, and this is was put you in your current predicament?  If so, and you interrupted it, did you try to assemble the array with the --backup-file option from the shrink operation?  If you didn't, please stop the array, and retry the assemble (with all 13 devices) and the --backup-file option.  Try twice, if needed, adding "--force" the second time.

If that works, sit tight until the reshape is complete.

If that was already tried, or doesn't change the situation, here's what I recommend:

Stop the array: "mdadm -S /dev/md0"

Recreate the array "mdadm -C /dev/md0 -l 5 -n 13 -e 0.90 -c 64 --assume-clean /dev/sd{k,d,l,m,a,b,e,n,f,g,h,i,j}"

The order in {} matters! The option "--assume-clean" is vital!

You will be warned that the members appear to be part of another array.  Continue.

Do *NOT* mount the array!

Try a non-destructive fsck: "fsck -n /dev/md0"

If that has a huge number of errors, stop the array, and recreate again, swapping /dev/sdd and /dev/sdn, then repeat the fsck:

"mdadm -C /dev/md0 -l 5 -n 13 -e 0.90 -c 64 --assume-clean /dev/sd{k,n,l,m,a,b,e,d,f,g,h,i,j}"

If you get a good, or mostly good fsck, you've found the right combination, and you can try the shrink operations again.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

(unknown)

[Dragon]

"No, it must be "Used Device Size" * 11 = 16116523456.  Try it without the 'k'."
-> was better:
mdadm /dev/md0 --grow --array-size=16116523456
mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Fri Jun 10 14:19:24 2011
     Raid Level : raid5
     Array Size : 16116523456 (15369.91 GiB 16503.32 GB)
  Used Dev Size : 1465138496 (1397.26 GiB 1500.30 GB)
   Raid Devices : 13
  Total Devices : 13
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Jun 10 16:49:37 2011
          State : clean
 Active Devices : 13
Working Devices : 13
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 8c4d8438:42aa49f9:a6d866f6:b6ea6b93 (local to host nassrv01)
         Events : 0.2

    Number   Major   Minor   RaidDevice State
       0       8      160        0      active sync   /dev/sdk
       1       8      208        1      active sync   /dev/sdn
       2       8      176        2      active sync   /dev/sdl
       3       8      192        3      active sync   /dev/sdm
       4       8        0        4      active sync   /dev/sda
       5       8       16        5      active sync   /dev/sdb
       6       8       64        6      active sync   /dev/sde
       7       8       48        7      active sync   /dev/sdd
       8       8       80        8      active sync   /dev/sdf
       9       8       96        9      active sync   /dev/sdg
      10       8      112       10      active sync   /dev/sdh
      11       8      128       11      active sync   /dev/sdi
      12       8      144       12      active sync   /dev/sdj

->fsck -n /dev/md0, was ok
->now:mdadm /dev/md0 --grow -n 12 --backup-file=/reshape.bak
->and after that, how become the disk out of the raid?
--

at this point i think i take the disk out of the raid, because i need the space of the disk.

I need another advise of you. While the computer is actualy build with 13 disk and i will become more data in the next month and the limit of power supply connecotors is reached i am looking forward to another solution. one possibility is to build up a better computer with more sata and sas connectors and add further raid-controller-cards. an other idea is to build a kind of cluster or dfs with two and later 3,4... computer. i read something about gluster.org. do you have a tip for me or experience in this?
-- 
NEU: FreePhone - kostenlos mobil telefonieren!			
Jetzt informieren: http://www.gmx.net/de/go/freephone


-- 
NEU: FreePhone - kostenlos mobil telefonieren!			
Jetzt informieren: http://www.gmx.net/de/go/freephone

Re:

[Phil Turmel]

On 06/10/2011 04:26 PM, Dragon wrote:
> "No, it must be "Used Device Size" * 11 = 16116523456.  Try it without the 'k'."
> -> was better:

[...]

> ->fsck -n /dev/md0, was ok
> ->now:mdadm /dev/md0 --grow -n 12 --backup-file=/reshape.bak
> ->and after that, how become the disk out of the raid?

Monitor your background reshape with "cat /proc/mdstat".

When the reshape is complete, the extra disk will be marked "spare".

Then you can use "mdadm --remove".

> at this point i think i take the disk out of the raid, because i need the space of the disk.

Understood, but you are living on the edge.  You have no backup, and only one drive of redundancy.  If one of your drives does fail, the odds of losing the whole array while replacing it is significant.  Your Samsung drives claim a non-recoverable read error rate of 1 per 1x10^15 bits.  Your eleven data disks contain 1.32x10^14 bits, all of which must be read during rebuild.  That means a _13%_ chance of total failure while replacing a failed drive.

I hope your 16T of data is not terribly important to you, or is otherwise replaceable.

> I need another advise of you. While the computer is actualy build with 13 disk and i will become more data in the next month and the limit of power supply connecotors is reached i am looking forward to another solution. one possibility is to build up a better computer with more sata and sas connectors and add further raid-controller-cards. an other idea is to build a kind of cluster or dfs with two and later 3,4... computer. i read something about gluster.org. do you have a tip for me or experience in this?

Unfortunately, no.  Although I skirt the edges in my engineering work, I'm primarily an end-user.  Both personal and work projects have relatively modest needs.  From the engineering side, I do recommend you spend extra on power supplies & UPS.

Phil

Re:

[Phil Turmel]

Hi Dragon,

On 06/18/2011 04:39 PM, Dragon wrote:
> Monitor your background reshape with "cat /proc/mdstat".
> 
> When the reshape is complete, the extra disk will be marked "spare".
> 
> Then you can use "mdadm --remove".
> -->after a view days the reshape was done and i take the disk out of the raid -> many thx for that

Good to hear.

>> at this point i think i take the disk out of the raid, because i need the space of
> the disk.
> 
> Understood, but you are living on the edge.  You have no backup, and only one drive
> of redundancy.  If one of your drives does fail, the odds of losing the whole array
> while replacing it is significant.  Your Samsung drives claim a non-recoverable read
> error rate of 1 per 1x10^15 bits.  Your eleven data disks contain 1.32x10^14 bits,
> all of which must be read during rebuild.  That means a _13%_ chance of total
> failure while replacing a failed drive.
> 
> I hope your 16T of data is not terribly important to you, or is otherwise replaceable.
> --> nice calculation, where do you have the data from?
> --> most of it is important, i will look for a better solution

The error rate is from Samsung, for your HD154UI drives:
http://www.samsung.com/latin_en/consumer/monitor-peripherals-printer/hard-disk-drives/internal/HD154UI/CKW/index.idx?pagetype=prd_detail&tab=specification

error rate = 1 / 1*10^15 = 1x10^-15

The rest comes from your setup:
11 disks * (1465138496 * 1024) bytes/disk * 8 bits/bytes = 1.32026560152e+14

% odds of failure = (data quantity * error rate) * 100%

[...]

> --> and than, ext4 max size is actually 16TB, what should i do?

I've been playing with XFS.  The only significant maintenance drawback I've identified is that it cannot be shrunk.  Not even offline.  It's not really holding me back, though, as I tend to layer LVM on top of my raid arrays, then allocate to specific volumes.  I always hold back a substantial fraction of the space for future use of "lvextend".

> --> for an end-user you have many knowledge about swraid ;)

Thank you.  I was a geek before I became an engineer :) .

Phil

Re:

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2814 bytes --]

On Sun, 25 Sep 2011 21:23:31 -0700 "Kenn" <kenn@kenn.us> wrote:

> I have a raid5 array that had a drive drop out, and resilvered the wrong
> drive when I put it back in, corrupting and destroying the raid.  I
> stopped the array at less than 1% resilvering and I'm in the process of
> making a dd-copy of the drive to recover the files.

I don't know what you mean by "resilvered".

> 
> (1) Is there anything diagnostic I can contribute to add more
> wrong-drive-resilvering protection to mdadm?  I have the command history
> showing everything I did, I have the five drives available for reading
> sectors, I haven't touched anything yet.

Yes, report the command history, and any relevant kernel logs, and the output
of "mdadm --examine" on all relevant devices.

NeilBrown


> 
> (2) Can I suggest improvements into resilvering?  Can I contribute code to
> implement them?  Such as resilver from the end of the drive back to the
> front, so if you notice the wrong drive resilvering, you can stop and not
> lose the MBR and the directory format structure that's stored in the first
> few sectors?  I'd also like to take a look at adding a raid mode where
> there's checksum in every stripe block so the system can detect corrupted
> disks and not resilver.  I'd also like to add a raid option where a
> resilvering need will be reported by email and needs to be started
> manually.  All to prevent what happened to me from happening again.
> 
> Thanks for your time.
> 
> Kenn Frank
> 
> P.S.  Setup:
> 
> # uname -a
> Linux teresa 2.6.26-2-686 #1 SMP Sat Jun 11 14:54:10 UTC 2011 i686 GNU/Linux
> 
> # mdadm --version
> mdadm - v2.6.7.2 - 14th November 2008
> 
> # mdadm --detail /dev/md3
> /dev/md3:
>         Version : 00.90
>   Creation Time : Thu Sep 22 16:23:50 2011
>      Raid Level : raid5
>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>    Raid Devices : 5
>   Total Devices : 4
> Preferred Minor : 3
>     Persistence : Superblock is persistent
> 
>     Update Time : Thu Sep 22 20:19:09 2011
>           State : clean, degraded
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
>          Events : 0.6
> 
>     Number   Major   Minor   RaidDevice State
>        0      33        1        0      active sync   /dev/hde1
>        1      56        1        1      active sync   /dev/hdi1
>        2       0        0        2      removed
>        3      57        1        3      active sync   /dev/hdk1
>        4      34        1        4      active sync   /dev/hdg1
> 
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re:

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]

On Mon, 26 Sep 2011 14:52:48 +1000
NeilBrown <neilb@suse.de> wrote:

> On Sun, 25 Sep 2011 21:23:31 -0700 "Kenn" <kenn@kenn.us> wrote:
> 
> > I have a raid5 array that had a drive drop out, and resilvered the wrong
> > drive when I put it back in, corrupting and destroying the raid.  I
> > stopped the array at less than 1% resilvering and I'm in the process of
> > making a dd-copy of the drive to recover the files.
> 
> I don't know what you mean by "resilvered".

At first I thought the initial poster just invented some peculiar funny word of his own, but it looks like it's from the ZFS circles:
https://encrypted.google.com/search?q=resilver+zfs
@Kenn; you probably mean 'resync' or 'rebuild', but no one ever calls those processes 'resilver' here, you'll get no google results and blank/unknowing/funny looks from people when using that term in relation to mdadm.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re:

[Kenn]

Replying.  I realize and I apologize I didn't create a subject.  I hope
this doesn't confuse majordomo.

> On Sun, 25 Sep 2011 21:23:31 -0700 "Kenn" <kenn@kenn.us> wrote:
>
>> I have a raid5 array that had a drive drop out, and resilvered the wrong
>> drive when I put it back in, corrupting and destroying the raid.  I
>> stopped the array at less than 1% resilvering and I'm in the process of
>> making a dd-copy of the drive to recover the files.
>
> I don't know what you mean by "resilvered".

Resilvering -- Rebuilding the array.  Lesser used term, sorry!

>
>>
>> (1) Is there anything diagnostic I can contribute to add more
>> wrong-drive-resilvering protection to mdadm?  I have the command history
>> showing everything I did, I have the five drives available for reading
>> sectors, I haven't touched anything yet.
>
> Yes, report the command history, and any relevant kernel logs, and the
> output
> of "mdadm --examine" on all relevant devices.
>
> NeilBrown

Awesome!  I hope this is useful.  It's really long, so I edited down the
logs and command history to what I thought were the important bits.  If
you want more, I can post unedited versions, please let me know.

### Command History ###

# The start of the sequence, removing sde from array
mdadm --examine /dev/sde
mdadm --detail /dev/md3
cat /proc/mdstat
mdadm /dev/md3 --remove /dev/sde1
mdadm /dev/md3 --remove /dev/sde
mdadm /dev/md3 --fail /dev/sde1
cat /proc/mdstat
mdadm --examine /dev/sde1
fdisk -l | grep 750
mdadm --examine /dev/sde1
mdadm --remove /dev/sde
mdadm /dev/md3 --remove /dev/sde
mdadm /dev/md3 --fail /dev/sde
fdisk /dev/sde
ls
vi /var/log/syslog
reboot
vi /var/log/syslog
reboot
mdadm --detail /dev/md3
mdadm --examine /dev/sde1
# Wiping sde
fdisk /dev/sde
newfs -t ext3 /dev/sde1
mkfs -t ext3 /dev/sde1
mkfs -t ext3 /dev/sde2
fdisk /dev/sde
mdadm --stop /dev/md3
# Putting sde back into array
mdadm --examine /dev/sde
mdadm --help
mdadm --misc --help
mdadm --zero-superblock /dev/sde
mdadm --query /dev/sde
mdadm --examine /dev/sde
mdadm --detail /dev/sde
mdadm --detail /dev/sde1
fdisk /dev/sde
mdadm --assemble --no-degraded /dev/md3  /dev/hde1 /dev/hdi1 /dev/sde1
/dev/hdk1 /dev/hdg1
cat /proc/mdstat
mdadm --stop /dev/md3
mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
missing /dev/hdk1 /dev/hdg1
mount -o ro /raid53
ls /raid53
umount /raid53
mdadm --stop /dev/md3
# The command that did the bad rebuild
mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
/dev/sde1 /dev/hdk1 /dev/hdg1
cat /proc/mdstat
mdadm --examine /dev/md3
mdadm --query /dev/md3
mdadm --detail /dev/md3
mount /raid53
mdadm --stop /dev/md3
# Trying to get the corrupted disk back up
mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
missing /dev/hdk1 /dev/hdg1
cat /proc/mdstat
mount /raid53
fsck -n /dev/md3



### KERNEL LOGS ###

# Me messing around with fdisk and mdadm creating new partitions to wipe
out sde
Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] 1465149168
512-byte hardware sectors (750156 MB)
Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Write
Protect is off
Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Mode
Sense: 00 3a 00 00
Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Write
cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 22 15:56:39 teresa kernel: [ 7897.778204]  sde: sde1 sde2
Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] 1465149168
512-byte hardware sectors (750156 MB)
Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Write
Protect is off
Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Mode
Sense: 00 3a 00 00
Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Write
cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 22 15:56:41 teresa kernel: [ 7899.848026]  sde: sde1 sde2
Sep 22 16:01:49 teresa kernel: [ 8207.733821] sd 5:0:0:0: [sde] 1465149168
512-byte hardware sectors (750156 MB)
Sep 22 16:01:49 teresa kernel: [ 8207.733919] sd 5:0:0:0: [sde] Write
Protect is off
Sep 22 16:01:49 teresa kernel: [ 8207.733943] sd 5:0:0:0: [sde] Mode
Sense: 00 3a 00 00
Sep 22 16:01:49 teresa kernel: [ 8207.734039] sd 5:0:0:0: [sde] Write
cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 22 16:01:49 teresa kernel: [ 8207.734083]  sde: sde1
Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] 1465149168
512-byte hardware sectors (750156 MB)
Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Write
Protect is off
Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Mode
Sense: 00 3a 00 00
Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Write
cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 22 16:01:51 teresa kernel: [ 8209.777260]  sde: sde1
Sep 22 16:02:09 teresa mdadm[2694]: DeviceDisappeared event detected on md
device /dev/md3
Sep 22 16:02:09 teresa kernel: [ 8227.781860] md: md3 stopped.
Sep 22 16:02:09 teresa kernel: [ 8227.781908] md: unbind<hde1>
Sep 22 16:02:09 teresa kernel: [ 8227.781937] md: export_rdev(hde1)
Sep 22 16:02:09 teresa kernel: [ 8227.782261] md: unbind<hdg1>
Sep 22 16:02:09 teresa kernel: [ 8227.782292] md: export_rdev(hdg1)
Sep 22 16:02:09 teresa kernel: [ 8227.782561] md: unbind<hdk1>
Sep 22 16:02:09 teresa kernel: [ 8227.782590] md: export_rdev(hdk1)
Sep 22 16:02:09 teresa kernel: [ 8227.782855] md: unbind<hdi1>
Sep 22 16:02:09 teresa kernel: [ 8227.782885] md: export_rdev(hdi1)
Sep 22 16:15:32 teresa smartd[2657]: Device: /dev/hda, Failed SMART usage
Attribute: 194 Temperature_Celsius.
Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/hdk, SMART Usage
Attribute: 194 Temperature_Celsius changed from 110 to 111
Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/sdb, SMART Usage
Attribute: 194 Temperature_Celsius changed from 113 to 116
Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/sdc, SMART Usage
Attribute: 190 Airflow_Temperature_Cel changed from 52 to 51
Sep 22 16:17:01 teresa /USR/SBIN/CRON[2965]: (root) CMD (   cd / &&
run-parts --report /etc/cron.hourly)
Sep 22 16:18:42 teresa kernel: [ 9220.400915] md: md3 stopped.
Sep 22 16:18:42 teresa kernel: [ 9220.411525] md: bind<hdi1>
Sep 22 16:18:42 teresa kernel: [ 9220.411884] md: bind<sde1>
Sep 22 16:18:42 teresa kernel: [ 9220.412577] md: bind<hdk1>
Sep 22 16:18:42 teresa kernel: [ 9220.413162] md: bind<hdg1>
Sep 22 16:18:42 teresa kernel: [ 9220.413750] md: bind<hde1>
Sep 22 16:18:42 teresa kernel: [ 9220.413855] md: kicking non-fresh sde1
from array!
Sep 22 16:18:42 teresa kernel: [ 9220.413887] md: unbind<sde1>
Sep 22 16:18:42 teresa kernel: [ 9220.413915] md: export_rdev(sde1)
Sep 22 16:18:42 teresa kernel: [ 9220.477393] raid5: device hde1
operational as raid disk 0
Sep 22 16:18:42 teresa kernel: [ 9220.477420] raid5: device hdg1
operational as raid disk 4
Sep 22 16:18:42 teresa kernel: [ 9220.477438] raid5: device hdk1
operational as raid disk 3
Sep 22 16:18:42 teresa kernel: [ 9220.477456] raid5: device hdi1
operational as raid disk 1
Sep 22 16:18:42 teresa kernel: [ 9220.478236] raid5: allocated 5252kB for md3
Sep 22 16:18:42 teresa kernel: [ 9220.478265] raid5: raid level 5 set md3
active with 4 out of 5 devices, algorithm 2
Sep 22 16:18:42 teresa kernel: [ 9220.478294] RAID5 conf printout:
Sep 22 16:18:42 teresa kernel: [ 9220.478309]  --- rd:5 wd:4
Sep 22 16:18:42 teresa kernel: [ 9220.478324]  disk 0, o:1, dev:hde1
Sep 22 16:18:42 teresa kernel: [ 9220.478339]  disk 1, o:1, dev:hdi1
Sep 22 16:18:42 teresa kernel: [ 9220.478354]  disk 3, o:1, dev:hdk1
Sep 22 16:18:42 teresa kernel: [ 9220.478369]  disk 4, o:1, dev:hdg1
# Me stopping md3
Sep 22 16:18:53 teresa mdadm[2694]: DeviceDisappeared event detected on md
device /dev/md3
Sep 22 16:18:53 teresa kernel: [ 9231.572348] md: md3 stopped.
Sep 22 16:18:53 teresa kernel: [ 9231.572394] md: unbind<hde1>
Sep 22 16:18:53 teresa kernel: [ 9231.572423] md: export_rdev(hde1)
Sep 22 16:18:53 teresa kernel: [ 9231.572728] md: unbind<hdg1>
Sep 22 16:18:53 teresa kernel: [ 9231.572758] md: export_rdev(hdg1)
Sep 22 16:18:53 teresa kernel: [ 9231.572988] md: unbind<hdk1>
Sep 22 16:18:53 teresa kernel: [ 9231.573015] md: export_rdev(hdk1)
Sep 22 16:18:53 teresa kernel: [ 9231.573243] md: unbind<hdi1>
Sep 22 16:18:53 teresa kernel: [ 9231.573270] md: export_rdev(hdi1)
# Me creating md3 with sde1 missing
Sep 22 16:19:51 teresa kernel: [ 9289.621646] md: bind<hde1>
Sep 22 16:19:51 teresa kernel: [ 9289.665268] md: bind<hdi1>
Sep 22 16:19:51 teresa kernel: [ 9289.695676] md: bind<hdk1>
Sep 22 16:19:51 teresa kernel: [ 9289.726906] md: bind<hdg1>
Sep 22 16:19:51 teresa kernel: [ 9289.809030] raid5: device hdg1
operational as raid disk 4
Sep 22 16:19:51 teresa kernel: [ 9289.809057] raid5: device hdk1
operational as raid disk 3
Sep 22 16:19:51 teresa kernel: [ 9289.809075] raid5: device hdi1
operational as raid disk 1
Sep 22 16:19:51 teresa kernel: [ 9289.809093] raid5: device hde1
operational as raid disk 0
Sep 22 16:19:51 teresa kernel: [ 9289.809821] raid5: allocated 5252kB for md3
Sep 22 16:19:51 teresa kernel: [ 9289.809850] raid5: raid level 5 set md3
active with 4 out of 5 devices, algorithm 2
Sep 22 16:19:51 teresa kernel: [ 9289.809877] RAID5 conf printout:
Sep 22 16:19:51 teresa kernel: [ 9289.809891]  --- rd:5 wd:4
Sep 22 16:19:51 teresa kernel: [ 9289.809907]  disk 0, o:1, dev:hde1
Sep 22 16:19:51 teresa kernel: [ 9289.809922]  disk 1, o:1, dev:hdi1
Sep 22 16:19:51 teresa kernel: [ 9289.809937]  disk 3, o:1, dev:hdk1
Sep 22 16:19:51 teresa kernel: [ 9289.809953]  disk 4, o:1, dev:hdg1
Sep 22 16:20:20 teresa kernel: [ 9318.486512] kjournald starting.  Commit
interval 5 seconds
Sep 22 16:20:20 teresa kernel: [ 9318.486512] EXT3-fs: mounted filesystem
with ordered data mode.
# Me stopping md3 again
Sep 22 16:20:42 teresa mdadm[2694]: DeviceDisappeared event detected on md
device /dev/md3
Sep 22 16:20:42 teresa kernel: [ 9340.300590] md: md3 stopped.
Sep 22 16:20:42 teresa kernel: [ 9340.300639] md: unbind<hdg1>
Sep 22 16:20:42 teresa kernel: [ 9340.300668] md: export_rdev(hdg1)
Sep 22 16:20:42 teresa kernel: [ 9340.300921] md: unbind<hdk1>
Sep 22 16:20:42 teresa kernel: [ 9340.300950] md: export_rdev(hdk1)
Sep 22 16:20:42 teresa kernel: [ 9340.301183] md: unbind<hdi1>
Sep 22 16:20:42 teresa kernel: [ 9340.301211] md: export_rdev(hdi1)
Sep 22 16:20:42 teresa kernel: [ 9340.301438] md: unbind<hde1>
Sep 22 16:20:42 teresa kernel: [ 9340.301465] md: export_rdev(hde1)
# This is me doing the fatal create, that recovers the wrong disk
Sep 22 16:21:39 teresa kernel: [ 9397.609864] md: bind<hde1>
Sep 22 16:21:39 teresa kernel: [ 9397.652426] md: bind<hdi1>
Sep 22 16:21:39 teresa kernel: [ 9397.673203] md: bind<sde1>
Sep 22 16:21:39 teresa kernel: [ 9397.699373] md: bind<hdk1>
Sep 22 16:21:39 teresa kernel: [ 9397.739372] md: bind<hdg1>
Sep 22 16:21:39 teresa kernel: [ 9397.801729] raid5: device hdk1
operational as raid disk 3
Sep 22 16:21:39 teresa kernel: [ 9397.801756] raid5: device sde1
operational as raid disk 2
Sep 22 16:21:39 teresa kernel: [ 9397.801774] raid5: device hdi1
operational as raid disk 1
Sep 22 16:21:39 teresa kernel: [ 9397.801793] raid5: device hde1
operational as raid disk 0
Sep 22 16:21:39 teresa kernel: [ 9397.802531] raid5: allocated 5252kB for md3
Sep 22 16:21:39 teresa kernel: [ 9397.802559] raid5: raid level 5 set md3
active with 4 out of 5 devices, algorithm 2
Sep 22 16:21:39 teresa kernel: [ 9397.802586] RAID5 conf printout:
Sep 22 16:21:39 teresa kernel: [ 9397.802600]  --- rd:5 wd:4
Sep 22 16:21:39 teresa kernel: [ 9397.802615]  disk 0, o:1, dev:hde1
Sep 22 16:21:39 teresa kernel: [ 9397.802631]  disk 1, o:1, dev:hdi1
Sep 22 16:21:39 teresa kernel: [ 9397.802646]  disk 2, o:1, dev:sde1
Sep 22 16:21:39 teresa kernel: [ 9397.802661]  disk 3, o:1, dev:hdk1
Sep 22 16:21:39 teresa kernel: [ 9397.838429] RAID5 conf printout:
Sep 22 16:21:39 teresa kernel: [ 9397.838454]  --- rd:5 wd:4
Sep 22 16:21:39 teresa kernel: [ 9397.838471]  disk 0, o:1, dev:hde1
Sep 22 16:21:39 teresa kernel: [ 9397.838486]  disk 1, o:1, dev:hdi1
Sep 22 16:21:39 teresa kernel: [ 9397.838502]  disk 2, o:1, dev:sde1
Sep 22 16:21:39 teresa kernel: [ 9397.838518]  disk 3, o:1, dev:hdk1
Sep 22 16:21:39 teresa kernel: [ 9397.838533]  disk 4, o:1, dev:hdg1
Sep 22 16:21:39 teresa mdadm[2694]: RebuildStarted event detected on md
device /dev/md3
Sep 22 16:21:39 teresa kernel: [ 9397.841822] md: recovery of RAID array md3
Sep 22 16:21:39 teresa kernel: [ 9397.841848] md: minimum _guaranteed_ 
speed: 1000 KB/sec/disk.
Sep 22 16:21:39 teresa kernel: [ 9397.841868] md: using maximum available
idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Sep 22 16:21:39 teresa kernel: [ 9397.841908] md: using 128k window, over
a total of 732571904 blocks.
Sep 22 16:22:33 teresa kernel: [ 9451.640192] EXT3-fs error (device md3):
ext3_check_descriptors: Block bitmap for group 3968 not in group (block
0)!
Sep 22 16:22:33 teresa kernel: [ 9451.750241] EXT3-fs: group descriptors
corrupted!
Sep 22 16:22:39 teresa kernel: [ 9458.079151] md: md_do_sync() got signal
... exiting
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: md3 stopped.
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdg1>
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdg1)
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdk1>
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdk1)
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<sde1>
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(sde1)
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdi1>
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdi1)
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hde1>
Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hde1)
Sep 22 16:22:39 teresa mdadm[2694]: DeviceDisappeared event detected on md
device /dev/md3
# Me trying to recreate md3 without sde
Sep 22 16:23:50 teresa kernel: [ 9529.065477] md: bind<hde1>
Sep 22 16:23:50 teresa kernel: [ 9529.107767] md: bind<hdi1>
Sep 22 16:23:50 teresa kernel: [ 9529.137743] md: bind<hdk1>
Sep 22 16:23:50 teresa kernel: [ 9529.177990] md: bind<hdg1>
Sep 22 16:23:51 teresa mdadm[2694]: RebuildFinished event detected on md
device /dev/md3
Sep 22 16:23:51 teresa kernel: [ 9529.240814] raid5: device hdg1
operational as raid disk 4
Sep 22 16:23:51 teresa kernel: [ 9529.241734] raid5: device hdk1
operational as raid disk 3
Sep 22 16:23:51 teresa kernel: [ 9529.241752] raid5: device hdi1
operational as raid disk 1
Sep 22 16:23:51 teresa kernel: [ 9529.241770] raid5: device hde1
operational as raid disk 0
Sep 22 16:23:51 teresa kernel: [ 9529.242520] raid5: allocated 5252kB for md3
Sep 22 16:23:51 teresa kernel: [ 9529.242547] raid5: raid level 5 set md3
active with 4 out of 5 devices, algorithm 2
Sep 22 16:23:51 teresa kernel: [ 9529.242574] RAID5 conf printout:
Sep 22 16:23:51 teresa kernel: [ 9529.242588]  --- rd:5 wd:4
Sep 22 16:23:51 teresa kernel: [ 9529.242603]  disk 0, o:1, dev:hde1
Sep 22 16:23:51 teresa kernel: [ 9529.242618]  disk 1, o:1, dev:hdi1
Sep 22 16:23:51 teresa kernel: [ 9529.242633]  disk 3, o:1, dev:hdk1
Sep 22 16:23:51 teresa kernel: [ 9529.242649]  disk 4, o:1, dev:hdg1
# And me trying a fsck -n or a mount
Sep 22 16:24:07 teresa kernel: [ 9545.326343] EXT3-fs error (device md3):
ext3_check_descriptors: Block bitmap for group 3968 not in group (block
0)!
Sep 22 16:24:07 teresa kernel: [ 9545.369071] EXT3-fs: group descriptors
corrupted!


### EXAMINES OF PARTITIONS ###

=== --examine /dev/hde1 ===
/dev/hde1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
  Creation Time : Thu Sep 22 16:23:50 2011
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
   Raid Devices : 5
  Total Devices : 4
Preferred Minor : 3

    Update Time : Sun Sep 25 22:11:22 2011
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : b7f6a3c0 - correct
         Events : 10

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0      33        1        0      active sync   /dev/hde1

   0     0      33        1        0      active sync   /dev/hde1
   1     1      56        1        1      active sync   /dev/hdi1
   2     2       0        0        2      faulty removed
   3     3      57        1        3      active sync   /dev/hdk1
   4     4      34        1        4      active sync   /dev/hdg1

=== --examine /dev/hdi1 ===
/dev/hdi1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
  Creation Time : Thu Sep 22 16:23:50 2011
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
   Raid Devices : 5
  Total Devices : 4
Preferred Minor : 3

    Update Time : Sun Sep 25 22:11:22 2011
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : b7f6a3d9 - correct
         Events : 10

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1      56        1        1      active sync   /dev/hdi1

   0     0      33        1        0      active sync   /dev/hde1
   1     1      56        1        1      active sync   /dev/hdi1
   2     2       0        0        2      faulty removed
   3     3      57        1        3      active sync   /dev/hdk1
   4     4      34        1        4      active sync   /dev/hdg1

=== --examine /dev/sde1 ===
/dev/sde1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : e6e3df36:1195239f:47f7b12e:9c2b2218 (local to host teresa)
  Creation Time : Thu Sep 22 16:21:39 2011
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 3

    Update Time : Thu Sep 22 16:22:39 2011
          State : clean
 Active Devices : 4
Working Devices : 5
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 4e69d679 - correct
         Events : 8

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       65        2      active sync   /dev/sde1

   0     0      33        1        0      active sync   /dev/hde1
   1     1      56        1        1      active sync   /dev/hdi1
   2     2       8       65        2      active sync   /dev/sde1
   3     3      57        1        3      active sync   /dev/hdk1
   4     4       0        0        4      faulty removed
   5     5      34        1        5      spare   /dev/hdg1

=== --examine /dev/hdk1 ===
/dev/hdk1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
  Creation Time : Thu Sep 22 16:23:50 2011
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
   Raid Devices : 5
  Total Devices : 4
Preferred Minor : 3

    Update Time : Sun Sep 25 22:11:22 2011
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : b7f6a3de - correct
         Events : 10

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3      57        1        3      active sync   /dev/hdk1

   0     0      33        1        0      active sync   /dev/hde1
   1     1      56        1        1      active sync   /dev/hdi1
   2     2       0        0        2      faulty removed
   3     3      57        1        3      active sync   /dev/hdk1
   4     4      34        1        4      active sync   /dev/hdg1

=== --examine /dev/hdg1 ===
/dev/hdg1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
  Creation Time : Thu Sep 22 16:23:50 2011
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
   Raid Devices : 5
  Total Devices : 4
Preferred Minor : 3

    Update Time : Sun Sep 25 22:11:22 2011
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 1
  Spare Devices : 0
       Checksum : b7f6a3c9 - correct
         Events : 10

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     4      34        1        4      active sync   /dev/hdg1

   0     0      33        1        0      active sync   /dev/hde1
   1     1      56        1        1      active sync   /dev/hdi1
   2     2       0        0        2      faulty removed
   3     3      57        1        3      active sync   /dev/hdk1
   4     4      34        1        4      active sync   /dev/hdg1




>
>
>>
>> (2) Can I suggest improvements into resilvering?  Can I contribute code
>> to
>> implement them?  Such as resilver from the end of the drive back to the
>> front, so if you notice the wrong drive resilvering, you can stop and
>> not
>> lose the MBR and the directory format structure that's stored in the
>> first
>> few sectors?  I'd also like to take a look at adding a raid mode where
>> there's checksum in every stripe block so the system can detect
>> corrupted
>> disks and not resilver.  I'd also like to add a raid option where a
>> resilvering need will be reported by email and needs to be started
>> manually.  All to prevent what happened to me from happening again.
>>
>> Thanks for your time.
>>
>> Kenn Frank
>>
>> P.S.  Setup:
>>
>> # uname -a
>> Linux teresa 2.6.26-2-686 #1 SMP Sat Jun 11 14:54:10 UTC 2011 i686
>> GNU/Linux
>>
>> # mdadm --version
>> mdadm - v2.6.7.2 - 14th November 2008
>>
>> # mdadm --detail /dev/md3
>> /dev/md3:
>>         Version : 00.90
>>   Creation Time : Thu Sep 22 16:23:50 2011
>>      Raid Level : raid5
>>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>>    Raid Devices : 5
>>   Total Devices : 4
>> Preferred Minor : 3
>>     Persistence : Superblock is persistent
>>
>>     Update Time : Thu Sep 22 20:19:09 2011
>>           State : clean, degraded
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 0
>>   Spare Devices : 0
>>
>>          Layout : left-symmetric
>>      Chunk Size : 64K
>>
>>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host
>> teresa)
>>          Events : 0.6
>>
>>     Number   Major   Minor   RaidDevice State
>>        0      33        1        0      active sync   /dev/hde1
>>        1      56        1        1      active sync   /dev/hdi1
>>        2       0        0        2      removed
>>        3      57        1        3      active sync   /dev/hdk1
>>        4      34        1        4      active sync   /dev/hdg1
>>
>>
>
>

Re:

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 26202 bytes --]

On Mon, 26 Sep 2011 00:42:23 -0700 "Kenn" <kenn@kenn.us> wrote:

> Replying.  I realize and I apologize I didn't create a subject.  I hope
> this doesn't confuse majordomo.
> 
> > On Sun, 25 Sep 2011 21:23:31 -0700 "Kenn" <kenn@kenn.us> wrote:
> >
> >> I have a raid5 array that had a drive drop out, and resilvered the wrong
> >> drive when I put it back in, corrupting and destroying the raid.  I
> >> stopped the array at less than 1% resilvering and I'm in the process of
> >> making a dd-copy of the drive to recover the files.
> >
> > I don't know what you mean by "resilvered".
> 
> Resilvering -- Rebuilding the array.  Lesser used term, sorry!

I see..

I guess that looking-glass mirrors have a silver backing and when it becomes
tarnished you might re-silver the mirror to make it better again.
So the name works as a poor pun for RAID1.  But I don't see how it applies
to RAID5....
No matter.

Basically you have messed up badly.
Recreating arrays should only be done as a last-ditch attempt to get data
back, and preferably with expert advice...

When you created the array with all devices present it effectively started
copying the corruption that you had deliberately (why??) placed on device 2
(sde) onto device 4 (counting from 0).
So now you have two devices that are corrupt in the early blocks.
There is not much you can do to fix that.

There is some chance that 'fsck' could find a backup superblock somewhere and
try to put the pieces back together.  But the 'mkfs' probably made a
substantial mess of important data structures so I don't consider you chances
very high.
Keeping sde out and just working with the remaining 4 is certainly your best
bet.

What made you think it would be a good idea to re-create the array when all
you wanted to do was trigger a resync/recovery??

NeilBrown


> 
> >
> >>
> >> (1) Is there anything diagnostic I can contribute to add more
> >> wrong-drive-resilvering protection to mdadm?  I have the command history
> >> showing everything I did, I have the five drives available for reading
> >> sectors, I haven't touched anything yet.
> >
> > Yes, report the command history, and any relevant kernel logs, and the
> > output
> > of "mdadm --examine" on all relevant devices.
> >
> > NeilBrown
> 
> Awesome!  I hope this is useful.  It's really long, so I edited down the
> logs and command history to what I thought were the important bits.  If
> you want more, I can post unedited versions, please let me know.
> 
> ### Command History ###
> 
> # The start of the sequence, removing sde from array
> mdadm --examine /dev/sde
> mdadm --detail /dev/md3
> cat /proc/mdstat
> mdadm /dev/md3 --remove /dev/sde1
> mdadm /dev/md3 --remove /dev/sde
> mdadm /dev/md3 --fail /dev/sde1
> cat /proc/mdstat
> mdadm --examine /dev/sde1
> fdisk -l | grep 750
> mdadm --examine /dev/sde1
> mdadm --remove /dev/sde
> mdadm /dev/md3 --remove /dev/sde
> mdadm /dev/md3 --fail /dev/sde
> fdisk /dev/sde
> ls
> vi /var/log/syslog
> reboot
> vi /var/log/syslog
> reboot
> mdadm --detail /dev/md3
> mdadm --examine /dev/sde1
> # Wiping sde
> fdisk /dev/sde
> newfs -t ext3 /dev/sde1
> mkfs -t ext3 /dev/sde1
> mkfs -t ext3 /dev/sde2
> fdisk /dev/sde
> mdadm --stop /dev/md3
> # Putting sde back into array
> mdadm --examine /dev/sde
> mdadm --help
> mdadm --misc --help
> mdadm --zero-superblock /dev/sde
> mdadm --query /dev/sde
> mdadm --examine /dev/sde
> mdadm --detail /dev/sde
> mdadm --detail /dev/sde1
> fdisk /dev/sde
> mdadm --assemble --no-degraded /dev/md3  /dev/hde1 /dev/hdi1 /dev/sde1
> /dev/hdk1 /dev/hdg1
> cat /proc/mdstat
> mdadm --stop /dev/md3
> mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
> missing /dev/hdk1 /dev/hdg1
> mount -o ro /raid53
> ls /raid53
> umount /raid53
> mdadm --stop /dev/md3
> # The command that did the bad rebuild
> mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
> /dev/sde1 /dev/hdk1 /dev/hdg1
> cat /proc/mdstat
> mdadm --examine /dev/md3
> mdadm --query /dev/md3
> mdadm --detail /dev/md3
> mount /raid53
> mdadm --stop /dev/md3
> # Trying to get the corrupted disk back up
> mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
> missing /dev/hdk1 /dev/hdg1
> cat /proc/mdstat
> mount /raid53
> fsck -n /dev/md3
> 
> 
> 
> ### KERNEL LOGS ###
> 
> # Me messing around with fdisk and mdadm creating new partitions to wipe
> out sde
> Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] 1465149168
> 512-byte hardware sectors (750156 MB)
> Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Write
> Protect is off
> Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Mode
> Sense: 00 3a 00 00
> Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Write
> cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Sep 22 15:56:39 teresa kernel: [ 7897.778204]  sde: sde1 sde2
> Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] 1465149168
> 512-byte hardware sectors (750156 MB)
> Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Write
> Protect is off
> Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Mode
> Sense: 00 3a 00 00
> Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Write
> cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Sep 22 15:56:41 teresa kernel: [ 7899.848026]  sde: sde1 sde2
> Sep 22 16:01:49 teresa kernel: [ 8207.733821] sd 5:0:0:0: [sde] 1465149168
> 512-byte hardware sectors (750156 MB)
> Sep 22 16:01:49 teresa kernel: [ 8207.733919] sd 5:0:0:0: [sde] Write
> Protect is off
> Sep 22 16:01:49 teresa kernel: [ 8207.733943] sd 5:0:0:0: [sde] Mode
> Sense: 00 3a 00 00
> Sep 22 16:01:49 teresa kernel: [ 8207.734039] sd 5:0:0:0: [sde] Write
> cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Sep 22 16:01:49 teresa kernel: [ 8207.734083]  sde: sde1
> Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] 1465149168
> 512-byte hardware sectors (750156 MB)
> Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Write
> Protect is off
> Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Mode
> Sense: 00 3a 00 00
> Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Write
> cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Sep 22 16:01:51 teresa kernel: [ 8209.777260]  sde: sde1
> Sep 22 16:02:09 teresa mdadm[2694]: DeviceDisappeared event detected on md
> device /dev/md3
> Sep 22 16:02:09 teresa kernel: [ 8227.781860] md: md3 stopped.
> Sep 22 16:02:09 teresa kernel: [ 8227.781908] md: unbind<hde1>
> Sep 22 16:02:09 teresa kernel: [ 8227.781937] md: export_rdev(hde1)
> Sep 22 16:02:09 teresa kernel: [ 8227.782261] md: unbind<hdg1>
> Sep 22 16:02:09 teresa kernel: [ 8227.782292] md: export_rdev(hdg1)
> Sep 22 16:02:09 teresa kernel: [ 8227.782561] md: unbind<hdk1>
> Sep 22 16:02:09 teresa kernel: [ 8227.782590] md: export_rdev(hdk1)
> Sep 22 16:02:09 teresa kernel: [ 8227.782855] md: unbind<hdi1>
> Sep 22 16:02:09 teresa kernel: [ 8227.782885] md: export_rdev(hdi1)
> Sep 22 16:15:32 teresa smartd[2657]: Device: /dev/hda, Failed SMART usage
> Attribute: 194 Temperature_Celsius.
> Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/hdk, SMART Usage
> Attribute: 194 Temperature_Celsius changed from 110 to 111
> Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/sdb, SMART Usage
> Attribute: 194 Temperature_Celsius changed from 113 to 116
> Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/sdc, SMART Usage
> Attribute: 190 Airflow_Temperature_Cel changed from 52 to 51
> Sep 22 16:17:01 teresa /USR/SBIN/CRON[2965]: (root) CMD (   cd / &&
> run-parts --report /etc/cron.hourly)
> Sep 22 16:18:42 teresa kernel: [ 9220.400915] md: md3 stopped.
> Sep 22 16:18:42 teresa kernel: [ 9220.411525] md: bind<hdi1>
> Sep 22 16:18:42 teresa kernel: [ 9220.411884] md: bind<sde1>
> Sep 22 16:18:42 teresa kernel: [ 9220.412577] md: bind<hdk1>
> Sep 22 16:18:42 teresa kernel: [ 9220.413162] md: bind<hdg1>
> Sep 22 16:18:42 teresa kernel: [ 9220.413750] md: bind<hde1>
> Sep 22 16:18:42 teresa kernel: [ 9220.413855] md: kicking non-fresh sde1
> from array!
> Sep 22 16:18:42 teresa kernel: [ 9220.413887] md: unbind<sde1>
> Sep 22 16:18:42 teresa kernel: [ 9220.413915] md: export_rdev(sde1)
> Sep 22 16:18:42 teresa kernel: [ 9220.477393] raid5: device hde1
> operational as raid disk 0
> Sep 22 16:18:42 teresa kernel: [ 9220.477420] raid5: device hdg1
> operational as raid disk 4
> Sep 22 16:18:42 teresa kernel: [ 9220.477438] raid5: device hdk1
> operational as raid disk 3
> Sep 22 16:18:42 teresa kernel: [ 9220.477456] raid5: device hdi1
> operational as raid disk 1
> Sep 22 16:18:42 teresa kernel: [ 9220.478236] raid5: allocated 5252kB for md3
> Sep 22 16:18:42 teresa kernel: [ 9220.478265] raid5: raid level 5 set md3
> active with 4 out of 5 devices, algorithm 2
> Sep 22 16:18:42 teresa kernel: [ 9220.478294] RAID5 conf printout:
> Sep 22 16:18:42 teresa kernel: [ 9220.478309]  --- rd:5 wd:4
> Sep 22 16:18:42 teresa kernel: [ 9220.478324]  disk 0, o:1, dev:hde1
> Sep 22 16:18:42 teresa kernel: [ 9220.478339]  disk 1, o:1, dev:hdi1
> Sep 22 16:18:42 teresa kernel: [ 9220.478354]  disk 3, o:1, dev:hdk1
> Sep 22 16:18:42 teresa kernel: [ 9220.478369]  disk 4, o:1, dev:hdg1
> # Me stopping md3
> Sep 22 16:18:53 teresa mdadm[2694]: DeviceDisappeared event detected on md
> device /dev/md3
> Sep 22 16:18:53 teresa kernel: [ 9231.572348] md: md3 stopped.
> Sep 22 16:18:53 teresa kernel: [ 9231.572394] md: unbind<hde1>
> Sep 22 16:18:53 teresa kernel: [ 9231.572423] md: export_rdev(hde1)
> Sep 22 16:18:53 teresa kernel: [ 9231.572728] md: unbind<hdg1>
> Sep 22 16:18:53 teresa kernel: [ 9231.572758] md: export_rdev(hdg1)
> Sep 22 16:18:53 teresa kernel: [ 9231.572988] md: unbind<hdk1>
> Sep 22 16:18:53 teresa kernel: [ 9231.573015] md: export_rdev(hdk1)
> Sep 22 16:18:53 teresa kernel: [ 9231.573243] md: unbind<hdi1>
> Sep 22 16:18:53 teresa kernel: [ 9231.573270] md: export_rdev(hdi1)
> # Me creating md3 with sde1 missing
> Sep 22 16:19:51 teresa kernel: [ 9289.621646] md: bind<hde1>
> Sep 22 16:19:51 teresa kernel: [ 9289.665268] md: bind<hdi1>
> Sep 22 16:19:51 teresa kernel: [ 9289.695676] md: bind<hdk1>
> Sep 22 16:19:51 teresa kernel: [ 9289.726906] md: bind<hdg1>
> Sep 22 16:19:51 teresa kernel: [ 9289.809030] raid5: device hdg1
> operational as raid disk 4
> Sep 22 16:19:51 teresa kernel: [ 9289.809057] raid5: device hdk1
> operational as raid disk 3
> Sep 22 16:19:51 teresa kernel: [ 9289.809075] raid5: device hdi1
> operational as raid disk 1
> Sep 22 16:19:51 teresa kernel: [ 9289.809093] raid5: device hde1
> operational as raid disk 0
> Sep 22 16:19:51 teresa kernel: [ 9289.809821] raid5: allocated 5252kB for md3
> Sep 22 16:19:51 teresa kernel: [ 9289.809850] raid5: raid level 5 set md3
> active with 4 out of 5 devices, algorithm 2
> Sep 22 16:19:51 teresa kernel: [ 9289.809877] RAID5 conf printout:
> Sep 22 16:19:51 teresa kernel: [ 9289.809891]  --- rd:5 wd:4
> Sep 22 16:19:51 teresa kernel: [ 9289.809907]  disk 0, o:1, dev:hde1
> Sep 22 16:19:51 teresa kernel: [ 9289.809922]  disk 1, o:1, dev:hdi1
> Sep 22 16:19:51 teresa kernel: [ 9289.809937]  disk 3, o:1, dev:hdk1
> Sep 22 16:19:51 teresa kernel: [ 9289.809953]  disk 4, o:1, dev:hdg1
> Sep 22 16:20:20 teresa kernel: [ 9318.486512] kjournald starting.  Commit
> interval 5 seconds
> Sep 22 16:20:20 teresa kernel: [ 9318.486512] EXT3-fs: mounted filesystem
> with ordered data mode.
> # Me stopping md3 again
> Sep 22 16:20:42 teresa mdadm[2694]: DeviceDisappeared event detected on md
> device /dev/md3
> Sep 22 16:20:42 teresa kernel: [ 9340.300590] md: md3 stopped.
> Sep 22 16:20:42 teresa kernel: [ 9340.300639] md: unbind<hdg1>
> Sep 22 16:20:42 teresa kernel: [ 9340.300668] md: export_rdev(hdg1)
> Sep 22 16:20:42 teresa kernel: [ 9340.300921] md: unbind<hdk1>
> Sep 22 16:20:42 teresa kernel: [ 9340.300950] md: export_rdev(hdk1)
> Sep 22 16:20:42 teresa kernel: [ 9340.301183] md: unbind<hdi1>
> Sep 22 16:20:42 teresa kernel: [ 9340.301211] md: export_rdev(hdi1)
> Sep 22 16:20:42 teresa kernel: [ 9340.301438] md: unbind<hde1>
> Sep 22 16:20:42 teresa kernel: [ 9340.301465] md: export_rdev(hde1)
> # This is me doing the fatal create, that recovers the wrong disk
> Sep 22 16:21:39 teresa kernel: [ 9397.609864] md: bind<hde1>
> Sep 22 16:21:39 teresa kernel: [ 9397.652426] md: bind<hdi1>
> Sep 22 16:21:39 teresa kernel: [ 9397.673203] md: bind<sde1>
> Sep 22 16:21:39 teresa kernel: [ 9397.699373] md: bind<hdk1>
> Sep 22 16:21:39 teresa kernel: [ 9397.739372] md: bind<hdg1>
> Sep 22 16:21:39 teresa kernel: [ 9397.801729] raid5: device hdk1
> operational as raid disk 3
> Sep 22 16:21:39 teresa kernel: [ 9397.801756] raid5: device sde1
> operational as raid disk 2
> Sep 22 16:21:39 teresa kernel: [ 9397.801774] raid5: device hdi1
> operational as raid disk 1
> Sep 22 16:21:39 teresa kernel: [ 9397.801793] raid5: device hde1
> operational as raid disk 0
> Sep 22 16:21:39 teresa kernel: [ 9397.802531] raid5: allocated 5252kB for md3
> Sep 22 16:21:39 teresa kernel: [ 9397.802559] raid5: raid level 5 set md3
> active with 4 out of 5 devices, algorithm 2
> Sep 22 16:21:39 teresa kernel: [ 9397.802586] RAID5 conf printout:
> Sep 22 16:21:39 teresa kernel: [ 9397.802600]  --- rd:5 wd:4
> Sep 22 16:21:39 teresa kernel: [ 9397.802615]  disk 0, o:1, dev:hde1
> Sep 22 16:21:39 teresa kernel: [ 9397.802631]  disk 1, o:1, dev:hdi1
> Sep 22 16:21:39 teresa kernel: [ 9397.802646]  disk 2, o:1, dev:sde1
> Sep 22 16:21:39 teresa kernel: [ 9397.802661]  disk 3, o:1, dev:hdk1
> Sep 22 16:21:39 teresa kernel: [ 9397.838429] RAID5 conf printout:
> Sep 22 16:21:39 teresa kernel: [ 9397.838454]  --- rd:5 wd:4
> Sep 22 16:21:39 teresa kernel: [ 9397.838471]  disk 0, o:1, dev:hde1
> Sep 22 16:21:39 teresa kernel: [ 9397.838486]  disk 1, o:1, dev:hdi1
> Sep 22 16:21:39 teresa kernel: [ 9397.838502]  disk 2, o:1, dev:sde1
> Sep 22 16:21:39 teresa kernel: [ 9397.838518]  disk 3, o:1, dev:hdk1
> Sep 22 16:21:39 teresa kernel: [ 9397.838533]  disk 4, o:1, dev:hdg1
> Sep 22 16:21:39 teresa mdadm[2694]: RebuildStarted event detected on md
> device /dev/md3
> Sep 22 16:21:39 teresa kernel: [ 9397.841822] md: recovery of RAID array md3
> Sep 22 16:21:39 teresa kernel: [ 9397.841848] md: minimum _guaranteed_ 
> speed: 1000 KB/sec/disk.
> Sep 22 16:21:39 teresa kernel: [ 9397.841868] md: using maximum available
> idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> Sep 22 16:21:39 teresa kernel: [ 9397.841908] md: using 128k window, over
> a total of 732571904 blocks.
> Sep 22 16:22:33 teresa kernel: [ 9451.640192] EXT3-fs error (device md3):
> ext3_check_descriptors: Block bitmap for group 3968 not in group (block
> 0)!
> Sep 22 16:22:33 teresa kernel: [ 9451.750241] EXT3-fs: group descriptors
> corrupted!
> Sep 22 16:22:39 teresa kernel: [ 9458.079151] md: md_do_sync() got signal
> ... exiting
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: md3 stopped.
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdg1>
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdg1)
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdk1>
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdk1)
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<sde1>
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(sde1)
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdi1>
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdi1)
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hde1>
> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hde1)
> Sep 22 16:22:39 teresa mdadm[2694]: DeviceDisappeared event detected on md
> device /dev/md3
> # Me trying to recreate md3 without sde
> Sep 22 16:23:50 teresa kernel: [ 9529.065477] md: bind<hde1>
> Sep 22 16:23:50 teresa kernel: [ 9529.107767] md: bind<hdi1>
> Sep 22 16:23:50 teresa kernel: [ 9529.137743] md: bind<hdk1>
> Sep 22 16:23:50 teresa kernel: [ 9529.177990] md: bind<hdg1>
> Sep 22 16:23:51 teresa mdadm[2694]: RebuildFinished event detected on md
> device /dev/md3
> Sep 22 16:23:51 teresa kernel: [ 9529.240814] raid5: device hdg1
> operational as raid disk 4
> Sep 22 16:23:51 teresa kernel: [ 9529.241734] raid5: device hdk1
> operational as raid disk 3
> Sep 22 16:23:51 teresa kernel: [ 9529.241752] raid5: device hdi1
> operational as raid disk 1
> Sep 22 16:23:51 teresa kernel: [ 9529.241770] raid5: device hde1
> operational as raid disk 0
> Sep 22 16:23:51 teresa kernel: [ 9529.242520] raid5: allocated 5252kB for md3
> Sep 22 16:23:51 teresa kernel: [ 9529.242547] raid5: raid level 5 set md3
> active with 4 out of 5 devices, algorithm 2
> Sep 22 16:23:51 teresa kernel: [ 9529.242574] RAID5 conf printout:
> Sep 22 16:23:51 teresa kernel: [ 9529.242588]  --- rd:5 wd:4
> Sep 22 16:23:51 teresa kernel: [ 9529.242603]  disk 0, o:1, dev:hde1
> Sep 22 16:23:51 teresa kernel: [ 9529.242618]  disk 1, o:1, dev:hdi1
> Sep 22 16:23:51 teresa kernel: [ 9529.242633]  disk 3, o:1, dev:hdk1
> Sep 22 16:23:51 teresa kernel: [ 9529.242649]  disk 4, o:1, dev:hdg1
> # And me trying a fsck -n or a mount
> Sep 22 16:24:07 teresa kernel: [ 9545.326343] EXT3-fs error (device md3):
> ext3_check_descriptors: Block bitmap for group 3968 not in group (block
> 0)!
> Sep 22 16:24:07 teresa kernel: [ 9545.369071] EXT3-fs: group descriptors
> corrupted!
> 
> 
> ### EXAMINES OF PARTITIONS ###
> 
> === --examine /dev/hde1 ===
> /dev/hde1:
>           Magic : a92b4efc
>         Version : 00.90.00
>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
>   Creation Time : Thu Sep 22 16:23:50 2011
>      Raid Level : raid5
>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>    Raid Devices : 5
>   Total Devices : 4
> Preferred Minor : 3
> 
>     Update Time : Sun Sep 25 22:11:22 2011
>           State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 1
>   Spare Devices : 0
>        Checksum : b7f6a3c0 - correct
>          Events : 10
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>       Number   Major   Minor   RaidDevice State
> this     0      33        1        0      active sync   /dev/hde1
> 
>    0     0      33        1        0      active sync   /dev/hde1
>    1     1      56        1        1      active sync   /dev/hdi1
>    2     2       0        0        2      faulty removed
>    3     3      57        1        3      active sync   /dev/hdk1
>    4     4      34        1        4      active sync   /dev/hdg1
> 
> === --examine /dev/hdi1 ===
> /dev/hdi1:
>           Magic : a92b4efc
>         Version : 00.90.00
>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
>   Creation Time : Thu Sep 22 16:23:50 2011
>      Raid Level : raid5
>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>    Raid Devices : 5
>   Total Devices : 4
> Preferred Minor : 3
> 
>     Update Time : Sun Sep 25 22:11:22 2011
>           State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 1
>   Spare Devices : 0
>        Checksum : b7f6a3d9 - correct
>          Events : 10
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>       Number   Major   Minor   RaidDevice State
> this     1      56        1        1      active sync   /dev/hdi1
> 
>    0     0      33        1        0      active sync   /dev/hde1
>    1     1      56        1        1      active sync   /dev/hdi1
>    2     2       0        0        2      faulty removed
>    3     3      57        1        3      active sync   /dev/hdk1
>    4     4      34        1        4      active sync   /dev/hdg1
> 
> === --examine /dev/sde1 ===
> /dev/sde1:
>           Magic : a92b4efc
>         Version : 00.90.00
>            UUID : e6e3df36:1195239f:47f7b12e:9c2b2218 (local to host teresa)
>   Creation Time : Thu Sep 22 16:21:39 2011
>      Raid Level : raid5
>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>    Raid Devices : 5
>   Total Devices : 5
> Preferred Minor : 3
> 
>     Update Time : Thu Sep 22 16:22:39 2011
>           State : clean
>  Active Devices : 4
> Working Devices : 5
>  Failed Devices : 1
>   Spare Devices : 1
>        Checksum : 4e69d679 - correct
>          Events : 8
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>       Number   Major   Minor   RaidDevice State
> this     2       8       65        2      active sync   /dev/sde1
> 
>    0     0      33        1        0      active sync   /dev/hde1
>    1     1      56        1        1      active sync   /dev/hdi1
>    2     2       8       65        2      active sync   /dev/sde1
>    3     3      57        1        3      active sync   /dev/hdk1
>    4     4       0        0        4      faulty removed
>    5     5      34        1        5      spare   /dev/hdg1
> 
> === --examine /dev/hdk1 ===
> /dev/hdk1:
>           Magic : a92b4efc
>         Version : 00.90.00
>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
>   Creation Time : Thu Sep 22 16:23:50 2011
>      Raid Level : raid5
>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>    Raid Devices : 5
>   Total Devices : 4
> Preferred Minor : 3
> 
>     Update Time : Sun Sep 25 22:11:22 2011
>           State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 1
>   Spare Devices : 0
>        Checksum : b7f6a3de - correct
>          Events : 10
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>       Number   Major   Minor   RaidDevice State
> this     3      57        1        3      active sync   /dev/hdk1
> 
>    0     0      33        1        0      active sync   /dev/hde1
>    1     1      56        1        1      active sync   /dev/hdi1
>    2     2       0        0        2      faulty removed
>    3     3      57        1        3      active sync   /dev/hdk1
>    4     4      34        1        4      active sync   /dev/hdg1
> 
> === --examine /dev/hdg1 ===
> /dev/hdg1:
>           Magic : a92b4efc
>         Version : 00.90.00
>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host teresa)
>   Creation Time : Thu Sep 22 16:23:50 2011
>      Raid Level : raid5
>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>    Raid Devices : 5
>   Total Devices : 4
> Preferred Minor : 3
> 
>     Update Time : Sun Sep 25 22:11:22 2011
>           State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 1
>   Spare Devices : 0
>        Checksum : b7f6a3c9 - correct
>          Events : 10
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>       Number   Major   Minor   RaidDevice State
> this     4      34        1        4      active sync   /dev/hdg1
> 
>    0     0      33        1        0      active sync   /dev/hde1
>    1     1      56        1        1      active sync   /dev/hdi1
>    2     2       0        0        2      faulty removed
>    3     3      57        1        3      active sync   /dev/hdk1
>    4     4      34        1        4      active sync   /dev/hdg1
> 
> 
> 
> 
> >
> >
> >>
> >> (2) Can I suggest improvements into resilvering?  Can I contribute code
> >> to
> >> implement them?  Such as resilver from the end of the drive back to the
> >> front, so if you notice the wrong drive resilvering, you can stop and
> >> not
> >> lose the MBR and the directory format structure that's stored in the
> >> first
> >> few sectors?  I'd also like to take a look at adding a raid mode where
> >> there's checksum in every stripe block so the system can detect
> >> corrupted
> >> disks and not resilver.  I'd also like to add a raid option where a
> >> resilvering need will be reported by email and needs to be started
> >> manually.  All to prevent what happened to me from happening again.
> >>
> >> Thanks for your time.
> >>
> >> Kenn Frank
> >>
> >> P.S.  Setup:
> >>
> >> # uname -a
> >> Linux teresa 2.6.26-2-686 #1 SMP Sat Jun 11 14:54:10 UTC 2011 i686
> >> GNU/Linux
> >>
> >> # mdadm --version
> >> mdadm - v2.6.7.2 - 14th November 2008
> >>
> >> # mdadm --detail /dev/md3
> >> /dev/md3:
> >>         Version : 00.90
> >>   Creation Time : Thu Sep 22 16:23:50 2011
> >>      Raid Level : raid5
> >>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
> >>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
> >>    Raid Devices : 5
> >>   Total Devices : 4
> >> Preferred Minor : 3
> >>     Persistence : Superblock is persistent
> >>
> >>     Update Time : Thu Sep 22 20:19:09 2011
> >>           State : clean, degraded
> >>  Active Devices : 4
> >> Working Devices : 4
> >>  Failed Devices : 0
> >>   Spare Devices : 0
> >>
> >>          Layout : left-symmetric
> >>      Chunk Size : 64K
> >>
> >>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host
> >> teresa)
> >>          Events : 0.6
> >>
> >>     Number   Major   Minor   RaidDevice State
> >>        0      33        1        0      active sync   /dev/hde1
> >>        1      56        1        1      active sync   /dev/hdi1
> >>        2       0        0        2      removed
> >>        3      57        1        3      active sync   /dev/hdk1
> >>        4      34        1        4      active sync   /dev/hdg1
> >>
> >>
> >
> >
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re:

[Kenn]

> On Mon, 26 Sep 2011 00:42:23 -0700 "Kenn" <kenn@kenn.us> wrote:
>
>> Replying.  I realize and I apologize I didn't create a subject.  I hope
>> this doesn't confuse majordomo.
>>
>> > On Sun, 25 Sep 2011 21:23:31 -0700 "Kenn" <kenn@kenn.us> wrote:
>> >
>> >> I have a raid5 array that had a drive drop out, and resilvered the
>> wrong
>> >> drive when I put it back in, corrupting and destroying the raid.  I
>> >> stopped the array at less than 1% resilvering and I'm in the process
>> of
>> >> making a dd-copy of the drive to recover the files.
>> >
>> > I don't know what you mean by "resilvered".
>>
>> Resilvering -- Rebuilding the array.  Lesser used term, sorry!
>
> I see..
>
> I guess that looking-glass mirrors have a silver backing and when it
> becomes
> tarnished you might re-silver the mirror to make it better again.
> So the name works as a poor pun for RAID1.  But I don't see how it applies
> to RAID5....
> No matter.
>
> Basically you have messed up badly.
> Recreating arrays should only be done as a last-ditch attempt to get data
> back, and preferably with expert advice...
>
> When you created the array with all devices present it effectively started
> copying the corruption that you had deliberately (why??) placed on device
> 2
> (sde) onto device 4 (counting from 0).
> So now you have two devices that are corrupt in the early blocks.
> There is not much you can do to fix that.
>
> There is some chance that 'fsck' could find a backup superblock somewhere
> and
> try to put the pieces back together.  But the 'mkfs' probably made a
> substantial mess of important data structures so I don't consider you
> chances
> very high.
> Keeping sde out and just working with the remaining 4 is certainly your
> best
> bet.
>
> What made you think it would be a good idea to re-create the array when
> all
> you wanted to do was trigger a resync/recovery??
>
> NeilBrown

Originally I had failed & removed sde from the array and then added it
back in, but no resilvering happened, it was just placed as raid device #
5 as an active (faulty?) spare, no rebuilding.  So I thought I'd have to
recreate the array to get it to rebuild.

Because my sde disk was only questionably healthy, if the problem was the
loose cable, I wanted to test the sde disk by having a complete rebuild
put onto it.   I was confident in all the other drives because when I
mounted the array without sde, I ran a complete md5sum scan and
everything's checksum was correct.  So I wanted to force a complete
rebuilding of the array on sde and the --zero-superblock was supposed to
render sde "new" to the array to force the rebuild onto sde.  I just did
the fsck and mkfs for good measure instead of spending the time of using
dd to zero every byte on the drive.  At the time because I thought if
--zero-superblock went wrong, md would reject a blank drive as a data
source for rebuilding and prevent resilvering.

So that brings up another point -- I've been reading through your blog,
and I acknowledge your thoughts on not having much benefit to checksums on
every block (http://neil.brown.name/blog/20110227114201), but sometimes
people like to having that extra lock on their door even though it takes
more effort to go in and out of their home.  In my five-drive array, if
the last five words were the checksums of the blocks on every drive, the
checksums off each drive could vote on trusting the blocks of every other
drive during the rebuild process, and prevent an idiot (me) from killing
his data.  It would force wasteful sectors on the drive, perhaps harm
performance by squeezing 2+n bytes out of each sector, but if someone
wants to protect their data as much as possible, it would be a welcome
option where performance is not a priority.

Also, the checksums do provide some protection: first, against against
partial media failure, which is a major flaw in raid 456 design according
to http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt , and checksum
voting could protect against the Atomicity/write-in-place flaw outlined in
http://en.wikipedia.org/wiki/RAID#Problems_with_RAID .

What do you think?

Kenn

>
>
>>
>> >
>> >>
>> >> (1) Is there anything diagnostic I can contribute to add more
>> >> wrong-drive-resilvering protection to mdadm?  I have the command
>> history
>> >> showing everything I did, I have the five drives available for
>> reading
>> >> sectors, I haven't touched anything yet.
>> >
>> > Yes, report the command history, and any relevant kernel logs, and the
>> > output
>> > of "mdadm --examine" on all relevant devices.
>> >
>> > NeilBrown
>>
>> Awesome!  I hope this is useful.  It's really long, so I edited down the
>> logs and command history to what I thought were the important bits.  If
>> you want more, I can post unedited versions, please let me know.
>>
>> ### Command History ###
>>
>> # The start of the sequence, removing sde from array
>> mdadm --examine /dev/sde
>> mdadm --detail /dev/md3
>> cat /proc/mdstat
>> mdadm /dev/md3 --remove /dev/sde1
>> mdadm /dev/md3 --remove /dev/sde
>> mdadm /dev/md3 --fail /dev/sde1
>> cat /proc/mdstat
>> mdadm --examine /dev/sde1
>> fdisk -l | grep 750
>> mdadm --examine /dev/sde1
>> mdadm --remove /dev/sde
>> mdadm /dev/md3 --remove /dev/sde
>> mdadm /dev/md3 --fail /dev/sde
>> fdisk /dev/sde
>> ls
>> vi /var/log/syslog
>> reboot
>> vi /var/log/syslog
>> reboot
>> mdadm --detail /dev/md3
>> mdadm --examine /dev/sde1
>> # Wiping sde
>> fdisk /dev/sde
>> newfs -t ext3 /dev/sde1
>> mkfs -t ext3 /dev/sde1
>> mkfs -t ext3 /dev/sde2
>> fdisk /dev/sde
>> mdadm --stop /dev/md3
>> # Putting sde back into array
>> mdadm --examine /dev/sde
>> mdadm --help
>> mdadm --misc --help
>> mdadm --zero-superblock /dev/sde
>> mdadm --query /dev/sde
>> mdadm --examine /dev/sde
>> mdadm --detail /dev/sde
>> mdadm --detail /dev/sde1
>> fdisk /dev/sde
>> mdadm --assemble --no-degraded /dev/md3  /dev/hde1 /dev/hdi1 /dev/sde1
>> /dev/hdk1 /dev/hdg1
>> cat /proc/mdstat
>> mdadm --stop /dev/md3
>> mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
>> missing /dev/hdk1 /dev/hdg1
>> mount -o ro /raid53
>> ls /raid53
>> umount /raid53
>> mdadm --stop /dev/md3
>> # The command that did the bad rebuild
>> mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
>> /dev/sde1 /dev/hdk1 /dev/hdg1
>> cat /proc/mdstat
>> mdadm --examine /dev/md3
>> mdadm --query /dev/md3
>> mdadm --detail /dev/md3
>> mount /raid53
>> mdadm --stop /dev/md3
>> # Trying to get the corrupted disk back up
>> mdadm --create /dev/md3 --level=5 --raid-devices=5  /dev/hde1 /dev/hdi1
>> missing /dev/hdk1 /dev/hdg1
>> cat /proc/mdstat
>> mount /raid53
>> fsck -n /dev/md3
>>
>>
>>
>> ### KERNEL LOGS ###
>>
>> # Me messing around with fdisk and mdadm creating new partitions to wipe
>> out sde
>> Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde]
>> 1465149168
>> 512-byte hardware sectors (750156 MB)
>> Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Write
>> Protect is off
>> Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Mode
>> Sense: 00 3a 00 00
>> Sep 22 15:56:39 teresa kernel: [ 7897.778204] sd 5:0:0:0: [sde] Write
>> cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Sep 22 15:56:39 teresa kernel: [ 7897.778204]  sde: sde1 sde2
>> Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde]
>> 1465149168
>> 512-byte hardware sectors (750156 MB)
>> Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Write
>> Protect is off
>> Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Mode
>> Sense: 00 3a 00 00
>> Sep 22 15:56:41 teresa kernel: [ 7899.848026] sd 5:0:0:0: [sde] Write
>> cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Sep 22 15:56:41 teresa kernel: [ 7899.848026]  sde: sde1 sde2
>> Sep 22 16:01:49 teresa kernel: [ 8207.733821] sd 5:0:0:0: [sde]
>> 1465149168
>> 512-byte hardware sectors (750156 MB)
>> Sep 22 16:01:49 teresa kernel: [ 8207.733919] sd 5:0:0:0: [sde] Write
>> Protect is off
>> Sep 22 16:01:49 teresa kernel: [ 8207.733943] sd 5:0:0:0: [sde] Mode
>> Sense: 00 3a 00 00
>> Sep 22 16:01:49 teresa kernel: [ 8207.734039] sd 5:0:0:0: [sde] Write
>> cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Sep 22 16:01:49 teresa kernel: [ 8207.734083]  sde: sde1
>> Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde]
>> 1465149168
>> 512-byte hardware sectors (750156 MB)
>> Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Write
>> Protect is off
>> Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Mode
>> Sense: 00 3a 00 00
>> Sep 22 16:01:51 teresa kernel: [ 8209.777260] sd 5:0:0:0: [sde] Write
>> cache: enabled, read cache: enabled, doesn't support DPO or FUA
>> Sep 22 16:01:51 teresa kernel: [ 8209.777260]  sde: sde1
>> Sep 22 16:02:09 teresa mdadm[2694]: DeviceDisappeared event detected on
>> md
>> device /dev/md3
>> Sep 22 16:02:09 teresa kernel: [ 8227.781860] md: md3 stopped.
>> Sep 22 16:02:09 teresa kernel: [ 8227.781908] md: unbind<hde1>
>> Sep 22 16:02:09 teresa kernel: [ 8227.781937] md: export_rdev(hde1)
>> Sep 22 16:02:09 teresa kernel: [ 8227.782261] md: unbind<hdg1>
>> Sep 22 16:02:09 teresa kernel: [ 8227.782292] md: export_rdev(hdg1)
>> Sep 22 16:02:09 teresa kernel: [ 8227.782561] md: unbind<hdk1>
>> Sep 22 16:02:09 teresa kernel: [ 8227.782590] md: export_rdev(hdk1)
>> Sep 22 16:02:09 teresa kernel: [ 8227.782855] md: unbind<hdi1>
>> Sep 22 16:02:09 teresa kernel: [ 8227.782885] md: export_rdev(hdi1)
>> Sep 22 16:15:32 teresa smartd[2657]: Device: /dev/hda, Failed SMART
>> usage
>> Attribute: 194 Temperature_Celsius.
>> Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/hdk, SMART Usage
>> Attribute: 194 Temperature_Celsius changed from 110 to 111
>> Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/sdb, SMART Usage
>> Attribute: 194 Temperature_Celsius changed from 113 to 116
>> Sep 22 16:15:33 teresa smartd[2657]: Device: /dev/sdc, SMART Usage
>> Attribute: 190 Airflow_Temperature_Cel changed from 52 to 51
>> Sep 22 16:17:01 teresa /USR/SBIN/CRON[2965]: (root) CMD (   cd / &&
>> run-parts --report /etc/cron.hourly)
>> Sep 22 16:18:42 teresa kernel: [ 9220.400915] md: md3 stopped.
>> Sep 22 16:18:42 teresa kernel: [ 9220.411525] md: bind<hdi1>
>> Sep 22 16:18:42 teresa kernel: [ 9220.411884] md: bind<sde1>
>> Sep 22 16:18:42 teresa kernel: [ 9220.412577] md: bind<hdk1>
>> Sep 22 16:18:42 teresa kernel: [ 9220.413162] md: bind<hdg1>
>> Sep 22 16:18:42 teresa kernel: [ 9220.413750] md: bind<hde1>
>> Sep 22 16:18:42 teresa kernel: [ 9220.413855] md: kicking non-fresh sde1
>> from array!
>> Sep 22 16:18:42 teresa kernel: [ 9220.413887] md: unbind<sde1>
>> Sep 22 16:18:42 teresa kernel: [ 9220.413915] md: export_rdev(sde1)
>> Sep 22 16:18:42 teresa kernel: [ 9220.477393] raid5: device hde1
>> operational as raid disk 0
>> Sep 22 16:18:42 teresa kernel: [ 9220.477420] raid5: device hdg1
>> operational as raid disk 4
>> Sep 22 16:18:42 teresa kernel: [ 9220.477438] raid5: device hdk1
>> operational as raid disk 3
>> Sep 22 16:18:42 teresa kernel: [ 9220.477456] raid5: device hdi1
>> operational as raid disk 1
>> Sep 22 16:18:42 teresa kernel: [ 9220.478236] raid5: allocated 5252kB
>> for md3
>> Sep 22 16:18:42 teresa kernel: [ 9220.478265] raid5: raid level 5 set
>> md3
>> active with 4 out of 5 devices, algorithm 2
>> Sep 22 16:18:42 teresa kernel: [ 9220.478294] RAID5 conf printout:
>> Sep 22 16:18:42 teresa kernel: [ 9220.478309]  --- rd:5 wd:4
>> Sep 22 16:18:42 teresa kernel: [ 9220.478324]  disk 0, o:1, dev:hde1
>> Sep 22 16:18:42 teresa kernel: [ 9220.478339]  disk 1, o:1, dev:hdi1
>> Sep 22 16:18:42 teresa kernel: [ 9220.478354]  disk 3, o:1, dev:hdk1
>> Sep 22 16:18:42 teresa kernel: [ 9220.478369]  disk 4, o:1, dev:hdg1
>> # Me stopping md3
>> Sep 22 16:18:53 teresa mdadm[2694]: DeviceDisappeared event detected on
>> md
>> device /dev/md3
>> Sep 22 16:18:53 teresa kernel: [ 9231.572348] md: md3 stopped.
>> Sep 22 16:18:53 teresa kernel: [ 9231.572394] md: unbind<hde1>
>> Sep 22 16:18:53 teresa kernel: [ 9231.572423] md: export_rdev(hde1)
>> Sep 22 16:18:53 teresa kernel: [ 9231.572728] md: unbind<hdg1>
>> Sep 22 16:18:53 teresa kernel: [ 9231.572758] md: export_rdev(hdg1)
>> Sep 22 16:18:53 teresa kernel: [ 9231.572988] md: unbind<hdk1>
>> Sep 22 16:18:53 teresa kernel: [ 9231.573015] md: export_rdev(hdk1)
>> Sep 22 16:18:53 teresa kernel: [ 9231.573243] md: unbind<hdi1>
>> Sep 22 16:18:53 teresa kernel: [ 9231.573270] md: export_rdev(hdi1)
>> # Me creating md3 with sde1 missing
>> Sep 22 16:19:51 teresa kernel: [ 9289.621646] md: bind<hde1>
>> Sep 22 16:19:51 teresa kernel: [ 9289.665268] md: bind<hdi1>
>> Sep 22 16:19:51 teresa kernel: [ 9289.695676] md: bind<hdk1>
>> Sep 22 16:19:51 teresa kernel: [ 9289.726906] md: bind<hdg1>
>> Sep 22 16:19:51 teresa kernel: [ 9289.809030] raid5: device hdg1
>> operational as raid disk 4
>> Sep 22 16:19:51 teresa kernel: [ 9289.809057] raid5: device hdk1
>> operational as raid disk 3
>> Sep 22 16:19:51 teresa kernel: [ 9289.809075] raid5: device hdi1
>> operational as raid disk 1
>> Sep 22 16:19:51 teresa kernel: [ 9289.809093] raid5: device hde1
>> operational as raid disk 0
>> Sep 22 16:19:51 teresa kernel: [ 9289.809821] raid5: allocated 5252kB
>> for md3
>> Sep 22 16:19:51 teresa kernel: [ 9289.809850] raid5: raid level 5 set
>> md3
>> active with 4 out of 5 devices, algorithm 2
>> Sep 22 16:19:51 teresa kernel: [ 9289.809877] RAID5 conf printout:
>> Sep 22 16:19:51 teresa kernel: [ 9289.809891]  --- rd:5 wd:4
>> Sep 22 16:19:51 teresa kernel: [ 9289.809907]  disk 0, o:1, dev:hde1
>> Sep 22 16:19:51 teresa kernel: [ 9289.809922]  disk 1, o:1, dev:hdi1
>> Sep 22 16:19:51 teresa kernel: [ 9289.809937]  disk 3, o:1, dev:hdk1
>> Sep 22 16:19:51 teresa kernel: [ 9289.809953]  disk 4, o:1, dev:hdg1
>> Sep 22 16:20:20 teresa kernel: [ 9318.486512] kjournald starting.
>> Commit
>> interval 5 seconds
>> Sep 22 16:20:20 teresa kernel: [ 9318.486512] EXT3-fs: mounted
>> filesystem
>> with ordered data mode.
>> # Me stopping md3 again
>> Sep 22 16:20:42 teresa mdadm[2694]: DeviceDisappeared event detected on
>> md
>> device /dev/md3
>> Sep 22 16:20:42 teresa kernel: [ 9340.300590] md: md3 stopped.
>> Sep 22 16:20:42 teresa kernel: [ 9340.300639] md: unbind<hdg1>
>> Sep 22 16:20:42 teresa kernel: [ 9340.300668] md: export_rdev(hdg1)
>> Sep 22 16:20:42 teresa kernel: [ 9340.300921] md: unbind<hdk1>
>> Sep 22 16:20:42 teresa kernel: [ 9340.300950] md: export_rdev(hdk1)
>> Sep 22 16:20:42 teresa kernel: [ 9340.301183] md: unbind<hdi1>
>> Sep 22 16:20:42 teresa kernel: [ 9340.301211] md: export_rdev(hdi1)
>> Sep 22 16:20:42 teresa kernel: [ 9340.301438] md: unbind<hde1>
>> Sep 22 16:20:42 teresa kernel: [ 9340.301465] md: export_rdev(hde1)
>> # This is me doing the fatal create, that recovers the wrong disk
>> Sep 22 16:21:39 teresa kernel: [ 9397.609864] md: bind<hde1>
>> Sep 22 16:21:39 teresa kernel: [ 9397.652426] md: bind<hdi1>
>> Sep 22 16:21:39 teresa kernel: [ 9397.673203] md: bind<sde1>
>> Sep 22 16:21:39 teresa kernel: [ 9397.699373] md: bind<hdk1>
>> Sep 22 16:21:39 teresa kernel: [ 9397.739372] md: bind<hdg1>
>> Sep 22 16:21:39 teresa kernel: [ 9397.801729] raid5: device hdk1
>> operational as raid disk 3
>> Sep 22 16:21:39 teresa kernel: [ 9397.801756] raid5: device sde1
>> operational as raid disk 2
>> Sep 22 16:21:39 teresa kernel: [ 9397.801774] raid5: device hdi1
>> operational as raid disk 1
>> Sep 22 16:21:39 teresa kernel: [ 9397.801793] raid5: device hde1
>> operational as raid disk 0
>> Sep 22 16:21:39 teresa kernel: [ 9397.802531] raid5: allocated 5252kB
>> for md3
>> Sep 22 16:21:39 teresa kernel: [ 9397.802559] raid5: raid level 5 set
>> md3
>> active with 4 out of 5 devices, algorithm 2
>> Sep 22 16:21:39 teresa kernel: [ 9397.802586] RAID5 conf printout:
>> Sep 22 16:21:39 teresa kernel: [ 9397.802600]  --- rd:5 wd:4
>> Sep 22 16:21:39 teresa kernel: [ 9397.802615]  disk 0, o:1, dev:hde1
>> Sep 22 16:21:39 teresa kernel: [ 9397.802631]  disk 1, o:1, dev:hdi1
>> Sep 22 16:21:39 teresa kernel: [ 9397.802646]  disk 2, o:1, dev:sde1
>> Sep 22 16:21:39 teresa kernel: [ 9397.802661]  disk 3, o:1, dev:hdk1
>> Sep 22 16:21:39 teresa kernel: [ 9397.838429] RAID5 conf printout:
>> Sep 22 16:21:39 teresa kernel: [ 9397.838454]  --- rd:5 wd:4
>> Sep 22 16:21:39 teresa kernel: [ 9397.838471]  disk 0, o:1, dev:hde1
>> Sep 22 16:21:39 teresa kernel: [ 9397.838486]  disk 1, o:1, dev:hdi1
>> Sep 22 16:21:39 teresa kernel: [ 9397.838502]  disk 2, o:1, dev:sde1
>> Sep 22 16:21:39 teresa kernel: [ 9397.838518]  disk 3, o:1, dev:hdk1
>> Sep 22 16:21:39 teresa kernel: [ 9397.838533]  disk 4, o:1, dev:hdg1
>> Sep 22 16:21:39 teresa mdadm[2694]: RebuildStarted event detected on md
>> device /dev/md3
>> Sep 22 16:21:39 teresa kernel: [ 9397.841822] md: recovery of RAID array
>> md3
>> Sep 22 16:21:39 teresa kernel: [ 9397.841848] md: minimum _guaranteed_
>> speed: 1000 KB/sec/disk.
>> Sep 22 16:21:39 teresa kernel: [ 9397.841868] md: using maximum
>> available
>> idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
>> Sep 22 16:21:39 teresa kernel: [ 9397.841908] md: using 128k window,
>> over
>> a total of 732571904 blocks.
>> Sep 22 16:22:33 teresa kernel: [ 9451.640192] EXT3-fs error (device
>> md3):
>> ext3_check_descriptors: Block bitmap for group 3968 not in group (block
>> 0)!
>> Sep 22 16:22:33 teresa kernel: [ 9451.750241] EXT3-fs: group descriptors
>> corrupted!
>> Sep 22 16:22:39 teresa kernel: [ 9458.079151] md: md_do_sync() got
>> signal
>> ... exiting
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: md3 stopped.
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdg1>
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdg1)
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdk1>
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdk1)
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<sde1>
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(sde1)
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hdi1>
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hdi1)
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: unbind<hde1>
>> Sep 22 16:22:39 teresa kernel: [ 9458.114590] md: export_rdev(hde1)
>> Sep 22 16:22:39 teresa mdadm[2694]: DeviceDisappeared event detected on
>> md
>> device /dev/md3
>> # Me trying to recreate md3 without sde
>> Sep 22 16:23:50 teresa kernel: [ 9529.065477] md: bind<hde1>
>> Sep 22 16:23:50 teresa kernel: [ 9529.107767] md: bind<hdi1>
>> Sep 22 16:23:50 teresa kernel: [ 9529.137743] md: bind<hdk1>
>> Sep 22 16:23:50 teresa kernel: [ 9529.177990] md: bind<hdg1>
>> Sep 22 16:23:51 teresa mdadm[2694]: RebuildFinished event detected on md
>> device /dev/md3
>> Sep 22 16:23:51 teresa kernel: [ 9529.240814] raid5: device hdg1
>> operational as raid disk 4
>> Sep 22 16:23:51 teresa kernel: [ 9529.241734] raid5: device hdk1
>> operational as raid disk 3
>> Sep 22 16:23:51 teresa kernel: [ 9529.241752] raid5: device hdi1
>> operational as raid disk 1
>> Sep 22 16:23:51 teresa kernel: [ 9529.241770] raid5: device hde1
>> operational as raid disk 0
>> Sep 22 16:23:51 teresa kernel: [ 9529.242520] raid5: allocated 5252kB
>> for md3
>> Sep 22 16:23:51 teresa kernel: [ 9529.242547] raid5: raid level 5 set
>> md3
>> active with 4 out of 5 devices, algorithm 2
>> Sep 22 16:23:51 teresa kernel: [ 9529.242574] RAID5 conf printout:
>> Sep 22 16:23:51 teresa kernel: [ 9529.242588]  --- rd:5 wd:4
>> Sep 22 16:23:51 teresa kernel: [ 9529.242603]  disk 0, o:1, dev:hde1
>> Sep 22 16:23:51 teresa kernel: [ 9529.242618]  disk 1, o:1, dev:hdi1
>> Sep 22 16:23:51 teresa kernel: [ 9529.242633]  disk 3, o:1, dev:hdk1
>> Sep 22 16:23:51 teresa kernel: [ 9529.242649]  disk 4, o:1, dev:hdg1
>> # And me trying a fsck -n or a mount
>> Sep 22 16:24:07 teresa kernel: [ 9545.326343] EXT3-fs error (device
>> md3):
>> ext3_check_descriptors: Block bitmap for group 3968 not in group (block
>> 0)!
>> Sep 22 16:24:07 teresa kernel: [ 9545.369071] EXT3-fs: group descriptors
>> corrupted!
>>
>>
>> ### EXAMINES OF PARTITIONS ###
>>
>> === --examine /dev/hde1 ===
>> /dev/hde1:
>>           Magic : a92b4efc
>>         Version : 00.90.00
>>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host
>> teresa)
>>   Creation Time : Thu Sep 22 16:23:50 2011
>>      Raid Level : raid5
>>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>>    Raid Devices : 5
>>   Total Devices : 4
>> Preferred Minor : 3
>>
>>     Update Time : Sun Sep 25 22:11:22 2011
>>           State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 1
>>   Spare Devices : 0
>>        Checksum : b7f6a3c0 - correct
>>          Events : 10
>>
>>          Layout : left-symmetric
>>      Chunk Size : 64K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     0      33        1        0      active sync   /dev/hde1
>>
>>    0     0      33        1        0      active sync   /dev/hde1
>>    1     1      56        1        1      active sync   /dev/hdi1
>>    2     2       0        0        2      faulty removed
>>    3     3      57        1        3      active sync   /dev/hdk1
>>    4     4      34        1        4      active sync   /dev/hdg1
>>
>> === --examine /dev/hdi1 ===
>> /dev/hdi1:
>>           Magic : a92b4efc
>>         Version : 00.90.00
>>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host
>> teresa)
>>   Creation Time : Thu Sep 22 16:23:50 2011
>>      Raid Level : raid5
>>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>>    Raid Devices : 5
>>   Total Devices : 4
>> Preferred Minor : 3
>>
>>     Update Time : Sun Sep 25 22:11:22 2011
>>           State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 1
>>   Spare Devices : 0
>>        Checksum : b7f6a3d9 - correct
>>          Events : 10
>>
>>          Layout : left-symmetric
>>      Chunk Size : 64K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     1      56        1        1      active sync   /dev/hdi1
>>
>>    0     0      33        1        0      active sync   /dev/hde1
>>    1     1      56        1        1      active sync   /dev/hdi1
>>    2     2       0        0        2      faulty removed
>>    3     3      57        1        3      active sync   /dev/hdk1
>>    4     4      34        1        4      active sync   /dev/hdg1
>>
>> === --examine /dev/sde1 ===
>> /dev/sde1:
>>           Magic : a92b4efc
>>         Version : 00.90.00
>>            UUID : e6e3df36:1195239f:47f7b12e:9c2b2218 (local to host
>> teresa)
>>   Creation Time : Thu Sep 22 16:21:39 2011
>>      Raid Level : raid5
>>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>>    Raid Devices : 5
>>   Total Devices : 5
>> Preferred Minor : 3
>>
>>     Update Time : Thu Sep 22 16:22:39 2011
>>           State : clean
>>  Active Devices : 4
>> Working Devices : 5
>>  Failed Devices : 1
>>   Spare Devices : 1
>>        Checksum : 4e69d679 - correct
>>          Events : 8
>>
>>          Layout : left-symmetric
>>      Chunk Size : 64K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     2       8       65        2      active sync   /dev/sde1
>>
>>    0     0      33        1        0      active sync   /dev/hde1
>>    1     1      56        1        1      active sync   /dev/hdi1
>>    2     2       8       65        2      active sync   /dev/sde1
>>    3     3      57        1        3      active sync   /dev/hdk1
>>    4     4       0        0        4      faulty removed
>>    5     5      34        1        5      spare   /dev/hdg1
>>
>> === --examine /dev/hdk1 ===
>> /dev/hdk1:
>>           Magic : a92b4efc
>>         Version : 00.90.00
>>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host
>> teresa)
>>   Creation Time : Thu Sep 22 16:23:50 2011
>>      Raid Level : raid5
>>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>>    Raid Devices : 5
>>   Total Devices : 4
>> Preferred Minor : 3
>>
>>     Update Time : Sun Sep 25 22:11:22 2011
>>           State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 1
>>   Spare Devices : 0
>>        Checksum : b7f6a3de - correct
>>          Events : 10
>>
>>          Layout : left-symmetric
>>      Chunk Size : 64K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     3      57        1        3      active sync   /dev/hdk1
>>
>>    0     0      33        1        0      active sync   /dev/hde1
>>    1     1      56        1        1      active sync   /dev/hdi1
>>    2     2       0        0        2      faulty removed
>>    3     3      57        1        3      active sync   /dev/hdk1
>>    4     4      34        1        4      active sync   /dev/hdg1
>>
>> === --examine /dev/hdg1 ===
>> /dev/hdg1:
>>           Magic : a92b4efc
>>         Version : 00.90.00
>>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host
>> teresa)
>>   Creation Time : Thu Sep 22 16:23:50 2011
>>      Raid Level : raid5
>>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>>    Raid Devices : 5
>>   Total Devices : 4
>> Preferred Minor : 3
>>
>>     Update Time : Sun Sep 25 22:11:22 2011
>>           State : clean
>>  Active Devices : 4
>> Working Devices : 4
>>  Failed Devices : 1
>>   Spare Devices : 0
>>        Checksum : b7f6a3c9 - correct
>>          Events : 10
>>
>>          Layout : left-symmetric
>>      Chunk Size : 64K
>>
>>       Number   Major   Minor   RaidDevice State
>> this     4      34        1        4      active sync   /dev/hdg1
>>
>>    0     0      33        1        0      active sync   /dev/hde1
>>    1     1      56        1        1      active sync   /dev/hdi1
>>    2     2       0        0        2      faulty removed
>>    3     3      57        1        3      active sync   /dev/hdk1
>>    4     4      34        1        4      active sync   /dev/hdg1
>>
>>
>>
>>
>> >
>> >
>> >>
>> >> (2) Can I suggest improvements into resilvering?  Can I contribute
>> code
>> >> to
>> >> implement them?  Such as resilver from the end of the drive back to
>> the
>> >> front, so if you notice the wrong drive resilvering, you can stop and
>> >> not
>> >> lose the MBR and the directory format structure that's stored in the
>> >> first
>> >> few sectors?  I'd also like to take a look at adding a raid mode
>> where
>> >> there's checksum in every stripe block so the system can detect
>> >> corrupted
>> >> disks and not resilver.  I'd also like to add a raid option where a
>> >> resilvering need will be reported by email and needs to be started
>> >> manually.  All to prevent what happened to me from happening again.
>> >>
>> >> Thanks for your time.
>> >>
>> >> Kenn Frank
>> >>
>> >> P.S.  Setup:
>> >>
>> >> # uname -a
>> >> Linux teresa 2.6.26-2-686 #1 SMP Sat Jun 11 14:54:10 UTC 2011 i686
>> >> GNU/Linux
>> >>
>> >> # mdadm --version
>> >> mdadm - v2.6.7.2 - 14th November 2008
>> >>
>> >> # mdadm --detail /dev/md3
>> >> /dev/md3:
>> >>         Version : 00.90
>> >>   Creation Time : Thu Sep 22 16:23:50 2011
>> >>      Raid Level : raid5
>> >>      Array Size : 2930287616 (2794.54 GiB 3000.61 GB)
>> >>   Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
>> >>    Raid Devices : 5
>> >>   Total Devices : 4
>> >> Preferred Minor : 3
>> >>     Persistence : Superblock is persistent
>> >>
>> >>     Update Time : Thu Sep 22 20:19:09 2011
>> >>           State : clean, degraded
>> >>  Active Devices : 4
>> >> Working Devices : 4
>> >>  Failed Devices : 0
>> >>   Spare Devices : 0
>> >>
>> >>          Layout : left-symmetric
>> >>      Chunk Size : 64K
>> >>
>> >>            UUID : ed1e6357:74e32684:47f7b12e:9c2b2218 (local to host
>> >> teresa)
>> >>          Events : 0.6
>> >>
>> >>     Number   Major   Minor   RaidDevice State
>> >>        0      33        1        0      active sync   /dev/hde1
>> >>        1      56        1        1      active sync   /dev/hdi1
>> >>        2       0        0        2      removed
>> >>        3      57        1        3      active sync   /dev/hdk1
>> >>        4      34        1        4      active sync   /dev/hdg1
>> >>
>> >>
>> >
>> >
>>
>
>

Re:

[David Brown]

On 26/09/11 20:04, Kenn wrote:
>> On Mon, 26 Sep 2011 00:42:23 -0700 "Kenn"<kenn@kenn.us>  wrote:
>>
>>> Replying.  I realize and I apologize I didn't create a subject.  I hope
>>> this doesn't confuse majordomo.
>>>
>>>> On Sun, 25 Sep 2011 21:23:31 -0700 "Kenn"<kenn@kenn.us>  wrote:
>>>>
>>>>> I have a raid5 array that had a drive drop out, and resilvered the
>>> wrong
>>>>> drive when I put it back in, corrupting and destroying the raid.  I
>>>>> stopped the array at less than 1% resilvering and I'm in the process
>>> of
>>>>> making a dd-copy of the drive to recover the files.
>>>>
>>>> I don't know what you mean by "resilvered".
>>>
>>> Resilvering -- Rebuilding the array.  Lesser used term, sorry!
>>
>> I see..
>>
>> I guess that looking-glass mirrors have a silver backing and when it
>> becomes
>> tarnished you might re-silver the mirror to make it better again.
>> So the name works as a poor pun for RAID1.  But I don't see how it applies
>> to RAID5....
>> No matter.
>>
>> Basically you have messed up badly.
>> Recreating arrays should only be done as a last-ditch attempt to get data
>> back, and preferably with expert advice...
>>
>> When you created the array with all devices present it effectively started
>> copying the corruption that you had deliberately (why??) placed on device
>> 2
>> (sde) onto device 4 (counting from 0).
>> So now you have two devices that are corrupt in the early blocks.
>> There is not much you can do to fix that.
>>
>> There is some chance that 'fsck' could find a backup superblock somewhere
>> and
>> try to put the pieces back together.  But the 'mkfs' probably made a
>> substantial mess of important data structures so I don't consider you
>> chances
>> very high.
>> Keeping sde out and just working with the remaining 4 is certainly your
>> best
>> bet.
>>
>> What made you think it would be a good idea to re-create the array when
>> all
>> you wanted to do was trigger a resync/recovery??
>>
>> NeilBrown
>
> Originally I had failed&  removed sde from the array and then added it
> back in, but no resilvering happened, it was just placed as raid device #
> 5 as an active (faulty?) spare, no rebuilding.  So I thought I'd have to
> recreate the array to get it to rebuild.
>
> Because my sde disk was only questionably healthy, if the problem was the
> loose cable, I wanted to test the sde disk by having a complete rebuild
> put onto it.   I was confident in all the other drives because when I
> mounted the array without sde, I ran a complete md5sum scan and
> everything's checksum was correct.  So I wanted to force a complete
> rebuilding of the array on sde and the --zero-superblock was supposed to
> render sde "new" to the array to force the rebuild onto sde.  I just did
> the fsck and mkfs for good measure instead of spending the time of using
> dd to zero every byte on the drive.  At the time because I thought if
> --zero-superblock went wrong, md would reject a blank drive as a data
> source for rebuilding and prevent resilvering.
>
> So that brings up another point -- I've been reading through your blog,
> and I acknowledge your thoughts on not having much benefit to checksums on
> every block (http://neil.brown.name/blog/20110227114201), but sometimes
> people like to having that extra lock on their door even though it takes
> more effort to go in and out of their home.  In my five-drive array, if
> the last five words were the checksums of the blocks on every drive, the
> checksums off each drive could vote on trusting the blocks of every other
> drive during the rebuild process, and prevent an idiot (me) from killing
> his data.  It would force wasteful sectors on the drive, perhaps harm
> performance by squeezing 2+n bytes out of each sector, but if someone
> wants to protect their data as much as possible, it would be a welcome
> option where performance is not a priority.
>
> Also, the checksums do provide some protection: first, against against
> partial media failure, which is a major flaw in raid 456 design according
> to http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt , and checksum
> voting could protect against the Atomicity/write-in-place flaw outlined in
> http://en.wikipedia.org/wiki/RAID#Problems_with_RAID .
>
> What do you think?
>
> Kenn

/raid/ protects against partial media flaws.  If one disk in a raid5 
stripe has a bad sector, that sector will be ignored and the missing 
data will be re-created from the other disks using the raid recovery 
algorithm.  If you want to have such protection even when doing a resync 
(as many people do), then use raid6 - it has two parity blocks.

As Neil points out in his blog, it is impossible to fully recover from a 
failure part way through a write - checksum voting or majority voting 
/may/ give you the right answer, but it may not.  If you need protection 
against that, you have to have filesystem level control (data logging 
and journalling as well as metafile journalling), or perhaps use raid 
systems with battery backed write caches.

Re:

[Kenn]

> On Mon, 26 Sep 2011 14:52:48 +1000
> NeilBrown <neilb@suse.de> wrote:
>
>> On Sun, 25 Sep 2011 21:23:31 -0700 "Kenn" <kenn@kenn.us> wrote:
>>
>> > I have a raid5 array that had a drive drop out, and resilvered the
>> wrong
>> > drive when I put it back in, corrupting and destroying the raid.  I
>> > stopped the array at less than 1% resilvering and I'm in the process
>> of
>> > making a dd-copy of the drive to recover the files.
>>
>> I don't know what you mean by "resilvered".
>
> At first I thought the initial poster just invented some peculiar funny
> word of his own, but it looks like it's from the ZFS circles:
> https://encrypted.google.com/search?q=resilver+zfs
> @Kenn; you probably mean 'resync' or 'rebuild', but no one ever calls
> those processes 'resilver' here, you'll get no google results and
> blank/unknowing/funny looks from people when using that term in relation
> to mdadm.

Good point, I am a very old unix user and my RAID terminology hasn't been
properly updated since college.  Resilver is mentioned here in wikipedia
for disk mirroring http://en.wikipedia.org/wiki/Disk_mirroring and I've
always used the word but it's not in the RAID page and I'll switch to
"rebuilding".

Thanks,
Kenn


>
> --
> With respect,
> Roman
>

Re:

[Phil Turmel]

Hi Maik,

On 12/16/2012 07:59 PM, Maik Purwin wrote:
> Hello,
> i make a misstake and disconnected 2 of my 6 disk in a software raid 5 on
> debian squeeze. After that the two disks reported as missing and spare so
> i have 4 on 4 in raid5.
> 
> after that i tried to add and re-add but without no efforts. Then i do this:
> 
> mdadm --assemble /dev/md2 --scan --force
> mdadm: failed to add /dev/sdd4 to /dev/md2: Device or resource busy
> mdadm: /dev/md2 assembled from 4 drives and 1 spare - not enough to start
> the array.
> 
> and now i didnt know to go on. i have fear to setup the raid new. I hope
> you can help.

You are in the right place.

Before doing anything else, it is vital that you collect and show
critical data about your array.

First, show the output of "mdadm -D /dev/md2"

Then, for all of the partitions involved, show "mdadm -E /dev/sdXN"

Finally, show "cat /proc/mdstat" and "dmesg".

Don't try to post them on a website--just make a big text e-mail.

Phil

Re:

[Phil Turmel]

On 12/24/2012 07:12 PM, bobzer wrote:
> Hi everyone,
> 
> i don't understand what happend (like i did nothing)
> the file look like there are here, i can browse, but can't read or copy

Two of your array members are failed.  Raid5 can only loose one.

> i'm sure the problem is obvious :
> 
> mdadm --detail /dev/md0
> /dev/md0:
>         Version : 1.2
>   Creation Time : Sun Mar  4 22:49:14 2012
>      Raid Level : raid5
>      Array Size : 3907021568 (3726.03 GiB 4000.79 GB)
>   Used Dev Size : 1953510784 (1863.01 GiB 2000.40 GB)
>    Raid Devices : 3
>   Total Devices : 3
>     Persistence : Superblock is persistent
> 
>     Update Time : Mon Dec 24 18:51:53 2012
>           State : clean, FAILED
>  Active Devices : 1
> Working Devices : 1
>  Failed Devices : 2
>   Spare Devices : 0
> 
>          Layout : left-symmetric
>      Chunk Size : 128K
> 
>            Name : debian:0  (local to host debian)
>            UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>          Events : 409
> 
>     Number   Major   Minor   RaidDevice State
>        3       8       17        0      active sync   /dev/sdb1
>        1       0        0        1      removed
>        2       0        0        2      removed
> 
>        1       8       33        -      faulty spare   /dev/sdc1
>        2       8       49        -      faulty spare   /dev/sdd1

It would be good to know *why* they failed, and in what order.

Please post your "dmesg", and the output of "mdadm -E /dev/sd[bcd]1".

> ls /dev/sd*
> /dev/sda  /dev/sda1  /dev/sda2  /dev/sda5  /dev/sda6  /dev/sda7
> /dev/sdb  /dev/sdb1  /dev/sdc  /dev/sdc1  /dev/sdd  /dev/sdd1
> 
> i thought about :
> mdadm --stop /dev/md0
> mdadm --assemble --force /dev/md0 /dev/sd[bcd]1

It'll be something like this.  Depends on the sequence of failures.

> but i don't know what i should do :-(
> thank you for your help
> 
> merry christmas

And to you. :-)

Phil

Re:

[Phil Turmel]

On 12/25/2012 07:16 PM, bobzer wrote:
> thanks to help me

No problem, but please *don't* top-post, and *do* trim replies.  Also,
use reply-to-all on kernel.org mailing lists.

> root@debian:~# mdadm -E /dev/sd[bcd]1
> /dev/sdb1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>            Name : debian:0  (local to host debian)
>   Creation Time : Sun Mar  4 22:49:14 2012
>      Raid Level : raid5
>    Raid Devices : 3
> 
>  Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>      Array Size : 7814043136 (3726.03 GiB 4000.79 GB)
>   Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : 5e71f69a:a78b0cd7:bbbb7ecb:cf81f9f6
> 
>     Update Time : Tue Dec 25 06:25:02 2012
>   Bad Block Log : 512 entries available at offset 2032 sectors
>        Checksum : 922ddaa8 - correct
>          Events : 413
> 
>          Layout : left-symmetric
>      Chunk Size : 128K
> 
>    Device Role : Active device 0
>    Array State : A.. ('A' == active, '.' == missing)
> mdadm: No md superblock detected on /dev/sdc1.
> mdadm: No md superblock detected on /dev/sdd1.

This is bad.  The two disks are still offline.  You must find and fix
the hardware problem that is keeping these two disks from communicating.
 Unless the two disks suffered a simultaneous power surge, the odds they
are OK is good.  But look at your cables, controller, or power supply.

> I would like to understand too
> dmesg (with the begin removed) show a lot of error :
> http://pastebin.com/D1D8AKF9

I browsed it: all attempts to communicate with those two drives failed.
 That must be fixed first.  Then we can help you recover the data.  If
you can plug the three drives into another machine, that would be the
simplest way to isolate the problem.

HTH,

Phil

Re:

[bobzer]

thank you
i just reboot to see the status of all my disk
and i saw that all the superblock didn't say the same thing
so i did :
mdadm --stop /dev/md0
mdadm --assemble --force /dev/md0 /dev/sd[bcd]1

and now it's perfectly working
thanks

i'm currently looking for how to do a good monitoring

On Tue, Dec 25, 2012 at 9:15 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/25/2012 07:16 PM, bobzer wrote:
>> thanks to help me
>
> No problem, but please *don't* top-post, and *do* trim replies.  Also,
> use reply-to-all on kernel.org mailing lists.
>
>> root@debian:~# mdadm -E /dev/sd[bcd]1
>> /dev/sdb1:
>>           Magic : a92b4efc
>>         Version : 1.2
>>     Feature Map : 0x0
>>      Array UUID : bf3c605b:9699aa55:d45119a2:7ba58d56
>>            Name : debian:0  (local to host debian)
>>   Creation Time : Sun Mar  4 22:49:14 2012
>>      Raid Level : raid5
>>    Raid Devices : 3
>>
>>  Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>>      Array Size : 7814043136 (3726.03 GiB 4000.79 GB)
>>   Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>>     Data Offset : 2048 sectors
>>    Super Offset : 8 sectors
>>           State : clean
>>     Device UUID : 5e71f69a:a78b0cd7:bbbb7ecb:cf81f9f6
>>
>>     Update Time : Tue Dec 25 06:25:02 2012
>>   Bad Block Log : 512 entries available at offset 2032 sectors
>>        Checksum : 922ddaa8 - correct
>>          Events : 413
>>
>>          Layout : left-symmetric
>>      Chunk Size : 128K
>>
>>    Device Role : Active device 0
>>    Array State : A.. ('A' == active, '.' == missing)
>> mdadm: No md superblock detected on /dev/sdc1.
>> mdadm: No md superblock detected on /dev/sdd1.
>
> This is bad.  The two disks are still offline.  You must find and fix
> the hardware problem that is keeping these two disks from communicating.
>  Unless the two disks suffered a simultaneous power surge, the odds they
> are OK is good.  But look at your cables, controller, or power supply.
>
>> I would like to understand too
>> dmesg (with the begin removed) show a lot of error :
>> http://pastebin.com/D1D8AKF9
>
> I browsed it: all attempts to communicate with those two drives failed.
>  That must be fixed first.  Then we can help you recover the data.  If
> you can plug the three drives into another machine, that would be the
> simplest way to isolate the problem.
>
> HTH,
>
> Phil

Re:

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1842 bytes --]

On Wed, 26 Nov 2014 12:38:44 -0600 Travis Williams <travis@euppc.com> wrote:

> Hello all,
> 
> I feel as though I must be missing something that I have had no luck
> finding all morning.
> 
> When setting up arrays with spares in a spare-group, I'm having no
> luck finding a way to get that information from mdadm or mdstat. This
> becomes an issue when trying to write out configs and the like, or
> simply trying to get a feel for how arrays are setup on a system.
> 
> Many tutorials/documentation/etc etc list using `mdadm --scan --detail
> >> /etc/mdadm/mdadm.conf` as a way to write out the running config for
> initialization at reboot.  There is never any of the spare-group
> information listed in that output. Is there another way to see what
> spare-group is included in a currently running array?
> 
> It also isn't listed in `mdadm --scan`, or by `cat /proc/mdstat`
> 
> I've primarily noticed this with Debian 7, with mdadm v3.2.5 - 18th
> May 2012. kernel 3.2.0-4.
> 
> When I modify the mdadm.conf myself and add the 'spare-group' setting
> myself, the arrays work as expected, but I haven't been able to find a
> way to KNOW that they are currently running that way without failing
> drives out to see. This nearly burned me after a restart in one
> instance that I caught out of dumb luck before anything of value was
> lost.
> 

mdadm.conf is the primary  location for spare-group information.
When "mdadm --monitor" is run, it reads that file and uses that information.
If you change the spare-group information in mdadm.conf, it would make sense
to restart "mdadm --monitor" so that it uses the updated information.

mdadm --scan --detail >> /etc/mdadm.conf

was only even meant to be a starting point - a guide.  You are still
responsible for your mdadm.conf file.

NeilBrown

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: Re:

[Peter Grandi]

>> I feel as though I must be missing something that I have had
>> no luck finding all morning.

Probably yes, ad the underlying insight is not explicitly
documented, it is left to the the reader of 'man mdadm.conf':

  "spare-group= The value is a textual name for a group of
    arrays. All arrays with the same spare-group name are
    considered to be part of the same group.
    The significance of a group of arrays is that mdadm will,
    when monitoring the arrays, move a spare drive from one
    array in a group to another array in that group if the first
    array had a failed or missing drive but no spare."

>> When setting up arrays with spares in a spare-group, I'm
>> having no luck finding a way to get that information from
>> mdadm or mdstat. This becomes an issue when trying to write
>> out configs and the like,

> mdadm.conf is the primary location for spare-group
> information.  When "mdadm --monitor" is run, it reads that
> file and uses that information.

A more detailed explanations is that MD RAID is divided in two
or arguably three components:

* MD kernel drivers: they *run* RAID sets, but not things like
  *creating* them or *maintaining* them. The MD kernel drivers
  only look at the MD member superblocks and do not look at
  'mdadm.conf' or act of their own initiative in changing RAID
  set membership, only the status of existing members listed in
  the superblocks.

* User space command 'mdadm': this creates MD RAID sets by
  writing "superblocks" that are recognized by the MD kernel
  drivers, and can maintain them when the user does explicit
  commands like '--add' or '--remove'. Options not provided on
  the command line are taken from 'mdadm.conf'.

* User space daemon 'mdadm --monitor': this automatically issues
  *some* 'mdadm' commands, based on the content of 'mdadm.conf'.

>> or simply trying to get a feel for how arrays are setup on a
>> system.

Specifically spare groups are not something that the MD kernel
drivers have any direct role in; the concept of "spare-group" is
only relevant to the 'mdadm --monitor' daemon.

Therefore as the reply above implies one cannot look at the
state of MD arrays as known to the kernel and figure out which
spares and MD arrays are in which spare group, it is something
that is handled entirely in user-space.

In recent version of MD RAID things get an additional dimension
of «how arrays are setup» in user-space as 'udev' too can be
configured to do things to MD RAID sets, which are described in
the 'POLICY' and related lines of 'mdadm.conf', and these too
are not recoverable from the information given by the MD kernel
drivers.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE:

[Apple-Free-Lotto]

You have won 760,889:00 GBP in Apple Free Lotto, without the sale of any tickets! Send. Full Name:. Mobile Number and Alternative Email Address. for details and instructions please contact Mr. Gilly Mann: Email: app.freeloto@foxmail.com

Re:

[Wols Lists]

On 06/11/16 21:00, Dennis Dataopslag wrote:
> Help wanted very much!

Quick response ...
> 
> My setup:
> Thecus N5550 NAS with 5 1TB drives installed.
> 
> MD0: RAID 5 config of 4 drives (SD[ABCD]2)
> MD10: RAID 1 config of all 5 drives (SD..1), system generated array
> MD50: RAID 1 config of 4 drives (SD[ABCD]3), system generated array
> 
> 1 drive (SDE) set as global hot spare.
> 
Bit late now, but you would probably have been better with raid-6.
> 
> What happened:
> This weekend I thought it might be a good idea to do a SMART test for
> the drives in my NAS.
> I started the test on 1 drive and after it ran for a while I started
> the other ones.
> While the test was running drive 3 failed. I got a message the RAID
> was degraded and started rebuilding. (My assumption is that at this
> moment the global hot spare will automatically be added to the array)
> 
> I stopped the SMART tests of all drives at this moment since it seemed
> logical to me the SMART test (or the outcomes) made the drive fail.
> In stopping the tests, drive 1 also failed!!
> I let it for a little but the admin interface kept telling me it was
> degraded, did not seem to take any actions to start rebuilding.

It can't - there's no spare drive to rebuild on, and there aren't enough
drives to build a working array.

> At this point I started googling and found I should remove and reseat
> the drives. This is also what I did but nothing seemd to happen.
> The turned up as new drives in the admin interface and I re-added them
> to the array, they were added as spares.
> Even after adding them the array didn't start rebuilding.
> I checked stat in mdadm and it told me clean FAILED opposed to the
> degraded in the admin interface.

Yup. You've only got two drives of a four-drive raid 5.

Where did you google? Did you read the linux raid wiki?

https://raid.wiki.kernel.org/index.php/Linux_Raid
> 
> I rebooted the NAS since it didn't seem to be doing anything I might interrupt.
> after rebooting it seemed as if the entire array had disappeared!!
> I started looking for options in MDADM and tried every "normal"option
> to rebuild the array (--assemble --scan for example)
> Unfortunately I cannot produce a complete list since I cannot find how
> to get it from the logging.
> 
> Finally I mdadm --create a new array with the original 4 drives with
> all the right settings. (Got them from 1 of the original volumes)

OUCH OUCH OUCH!

Are you sure you've got the right settings? A lot of "hidden" settings
have changed their values over the years. Do you know which mdadm was
used to create the array in the first place?

> The creation worked but after creation it doesn't seem to have a valid
> partition table. This is the point where I realized I probably fucked
> it up big-time and should call in the help squad!!!
> What I think went wrong is that I re-created an array with the
> original 4 drives from before the first failure but the hot-spare was
> already added?

Nope. You've probably used a newer version of mdadm. That's assuming the
array is still all the original drives. If some of them have been
replaced you've got a still messier problem.
> 
> The most important data from the array is saved in an offline backup
> luckily but I would very much like it if there is any way I could
> restore the data from the array.
> 
> Is there any way I could get it back online?

You're looking at a big forensic job. I've moved the relevant page to
the archaeology area - probably a bit too soon - but you need to read
the following page

https://raid.wiki.kernel.org/index.php/Reconstruction

Especially the bit about overlays. And wait for the experts to chime in
about how to do a hexdump and work out the values you need to pass to
mdadm to get the array back. It's a lot of work and you could be looking
at a week what with the delays as you wait for replies.

I think it's recoverable. Is it worth it?

Cheers,
Wol

Re:

[Wols Lists]

On 07/11/16 16:50, Wols Lists wrote:
> You're looking at a big forensic job. I've moved the relevant page to
> the archaeology area - probably a bit too soon - but you need to read
> the following page
> 
> https://raid.wiki.kernel.org/index.php/Reconstruction
> 
> Especially the bit about overlays. And wait for the experts to chime in
> about how to do a hexdump and work out the values you need to pass to
> mdadm to get the array back. It's a lot of work and you could be looking
> at a week what with the delays as you wait for replies.

Whoops, sorry. Wrong page, you need this one ...

https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID

Cheers,
Wol

Re:

[Dennis Dataopslag]

CHeers for the reaction and sorry for my late response, I've been out
for business.

Trying to rebuild this RAID is definately worth it for me. The
learning experience alone already makes it worth.

I did read the wiki page and tried several steps that are on there but
it didn't seem to get me out of trouble.

I used this information from the drive, obviously didn't search for
any "hidden" settings:
" Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 36fdeb4b:c5360009:0958ad1e:17da451b
           Name : TRD106:0  (local to host TRD106)
  Creation Time : Fri Oct 10 12:27:27 2014
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1948250112 (929.00 GiB 997.50 GB)
     Array Size : 5844750336 (2786.99 GiB 2992.51 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b49e2752:d37dac6c:8764c52a:372277bd

    Update Time : Sat Nov  5 14:40:33 2016
       Checksum : d47a9ad4 - correct
         Events : 14934

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing)"

Anybody that can give me a little extra push?

On 06/11/16 21:00, Dennis Dataopslag wrote:
> Help wanted very much!

Quick response ...
>
> My setup:
> Thecus N5550 NAS with 5 1TB drives installed.
>
> MD0: RAID 5 config of 4 drives (SD[ABCD]2)
> MD10: RAID 1 config of all 5 drives (SD..1), system generated array
> MD50: RAID 1 config of 4 drives (SD[ABCD]3), system generated array
>
> 1 drive (SDE) set as global hot spare.
>
Bit late now, but you would probably have been better with raid-6.
>
> What happened:
> This weekend I thought it might be a good idea to do a SMART test for
> the drives in my NAS.
> I started the test on 1 drive and after it ran for a while I started
> the other ones.
> While the test was running drive 3 failed. I got a message the RAID
> was degraded and started rebuilding. (My assumption is that at this
> moment the global hot spare will automatically be added to the array)
>
> I stopped the SMART tests of all drives at this moment since it seemed
> logical to me the SMART test (or the outcomes) made the drive fail.
> In stopping the tests, drive 1 also failed!!
> I let it for a little but the admin interface kept telling me it was
> degraded, did not seem to take any actions to start rebuilding.

It can't - there's no spare drive to rebuild on, and there aren't enough
drives to build a working array.

> At this point I started googling and found I should remove and reseat
> the drives. This is also what I did but nothing seemd to happen.
> The turned up as new drives in the admin interface and I re-added them
> to the array, they were added as spares.
> Even after adding them the array didn't start rebuilding.
> I checked stat in mdadm and it told me clean FAILED opposed to the
> degraded in the admin interface.

Yup. You've only got two drives of a four-drive raid 5.

Where did you google? Did you read the linux raid wiki?

https://raid.wiki.kernel.org/index.php/Linux_Raid
>
> I rebooted the NAS since it didn't seem to be doing anything I might interrupt.
> after rebooting it seemed as if the entire array had disappeared!!
> I started looking for options in MDADM and tried every "normal"option
> to rebuild the array (--assemble --scan for example)
> Unfortunately I cannot produce a complete list since I cannot find how
> to get it from the logging.
>
> Finally I mdadm --create a new array with the original 4 drives with
> all the right settings. (Got them from 1 of the original volumes)

OUCH OUCH OUCH!

Are you sure you've got the right settings? A lot of "hidden" settings
have changed their values over the years. Do you know which mdadm was
used to create the array in the first place?

> The creation worked but after creation it doesn't seem to have a valid
> partition table. This is the point where I realized I probably fucked
> it up big-time and should call in the help squad!!!
> What I think went wrong is that I re-created an array with the
> original 4 drives from before the first failure but the hot-spare was
> already added?

Nope. You've probably used a newer version of mdadm. That's assuming the
array is still all the original drives. If some of them have been
replaced you've got a still messier problem.
>
> The most important data from the array is saved in an offline backup
> luckily but I would very much like it if there is any way I could
> restore the data from the array.
>
> Is there any way I could get it back online?

You're looking at a big forensic job. I've moved the relevant page to
the archaeology area - probably a bit too soon - but you need to read
the following page

https://raid.wiki.kernel.org/index.php/Reconstruction

Especially the bit about overlays. And wait for the experts to chime in
about how to do a hexdump and work out the values you need to pass to
mdadm to get the array back. It's a lot of work and you could be looking
at a week what with the delays as you wait for replies.

I think it's recoverable. Is it worth it?

Cheers,
Wol

On Sun, Nov 6, 2016 at 10:00 PM, Dennis Dataopslag
<dennisdataopslag@gmail.com> wrote:
> Help wanted very much!
>
> My setup:
> Thecus N5550 NAS with 5 1TB drives installed.
>
> MD0: RAID 5 config of 4 drives (SD[ABCD]2)
> MD10: RAID 1 config of all 5 drives (SD..1), system generated array
> MD50: RAID 1 config of 4 drives (SD[ABCD]3), system generated array
>
> 1 drive (SDE) set as global hot spare.
>
>
> What happened:
> This weekend I thought it might be a good idea to do a SMART test for
> the drives in my NAS.
> I started the test on 1 drive and after it ran for a while I started
> the other ones.
> While the test was running drive 3 failed. I got a message the RAID
> was degraded and started rebuilding. (My assumption is that at this
> moment the global hot spare will automatically be added to the array)
>
> I stopped the SMART tests of all drives at this moment since it seemed
> logical to me the SMART test (or the outcomes) made the drive fail.
> In stopping the tests, drive 1 also failed!!
> I let it for a little but the admin interface kept telling me it was
> degraded, did not seem to take any actions to start rebuilding.
> At this point I started googling and found I should remove and reseat
> the drives. This is also what I did but nothing seemd to happen.
> The turned up as new drives in the admin interface and I re-added them
> to the array, they were added as spares.
> Even after adding them the array didn't start rebuilding.
> I checked stat in mdadm and it told me clean FAILED opposed to the
> degraded in the admin interface.
>
> I rebooted the NAS since it didn't seem to be doing anything I might interrupt.
> after rebooting it seemed as if the entire array had disappeared!!
> I started looking for options in MDADM and tried every "normal"option
> to rebuild the array (--assemble --scan for example)
> Unfortunately I cannot produce a complete list since I cannot find how
> to get it from the logging.
>
> Finally I mdadm --create a new array with the original 4 drives with
> all the right settings. (Got them from 1 of the original volumes)
> The creation worked but after creation it doesn't seem to have a valid
> partition table. This is the point where I realized I probably fucked
> it up big-time and should call in the help squad!!!
> What I think went wrong is that I re-created an array with the
> original 4 drives from before the first failure but the hot-spare was
> already added?
>
> The most important data from the array is saved in an offline backup
> luckily but I would very much like it if there is any way I could
> restore the data from the array.
>
> Is there any way I could get it back online?

Re:

[Wols Lists]

On 17/11/16 20:33, Dennis Dataopslag wrote:
> CHeers for the reaction and sorry for my late response, I've been out
> for business.
> 
> Trying to rebuild this RAID is definately worth it for me. The
> learning experience alone already makes it worth.
> 
> I did read the wiki page and tried several steps that are on there but
> it didn't seem to get me out of trouble.
> 
> I used this information from the drive, obviously didn't search for
> any "hidden" settings:
> " Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 36fdeb4b:c5360009:0958ad1e:17da451b
>            Name : TRD106:0  (local to host TRD106)
>   Creation Time : Fri Oct 10 12:27:27 2014
>      Raid Level : raid5
>    Raid Devices : 4
> 
>  Avail Dev Size : 1948250112 (929.00 GiB 997.50 GB)
>      Array Size : 5844750336 (2786.99 GiB 2992.51 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : b49e2752:d37dac6c:8764c52a:372277bd
> 
>     Update Time : Sat Nov  5 14:40:33 2016
>        Checksum : d47a9ad4 - correct
>          Events : 14934
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 0
>    Array State : AAAA ('A' == active, '.' == missing)"
> 
> Anybody that can give me a little extra push?
> 
Others will be able to help better than me, but you might want to look
for the thread "RAID10 with 2 drives auto-assembled as RAID1".

This will give you some information about how to run hexdump and find
where your filesystems are on the array.

There's plenty of other threads with this sort of information, but this
will give you a starting point. If Phil Turmel sees this, he'll chime in
with better detail.

Cheers,
Wol

RE:

[Qin's Yanjun]

How are you today and your family? I require your attention and honest
co-operation about some issues which i will really want to discuss with you
which.  Looking forward to read from you soon.  

Qin's


______________________________

Sky Silk, http://aknet.kz

Re:

[Scott Ellentuch]

DOH! Stared at it for a while... Thanks.

Tuc

On Thu, Apr 13, 2017 at 12:22 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Thu, Apr 13, 2017 at 8:58 AM, Scott Ellentuch <tuctboh@gmail.com> wrote:
>>
>> for disk in a b c d g h i j k l m n
>> do
>>
>>   disklist="${disklist} /dev/sd${disk}1"
>>
>> done
>>
>> mdadm --create --verbose /dev/md2 --level=5 --raid=devices=12  ${disklist}
>>
>> But its telling me :
>>
>> mdadm: invalid number of raid devices: devices=12
>>
>>
>> I can't find any definition of a limit anywhere.
>>
>> Thank you, Tuc
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> Try
>
> --raid-devices=12
>
> not
>
> --raid=devices=12

Re:

[H.A]

With profound love in my heart, I Kindly Oblige your interest to very important proposal.. It is Truly Divine and require your utmost attention..........

S hlubokou láskou v mém srdci, Laskave jsem prinutit svuj zájem k návrhu .. Je velmi duležité, skutecne Divine a vyžadují vaši nejvyšší pozornost.

  Kontaktujte me prímo pres: helenaroberts99@gmail.com pro úplné podrobnosti.complete.


HELINA .A ROBERTS

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re:

[Amos Kalonzo]

Attn:

I am wondering why You haven't respond to my email for some days now.
reference to my client's contract balance payment of (11.7M,USD)
Kindly get back to me for more details.

Best Regards

Amos Kalonzo

Re;

[test02]

Congratulations!!!


As part of my humanitarian individual support during this hard times of fighting the Corona Virus (Convid-19); your email account was selected for a Donation of $1,000,000.00 USD for charity and community medical support in your area. 
Please contact us for more information on charles_jackson001@yahoo.com.com


Send Your Response To: charles_jackson001@yahoo.com


Best Regards,

Charles .W. Jackson Jr

-- 
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re:

[Alex Anadi]

Attention: Sir/Madam,

Compliments of the season.

I am Mr Alex Anadi a senior staff of Computer Telex Dept of central
bank of Nigeria.

I decided to contact you because of the prevailing security report
reaching my office and the intense nature of polity in Nigeria.

This is to inform you about the recent plan of federal government of
Nigeria to send your fund to you via diplomatic immunity CASH DELIVERY
SYSTEM valued at $10.6 Million United states dollars only, contact me
for further details.

Regards,
Mr Alex Anadi.