* array went wonky
@ 2013-04-13 5:41 Gimpbully
2013-04-13 14:20 ` Sam Bingner
0 siblings, 1 reply; 14+ messages in thread
From: Gimpbully @ 2013-04-13 5:41 UTC (permalink / raw)
To: linux-raid
Good Evening Folks,
Alright, I have a 5 disk raid5, it's worked for years. Today a disk went "click click" and I dropped the cold spare I had in and started a rebuild. Everything was great until I checked around 35% rebuild and everything was in the toilet. Here is the current -E info. Any advise would be *amazingly* appreciated. (I can't believe I didn't just put the spare in and just go RAID5...).
/dev/sda1:
Events : 25952
this 4 8 1 4 active sync /dev/sda1
0 0 8 33 0 active sync /dev/sdc1
2 2 8 65 2 active sync /dev/sde1
4 4 8 1 4 active sync /dev/sda1
5 5 8 81 5 spare /dev/sdf1
/dev/sdb1:
Events : 25944
this 1 8 17 1 active sync /dev/sdb1
0 0 8 33 0 active sync /dev/sdc1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 65 2 active sync /dev/sde1
4 4 8 1 4 active sync /dev/sda1
5 5 8 81 5 spare /dev/sdf1
/dev/sdc1:
Events : 25952
this 0 8 33 0 active sync /dev/sdc1
0 0 8 33 0 active sync /dev/sdc1
2 2 8 65 2 active sync /dev/sde1
4 4 8 1 4 active sync /dev/sda1
5 5 8 81 5 spare /dev/sdf1
/dev/sde1:
Events : 25952
this 2 8 65 2 active sync /dev/sde1
0 0 8 33 0 active sync /dev/sdc1
2 2 8 65 2 active sync /dev/sde1
4 4 8 1 4 active sync /dev/sda1
5 5 8 81 5 spare /dev/sdf1
/dev/sdf1:
Events : 25952
this 5 8 81 5 spare /dev/sdf1
0 0 8 33 0 active sync /dev/sdc1
2 2 8 65 2 active sync /dev/sde1
4 4 8 1 4 active sync /dev/sda1
5 5 8 81 5 spare /dev/sdf1
--
~~~~~~~~~
He hates these cans
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: array went wonky 2013-04-13 5:41 array went wonky Gimpbully @ 2013-04-13 14:20 ` Sam Bingner 2013-04-13 15:46 ` Robin Hill 2013-04-30 2:33 ` Gimpbully 0 siblings, 2 replies; 14+ messages in thread From: Sam Bingner @ 2013-04-13 14:20 UTC (permalink / raw) To: Gimpbully; +Cc: <linux-raid@vger.kernel.org> On Apr 12, 2013, at 7:41 PM, Gimpbully <gimpbully@gmail.com> wrote: > Good Evening Folks, > Alright, I have a 5 disk raid5, it's worked for years. Today a disk went "click click" and I dropped the cold spare I had in and started a rebuild. Everything was great until I checked around 35% rebuild and everything was in the toilet. Here is the current -E info. Any advise would be *amazingly* appreciated. (I can't believe I didn't just put the spare in and just go RAID5...). > > /dev/sda1: > Events : 25952 > this 4 8 1 4 active sync /dev/sda1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sdb1: > Events : 25944 > this 1 8 17 1 active sync /dev/sdb1 > 0 0 8 33 0 active sync /dev/sdc1 > 1 1 8 17 1 active sync /dev/sdb1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sdc1: > Events : 25952 > this 0 8 33 0 active sync /dev/sdc1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sde1: > Events : 25952 > this 2 8 65 2 active sync /dev/sde1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sdf1: > Events : 25952 > this 5 8 81 5 spare /dev/sdf1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > > -- > I suspect that your sdb drive is also bad... you should try to copy it to a new drive. I would suggest GNU ddrescue (don't forget to use the logfile feature). At this point I would actually suggest making a copy of all the drives (except the spare)... After that you can try to recreate the array with the proper order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add the spare in again depending on if you were able to recover all the data wih GNU ddrescue. You should use the check function for arrays to find such errors, and of course it would be best if you have a backup - then you can just restore from that. Sam ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-04-13 14:20 ` Sam Bingner @ 2013-04-13 15:46 ` Robin Hill 2013-04-13 16:09 ` John White 2013-04-30 2:33 ` Gimpbully 1 sibling, 1 reply; 14+ messages in thread From: Robin Hill @ 2013-04-13 15:46 UTC (permalink / raw) To: Gimpbully; +Cc: <linux-raid@vger.kernel.org> [-- Attachment #1: Type: text/plain, Size: 3769 bytes --] On Sat Apr 13, 2013 at 02:20:50 +0000, Sam Bingner wrote: > On Apr 12, 2013, at 7:41 PM, Gimpbully <gimpbully@gmail.com> wrote: > > > Good Evening Folks, > > Alright, I have a 5 disk raid5, it's worked for years. Today a > > disk went "click click" and I dropped the cold spare I had in > > and started a rebuild. Everything was great until I checked > > around 35% rebuild and everything was in the toilet. Here is > > the current -E info. Any advise would be *amazingly* > > appreciated. (I can't believe I didn't just put the spare in > > and just go RAID5...). > > > > /dev/sda1: > > Events : 25952 > > this 4 8 1 4 active sync /dev/sda1 > > 0 0 8 33 0 active sync /dev/sdc1 > > 2 2 8 65 2 active sync /dev/sde1 > > 4 4 8 1 4 active sync /dev/sda1 > > 5 5 8 81 5 spare /dev/sdf1 > > /dev/sdb1: > > Events : 25944 > > this 1 8 17 1 active sync /dev/sdb1 > > 0 0 8 33 0 active sync /dev/sdc1 > > 1 1 8 17 1 active sync /dev/sdb1 > > 2 2 8 65 2 active sync /dev/sde1 > > 4 4 8 1 4 active sync /dev/sda1 > > 5 5 8 81 5 spare /dev/sdf1 > > /dev/sdc1: > > Events : 25952 > > this 0 8 33 0 active sync /dev/sdc1 > > 0 0 8 33 0 active sync /dev/sdc1 > > 2 2 8 65 2 active sync /dev/sde1 > > 4 4 8 1 4 active sync /dev/sda1 > > 5 5 8 81 5 spare /dev/sdf1 > > /dev/sde1: > > Events : 25952 > > this 2 8 65 2 active sync /dev/sde1 > > 0 0 8 33 0 active sync /dev/sdc1 > > 2 2 8 65 2 active sync /dev/sde1 > > 4 4 8 1 4 active sync /dev/sda1 > > 5 5 8 81 5 spare /dev/sdf1 > > /dev/sdf1: > > Events : 25952 > > this 5 8 81 5 spare /dev/sdf1 > > 0 0 8 33 0 active sync /dev/sdc1 > > 2 2 8 65 2 active sync /dev/sde1 > > 4 4 8 1 4 active sync /dev/sda1 > > 5 5 8 81 5 spare /dev/sdf1 > > > > -- > > > > I suspect that your sdb drive is also bad... you should try to copy it > to a new drive. I would suggest GNU ddrescue (don't forget to use the > logfile feature). At this point I would actually suggest making a > copy of all the drives (except the spare)... > Yes, looks like a read error on sdb - this may be just a transient issue, or could be a sign of pending failure. Once you've got it copied, you can do a more thorough check (SMART tests, badblocks, etc). > After that you can try to recreate the array with the proper order > (sdc1, sdb1, sde1, missing, sda1) and copy data off or add the spare > in again depending on if you were able to recover all the data wih GNU > ddrescue. > No! Definitely DO NOT recreate the array. A force assemble should get the array back up, without any risk of differing data offsets, metadata formats, etc. Cheers, Robin -- ___ ( ' } | Robin Hill <robin@robinhill.me.uk> | / / ) | Little Jim says .... | // !! | "He fallen in de water !!" | [-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-04-13 15:46 ` Robin Hill @ 2013-04-13 16:09 ` John White 0 siblings, 0 replies; 14+ messages in thread From: John White @ 2013-04-13 16:09 UTC (permalink / raw) To: Robin Hill; +Cc: <linux-raid@vger.kernel.org> Force assembles call out missing superblocks... I think my best hope appears to be a ddrescue to a new disk... Sent from my phone On Apr 13, 2013, at 8:46 AM, Robin Hill <robin@robinhill.me.uk> wrote: > On Sat Apr 13, 2013 at 02:20:50 +0000, Sam Bingner wrote: > >> On Apr 12, 2013, at 7:41 PM, Gimpbully <gimpbully@gmail.com> wrote: >> >>> Good Evening Folks, >>> Alright, I have a 5 disk raid5, it's worked for years. Today a >>> disk went "click click" and I dropped the cold spare I had in >>> and started a rebuild. Everything was great until I checked >>> around 35% rebuild and everything was in the toilet. Here is >>> the current -E info. Any advise would be *amazingly* >>> appreciated. (I can't believe I didn't just put the spare in >>> and just go RAID5...). >>> >>> /dev/sda1: >>> Events : 25952 >>> this 4 8 1 4 active sync /dev/sda1 >>> 0 0 8 33 0 active sync /dev/sdc1 >>> 2 2 8 65 2 active sync /dev/sde1 >>> 4 4 8 1 4 active sync /dev/sda1 >>> 5 5 8 81 5 spare /dev/sdf1 >>> /dev/sdb1: >>> Events : 25944 >>> this 1 8 17 1 active sync /dev/sdb1 >>> 0 0 8 33 0 active sync /dev/sdc1 >>> 1 1 8 17 1 active sync /dev/sdb1 >>> 2 2 8 65 2 active sync /dev/sde1 >>> 4 4 8 1 4 active sync /dev/sda1 >>> 5 5 8 81 5 spare /dev/sdf1 >>> /dev/sdc1: >>> Events : 25952 >>> this 0 8 33 0 active sync /dev/sdc1 >>> 0 0 8 33 0 active sync /dev/sdc1 >>> 2 2 8 65 2 active sync /dev/sde1 >>> 4 4 8 1 4 active sync /dev/sda1 >>> 5 5 8 81 5 spare /dev/sdf1 >>> /dev/sde1: >>> Events : 25952 >>> this 2 8 65 2 active sync /dev/sde1 >>> 0 0 8 33 0 active sync /dev/sdc1 >>> 2 2 8 65 2 active sync /dev/sde1 >>> 4 4 8 1 4 active sync /dev/sda1 >>> 5 5 8 81 5 spare /dev/sdf1 >>> /dev/sdf1: >>> Events : 25952 >>> this 5 8 81 5 spare /dev/sdf1 >>> 0 0 8 33 0 active sync /dev/sdc1 >>> 2 2 8 65 2 active sync /dev/sde1 >>> 4 4 8 1 4 active sync /dev/sda1 >>> 5 5 8 81 5 spare /dev/sdf1 >>> >>> -- >>> >> >> I suspect that your sdb drive is also bad... you should try to copy it >> to a new drive. I would suggest GNU ddrescue (don't forget to use the >> logfile feature). At this point I would actually suggest making a >> copy of all the drives (except the spare)... >> > Yes, looks like a read error on sdb - this may be just a transient > issue, or could be a sign of pending failure. Once you've got it copied, > you can do a more thorough check (SMART tests, badblocks, etc). > >> After that you can try to recreate the array with the proper order >> (sdc1, sdb1, sde1, missing, sda1) and copy data off or add the spare >> in again depending on if you were able to recover all the data wih GNU >> ddrescue. >> > No! Definitely DO NOT recreate the array. A force assemble should get > the array back up, without any risk of differing data offsets, metadata > formats, etc. > > Cheers, > Robin > -- > ___ > ( ' } | Robin Hill <robin@robinhill.me.uk> | > / / ) | Little Jim says .... | > // !! | "He fallen in de water !!" | ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-04-13 14:20 ` Sam Bingner 2013-04-13 15:46 ` Robin Hill @ 2013-04-30 2:33 ` Gimpbully 2013-04-30 6:20 ` Sam Bingner 1 sibling, 1 reply; 14+ messages in thread From: Gimpbully @ 2013-04-30 2:33 UTC (permalink / raw) To: Sam Bingner; +Cc: <linux-raid@vger.kernel.org> On Apr 13, 2013, at 7:20 AM, Sam Bingner <sam@bingner.com> wrote: > After that you can try to recreate the array with the proper order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add the spare in again depending on if you were able to recover all the data wih GNU ddrescue. What do you mean recreate? what's the specific command? something like: mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1 ? ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-04-30 2:33 ` Gimpbully @ 2013-04-30 6:20 ` Sam Bingner 2013-04-30 14:45 ` Phil Turmel 0 siblings, 1 reply; 14+ messages in thread From: Sam Bingner @ 2013-04-30 6:20 UTC (permalink / raw) To: Gimpbully; +Cc: <linux-raid@vger.kernel.org> On Apr 29, 2013, at 4:33 PM, "Gimpbully" <gimpbully@gmail.com> wrote: > > On Apr 13, 2013, at 7:20 AM, Sam Bingner <sam@bingner.com> wrote: > >> After that you can try to recreate the array with the proper order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add the spare in again depending on if you were able to recover all the data wih GNU ddrescue. > > > What do you mean recreate? what's the specific command? something like: > mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1 Don't recreate it - I said the wrong thing... You want to do an assemble on them with force if possible... Recreate is last ditch and make sure you have another copy if you do the previous command in case it doesn't work right due to offsets etc... try: mdadm --stop /dev/md127 mdadm --assemble --force /dev/md127 /dev/sd{c,b,e,a}1 If you DO need to recreate it, what you showed looks correct. Sam ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-04-30 6:20 ` Sam Bingner @ 2013-04-30 14:45 ` Phil Turmel 2013-04-30 14:48 ` Gimpbully 0 siblings, 1 reply; 14+ messages in thread From: Phil Turmel @ 2013-04-30 14:45 UTC (permalink / raw) To: Sam Bingner; +Cc: Gimpbully, <linux-raid@vger.kernel.org> On 04/30/2013 02:20 AM, Sam Bingner wrote: > On Apr 29, 2013, at 4:33 PM, "Gimpbully" <gimpbully@gmail.com> > wrote: >> On Apr 13, 2013, at 7:20 AM, Sam Bingner <sam@bingner.com> wrote: >> >>> After that you can try to recreate the array with the proper >>> order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add >>> the spare in again depending on if you were able to recover all >>> the data wih GNU ddrescue. >> >> >> What do you mean recreate? what's the specific command? something >> like: >> mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 >> /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1 > > Don't recreate it - I said the wrong thing... You want to do an > assemble on them with force if possible... Recreate is last ditch and > make sure you have another copy if you do the previous command in > case it doesn't work right due to offsets etc... > > try: > mdadm --stop /dev/md127 > mdadm --assemble --force /dev/md127 /dev/sd{c,b,e,a}1 Yes. > If you DO need to recreate it, what you showed looks correct. NO! The OP has *not* shared sufficient information on the array members to say that. Since it has "worked for years", the odds of an offset error is *very* high. Chunk size defaults are also likely to be different. *Complete* output of "mdadm -E" for the array members is needed before any "--create" operation is attempted. Plus the distro info, kernel version, and mdadm version . Phil ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-04-30 14:45 ` Phil Turmel @ 2013-04-30 14:48 ` Gimpbully 2013-05-01 18:59 ` Gimpbully 0 siblings, 1 reply; 14+ messages in thread From: Gimpbully @ 2013-04-30 14:48 UTC (permalink / raw) To: Phil Turmel; +Cc: Sam Bingner, <linux-raid@vger.kernel.org> I have no intentions of recreating at all. I fully understand the implications (I've had stripe orders go bad on truly massive scales before, I don't ever wanna experience that again), I just don't know mdadm as well as other vendor storage. I was able to ddrescue a copy of sdb with no errors. I assembled the array and got a file system metadata dump (this is HUGE) and it's currently rebuilding before I even look at data. Thank you all for your help, patience was absolutely key here. On Apr 30, 2013, at 7:45 AM, Phil Turmel <philip@turmel.org> wrote: > On 04/30/2013 02:20 AM, Sam Bingner wrote: >> On Apr 29, 2013, at 4:33 PM, "Gimpbully" <gimpbully@gmail.com> >> wrote: >>> On Apr 13, 2013, at 7:20 AM, Sam Bingner <sam@bingner.com> wrote: >>> >>>> After that you can try to recreate the array with the proper >>>> order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add >>>> the spare in again depending on if you were able to recover all >>>> the data wih GNU ddrescue. >>> >>> >>> What do you mean recreate? what's the specific command? something >>> like: >>> mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 >>> /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1 >> >> Don't recreate it - I said the wrong thing... You want to do an >> assemble on them with force if possible... Recreate is last ditch and >> make sure you have another copy if you do the previous command in >> case it doesn't work right due to offsets etc... >> >> try: >> mdadm --stop /dev/md127 >> mdadm --assemble --force /dev/md127 /dev/sd{c,b,e,a}1 > > Yes. > >> If you DO need to recreate it, what you showed looks correct. > > NO! > > The OP has *not* shared sufficient information on the array members to > say that. Since it has "worked for years", the odds of an offset error > is *very* high. Chunk size defaults are also likely to be different. > > *Complete* output of "mdadm -E" for the array members is needed before > any "--create" operation is attempted. Plus the distro info, kernel > version, and mdadm version . > > Phil ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-04-30 14:48 ` Gimpbully @ 2013-05-01 18:59 ` Gimpbully 2013-05-02 14:18 ` Gimpbully 0 siblings, 1 reply; 14+ messages in thread From: Gimpbully @ 2013-05-01 18:59 UTC (permalink / raw) To: Phil Turmel; +Cc: Sam Bingner, <linux-raid@vger.kernel.org> Alright, so I've ddrescued 2 disks now: sda - clean sdb - SMART errors, kicked out of array sdc - SMART errors, kicked out of array sde - ddrescued copy of sdb sdf - original failed disk, replaced. is now a ddrescued copy of sdc So I run eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 mdadm: cannot open device /dev/sdf1: Device or resource busy mdadm: /dev/sdf1 has no superblock - assembly aborted eduardo ~ # Any ideas? On Apr 30, 2013, at 7:48 AM, Gimpbully <gimpbully@gmail.com> wrote: > I have no intentions of recreating at all. I fully understand the implications (I've had stripe orders go bad on truly massive scales before, I don't ever wanna experience that again), I just don't know mdadm as well as other vendor storage. > > I was able to ddrescue a copy of sdb with no errors. I assembled the array and got a file system metadata dump (this is HUGE) and it's currently rebuilding before I even look at data. Thank you all for your help, patience was absolutely key here. > > > On Apr 30, 2013, at 7:45 AM, Phil Turmel <philip@turmel.org> wrote: > >> On 04/30/2013 02:20 AM, Sam Bingner wrote: >>> On Apr 29, 2013, at 4:33 PM, "Gimpbully" <gimpbully@gmail.com> >>> wrote: >>>> On Apr 13, 2013, at 7:20 AM, Sam Bingner <sam@bingner.com> wrote: >>>> >>>>> After that you can try to recreate the array with the proper >>>>> order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add >>>>> the spare in again depending on if you were able to recover all >>>>> the data wih GNU ddrescue. >>>> >>>> >>>> What do you mean recreate? what's the specific command? something >>>> like: >>>> mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 >>>> /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1 >>> >>> Don't recreate it - I said the wrong thing... You want to do an >>> assemble on them with force if possible... Recreate is last ditch and >>> make sure you have another copy if you do the previous command in >>> case it doesn't work right due to offsets etc... >>> >>> try: >>> mdadm --stop /dev/md127 >>> mdadm --assemble --force /dev/md127 /dev/sd{c,b,e,a}1 >> >> Yes. >> >>> If you DO need to recreate it, what you showed looks correct. >> >> NO! >> >> The OP has *not* shared sufficient information on the array members to >> say that. Since it has "worked for years", the odds of an offset error >> is *very* high. Chunk size defaults are also likely to be different. >> >> *Complete* output of "mdadm -E" for the array members is needed before >> any "--create" operation is attempted. Plus the distro info, kernel >> version, and mdadm version . >> >> Phil > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-05-01 18:59 ` Gimpbully @ 2013-05-02 14:18 ` Gimpbully 2013-05-02 14:29 ` Gimpbully 0 siblings, 1 reply; 14+ messages in thread From: Gimpbully @ 2013-05-02 14:18 UTC (permalink / raw) To: Phil Turmel; +Cc: Sam Bingner, <linux-raid@vger.kernel.org> This is very odd. When trying to force an assemble, it's syaing no superblock on /dev/sdf1, but the --examine output appears to say otherwise… Does anyone have any idea how to do this without doing a re-creation? eduardo ~ # mdadm --examine /dev/sd{f,g,e,a}1 | egrep 'Event|/dev/sd' /dev/sdf1: Events : 25979 this 5 8 81 5 spare /dev/sdf1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sdg1: Events : 25971 this 1 8 17 1 active sync /dev/sdb1 0 0 8 33 0 active sync /dev/sdc1 1 1 8 17 1 active sync /dev/sdb1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sde1: Events : 25979 this 2 8 65 2 active sync /dev/sde1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sda1: Events : 25979 this 4 8 1 4 active sync /dev/sda1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 eduardo ~ # On May 1, 2013, at 11:59 AM, Gimpbully <gimpbully@gmail.com> wrote: > Alright, so I've ddrescued 2 disks now: > sda - clean > sdb - SMART errors, kicked out of array > sdc - SMART errors, kicked out of array > sde - ddrescued copy of sdb > sdf - original failed disk, replaced. is now a ddrescued copy of sdc > > So I run > > eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 > mdadm: cannot open device /dev/sdf1: Device or resource busy > mdadm: /dev/sdf1 has no superblock - assembly aborted > eduardo ~ # > > Any ideas? > > On Apr 30, 2013, at 7:48 AM, Gimpbully <gimpbully@gmail.com> wrote: > >> I have no intentions of recreating at all. I fully understand the implications (I've had stripe orders go bad on truly massive scales before, I don't ever wanna experience that again), I just don't know mdadm as well as other vendor storage. >> >> I was able to ddrescue a copy of sdb with no errors. I assembled the array and got a file system metadata dump (this is HUGE) and it's currently rebuilding before I even look at data. Thank you all for your help, patience was absolutely key here. >> >> >> On Apr 30, 2013, at 7:45 AM, Phil Turmel <philip@turmel.org> wrote: >> >>> On 04/30/2013 02:20 AM, Sam Bingner wrote: >>>> On Apr 29, 2013, at 4:33 PM, "Gimpbully" <gimpbully@gmail.com> >>>> wrote: >>>>> On Apr 13, 2013, at 7:20 AM, Sam Bingner <sam@bingner.com> wrote: >>>>> >>>>>> After that you can try to recreate the array with the proper >>>>>> order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add >>>>>> the spare in again depending on if you were able to recover all >>>>>> the data wih GNU ddrescue. >>>>> >>>>> >>>>> What do you mean recreate? what's the specific command? something >>>>> like: >>>>> mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 >>>>> /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1 >>>> >>>> Don't recreate it - I said the wrong thing... You want to do an >>>> assemble on them with force if possible... Recreate is last ditch and >>>> make sure you have another copy if you do the previous command in >>>> case it doesn't work right due to offsets etc... >>>> >>>> try: >>>> mdadm --stop /dev/md127 >>>> mdadm --assemble --force /dev/md127 /dev/sd{c,b,e,a}1 >>> >>> Yes. >>> >>>> If you DO need to recreate it, what you showed looks correct. >>> >>> NO! >>> >>> The OP has *not* shared sufficient information on the array members to >>> say that. Since it has "worked for years", the odds of an offset error >>> is *very* high. Chunk size defaults are also likely to be different. >>> >>> *Complete* output of "mdadm -E" for the array members is needed before >>> any "--create" operation is attempted. Plus the distro info, kernel >>> version, and mdadm version . >>> >>> Phil >> > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-05-02 14:18 ` Gimpbully @ 2013-05-02 14:29 ` Gimpbully 2013-05-03 6:47 ` Gimpbully 2013-05-03 7:40 ` Robin Hill 0 siblings, 2 replies; 14+ messages in thread From: Gimpbully @ 2013-05-02 14:29 UTC (permalink / raw) To: Phil Turmel; +Cc: Sam Bingner, <linux-raid@vger.kernel.org> Sorry, I realize I had md127 started. So I stopped it and did the assemble and got the following (the problem is sdf isn't supposed to be a spare at all, it's supposed to be a ddrescue'd sdc: eduardo ~ # mdadm --stop /dev/md127 mdadm: stopped /dev/md127 eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 mdadm: /dev/md127 assembled from 3 drives and 1 spare - not enough to start the array. eduardo ~ # eduardo ~ # mdadm --examine /dev/sd{f,g,e,a}1 | egrep 'Event|/dev/sd' /dev/sdf1: Events : 25979 this 5 8 81 5 spare /dev/sdf1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sdg1: Events : 25979 this 1 8 17 1 active sync /dev/sdb1 0 0 8 33 0 active sync /dev/sdc1 1 1 8 17 1 active sync /dev/sdb1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sde1: Events : 25979 this 2 8 65 2 active sync /dev/sde1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 /dev/sda1: Events : 25979 this 4 8 1 4 active sync /dev/sda1 0 0 8 33 0 active sync /dev/sdc1 2 2 8 65 2 active sync /dev/sde1 4 4 8 1 4 active sync /dev/sda1 5 5 8 81 5 spare /dev/sdf1 On May 2, 2013, at 7:18 AM, Gimpbully <gimpbully@gmail.com> wrote: > This is very odd. When trying to force an assemble, it's syaing no superblock on /dev/sdf1, but the --examine output appears to say otherwise… Does anyone have any idea how to do this without doing a re-creation? > > > eduardo ~ # mdadm --examine /dev/sd{f,g,e,a}1 | egrep 'Event|/dev/sd' > /dev/sdf1: > Events : 25979 > this 5 8 81 5 spare /dev/sdf1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sdg1: > Events : 25971 > this 1 8 17 1 active sync /dev/sdb1 > 0 0 8 33 0 active sync /dev/sdc1 > 1 1 8 17 1 active sync /dev/sdb1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sde1: > Events : 25979 > this 2 8 65 2 active sync /dev/sde1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sda1: > Events : 25979 > this 4 8 1 4 active sync /dev/sda1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > eduardo ~ # > > > > On May 1, 2013, at 11:59 AM, Gimpbully <gimpbully@gmail.com> wrote: > >> Alright, so I've ddrescued 2 disks now: >> sda - clean >> sdb - SMART errors, kicked out of array >> sdc - SMART errors, kicked out of array >> sde - ddrescued copy of sdb >> sdf - original failed disk, replaced. is now a ddrescued copy of sdc >> >> So I run >> >> eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 >> mdadm: cannot open device /dev/sdf1: Device or resource busy >> mdadm: /dev/sdf1 has no superblock - assembly aborted >> eduardo ~ # >> >> Any ideas? >> >> On Apr 30, 2013, at 7:48 AM, Gimpbully <gimpbully@gmail.com> wrote: >> >>> I have no intentions of recreating at all. I fully understand the implications (I've had stripe orders go bad on truly massive scales before, I don't ever wanna experience that again), I just don't know mdadm as well as other vendor storage. >>> >>> I was able to ddrescue a copy of sdb with no errors. I assembled the array and got a file system metadata dump (this is HUGE) and it's currently rebuilding before I even look at data. Thank you all for your help, patience was absolutely key here. >>> >>> >>> On Apr 30, 2013, at 7:45 AM, Phil Turmel <philip@turmel.org> wrote: >>> >>>> On 04/30/2013 02:20 AM, Sam Bingner wrote: >>>>> On Apr 29, 2013, at 4:33 PM, "Gimpbully" <gimpbully@gmail.com> >>>>> wrote: >>>>>> On Apr 13, 2013, at 7:20 AM, Sam Bingner <sam@bingner.com> wrote: >>>>>> >>>>>>> After that you can try to recreate the array with the proper >>>>>>> order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add >>>>>>> the spare in again depending on if you were able to recover all >>>>>>> the data wih GNU ddrescue. >>>>>> >>>>>> >>>>>> What do you mean recreate? what's the specific command? something >>>>>> like: >>>>>> mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 >>>>>> /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1 >>>>> >>>>> Don't recreate it - I said the wrong thing... You want to do an >>>>> assemble on them with force if possible... Recreate is last ditch and >>>>> make sure you have another copy if you do the previous command in >>>>> case it doesn't work right due to offsets etc... >>>>> >>>>> try: >>>>> mdadm --stop /dev/md127 >>>>> mdadm --assemble --force /dev/md127 /dev/sd{c,b,e,a}1 >>>> >>>> Yes. >>>> >>>>> If you DO need to recreate it, what you showed looks correct. >>>> >>>> NO! >>>> >>>> The OP has *not* shared sufficient information on the array members to >>>> say that. Since it has "worked for years", the odds of an offset error >>>> is *very* high. Chunk size defaults are also likely to be different. >>>> >>>> *Complete* output of "mdadm -E" for the array members is needed before >>>> any "--create" operation is attempted. Plus the distro info, kernel >>>> version, and mdadm version . >>>> >>>> Phil >>> >> > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-05-02 14:29 ` Gimpbully @ 2013-05-03 6:47 ` Gimpbully 2013-05-03 14:24 ` Phil Turmel 2013-05-03 7:40 ` Robin Hill 1 sibling, 1 reply; 14+ messages in thread From: Gimpbully @ 2013-05-03 6:47 UTC (permalink / raw) To: Phil Turmel; +Cc: Sam Bingner, <linux-raid@vger.kernel.org> Anyone? I'm so close I can taste it. It looks like the superblocks on the other disks think that sdf1 is still defined as a spare, but it's a ddrescue of what used to be /dev/sdc. Is there a way to update the superblocks? On May 2, 2013, at 7:29 AM, Gimpbully <gimpbully@gmail.com> wrote: > Sorry, I realize I had md127 started. So I stopped it and did the assemble and got the following (the problem is sdf isn't supposed to be a spare at all, it's supposed to be a ddrescue'd sdc: > > eduardo ~ # mdadm --stop /dev/md127 > mdadm: stopped /dev/md127 > eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 > mdadm: /dev/md127 assembled from 3 drives and 1 spare - not enough to start the array. > eduardo ~ # > > > eduardo ~ # mdadm --examine /dev/sd{f,g,e,a}1 | egrep 'Event|/dev/sd' > /dev/sdf1: > Events : 25979 > this 5 8 81 5 spare /dev/sdf1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sdg1: > Events : 25979 > this 1 8 17 1 active sync /dev/sdb1 > 0 0 8 33 0 active sync /dev/sdc1 > 1 1 8 17 1 active sync /dev/sdb1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sde1: > Events : 25979 > this 2 8 65 2 active sync /dev/sde1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sda1: > Events : 25979 > this 4 8 1 4 active sync /dev/sda1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > > > > > > On May 2, 2013, at 7:18 AM, Gimpbully <gimpbully@gmail.com> wrote: > >> This is very odd. When trying to force an assemble, it's syaing no superblock on /dev/sdf1, but the --examine output appears to say otherwise… Does anyone have any idea how to do this without doing a re-creation? >> >> >> eduardo ~ # mdadm --examine /dev/sd{f,g,e,a}1 | egrep 'Event|/dev/sd' >> /dev/sdf1: >> Events : 25979 >> this 5 8 81 5 spare /dev/sdf1 >> 0 0 8 33 0 active sync /dev/sdc1 >> 2 2 8 65 2 active sync /dev/sde1 >> 4 4 8 1 4 active sync /dev/sda1 >> 5 5 8 81 5 spare /dev/sdf1 >> /dev/sdg1: >> Events : 25971 >> this 1 8 17 1 active sync /dev/sdb1 >> 0 0 8 33 0 active sync /dev/sdc1 >> 1 1 8 17 1 active sync /dev/sdb1 >> 2 2 8 65 2 active sync /dev/sde1 >> 4 4 8 1 4 active sync /dev/sda1 >> 5 5 8 81 5 spare /dev/sdf1 >> /dev/sde1: >> Events : 25979 >> this 2 8 65 2 active sync /dev/sde1 >> 0 0 8 33 0 active sync /dev/sdc1 >> 2 2 8 65 2 active sync /dev/sde1 >> 4 4 8 1 4 active sync /dev/sda1 >> 5 5 8 81 5 spare /dev/sdf1 >> /dev/sda1: >> Events : 25979 >> this 4 8 1 4 active sync /dev/sda1 >> 0 0 8 33 0 active sync /dev/sdc1 >> 2 2 8 65 2 active sync /dev/sde1 >> 4 4 8 1 4 active sync /dev/sda1 >> 5 5 8 81 5 spare /dev/sdf1 >> eduardo ~ # >> >> >> >> On May 1, 2013, at 11:59 AM, Gimpbully <gimpbully@gmail.com> wrote: >> >>> Alright, so I've ddrescued 2 disks now: >>> sda - clean >>> sdb - SMART errors, kicked out of array >>> sdc - SMART errors, kicked out of array >>> sde - ddrescued copy of sdb >>> sdf - original failed disk, replaced. is now a ddrescued copy of sdc >>> >>> So I run >>> >>> eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 >>> mdadm: cannot open device /dev/sdf1: Device or resource busy >>> mdadm: /dev/sdf1 has no superblock - assembly aborted >>> eduardo ~ # >>> >>> Any ideas? >>> >>> On Apr 30, 2013, at 7:48 AM, Gimpbully <gimpbully@gmail.com> wrote: >>> >>>> I have no intentions of recreating at all. I fully understand the implications (I've had stripe orders go bad on truly massive scales before, I don't ever wanna experience that again), I just don't know mdadm as well as other vendor storage. >>>> >>>> I was able to ddrescue a copy of sdb with no errors. I assembled the array and got a file system metadata dump (this is HUGE) and it's currently rebuilding before I even look at data. Thank you all for your help, patience was absolutely key here. >>>> >>>> >>>> On Apr 30, 2013, at 7:45 AM, Phil Turmel <philip@turmel.org> wrote: >>>> >>>>> On 04/30/2013 02:20 AM, Sam Bingner wrote: >>>>>> On Apr 29, 2013, at 4:33 PM, "Gimpbully" <gimpbully@gmail.com> >>>>>> wrote: >>>>>>> On Apr 13, 2013, at 7:20 AM, Sam Bingner <sam@bingner.com> wrote: >>>>>>> >>>>>>>> After that you can try to recreate the array with the proper >>>>>>>> order (sdc1, sdb1, sde1, missing, sda1) and copy data off or add >>>>>>>> the spare in again depending on if you were able to recover all >>>>>>>> the data wih GNU ddrescue. >>>>>>> >>>>>>> >>>>>>> What do you mean recreate? what's the specific command? something >>>>>>> like: >>>>>>> mdadm --create --assume-clean --level=5 --raid-devices=5 /dev/md127 >>>>>>> /dev/sdc1 /dev/sdb1 /sdv/sde1 missing /dev/sda1 >>>>>> >>>>>> Don't recreate it - I said the wrong thing... You want to do an >>>>>> assemble on them with force if possible... Recreate is last ditch and >>>>>> make sure you have another copy if you do the previous command in >>>>>> case it doesn't work right due to offsets etc... >>>>>> >>>>>> try: >>>>>> mdadm --stop /dev/md127 >>>>>> mdadm --assemble --force /dev/md127 /dev/sd{c,b,e,a}1 >>>>> >>>>> Yes. >>>>> >>>>>> If you DO need to recreate it, what you showed looks correct. >>>>> >>>>> NO! >>>>> >>>>> The OP has *not* shared sufficient information on the array members to >>>>> say that. Since it has "worked for years", the odds of an offset error >>>>> is *very* high. Chunk size defaults are also likely to be different. >>>>> >>>>> *Complete* output of "mdadm -E" for the array members is needed before >>>>> any "--create" operation is attempted. Plus the distro info, kernel >>>>> version, and mdadm version . >>>>> >>>>> Phil >>>> >>> >> > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-05-03 6:47 ` Gimpbully @ 2013-05-03 14:24 ` Phil Turmel 0 siblings, 0 replies; 14+ messages in thread From: Phil Turmel @ 2013-05-03 14:24 UTC (permalink / raw) To: Gimpbully; +Cc: Sam Bingner, <linux-raid@vger.kernel.org> On 05/03/2013 02:47 AM, Gimpbully wrote: > Anyone? I'm so close I can taste it. It looks like the superblocks > on the other disks think that sdf1 is still defined as a spare, but > it's a ddrescue of what used to be /dev/sdc. Is there a way to > update the superblocks? Sorry, been busy. Unfortunately, once a member is marked "spare", it simply doesn't know any more what role it used to have. So "--assemble" won't get you there. You will need to use "--create --assume-clean" or directly edit the offending superblock. > *Complete* output of "mdadm -E" for the array members is needed before > any "--create" operation is attempted. Plus the distro info, kernel > version, and mdadm version . I would also like to see "smartctl -x" output for each underlying disk. It would be good if we understood *why* your array "went wonky" so that it doesn't happen again after reassembling your array. Your situation "smells" like a timeout mismatch and/or lack of scrubbing. Finally, in the interest of helping others help you, please don't top-post, and *do* trim your quotes. It greatly helps when coming back to a thread after a few days. Phil ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: array went wonky 2013-05-02 14:29 ` Gimpbully 2013-05-03 6:47 ` Gimpbully @ 2013-05-03 7:40 ` Robin Hill 1 sibling, 0 replies; 14+ messages in thread From: Robin Hill @ 2013-05-03 7:40 UTC (permalink / raw) To: Gimpbully; +Cc: Phil Turmel, Sam Bingner, <linux-raid@vger.kernel.org> [-- Attachment #1: Type: text/plain, Size: 2791 bytes --] On Thu May 02, 2013 at 07:29:38AM -0700, Gimpbully wrote: > Sorry, I realize I had md127 started. So I stopped it and did the > assemble and got the following (the problem is sdf isn't supposed to > be a spare at all, it's supposed to be a ddrescue'd sdc: > > eduardo ~ # mdadm --stop /dev/md127 > mdadm: stopped /dev/md127 > eduardo ~ # mdadm --assemble --force /dev/md127 /dev/sd{f,g,e,a}1 > mdadm: /dev/md127 assembled from 3 drives and 1 spare - not enough to start the array. > eduardo ~ # > > > eduardo ~ # mdadm --examine /dev/sd{f,g,e,a}1 | egrep 'Event|/dev/sd' > /dev/sdf1: > Events : 25979 > this 5 8 81 5 spare /dev/sdf1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sdg1: > Events : 25979 > this 1 8 17 1 active sync /dev/sdb1 > 0 0 8 33 0 active sync /dev/sdc1 > 1 1 8 17 1 active sync /dev/sdb1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sde1: > Events : 25979 > this 2 8 65 2 active sync /dev/sde1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > /dev/sda1: > Events : 25979 > this 4 8 1 4 active sync /dev/sda1 > 0 0 8 33 0 active sync /dev/sdc1 > 2 2 8 65 2 active sync /dev/sde1 > 4 4 8 1 4 active sync /dev/sda1 > 5 5 8 81 5 spare /dev/sdf1 > Okay, so we have: sda - original disk sde - original disk sdf - supposed to be ddrescue of original sdc sdg - ddrescue of original sdb From the metadata, it would appear that sdf is not in fact a ddrescue of sdc, so I'd suggest going back and redoing that. Check afterwards that the metadata reported by "mdadm --examine" on the two disks matches. Cheers, Robin -- ___ ( ' } | Robin Hill <robin@robinhill.me.uk> | / / ) | Little Jim says .... | // !! | "He fallen in de water !!" | [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2013-05-03 14:24 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-04-13 5:41 array went wonky Gimpbully 2013-04-13 14:20 ` Sam Bingner 2013-04-13 15:46 ` Robin Hill 2013-04-13 16:09 ` John White 2013-04-30 2:33 ` Gimpbully 2013-04-30 6:20 ` Sam Bingner 2013-04-30 14:45 ` Phil Turmel 2013-04-30 14:48 ` Gimpbully 2013-05-01 18:59 ` Gimpbully 2013-05-02 14:18 ` Gimpbully 2013-05-02 14:29 ` Gimpbully 2013-05-03 6:47 ` Gimpbully 2013-05-03 14:24 ` Phil Turmel 2013-05-03 7:40 ` Robin Hill
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).