* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Wolfgang Denk @ 2016-05-16 11:05 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20160516102418.GA2347@metamorpher.de>
Dear Andreas,
In message <20160516102418.GA2347@metamorpher.de> you wrote:
> On Mon, May 16, 2016 at 12:06:42PM +0200, Wolfgang Denk wrote:
> > Hm... is --data-offset the only parameter I can play with?
>
> There is also --layout=
The output of the initial create command was
mdadm: layout defaults to left-symmetric
mdadm: size set to 976762448K
mdadm: array /dev/md2 started.
so this should be known...
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I think animal testing is a terrible idea; they get all nervous and
give the wrong answers.
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Andreas Klauer @ 2016-05-16 10:24 UTC (permalink / raw)
To: Wolfgang Denk; +Cc: linux-raid
In-Reply-To: <20160516100642.D4B4D10035C@atlas.denx.de>
On Mon, May 16, 2016 at 12:06:42PM +0200, Wolfgang Denk wrote:
> Hm... is --data-offset the only parameter I can play with?
There is also --layout=
Regards
Andreas Klauer
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Wolfgang Denk @ 2016-05-16 10:06 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20160516083903.GA29380@EIS.leimen.priv>
Dear Andreas,
In message <20160516083903.GA29380@EIS.leimen.priv> you wrote:
>
> You are using your disks in alphabetical order, are you sure this is
> the same order your RAID originally used? Maybe the drive letters
> changed?
Yes, this is one thing I am absolutely sure about. I have the mapping
of the disk serial numbers from initial install, and I verified that
the drive order is still the same.
> You found LABELONE on sda, which is your first drive (Device Role 0)
> in your RAID (see mdadm --examine after you create it), but when I
> create a new RAID based on loop devices, pvcreate and vgcreate it,
> the LABELONE actually appears on the 2nd drive (Device Role 1).
Thi sis strage; I see you are using 6 disks and the same stripe size,
so I would also expect a layout like yours. OK, I need to experiment
a bit...
> So I assume in the default left-symmetric layout in RAID6, the first chunk
> of the first disk actually ends up being a parity chunk...? I'm not too
> sure about this either right now. ;)
Hm... is --data-offset the only parameter I can play with? (except for
variations of drive order, which appear to make no sense to me as I'm
sure I have it correct).
Then I could also start a brute force approach and just try out all
possible values until I gind a match ;-)
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
[Braddock:] Mr. Churchill, you are drunk.
[Churchill:] And you madam, are ugly. But I shall be sober tomorrow.
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Andreas Klauer @ 2016-05-16 8:39 UTC (permalink / raw)
To: Wolfgang Denk; +Cc: linux-raid
In-Reply-To: <20160515231046.E3685100A8E@atlas.denx.de>
On Mon, May 16, 2016 at 01:10:46AM +0200, Wolfgang Denk wrote:
> NO, I think it's me to blame. If I read the bash history correctly,
> I made a mistake (confusing $DEVICES and $devices and $overlays)
> and ran the --zero-superblock on the real disk devices, not the overlays.
Ooh. :(
> On the other hand, that should not make a real difference if it worked
> for the overlays. Well, it does...
About that, I don't know either.
> That would be the --data-offset= parameter?
Yes.
> It does - but I'm confused as I get a number of different values from
> the devices:
You can ignore those that don't even align to 512 bytes.
> So I have 5 numbers; minus 512 and converted to kB gives:
>
> Of these, only 136 appears to make sense. But running
>
> # mdadm --create --verbose /dev/md2 --metadata=1.2 --level=6 --raid-devices=6 --chunk=16 --assume-clean --data-offset=136 /dev/mapper/sd?
>
> ...creates an array, but LVM does not recognise it.
So, now here's a puzzle.
First, you can use hexdump after all to have a look at the first chunk
(assuming the 136KiB you found is actually the data offset).
dd bs=136K skip=1 if=/dev/mapper/sda | hexdump -C | less
(same for sd?)
LVM metadata is in plaintext, example:
| 00001200 53 53 44 20 7b 0a 69 64 20 3d 20 22 74 58 4a 43 |SSD {.id = "tXJC|
| 00001210 77 31 2d 71 51 69 6e 2d 4b 78 31 6b 2d 30 65 78 |w1-qQin-Kx1k-0ex|
| 00001220 79 2d 32 6e 4d 76 2d 6a 63 57 78 2d 4f 48 70 76 |y-2nMv-jcWx-OHpv|
| 00001230 76 69 22 0a 73 65 71 6e 6f 20 3d 20 36 38 0a 66 |vi".seqno = 68.f|
| 00001240 6f 72 6d 61 74 20 3d 20 22 6c 76 6d 32 22 0a 73 |ormat = "lvm2".s|
| 00001250 74 61 74 75 73 20 3d 20 5b 22 52 45 53 49 5a 45 |tatus = ["RESIZE|
| 00001260 41 42 4c 45 22 2c 20 22 52 45 41 44 22 2c 20 22 |ABLE", "READ", "|
| 00001270 57 52 49 54 45 22 5d 0a 66 6c 61 67 73 20 3d 20 |WRITE"].flags = |
For me this starts at offset 0x1200 (roughly 4K) should be well within
your 16K chunk. It should look similar for you on one of your disks if
the offset is correct.
You are using your disks in alphabetical order, are you sure this is
the same order your RAID originally used? Maybe the drive letters
changed?
You found LABELONE on sda, which is your first drive (Device Role 0)
in your RAID (see mdadm --examine after you create it), but when I
create a new RAID based on loop devices, pvcreate and vgcreate it,
the LABELONE actually appears on the 2nd drive (Device Role 1).
| # cat /proc/mdstat
| Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
| md42 : active raid6 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
| 64960 blocks super 1.2 level 6, 16k chunk, algorithm 2 [6/6] [UUUUUU]
| # strings -t d -n 8 /dev/loop0 | grep LABELONE
| # strings -t d -n 8 /dev/loop1 | grep LABELONE
| 139776 LABELONE
| # dd bs=136K skip=1 if=/dev/loop1 | hexdump -C -n $((16*1024)) | head
| 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| *
| 00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
| 00000210 e0 3d e7 de 20 00 00 00 4c 56 4d 32 20 30 30 31 |.=.. ...LVM2 001|
| 00000220 58 48 67 41 68 45 70 76 4c 53 36 61 41 62 4d 77 |XHgAhEpvLS6aAbMw|
| 00000230 50 6e 4c 57 4c 64 4a 46 6d 36 30 54 48 66 6d 75 |PnLWLdJFm60THfmu|
So I assume in the default left-symmetric layout in RAID6, the first chunk
of the first disk actually ends up being a parity chunk...? I'm not too
sure about this either right now. ;)
Regards
Andreas Klauer
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Wolfgang Denk @ 2016-05-15 23:10 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20160515203446.GA13218@EIS.leimen.priv>
Dear Andreas,
In message <20160515203446.GA13218@EIS.leimen.priv> you wrote:
>
> That's strange, if it worked once, it should work twice (if no changes
> were made outside of the overlays). If you have created the udev rules
> file, just in case there is a side effect, remove it again ...
NO, I think it's me to blame. If I read the bash history correctly,
I made a mistake (confusing $DEVICES and $devices and $overlays)
and ran the --zero-superblock on the real disk devices, not the overlays.
On the other hand, that should not make a real difference if it worked
for the overlays. Well, it does...
> What does mdadm --examine /dev/mapper/sda say before and after
> you --zero-superblock? (is it the same for the other disks?)
--zero-superblock now porperly reports that it cannot find a
superblock...
> If it's LVM, you could search the first few hundred megs of your devices for
> the LVM header, that should be the data offset you're looking for.
Ok, but...
> That's LABELONE at offset 0x200 and you have to substract 512 bytes from it,
> so this is actually what it would look like for offset 0.
That would be the --data-offset= parameter?
> Alternatively this can also be done using
>
> dd bs=1M count=1024 if=... | strings -t d -n 8 | grep LABELONE
> 512 LABELONE
>
> if your version of strings supports this option.
It does - but I'm confused as I get a number of different values from
the devices:
# for i in /dev/mapper/sd? ; do
> echo $i
> dd bs=1M count=1024 if=$i | strings -t d -n 8 | grep LABELONE
> done
/dev/mapper/sda
139776 LABELONE
...
/dev/mapper/sdb
...
/dev/mapper/sdc
396826104 LABELONE
...
/dev/mapper/sdd
...
/dev/mapper/sde
387503608 LABELONE
389663524 LABELONE
...
/dev/mapper/sdf
398969636 LABELONE
...
So I have 5 numbers; minus 512 and converted to kB gives:
139776 -> 139264 -> 136
396826104 -> 396825592 -> 887524.99
387503608 -> 387503096 -> 878410.99
389663524 -> 389663012 -> 380530.28
398969636 -> 398969124 -> 889618.28
Of these, only 136 appears to make sense. But running
# mdadm --create --verbose /dev/md2 --metadata=1.2 --level=6 --raid-devices=6 --chunk=16 --assume-clean --data-offset=136 /dev/mapper/sd?
mdadm: layout defaults to left-symmetric
mdadm: size set to 976762448K
mdadm: automatically enabling write-intent bitmap on large array
mdadm: array /dev/md2 started.
...creates an array, but LVM does not recognise it.
Searching the resulting /dev/md2 for the LABELONE does not find any.
Am I misinterpreting your information?
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
The sixth sick sheik’s sixth sheep’s sick.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Andreas Klauer @ 2016-05-15 20:34 UTC (permalink / raw)
To: Wolfgang Denk; +Cc: linux-raid
In-Reply-To: <20160515193516.2C457100879@atlas.denx.de>
On Sun, May 15, 2016 at 09:35:16PM +0200, Wolfgang Denk wrote:
> So I tried to repeat the same procedure, but it does not work any
> more: after erasing the superblocks my attempts to assemble the array
> now give only:
That's strange, if it worked once, it should work twice (if no changes
were made outside of the overlays). If you have created the udev rules
file, just in case there is a side effect, remove it again ...
> # mdadm --assemble /dev/md2 --metadata=1.2 $overlays
> mdadm: no RAID superblock on /dev/mapper/sda
> mdadm: /dev/mapper/sda has no superblock - assembly aborted
What does mdadm --examine /dev/mapper/sda say before and after
you --zero-superblock? (is it the same for the other disks?)
> OK, I can recreate the array, but LVM does not recognize it.
>
> You mentioned I had to play around with the offsets - do you have any
> idea which values would be reasonable to try out?
If it's LVM, you could search the first few hundred megs of your devices for
the LVM header, that should be the data offset you're looking for.
hexdump -C -n $((1024*1024*1024)) /dev/mapper/sda | less
and then search for LABELONE
/LABELONE
| 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| *
| 00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
| 00000210 c1 11 05 3d 20 00 00 00 4c 56 4d 32 20 30 30 31 |...= ...LVM2 001|
| 00000220 36 46 41 75 32 49 70 65 38 62 75 50 31 46 32 75 |6FAu2Ipe8buP1F2u|
| 00000230 38 4a 41 34 41 45 64 51 54 49 49 6b 4f 76 45 35 |8JA4AEdQTIIkOvE5|
That's LABELONE at offset 0x200 and you have to substract 512 bytes from it,
so this is actually what it would look like for offset 0.
Alternatively this can also be done using
dd bs=1M count=1024 if=... | strings -t d -n 8 | grep LABELONE
512 LABELONE
if your version of strings supports this option.
Regards
Andreas Klauer
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Wolfgang Denk @ 2016-05-15 19:35 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20160515183128.GA12823@EIS.leimen.priv>
Dear Andreas,
In message <20160515183128.GA12823@EIS.leimen.priv> you wrote:
> On Sun, May 15, 2016 at 08:25:24PM +0200, Wolfgang Denk wrote:
> > After creating the overlys, the system would automatically start the
> > (incorrect) RAID arrays.
>
> That should be courtesy of udev, see if you have a
>
> /lib/udev/rules.d/64-md-raid-assembly.rules
>
> and if that's the case you can temporarily disable them by
>
> touch /etc/udev/rules.d/64-md-raid-assembly.rules
>
> and (later) re-enable by rm'ing the /etc/ file.
Thanks.
> > Recovering my data was (fortunately) simple:
Unfortunately my luck did not last very long. While copying the first
file system from the recovered array, the system crashed - can't tell
why, when I got to the console it was all black :-(
So I tried to repeat the same procedure, but it does not work any
more: after erasing the superblocks my attempts to assemble the array
now give only:
# mdadm --assemble /dev/md2 --metadata=1.2 $overlays
mdadm: no RAID superblock on /dev/mapper/sda
mdadm: /dev/mapper/sda has no superblock - assembly aborted
[which is what I had initially expected; I have no idea why it worked
once, but not a second time.]
OK, I can recreate the array, but LVM does not recognize it.
You mentioned I had to play around with the offsets - do you have any
idea which values would be reasonable to try out?
Thanks in advance - once more...
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
You can't depend on your eyes when your imagination is out of focus.
- Mark Twain
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Andreas Klauer @ 2016-05-15 18:31 UTC (permalink / raw)
To: Wolfgang Denk; +Cc: linux-raid
In-Reply-To: <20160515182524.7B12D100A8E@atlas.denx.de>
On Sun, May 15, 2016 at 08:25:24PM +0200, Wolfgang Denk wrote:
> After creating the overlys, the system would automatically start the
> (incorrect) RAID arrays.
That should be courtesy of udev, see if you have a
/lib/udev/rules.d/64-md-raid-assembly.rules
and if that's the case you can temporarily disable them by
touch /etc/udev/rules.d/64-md-raid-assembly.rules
and (later) re-enable by rm'ing the /etc/ file.
> Recovering my data was (fortunately) simple:
Congrats!
Regards
Andreas Klauer
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller [SOLVED]
From: Wolfgang Denk @ 2016-05-15 18:25 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20160515153121.GA11365@EIS.leimen.priv>
Dear Andreas,
thanks for the quyick reply, and on a Sunday...
In message <20160515153121.GA11365@EIS.leimen.priv> you wrote:
> On Sun, May 15, 2016 at 03:37:40PM +0200, Wolfgang Denk wrote:
> > Trying to follow the overlay method in [1], I run into errors; guess I
> > must be missing something:
> >
> > [1] https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> I think you mixed two approaches to the same thing, the wiki shows
> a) how to create overlays manually and b) offers some convenience
> functions that do the same thing (the overlay create remove functions,
> you define those functions once and then you can repeatedly call them,
> basically giving you two commands overlay_create and overlay_remove).
Yes, you are right. I now realized this, too.
> And then call 'overlay_create' when you want your overlays,
> and 'overlay_remove; overlay create' when an experiment
> failed and you want to reset them to their original state.
>
> At the time you remove the overlays, all things using them
> must also be gone, so mdadm --stop before overlay_remove.
> (And make sure no raid is running for the disks you're
> overlaying...)
Thanks - this was the key that got me working.
After creating the overlys, the system would automatically start the
(incorrect) RAID arrays. After manually stopping these, I had write
access to the overlays.
Recovering my data was (fortunately) simple:
1) I zeroed the incorrect superblocks on all devices:
# mdadm --zero-superblock /dev/mapper/sda
# mdadm --zero-superblock /dev/mapper/sdb
# mdadm --zero-superblock /dev/mapper/sdc
# mdadm --zero-superblock /dev/mapper/sdd
# mdadm --zero-superblock /dev/mapper/sde
# mdadm --zero-superblock /dev/mapper/sdf
2) Then I forced an assemble of the array:
# mdadm --assemble --force --verbose /dev/md2 --metadata=1.2
# $overlays
mdadm: looking for devices for /dev/md2
mdadm: /dev/mapper/sda is identified as a member of /dev/md2, slot 1.
mdadm: /dev/mapper/sdb is identified as a member of /dev/md2, slot 0.
mdadm: /dev/mapper/sdc is identified as a member of /dev/md2, slot 2.
mdadm: /dev/mapper/sdd is identified as a member of /dev/md2, slot 3.
mdadm: /dev/mapper/sde is identified as a member of /dev/md2, slot 5.
mdadm: /dev/mapper/sdf is identified as a member of /dev/md2, slot 4.
mdadm: added /dev/mapper/sda to /dev/md2 as 1
mdadm: added /dev/mapper/sdc to /dev/md2 as 2
mdadm: added /dev/mapper/sdd to /dev/md2 as 3
mdadm: added /dev/mapper/sdf to /dev/md2 as 4
mdadm: added /dev/mapper/sde to /dev/md2 as 5
mdadm: added /dev/mapper/sdb to /dev/md2 as 0
mdadm: /dev/md2 has been started with 6 drives.
And me was happy again.
I owe you a beer or two. Please don't hesitate to remind me whenever
we meet... Thanks again.
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"You can have my Unix system when you pry it from my cold, dead
fingers." - Cal Keegan
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller
From: Andreas Klauer @ 2016-05-15 15:31 UTC (permalink / raw)
To: Wolfgang Denk; +Cc: linux-raid
In-Reply-To: <20160515133740.85EC3100879@atlas.denx.de>
On Sun, May 15, 2016 at 03:37:40PM +0200, Wolfgang Denk wrote:
> Trying to follow the overlay method in [1], I run into errors; guess I
> must be missing something:
>
> [1] https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
I think you mixed two approaches to the same thing, the wiki shows
a) how to create overlays manually and b) offers some convenience
functions that do the same thing (the overlay create remove functions,
you define those functions once and then you can repeatedly call them,
basically giving you two commands overlay_create and overlay_remove).
It should work if you use only this part:
> # devices="/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf"
> # overlay_create()
> > {
> > free=$((`stat -c '%a*%S/1024/1024' -f .`))
> > echo free ${free}M
> > overlays=""
> > overlay_remove
> > for d in $devices; do
> > b=$(basename $d)
> > size_bkl=$(blockdev --getsz $d) # in 512 blocks/sectors
> > # reserve 1M space for snapshot header
> > # ext3 max file length is 2TB
> > truncate -s$((((size_bkl+1)/2)+1024))K $b.ovr || (echo "Do you use ext4?"; return 1)
> > loop=$(losetup -f --show -- $b.ovr)
> > # https://www.kernel.org/doc/Documentation/device-mapper/snapshot.txt
> > dmsetup create $b --table "0 $size_bkl snapshot $d $loop P 8"
> > echo $d $((size_bkl/2048))M $loop /dev/mapper/$b
> > overlays="$overlays /dev/mapper/$b"
> > done
> > overlays=${overlays# }
> > }
> # overlay_remove()
> > {
> > for d in $devices; do
> > b=$(basename $d)
> > [ -e /dev/mapper/$b ] && dmsetup remove $b && echo /dev/mapper/$b
> > if [ -e $b.ovr ]; then
> > echo $b.ovr
> > l=$(losetup -j $b.ovr | cut -d : -f1)
> > echo $l
> > [ -n "$l" ] && losetup -d $(losetup -j $b.ovr | cut -d : -f1)
> > rm -f $b.ovr &> /dev/null
> > fi
> > done
> > }
And then call 'overlay_create' when you want your overlays,
and 'overlay_remove; overlay create' when an experiment
failed and you want to reset them to their original state.
At the time you remove the overlays, all things using them
must also be gone, so mdadm --stop before overlay_remove.
(And make sure no raid is running for the disks you're
overlaying...)
As for your controller, I don't know this controller. If it's a HW-RAID
that passes individual disks through as RAID-0, usually some sectors of
the disk are missing (controller has to keep RAID-0 metadata somewhere)
and that alone might be enough to damage your old setup in some way.
I prefer "dumb" controllers that pass through disks the way they are.
You showed --detail output of your old RAID; that's already very good,
is there --examine output by any chance? --detail doesn't contain some
things such as data offsets, and the ones mdadm picks by default have
changed a lot, so the same --create command won't actually produce
the same RAID. If your old RAID metadata is actually lost, if you wish
to experiment with --create on the overlay, you'll have to specify all
variables you know and guess the variables you don't know...
Regards
Andreas Klauer
^ permalink raw reply
* Re: MD RAID6 corrupted by Avago 9260-4i controller
From: Wolfgang Denk @ 2016-05-15 13:37 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20160515124534.A42D0100879@atlas.denx.de>
Hi again,
In message <20160515124534.A42D0100879@atlas.denx.de> I wrote:
>
> I managed to kill a RAID6...
...
Trying to follow the overlay method in [1], I run into errors; guess I
must be missing something:
[1] https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
# DEVICES="/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf"
# ls /dev/loop*
/dev/loop-control /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4
# parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES
# ls /dev/loop*
/dev/loop-control /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6
# parallel truncate -s4000G overlay-{/} ::: $DEVICES
# parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES
# OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES)
# echo $OVERLAYS
/dev/mapper/sda /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf
# dmsetup status
castor2-git_backup: 0 67108864 linear
live-base: 0 12582912 linear
castor2-f22: 0 67108864 linear
castor2-root: 0 134217728 linear
castor2-f19: 0 100663296 linear
castor2-f19: 100663296 33554432 linear
castor2-f21: 0 134217728 linear
castor2-f18: 0 67108864 linear
castor2-f20: 0 67108864 linear
sdf: 0 1953525168 snapshot 16/8388608000 16
sde: 0 1953525168 snapshot 16/8388608000 16
sdd: 0 1953525168 snapshot 16/8388608000 16
live-osimg-min: 0 12582912 snapshot 3688/3688 24
sdc: 0 1953525168 snapshot 16/8388608000 16
live-rw: 0 12582912 snapshot 462664/1048576 1816
sdb: 0 1953525168 snapshot 16/8388608000 16
sda: 0 1953525168 snapshot 16/8388608000 16
# devices="/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf"
# overlay_create()
> {
> free=$((`stat -c '%a*%S/1024/1024' -f .`))
> echo free ${free}M
> overlays=""
> overlay_remove
> for d in $devices; do
> b=$(basename $d)
> size_bkl=$(blockdev --getsz $d) # in 512 blocks/sectors
> # reserve 1M space for snapshot header
> # ext3 max file length is 2TB
> truncate -s$((((size_bkl+1)/2)+1024))K $b.ovr || (echo "Do you use ext4?"; return 1)
> loop=$(losetup -f --show -- $b.ovr)
> # https://www.kernel.org/doc/Documentation/device-mapper/snapshot.txt
> dmsetup create $b --table "0 $size_bkl snapshot $d $loop P 8"
> echo $d $((size_bkl/2048))M $loop /dev/mapper/$b
> overlays="$overlays /dev/mapper/$b"
> done
> overlays=${overlays# }
> }
# overlay_remove()
> {
> for d in $devices; do
> b=$(basename $d)
> [ -e /dev/mapper/$b ] && dmsetup remove $b && echo /dev/mapper/$b
> if [ -e $b.ovr ]; then
> echo $b.ovr
> l=$(losetup -j $b.ovr | cut -d : -f1)
> echo $l
> [ -n "$l" ] && losetup -d $(losetup -j $b.ovr | cut -d : -f1)
> rm -f $b.ovr &> /dev/null
> fi
> done
> }
# echo $OVERLAYS
/dev/mapper/sda /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf
# mdadm --create --force --verbose /dev/md2 --metadata=1.2 --level=6 --raid-devices=6 --chunk=16 --assume-clean $OVERLAYS
mdadm: layout defaults to left-symmetric
mdadm: super1.x cannot open /dev/mapper/sda: Device or resource busy
mdadm: /dev/mapper/sda is not suitable for this array.
mdadm: super1.x cannot open /dev/mapper/sdb: Device or resource busy
mdadm: /dev/mapper/sdb is not suitable for this array.
mdadm: super1.x cannot open /dev/mapper/sdc: Device or resource busy
mdadm: /dev/mapper/sdc is not suitable for this array.
mdadm: super1.x cannot open /dev/mapper/sdd: Device or resource busy
mdadm: /dev/mapper/sdd is not suitable for this array.
mdadm: super1.x cannot open /dev/mapper/sde: Device or resource busy
mdadm: /dev/mapper/sde is not suitable for this array.
mdadm: super1.x cannot open /dev/mapper/sdf: Device or resource busy
mdadm: /dev/mapper/sdf is not suitable for this array.
mdadm: create aborted
mdadm --assemble --force /dev/md2 $OVERLAYS
mdadm: /dev/mapper/sda is busy - skipping
mdadm: /dev/mapper/sdb is busy - skipping
mdadm: /dev/mapper/sdc is busy - skipping
mdadm: /dev/mapper/sdd is busy - skipping
mdadm: /dev/mapper/sde is busy - skipping
mdadm: /dev/mapper/sdf is busy - skipping
# overlay_create
free 1843M
device-mapper: remove ioctl on sda failed: Device or resource busy
Command failed
device-mapper: remove ioctl on sdb failed: Device or resource busy
Command failed
device-mapper: remove ioctl on sdc failed: Device or resource busy
Command failed
device-mapper: remove ioctl on sdd failed: Device or resource busy
Command failed
device-mapper: remove ioctl on sde failed: Device or resource busy
Command failed
device-mapper: remove ioctl on sdf failed: Device or resource busy
Command failed
device-mapper: create ioctl on sda failed: Device or resource busy
Command failed
/dev/sda 953869M /dev/loop11 /dev/mapper/sda
device-mapper: create ioctl on sdb failed: Device or resource busy
Command failed
/dev/sdb 953869M /dev/loop12 /dev/mapper/sdb
device-mapper: create ioctl on sdc failed: Device or resource busy
Command failed
/dev/sdc 953869M /dev/loop13 /dev/mapper/sdc
device-mapper: create ioctl on sdd failed: Device or resource busy
Command failed
/dev/sdd 953869M /dev/loop14 /dev/mapper/sdd
device-mapper: create ioctl on sde failed: Device or resource busy
Command failed
/dev/sde 953869M /dev/loop15 /dev/mapper/sde
device-mapper: create ioctl on sdf failed: Device or resource busy
Command failed
/dev/sdf 953869M /dev/loop16 /dev/mapper/sdf
What am I doing wrong?
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Do we define evil as the absence of goodness? It seems only logical
that shit happens--we discover this by the process of elimination."
-- Larry Wall
^ permalink raw reply
* MD RAID6 corrupted by Avago 9260-4i controller
From: Wolfgang Denk @ 2016-05-15 12:45 UTC (permalink / raw)
To: linux-raid
Hi,
I managed to kill a RAID6... My old server mainboard died, the new one
did not have PCI-X any more, so I bought it with an Avago 9260-4i, of
course after asking (but not verifying in the net, sic) that I can
export the disks as plain JBOD. Well, you cannot. So I played around
with a set of spare disks and realized that you can configure a RAID0
consisting of a single disk drive, and when you skip the initialization
of the array, it basically does what I need. OK, so I added my real
disks, and things looked fine. Then I added some new disks and
decided to set them up as a HW RAID6 to compare performance. But,
when I intended to start initialization of this new RAID6, the Avago
firmware silently also initalized all my previously untouched old
disks, and boom!
The original array was created this way:
# mdadm --create --verbose /dev/md2 --metadata=1.2 --level=6 \
--raid-devices=6 --chunk=16 --assume-clean /dev/sd[abefgh]
mdadm: layout defaults to left-symmetric
mdadm: size set to 976762448K
mdadm: array /dev/md2 started.
# mdadm -Q --detail /dev/md2
/dev/md2:
Version : 1.02
Creation Time : Tue Jan 18 12:38:15 2011
Raid Level : raid6
Array Size : 3907049792 (3726.05 GiB 4000.82 GB)
Used Dev Size : 1953524896 (1863.03 GiB 2000.41 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Tue Jan 18 12:38:15 2011
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Chunk Size : 16K
Name : 2
UUID : 7ae2c7ac:74b4b307:69c2de0e:a2735e73
Events : 0
Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
1 8 16 1 active sync /dev/sdb
2 8 64 2 active sync /dev/sde
3 8 80 3 active sync /dev/sdf
4 8 96 4 active sync /dev/sdg
5 8 112 5 active sync /dev/sdh
Now, I see this instead:
# cat /proc/mdstat
Personalities : [raid0] [raid1]
md120 : active raid0 sdf[0]
976224256 blocks super external:/md0/5 256k chunks
md121 : active raid0 sde[0]
976224256 blocks super external:/md0/4 256k chunks
md122 : active raid0 sdd[0]
976224256 blocks super external:/md0/3 256k chunks
md123 : active raid0 sdc[0]
976224256 blocks super external:/md0/2 256k chunks
md124 : active raid0 sdb[0]
976224256 blocks super external:/md0/1 256k chunks
md125 : active raid0 sda[0]
976224256 blocks super external:/md0/0 256k chunks
md126 : inactive sda[5](S) sdf[4](S) sde[3](S) sdd[2](S) sdc[1](S) sdb[0](S)
3229968 blocks super external:ddf
# mdadm -Q --detail /dev/md126
/dev/md126:
Version : ddf
Raid Level : container
Total Devices : 6
Working Devices : 6
Container GUID : 4C534920:20202020:10000079:10009260:446872B3:105E9355
(LSI 05/14/16 08:23:15)
Seq : 00000019
Virtual Disks : 11
Member Arrays : /dev/md120 /dev/md121 /dev/md122 /dev/md123 /dev/md124 /dev/md125
Number Major Minor RaidDevice
0 8 16 - /dev/sdb
1 8 32 - /dev/sdc
2 8 48 - /dev/sdd
3 8 64 - /dev/sde
4 8 80 - /dev/sdf
5 8 0 - /dev/sda
# mdadm -Q --detail /dev/md120
/dev/md120:
Container : /dev/md0, member 5
Raid Level : raid0
Array Size : 976224256 (931.00 GiB 999.65 GB)
Raid Devices : 1
Total Devices : 1
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K
Container GUID : 4C534920:20202020:10000079:10009260:446872B3:105E9355
(LSI 05/14/16 08:23:15)
Seq : 00000019
Virtual Disks : 11
Number Major Minor RaidDevice State
0 8 80 0 active sync /dev/sdf
# mdadm -Q --detail /dev/md121
/dev/md121:
Container : /dev/md0, member 4
Raid Level : raid0
Array Size : 976224256 (931.00 GiB 999.65 GB)
Raid Devices : 1
Total Devices : 1
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K
Container GUID : 4C534920:20202020:10000079:10009260:446872B3:105E9355
(LSI 05/14/16 08:23:15)
Seq : 00000019
Virtual Disks : 11
Number Major Minor RaidDevice State
0 8 64 0 active sync /dev/sde
# mdadm -Q --detail /dev/md122
/dev/md122:
Container : /dev/md0, member 3
Raid Level : raid0
Array Size : 976224256 (931.00 GiB 999.65 GB)
Raid Devices : 1
Total Devices : 1
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K
Container GUID : 4C534920:20202020:10000079:10009260:446872B3:105E9355
(LSI 05/14/16 08:23:15)
Seq : 00000019
Virtual Disks : 11
Number Major Minor RaidDevice State
0 8 48 0 active sync /dev/sdd
# mdadm -Q --detail /dev/md123
/dev/md123:
Container : /dev/md0, member 2
Raid Level : raid0
Array Size : 976224256 (931.00 GiB 999.65 GB)
Raid Devices : 1
Total Devices : 1
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K
Container GUID : 4C534920:20202020:10000079:10009260:446872B3:105E9355
(LSI 05/14/16 08:23:15)
Seq : 00000019
Virtual Disks : 11
Number Major Minor RaidDevice State
0 8 32 0 active sync /dev/sdc
# mdadm -Q --detail /dev/md124
/dev/md124:
Container : /dev/md0, member 1
Raid Level : raid0
Array Size : 976224256 (931.00 GiB 999.65 GB)
Raid Devices : 1
Total Devices : 1
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K
Container GUID : 4C534920:20202020:10000079:10009260:446872B3:105E9355
(LSI 05/14/16 08:23:15)
Seq : 00000019
Virtual Disks : 11
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
# mdadm -Q --detail /dev/md125
/dev/md125:
Container : /dev/md0, member 0
Raid Level : raid0
Array Size : 976224256 (931.00 GiB 999.65 GB)
Raid Devices : 1
Total Devices : 1
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Chunk Size : 256K
Container GUID : 4C534920:20202020:10000079:10009260:446872B3:105E9355
(LSI 05/14/16 08:23:15)
Seq : 00000019
Virtual Disks : 11
Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
Yes, I know I was stupid, but can anybody help? Is there a way to get
the old RAID6 setup running, just to recover the data (we have
backups on tape, but I figure the restore takes long....)
For the record: there were also two disks which had partitions which
were used for 2 x RAID1 arrays; these survived the Avago's firmware
initialization:
# mdadm -Q --detail /dev/md126
/dev/md126:
Version : 1.0
Creation Time : Fri Jan 21 11:34:46 2011
Raid Level : raid1
Array Size : 262132 (256.03 MiB 268.42 MB)
Used Dev Size : 262132 (256.03 MiB 268.42 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sun May 15 08:10:48 2016
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : localhost.localdomain:3
UUID : 28815077:9fe434a1:7fbd6fbb:46816ee0
Events : 847
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 113 1 active sync /dev/sdh1
# mdadm -Q --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Wed Jan 19 07:28:49 2011
Raid Level : raid1
Array Size : 970206800 (925.26 GiB 993.49 GB)
Used Dev Size : 970206800 (925.26 GiB 993.49 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Sun May 15 08:41:23 2016
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : castor.denx.de:4
UUID : 0551c50c:30e757d4:83368de2:9a8ff1e1
Events : 38662
Number Major Minor RaidDevice State
2 8 99 0 active sync /dev/sdg3
3 8 115 1 active sync /dev/sdh3
Thanks in advance!
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I am more bored than you could ever possibly be. Go back to work.
^ permalink raw reply
* Email
From: Richard Sun @ 2016-05-15 10:05 UTC (permalink / raw)
Hello,
My name is Mr. Richard Sun from Hong Kong. I want you to be my partner
in a business project. Contact me back via my private e-mail address for more
details;
ricadtang@ymail.com
Thank you.
Mr. Richard Sun.
^ permalink raw reply
* Re: recovering failed and unrecognizable RAID5 during mdadm --grow without backup
From: Claudiu Rad-Lohanel @ 2016-05-13 15:33 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Phil Turmel, linux-raid
In-Reply-To: <20160513143923.GA23787@EIS.leimen.priv>
On 5/13/2016 5:39 PM, Andreas Klauer wrote:
> On Fri, May 13, 2016 at 05:26:02PM +0300, Claudiu Rad-Lohanel wrote:
>> 1. i can't find any documentation anywhere about the --revert-reshape
>> option. how should the command look like? is it available in v3.3.2?
> IIRC it's --assemble --update=revert-reshape and it's not mentioned
> in the man page...
thanks. it would seem to do it, but, hangs, doesn't want to advance:
root@rescue /mnt # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : active raid5 dm-1[0] dm-3[4] dm-2[2] dm-0[1]
11668750848 blocks super 1.2 level 5, 64k chunk, algorithm 2
[4/4] [UUUU]
[===================>.] reshape = 99.9% (3889567232/3889583616)
finish=7338.6min speed=0K/sec
--
jazzman
^ permalink raw reply
* Re: [RFC] super1: error handling for super-block loading
From: Gioh Kim @ 2016-05-13 15:15 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid, Elmar Gerdes, Jinpu Wang
In-Reply-To: <wrfj4ma39i4l.fsf@redhat.com>
On 12.05.2016 21:43, Jes Sorensen wrote:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> Gioh Kim <gi-oh.kim@profitbricks.com> writes:
>> This is definitly not the right way to solve this problem. Error codes
>> are negative, and zero should _always_ mean success. Here you suddenly
>> introduced a new meaning to positive values of rv.
>>
>> I agree handling the error case needs to be fixed, so a better way to
>> solve this would be to bail out when the load_super() call fails and
>> stop there, the same way it does if add_internal_bitmap() fails.
>>
>> Ie. make it do something like this:
>>
>> rv = st->ss->load_super(st, fd2, NULL)==0) {
>> if (!rv) {
>> if (st->ss->add_internal_bitmap(
>> ....
>> } else {
>> pr_err("failed to load super-block.\n");
>> close(fd2);
>> return 1;
>> }
>>
>> Actually looking at that code, there's a couple of things to do to clean
>> it up and make it more readable.
> So I started looking at this, and ended up making a bunch of changes
> which should both resolve the issue you encountered and also cleans up
> this part of the code. I had to change add_internal_bitmap() to return 0
> on success, in order to get it cleaned up. Looks like we have a pile of
> inconsistencies on the 0 vs 1 as success returns ... guess we won't get
> bored.
>
> I just pushed this into git - let me know if it doesn't work for you.
GREATE!!
I pulled your patch and checked it works fine.
Following is how I tested.
1. I set one device as faulty.
# cat /proc/mdstat
Personalities : [raid1]
md101 : active raid1 dm-6[1] dm-1[0](F)
16384 blocks super 1.2 [2/1] [_U]
unused devices: <none>
2. I disconnected storage.
3. recover bitmap
# mdadm --grow --bitmap=internal /dev/md101
Segmentation fault
4. new mdadm including your patch
# mdadm --grow --bitmap=internal /dev/md101
failed to load super-block.
# echo $?
1
Thank you very much.
Have a nice weekend!
--
Best regards,
Gioh Kim
^ permalink raw reply
* Re: recovering failed and unrecognizable RAID5 during mdadm --grow without backup
From: Andreas Klauer @ 2016-05-13 14:39 UTC (permalink / raw)
To: Claudiu Rad-Lohanel; +Cc: Phil Turmel, linux-raid
In-Reply-To: <3b4ae5f5-af19-744b-cab3-aad317ea3a82@misalpina.net>
On Fri, May 13, 2016 at 05:26:02PM +0300, Claudiu Rad-Lohanel wrote:
> 1. i can't find any documentation anywhere about the --revert-reshape
> option. how should the command look like? is it available in v3.3.2?
IIRC it's --assemble --update=revert-reshape and it's not mentioned
in the man page...
> 2. is it safe?
Theoretically...
You could do this with overlays.
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
That's what I'd also recommend when running fsck...
all of these things write on your disks without providing an undo function.
Regards
Andreas Klauer
^ permalink raw reply
* Re: recovering failed and unrecognizable RAID5 during mdadm --grow without backup
From: Claudiu Rad-Lohanel @ 2016-05-13 14:26 UTC (permalink / raw)
To: Phil Turmel, linux-raid
In-Reply-To: <5735E07F.2030802@turmel.org>
On 5/13/2016 5:11 PM, Phil Turmel wrote:
> On 05/13/2016 10:04 AM, Phil Turmel wrote:
>> On 05/12/2016 05:37 PM, Claudiu Rad wrote:
>>
>>> how can i safely stop this reshape and assuming my / partition inside
>>> the array is sane enough restart the actual server normally after
>>> fsck-ing all volumes?
>> Well, your root is inside the array. So you won't be able to boot
>> without the array assembling in the initramfs, which needs manual
>> intervention to supply the backup file.
> Actually, if it still isn't too far into the reshape, you could use
> --revert-reshape. Then it'll reshape back to the original chunk size
> what it has done so far. That might be quicker than finishing the
> reshape. Then you could reboot into your normal OS.
this is interesting.
1. i can't find any documentation anywhere about the --revert-reshape
option. how should the command look like? is it available in v3.3.2?
2. is it safe?
--
jazzman
^ permalink raw reply
* Re: recovering failed and unrecognizable RAID5 during mdadm --grow without backup
From: Phil Turmel @ 2016-05-13 14:11 UTC (permalink / raw)
To: Claudiu Rad, linux-raid
In-Reply-To: <5735DF05.8080706@turmel.org>
On 05/13/2016 10:04 AM, Phil Turmel wrote:
> On 05/12/2016 05:37 PM, Claudiu Rad wrote:
>
>> how can i safely stop this reshape and assuming my / partition inside
>> the array is sane enough restart the actual server normally after
>> fsck-ing all volumes?
>
> Well, your root is inside the array. So you won't be able to boot
> without the array assembling in the initramfs, which needs manual
> intervention to supply the backup file.
Actually, if it still isn't too far into the reshape, you could use
--revert-reshape. Then it'll reshape back to the original chunk size
what it has done so far. That might be quicker than finishing the
reshape. Then you could reboot into your normal OS.
Whether you finish reshaping, or unreshaping, you need to not be
reshaping at all when you boot into your normal OS, if you can't do the
initramfs console.
Phil
^ permalink raw reply
* Re: recovering failed and unrecognizable RAID5 during mdadm --grow without backup
From: Phil Turmel @ 2016-05-13 14:04 UTC (permalink / raw)
To: Claudiu Rad, linux-raid
In-Reply-To: <7cf56631-7909-6a92-f0b2-05dd02722ee8@misalpina.net>
On 05/12/2016 05:37 PM, Claudiu Rad wrote:
> how can i safely stop this reshape and assuming my / partition inside
> the array is sane enough restart the actual server normally after
> fsck-ing all volumes?
Well, your root is inside the array. So you won't be able to boot
without the array assembling in the initramfs, which needs manual
intervention to supply the backup file.
If your hosting service provides access to the boot console, you could
do this with "rd.shell" or whatever kernel option tells your initramfs
to drop into a rescue console. Then you could manually assemble and
resume booting. This would be true even if you had properly specified a
backup file outside the array.
I suspect this was one of the motivations for the enhancement to modern
mdadm & kernels to avoid the backup file for many cases by manipulating
the data offset. Such a reshape in progress can be automatically
assembled without special options during boot. Plus it is faster -- one
third fewer I/O operations.
Phil
^ permalink raw reply
* Re: recovering failed and unrecognizable RAID5 during mdadm --grow without backup
From: Phil Turmel @ 2016-05-12 20:23 UTC (permalink / raw)
To: Claudiu Rad-Lohanel, linux-raid
In-Reply-To: <f06998d6-8e1e-c8ea-a22c-1307935092c3@misalpina.net>
On 05/12/2016 04:09 PM, Claudiu Rad-Lohanel wrote:
>
>
> On 5/12/2016 9:58 PM, Phil Turmel wrote:
>> Please show the examine for the individual partitions of the raid5:
>>
>> mdadm --examine /dev/sd[a-d]3
>>
>
> root@rescue ~ # mdadm --examine /dev/sd[a-d]3
> /dev/sda3:
Ok. Nothing outlandish.
>> You will need to manually assemble (not create !) your array with a
>> backup file outside the raid5, and the --invalid-backup option to
>> abandon the backup file you can't get to. You will likely have some
>> unavoidable corruption at the reshape position due to this.
>
> i am waiting for your input on this and how to continue. it seems that i
> actually set new chunk size to 64K not 128K as i was remembering.
> clearly i wasn't with a clear mind when i did all this..
> should i be worried that reshape position is so at the beginning of the
> volume? maybe LVM vg0 metadata lost? (i am just assuming, don't know
> much about how and where LVM stores info about its volumes).
It just didn't get very far.
> the backup file is there, inside the array, if i could reach it somehow
> i could feed it to mdadm and would probably go well afterwards.
No way to get to it without assembling, and you can't assemble
error-free without it. Sorry.
> anyway, if just data is lost, i don't care, what are really important
> are some LVM volumes probably placed much further inside the array.
They are likely to be fine, then.
> thank you phil!
You're welcome.
You should mount your /boot array somewhere convenient, then:
mdadm -Av /dev/md3 --invalid-backup \
--backup-file=/mount/path/to/boot/newbackupfile \
/dev/sd[a-d]3
If that fails, repeat with the --force option included. If that fails,
show us everything it prints out.
If it succeeds, the reshape will be continuing in the background. While
that is going on, you may mount the array and grab backups of the most
critical content. Just in case :-)
It will probably take a very long time. Look at /proc/mdstat to check
the progress.
Phil
^ permalink raw reply
* Re: recovering failed and unrecognizable RAID5 during mdadm --grow without backup
From: Claudiu Rad-Lohanel @ 2016-05-12 20:09 UTC (permalink / raw)
To: Phil Turmel, linux-raid
In-Reply-To: <5734D23D.6030400@turmel.org>
On 5/12/2016 9:58 PM, Phil Turmel wrote:
> Please show the examine for the individual partitions of the raid5:
>
> mdadm --examine /dev/sd[a-d]3
>
root@rescue ~ # mdadm --examine /dev/sd[a-d]3
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : a935894f:be435fc0:589c1c7f:d5454b43
Name : rescue:2 (local to host rescue)
Creation Time : Mon Apr 14 15:22:47 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7779167887 (3709.40 GiB 3982.93 GB)
Array Size : 11668750848 (11128.19 GiB 11948.80 GB)
Used Dev Size : 7779167232 (3709.40 GiB 3982.93 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=655 sectors
State : active
Device UUID : 9bd5271f:9cb24f1f:f27b2d29:71320066
Reshape pos'n : 49152 (48.01 MiB 50.33 MB)
New Chunksize : 64K
Update Time : Wed May 11 16:19:38 2016
Checksum : 286cd938 - correct
Events : 11526
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : a935894f:be435fc0:589c1c7f:d5454b43
Name : rescue:2 (local to host rescue)
Creation Time : Mon Apr 14 15:22:47 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7779167887 (3709.40 GiB 3982.93 GB)
Array Size : 11668750848 (11128.19 GiB 11948.80 GB)
Used Dev Size : 7779167232 (3709.40 GiB 3982.93 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=655 sectors
State : active
Device UUID : fe992c5f:cf125d01:9bb8e3f7:572aef37
Reshape pos'n : 49152 (48.01 MiB 50.33 MB)
New Chunksize : 64K
Update Time : Wed May 11 16:19:38 2016
Checksum : eb24325e - correct
Events : 11526
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : a935894f:be435fc0:589c1c7f:d5454b43
Name : rescue:2 (local to host rescue)
Creation Time : Mon Apr 14 15:22:47 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7779167887 (3709.40 GiB 3982.93 GB)
Array Size : 11668750848 (11128.19 GiB 11948.80 GB)
Used Dev Size : 7779167232 (3709.40 GiB 3982.93 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=655 sectors
State : active
Device UUID : 0eb93951:876cbbad:46c6004c:0101f3ca
Reshape pos'n : 49152 (48.01 MiB 50.33 MB)
New Chunksize : 64K
Update Time : Wed May 11 16:19:38 2016
Checksum : 70b08f7d - correct
Events : 11526
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : a935894f:be435fc0:589c1c7f:d5454b43
Name : rescue:2 (local to host rescue)
Creation Time : Mon Apr 14 15:22:47 2014
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 7779167887 (3709.40 GiB 3982.93 GB)
Array Size : 11668750848 (11128.19 GiB 11948.80 GB)
Used Dev Size : 7779167232 (3709.40 GiB 3982.93 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=655 sectors
State : active
Device UUID : 957d7ddb:dc6de4e7:feb6fb1f:7776adcc
Reshape pos'n : 49152 (48.01 MiB 50.33 MB)
New Chunksize : 64K
Update Time : Wed May 11 16:19:38 2016
Checksum : ad2bb8a - correct
Events : 11526
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
> You will need to manually assemble (not create !) your array with a
> backup file outside the raid5, and the --invalid-backup option to
> abandon the backup file you can't get to. You will likely have some
> unavoidable corruption at the reshape position due to this.
i am waiting for your input on this and how to continue. it seems that i
actually set new chunk size to 64K not 128K as i was remembering.
clearly i wasn't with a clear mind when i did all this..
should i be worried that reshape position is so at the beginning of the
volume? maybe LVM vg0 metadata lost? (i am just assuming, don't know
much about how and where LVM stores info about its volumes).
the backup file is there, inside the array, if i could reach it somehow
i could feed it to mdadm and would probably go well afterwards.
anyway, if just data is lost, i don't care, what are really important
are some LVM volumes probably placed much further inside the array.
thank you phil!
--
jazzman
^ permalink raw reply
* Re: [PATCH] super1: add more checks for NodeNumUpdate option
From: Jes Sorensen @ 2016-05-12 19:45 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-raid
In-Reply-To: <1462959096-11272-1-git-send-email-gqjiang@suse.com>
Guoqing Jiang <gqjiang@suse.com> writes:
> There are some cases which didn't need to check the space
> is enough or not for NodeNumUpdate option.
>
> 1. for array which does not have clustered bitmap.
> 2. "--nodes" parameter is 0 (eg, add a disk to clustered raid).
> 3. if "--nodes" parameter is set to a smaller num than
> current bms->nodes.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> super1.c | 22 +++++++++++++++++++---
> 1 file changed, 19 insertions(+), 3 deletions(-)
Applied!
Thanks,
Jes
^ permalink raw reply
* Re: [RFC] super1: error handling for super-block loading
From: Jes Sorensen @ 2016-05-12 19:43 UTC (permalink / raw)
To: Gioh Kim; +Cc: linux-raid, Elmar Gerdes, Jinpu Wang
In-Reply-To: <wrfj8tzfb0r0.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Gioh Kim <gi-oh.kim@profitbricks.com> writes:
> This is definitly not the right way to solve this problem. Error codes
> are negative, and zero should _always_ mean success. Here you suddenly
> introduced a new meaning to positive values of rv.
>
> I agree handling the error case needs to be fixed, so a better way to
> solve this would be to bail out when the load_super() call fails and
> stop there, the same way it does if add_internal_bitmap() fails.
>
> Ie. make it do something like this:
>
> rv = st->ss->load_super(st, fd2, NULL)==0) {
> if (!rv) {
> if (st->ss->add_internal_bitmap(
> ....
> } else {
> pr_err("failed to load super-block.\n");
> close(fd2);
> return 1;
> }
>
> Actually looking at that code, there's a couple of things to do to clean
> it up and make it more readable.
So I started looking at this, and ended up making a bunch of changes
which should both resolve the issue you encountered and also cleans up
this part of the code. I had to change add_internal_bitmap() to return 0
on success, in order to get it cleaned up. Looks like we have a pile of
inconsistencies on the 0 vs 1 as success returns ... guess we won't get
bored.
I just pushed this into git - let me know if it doesn't work for you.
Cheers,
Jes
^ permalink raw reply
* Re: recovering failed and unrecognizable RAID5 during mdadm --grow without backup
From: Phil Turmel @ 2016-05-12 18:58 UTC (permalink / raw)
To: Claudiu Rad, linux-raid
In-Reply-To: <1b26650f-0b2c-2485-f781-e3fd26340741@misalpina.net>
On 05/12/2016 02:22 AM, Claudiu Rad wrote:
> hello all,
Please show the examine for the individual partitions of the raid5:
mdadm --examine /dev/sd[a-d]3
{ Replace the '3' if appropriate. You don't say what partition numbers
your raid5 is on. }
You will need to manually assemble (not create !) your array with a
backup file outside the raid5, and the --invalid-backup option to
abandon the backup file you can't get to. You will likely have some
unavoidable corruption at the reshape position due to this.
Phil
^ permalink raw reply
* Re: [RFC] super1: error handling for super-block loading
From: Jes Sorensen @ 2016-05-12 18:16 UTC (permalink / raw)
To: Gioh Kim; +Cc: linux-raid, Elmar Gerdes, Jinpu Wang
In-Reply-To: <5734BFA3.1070405@profitbricks.com>
Gioh Kim <gi-oh.kim@profitbricks.com> writes:
> I'm not sure yet why mdadm failed to load super-block of disks.
> I checked the kernel log and found I/O error from disks.
> Anyway mdadm needs to handle that error case.
>
> Please review following patch.
>
> ------------------------------------------- 8<
> -------------------------------------------------------------
> From 8cacf56b2d630c7e74bad942779ff7ed5f516d26 Mon Sep 17 00:00:00 2001
> From: Gioh Kim <gi-oh.kim@profitbricks.com>
> Date: Thu, 12 May 2016 19:09:45 +0200
> Subject: [PATCH] super1: error handling for super-block loading
>
> Loading super-block can fail if all sub-devices are faulty
> or have I/O errors.
>
> Signed-off-by: Gioh Kim <gi-oh.kim@profitbricks.com>
> ---
> Grow.c | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/Grow.c b/Grow.c
> index f58c753..fa08522 100755
> --- a/Grow.c
> +++ b/Grow.c
> @@ -389,7 +389,7 @@ int Grow_addbitmap(char *devname, int fd, struct
> context *c, struct shape *s)
> }
> if (strcmp(s->bitmap_file, "internal") == 0 ||
> strcmp(s->bitmap_file, "clustered") == 0) {
> - int rv;
> + int rv = 0;
> int d;
> int offset_setable = 0;
> struct mdinfo *mdi;
> @@ -419,6 +419,7 @@ int Grow_addbitmap(char *devname, int fd, struct
> context *c, struct shape *s)
> if (fd2 < 0)
> continue;
> if (st->ss->load_super(st, fd2, NULL)==0) {
> + rv++;
> if (st->ss->add_internal_bitmap(
> st,
> &s->bitmap_chunk, c->delay, s->write_behind,
> @@ -435,6 +436,10 @@ int Grow_addbitmap(char *devname, int fd, struct
> context *c, struct shape *s)
> close(fd2);
> }
> }
> + if (rv == 0) {
> + pr_err("failed to load super-block.\n");
> + return 1;
> + }
> if (offset_setable) {
> st->ss->getinfo_super(st, mdi, NULL);
> sysfs_init(mdi, fd, NULL);
This is definitly not the right way to solve this problem. Error codes
are negative, and zero should _always_ mean success. Here you suddenly
introduced a new meaning to positive values of rv.
I agree handling the error case needs to be fixed, so a better way to
solve this would be to bail out when the load_super() call fails and
stop there, the same way it does if add_internal_bitmap() fails.
Ie. make it do something like this:
rv = st->ss->load_super(st, fd2, NULL)==0) {
if (!rv) {
if (st->ss->add_internal_bitmap(
....
} else {
pr_err("failed to load super-block.\n");
close(fd2);
return 1;
}
Actually looking at that code, there's a couple of things to do to clean
it up and make it more readable.
Cheers,
Jes
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox