* Re: [PATCH] Add '--nodes' option in GROW mode
From: Guoqing Jiang @ 2016-03-23 11:38 UTC (permalink / raw)
To: Jes Sorensen, zhilong; +Cc: linux-raid
In-Reply-To: <wrfjshzi9z1l.fsf@redhat.com>
On 03/23/2016 01:56 AM, Jes Sorensen wrote:
> zhilong <zlliu@suse.com> writes:
>> Bug description:
>> # mdadm -Cv /dev/md0 --bitmap=clustered -l1 -n2 /dev/sdd /dev/sdb
>> --nodes=6 --assume-clean
>> # mdadm --grow /dev/md0 --bitmap=none
>> # mdadm --grow /dev/md0 --bitmap=clustered
>> -> cluster nodes would be changed to 4, here is my method to
>> resolve it.
>> # mdadm --grow /dev/md0 --bitmap=clustered --nodes=6
>>
>> Patch:
>>
>> Cluster nodes is set 4 as default when switch bitmap from
>> none to clustered under GROW mode, so add '--nodes'
>> option in GROW mode.
>>
>> Signed-off-by: Zhilong Liu <zlliu@suse.com>
>> ---
>> mdadm.c | 1 +
>> 1 file changed, 1 insertion(+)
> Hi,
>
> I have a question about this, since I simply do not have the knowledge
> about clustering to judge this approach.
>
> If you remove the bitmap, does that 'uncluster' the array? If not,
Yes, remove the bitmap would 'uncluster' the array since leave()
will be called in kernel if switch the bitmap from 'clustered' to
'none'.
> shouldn't mdadm be fixed to not set #nodes to 4 when adding a clustered
> bitmap, if a number of cluster nodes is already present?
The nodes is set 4 by default if the 'nodes' parameter is not specified,
And it could be overwrite if re-create the clustered bitmap.
Thanks,
Guoqing
^ permalink raw reply
* Re: [PATCH] Grow: goto release if Manage_subdevs failed
From: Guoqing Jiang @ 2016-03-23 1:13 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <wrfjwpou9z4r.fsf@redhat.com>
On 03/23/2016 01:54 AM, Jes Sorensen wrote:
> Guoqing Jiang <gqjiang@suse.com> writes:
>> If failure happened when add disk to array
>> by grow mode, need to goto release instead
>> of continue the reshape.
>>
>> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
>> ---
>> Grow.c | 5 +++--
>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/Grow.c b/Grow.c
>> index cf2750a..1767a1e 100755
>> --- a/Grow.c
>> +++ b/Grow.c
>> @@ -3003,8 +3003,9 @@ static int reshape_array(char *container, int fd, char *devname,
>> * level and frozen, we can safely add them.
>> */
>> if (devlist)
>> - Manage_subdevs(devname, fd, devlist, verbose,
>> - 0,NULL, 0);
>> + if (Manage_subdevs(devname, fd, devlist, verbose,
>> + 0,NULL, 0))
>> + goto release;
>>
>> if (reshape.backup_blocks == 0 && data_offset != INVALID_SECTORS)
>> reshape.backup_blocks = reshape.before.data_disks * info->array.chunk_size/512;
> Applied with minor mod to add brackets to the outer if() case - I hope
> you are OK with this.
Thanks, I am fine with it :).
Regards,
Guoqing
>
> Cheers,
> Jes
>
> From 31dbeda730ab1c25040f2e9b5bc49008b8519adf Mon Sep 17 00:00:00 2001
> From: Guoqing Jiang <gqjiang@suse.com>
> Date: Tue, 22 Mar 2016 17:03:44 +0800
> Subject: [PATCH] Grow: goto release if Manage_subdevs failed
>
> If failure happened when add disk to array
> by grow mode, need to goto release instead
> of continue the reshape.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> ---
> Grow.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/Grow.c b/Grow.c
> index 3204135..5953db2 100755
> --- a/Grow.c
> +++ b/Grow.c
> @@ -3009,9 +3009,11 @@ static int reshape_array(char *container, int fd, char *devname,
> * array. Now that the array has been changed to the right
> * level and frozen, we can safely add them.
> */
> - if (devlist)
> - Manage_subdevs(devname, fd, devlist, verbose,
> - 0,NULL, 0);
> + if (devlist) {
> + if (Manage_subdevs(devname, fd, devlist, verbose,
> + 0, NULL, 0))
> + goto release;
> + }
>
> if (reshape.backup_blocks == 0 && data_offset != INVALID_SECTORS)
> reshape.backup_blocks = reshape.before.data_disks * info->array.chunk_size/512;
^ permalink raw reply
* Re: [PATCH] Add '--nodes' option in GROW mode
From: Jes Sorensen @ 2016-03-22 17:56 UTC (permalink / raw)
To: zhilong; +Cc: linux-raid
In-Reply-To: <56F13020.8020900@suse.com>
zhilong <zlliu@suse.com> writes:
> Bug description:
> # mdadm -Cv /dev/md0 --bitmap=clustered -l1 -n2 /dev/sdd /dev/sdb
> --nodes=6 --assume-clean
> # mdadm --grow /dev/md0 --bitmap=none
> # mdadm --grow /dev/md0 --bitmap=clustered
> -> cluster nodes would be changed to 4, here is my method to
> resolve it.
> # mdadm --grow /dev/md0 --bitmap=clustered --nodes=6
>
> Patch:
>
> Cluster nodes is set 4 as default when switch bitmap from
> none to clustered under GROW mode, so add '--nodes'
> option in GROW mode.
>
> Signed-off-by: Zhilong Liu <zlliu@suse.com>
> ---
> mdadm.c | 1 +
> 1 file changed, 1 insertion(+)
Hi,
I have a question about this, since I simply do not have the knowledge
about clustering to judge this approach.
If you remove the bitmap, does that 'uncluster' the array? If not,
shouldn't mdadm be fixed to not set #nodes to 4 when adding a clustered
bitmap, if a number of cluster nodes is already present?
Thanks,
Jes
>
> diff --git a/mdadm.c b/mdadm.c
> index 51e16f3..b542b8c 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -592,6 +592,7 @@ int main(int argc, char *argv[])
> ident.raid_disks = s.raiddisks;
> continue;
> case O(ASSEMBLE, Nodes):
> + case O(GROW, Nodes):
> case O(CREATE, Nodes):
> c.nodes = parse_num(optarg);
> if (c.nodes <= 0) {
^ permalink raw reply
* Re: [PATCH] Grow: goto release if Manage_subdevs failed
From: Jes Sorensen @ 2016-03-22 17:54 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-raid
In-Reply-To: <1458637424-13887-1-git-send-email-gqjiang@suse.com>
Guoqing Jiang <gqjiang@suse.com> writes:
> If failure happened when add disk to array
> by grow mode, need to goto release instead
> of continue the reshape.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> Grow.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/Grow.c b/Grow.c
> index cf2750a..1767a1e 100755
> --- a/Grow.c
> +++ b/Grow.c
> @@ -3003,8 +3003,9 @@ static int reshape_array(char *container, int fd, char *devname,
> * level and frozen, we can safely add them.
> */
> if (devlist)
> - Manage_subdevs(devname, fd, devlist, verbose,
> - 0,NULL, 0);
> + if (Manage_subdevs(devname, fd, devlist, verbose,
> + 0,NULL, 0))
> + goto release;
>
> if (reshape.backup_blocks == 0 && data_offset != INVALID_SECTORS)
> reshape.backup_blocks = reshape.before.data_disks * info->array.chunk_size/512;
Applied with minor mod to add brackets to the outer if() case - I hope
you are OK with this.
Cheers,
Jes
From 31dbeda730ab1c25040f2e9b5bc49008b8519adf Mon Sep 17 00:00:00 2001
From: Guoqing Jiang <gqjiang@suse.com>
Date: Tue, 22 Mar 2016 17:03:44 +0800
Subject: [PATCH] Grow: goto release if Manage_subdevs failed
If failure happened when add disk to array
by grow mode, need to goto release instead
of continue the reshape.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
Grow.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/Grow.c b/Grow.c
index 3204135..5953db2 100755
--- a/Grow.c
+++ b/Grow.c
@@ -3009,9 +3009,11 @@ static int reshape_array(char *container, int fd, char *devname,
* array. Now that the array has been changed to the right
* level and frozen, we can safely add them.
*/
- if (devlist)
- Manage_subdevs(devname, fd, devlist, verbose,
- 0,NULL, 0);
+ if (devlist) {
+ if (Manage_subdevs(devname, fd, devlist, verbose,
+ 0, NULL, 0))
+ goto release;
+ }
if (reshape.backup_blocks == 0 && data_offset != INVALID_SECTORS)
reshape.backup_blocks = reshape.before.data_disks * info->array.chunk_size/512;
--
2.5.5
^ permalink raw reply related
* Hi linux
From: tirumalareddy marri @ 2016-03-22 13:59 UTC (permalink / raw)
To: linux raid
hiya linux
http://www.katielister.co.uk/physical.php?coast=pbay13az2nhbn57d
tirumalareddy marri
^ permalink raw reply
* [PATCH] Add '--nodes' option in GROW mode
From: zhilong @ 2016-03-22 11:44 UTC (permalink / raw)
To: linux-raid
Bug description:
# mdadm -Cv /dev/md0 --bitmap=clustered -l1 -n2 /dev/sdd /dev/sdb
--nodes=6 --assume-clean
# mdadm --grow /dev/md0 --bitmap=none
# mdadm --grow /dev/md0 --bitmap=clustered
-> cluster nodes would be changed to 4, here is my method to
resolve it.
# mdadm --grow /dev/md0 --bitmap=clustered --nodes=6
Patch:
Cluster nodes is set 4 as default when switch bitmap from
none to clustered under GROW mode, so add '--nodes'
option in GROW mode.
Signed-off-by: Zhilong Liu <zlliu@suse.com>
---
mdadm.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/mdadm.c b/mdadm.c
index 51e16f3..b542b8c 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -592,6 +592,7 @@ int main(int argc, char *argv[])
ident.raid_disks = s.raiddisks;
continue;
case O(ASSEMBLE, Nodes):
+ case O(GROW, Nodes):
case O(CREATE, Nodes):
c.nodes = parse_num(optarg);
if (c.nodes <= 0) {
--
2.1.4
^ permalink raw reply related
* [PATCH] Grow: goto release if Manage_subdevs failed
From: Guoqing Jiang @ 2016-03-22 9:03 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid, Guoqing Jiang
If failure happened when add disk to array
by grow mode, need to goto release instead
of continue the reshape.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Grow.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/Grow.c b/Grow.c
index cf2750a..1767a1e 100755
--- a/Grow.c
+++ b/Grow.c
@@ -3003,8 +3003,9 @@ static int reshape_array(char *container, int fd, char *devname,
* level and frozen, we can safely add them.
*/
if (devlist)
- Manage_subdevs(devname, fd, devlist, verbose,
- 0,NULL, 0);
+ if (Manage_subdevs(devname, fd, devlist, verbose,
+ 0,NULL, 0))
+ goto release;
if (reshape.backup_blocks == 0 && data_offset != INVALID_SECTORS)
reshape.backup_blocks = reshape.before.data_disks * info->array.chunk_size/512;
--
2.6.2
^ permalink raw reply related
* Re: problems with dm-raid 6
From: Chris Murphy @ 2016-03-22 4:22 UTC (permalink / raw)
To: Chris Murphy; +Cc: Andreas Klauer, Patrick Tschackert, linux-raid
In-Reply-To: <CAJCQCtRELEP8WxVpiS_E7i7raz+JbPOYTGvVvswCVHW__BC+nA@mail.gmail.com>
On Mon, Mar 21, 2016 at 9:53 PM, Chris Murphy <lists@colorremedies.com> wrote:
So even though
> there's a duplicate of the chunk tree, since there's only one logical
> address in the super for the chunk tree, if that logical address
> doesn't resolve to a sane physical location, there's no way to find
> out where any copies are. So it's stuck.
Small clarification. The Btrfs superblock contains four backups. But
in Patrick's case, all four backups contain the same chunk root
address as the primary entry in the super (all five copies are the
same address) so... yeah stuck. Basically that address has to become
viable to start to get access to one or more copies of the filesystem,
including the root tree which is also a logical address. So none of
the trees can be sorted out without the chunk tree. There is a chunk
recover option, but I'm very skeptical of it being able to do anything
when the btrfs-find-root command fails totally for Patrick. There's
just not enough information available.
So I think the key is to get at the least n-2 drives up in a degraded
array and then yes indeed we'll have magic, or unicorn poop cookies,
or whatever...
--
Chris Murphy
^ permalink raw reply
* Re: problems with dm-raid 6
From: Chris Murphy @ 2016-03-22 3:53 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Patrick Tschackert, Chris Murphy, linux-raid
In-Reply-To: <20160321230456.GA16913@EIS>
On Mon, Mar 21, 2016 at 5:04 PM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> Of course you can also attempt to repair btrfs directly but
> if btrfs redundancy is not equal to RAID-6 then it won't be
> able to fix. (I think you already cleared that point on
> the btrfs mailing list and would not be asking here if
> btrfs had the magic ability to recover)
No such unicorns in Btrfs.
In the Btrfs thread, I suggested the md scrub "check", and when I saw
that high count I was like, no Patrick, do not change anything, don't
run any Btrfs repair tools, go talk to the linux-raid@ folks. Those
mismatches are not read errors. Those are discrepancies between data
and parity strips (chunks for md folks, I use SNIA strip terminology
because in Btrfs a chunk is a kind of super-extent or collection of
extents, either metadata or data).
Patrick, do you remember what your mkfs options were?
If default, it will be single profile for data chunks, and DUP for
metadata chunks. That means there's only one copy of data extents, and
two copies of metadata. The file system is the metadata including all
checksums for data and metadata, and all trees. The problem though is
there's a possible single source of failure in really narrow cases.
Everything in Btrfs land is a logical address. if you do filefrag on a
file, the physical address * 4096 is the logical address within
Btrfs's address space, it's not a physical block. That address has to
be referenced in the chunk tree, which is what references two things:
what device(s) *and* what sector on those devices. So even though
there's a duplicate of the chunk tree, since there's only one logical
address in the super for the chunk tree, if that logical address
doesn't resolve to a sane physical location, there's no way to find
out where any copies are. So it's stuck.
Per the Btrfs FAQ:
------
There are three superblocks: the first one is located at 64K, the
second one at 64M, the third one at 256GB. The following lines reset
the magic string on all the three superblocks
# dd if=/dev/zero bs=1 count=8 of=/dev/sda seek=$((64*1024+64))
# dd if=/dev/zero bs=1 count=8 of=/dev/sda seek=$((64*1024*1024+64))
# dd if=/dev/zero bs=1 count=8 of=/dev/sda seek=$((256*1024*1024*1024+64))
------
So if someone can do the math and figure out what physical devices
those supers might be on, with a 64K chunk and 9 devices, might be
funny if they end up all on one drive... tragically funny. Hopefully
they're on multiple drives thought, and this suggests at least the
critical minimum number of drives are still sane and this can be
recovered even minus two drives.
If there's more than two drives toasted, then I don't think Btrfs can
help at all - same as any other file system.
--
Chris Murphy
^ permalink raw reply
* Re: problems with dm-raid 6
From: Adam Goryachev @ 2016-03-21 23:48 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Patrick Tschackert, lists, linux-raid
In-Reply-To: <20160321231533.GB16913@EIS>
On 22/03/16 10:15, Andreas Klauer wrote:
> On Tue, Mar 22, 2016 at 09:54:54AM +1100, Adam Goryachev wrote:
>> To my untrained eye, it looks like maybe the "first" drive in your array is correct, and
>> hence the first block returns the correct data so you can access the
>> LUKS, but the second (or third, or fourth) is damaged, and thats why you
>> can't read the filesystem inside the LUKS.
> This is a "problem" you get with arrays of many disks, if you "forget"
> the correct drive order and then "create" the RAID anew, it might
> result in a perfectly mountable filesystem but with errors down the
> way since the first "wrong" data may appear outside the filesystems
> immediate metadata zone, if two later disks switched places.
>
> However the OP only uses 64K chunksize, so that gives a lot less
> valid data than you'd get with 512K chunks. The LUKS header is already
> larger than 64K so if there is really bad data on one of the disks
> throughout, it's already quite lucky for the LUKS header to have
> survived. May be a good idea to grab a backup of that header while
> it's still working anyhow.
>
> The one disk full of bad data theory might not even be correct,
> maybe a sync started, and somehow the disk got accepted as fully
> synced even though it didn't... because the controller silently
> ignored all writes? Mysterious selective hardware failure?
>
>> Once you can do that, then either the filesystem will "Just Work" or
>> else you might need to do a repair depending on what exactly went wrong,
>> and how much was written during that time.
> Hope dies last.
>
> If btrfs stored data the same way a traditional filesystem would,
> uncompressed unencrypted unfragmented, you could hunt the raw data
> of your md for magic headers of known large files and see if you
> can tell in more detail the type of damage.
>
> For example if you could find a large megapixel JPEG image like
> that and were able to load it but it would appear corrupted at
> some point, the point of corruption might point you to the
> disk you no longer want to be in your array.
>
> But I don't know enough/anything about btrfs so not sure if viable.
All of that is true, but it is a LUKS (encrypted) partition, so even if
the filesystem format was simple, you wouldn't be able to work it out
because all the data is encrypted. (At least, that is the point of data
encryption right.... in practice maybe there are still some options, but
I'm definitely out of my depth there).
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: problems with dm-raid 6
From: Andreas Klauer @ 2016-03-21 23:15 UTC (permalink / raw)
To: Adam Goryachev; +Cc: Patrick Tschackert, lists, linux-raid
In-Reply-To: <56F07BBE.1010306@websitemanagers.com.au>
On Tue, Mar 22, 2016 at 09:54:54AM +1100, Adam Goryachev wrote:
> To my untrained eye, it looks like maybe the "first" drive in your array is correct, and
> hence the first block returns the correct data so you can access the
> LUKS, but the second (or third, or fourth) is damaged, and thats why you
> can't read the filesystem inside the LUKS.
This is a "problem" you get with arrays of many disks, if you "forget"
the correct drive order and then "create" the RAID anew, it might
result in a perfectly mountable filesystem but with errors down the
way since the first "wrong" data may appear outside the filesystems
immediate metadata zone, if two later disks switched places.
However the OP only uses 64K chunksize, so that gives a lot less
valid data than you'd get with 512K chunks. The LUKS header is already
larger than 64K so if there is really bad data on one of the disks
throughout, it's already quite lucky for the LUKS header to have
survived. May be a good idea to grab a backup of that header while
it's still working anyhow.
The one disk full of bad data theory might not even be correct,
maybe a sync started, and somehow the disk got accepted as fully
synced even though it didn't... because the controller silently
ignored all writes? Mysterious selective hardware failure?
> Once you can do that, then either the filesystem will "Just Work" or
> else you might need to do a repair depending on what exactly went wrong,
> and how much was written during that time.
Hope dies last.
If btrfs stored data the same way a traditional filesystem would,
uncompressed unencrypted unfragmented, you could hunt the raw data
of your md for magic headers of known large files and see if you
can tell in more detail the type of damage.
For example if you could find a large megapixel JPEG image like
that and were able to load it but it would appear corrupted at
some point, the point of corruption might point you to the
disk you no longer want to be in your array.
But I don't know enough/anything about btrfs so not sure if viable.
Regards,
Andreas Klauer
^ permalink raw reply
* Re: problems with dm-raid 6
From: Andreas Klauer @ 2016-03-21 23:04 UTC (permalink / raw)
To: Patrick Tschackert; +Cc: lists, linux-raid
In-Reply-To: <trinity-226526b2-6058-40bf-a276-9c1b47a80b3b-1458600120646@3capp-gmx-bs35>
On Mon, Mar 21, 2016 at 11:42:00PM +0100, Patrick Tschackert wrote:
> Based on your advice, I'll use an overlay on the array
Below the array. Assemble an array of overlay disks with two missing.
One of the missing disks should be that spare you synced in, because
that's one of the prime suspects... actually if it was synced to it
should not be the cause of mismatches, but if there are mismatches
then it's likely the data that was synced to it is not good.
Although RAID is able to detect mismatches, it can not tell
which is the valid data and which is the invalid data so ...
Definitely assemble without that disk and without each other
disk in turn; so for each attempt two disks missing from
your RAID-6, hopefully one of them is a bad egg and with it
gone, things will look better to btrfs than they do now.
The ultimate cause of those many mismatches remains a mystery.
Of course you can also attempt to repair btrfs directly but
if btrfs redundancy is not equal to RAID-6 then it won't be
able to fix. (I think you already cleared that point on
the btrfs mailing list and would not be asking here if
btrfs had the magic ability to recover)
Regards,
Andreas
^ permalink raw reply
* Re: problems with dm-raid 6
From: Adam Goryachev @ 2016-03-21 22:54 UTC (permalink / raw)
To: Patrick Tschackert, lists, Andreas.Klauer; +Cc: linux-raid
In-Reply-To: <trinity-226526b2-6058-40bf-a276-9c1b47a80b3b-1458600120646@3capp-gmx-bs35>
On 22/03/16 09:42, Patrick Tschackert wrote:
> Hi Chris,
>
>> From what I understand, no, the smartctl are after the scrub check.
> That's correct, I ran those shortly before sending the OP.
>
>> But it's an open question how long after device failure he
>> actually noticed it before doing the rebuild and how he did that
>> rebuild;
> I noticed something was wrong directy after the reboot. I opened the LUKS container (which worked) and tried to mount the btrfs filesys, which didn't. After noticing the raid was in the state "clean, degraded", I triggered the restore by running mdadm --run, because i already had two hotspares present in the raid. So one of the hotspares was used to do the restore.
> After that, i ran mdadm --readwrite /dev/md0, because the raid was set to read only. The problem with mounting the btrfs filesys still occurred.
>
>> But at this point we need to hear back from Patrick.
> Sorry for taking so long, long day at work :(
>
>> You should not assemble the original RAID anymore anyhow, anything
>> you write to the array at this point will likely only increase
>> damages. The overlay allows you to experiment in read-"write"
>> mode without actually changing anything on your disks.
>> Agreed. If it turns out there are some repairs needed within Btrfs
>> it's better with the overlay because it's unclear based on the errors
>> thus far what repair step to use next, and some of these repair
>> attempts can still sometimes make things worse (which are of course
>> bugs, but nevertheless...)
> I'll look into overlays and try that tomorrow, it's too late and i don't want to further screw this up by doing this half asleep :/
> Based on your advice, I'll use an overlay on the array and then try to fix the btrfs filesystem. If I understand correctly, the dm overlay file would enable me to revert to the current state, in case the btrfs repair goes wrong?
>
> Regards,
> Patrick
I think the suggestion is to use overlays on each individual
drive/partition that is used in the RAID array, and then try/test adding
a subset of those drives to assemble the raid array. That way, any
writes to the raid will not damage the underlying data. To my untrained
eye, it looks like maybe the "first" drive in your array is correct, and
hence the first block returns the correct data so you can access the
LUKS, but the second (or third, or fourth) is damaged, and thats why you
can't read the filesystem inside the LUKS. Hence, try swapping the order
of the disks and/or leaving different disks out, and see if you can then
read both LUKS and the filesystem inside it.
Once you can do that, then either the filesystem will "Just Work" or
else you might need to do a repair depending on what exactly went wrong,
and how much was written during that time.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: problems with dm-raid 6
From: Patrick Tschackert @ 2016-03-21 22:42 UTC (permalink / raw)
To: lists, Andreas.Klauer; +Cc: linux-raid
Hi Chris,
> From what I understand, no, the smartctl are after the scrub check.
That's correct, I ran those shortly before sending the OP.
> But it's an open question how long after device failure he
> actually noticed it before doing the rebuild and how he did that
> rebuild;
I noticed something was wrong directy after the reboot. I opened the LUKS container (which worked) and tried to mount the btrfs filesys, which didn't. After noticing the raid was in the state "clean, degraded", I triggered the restore by running mdadm --run, because i already had two hotspares present in the raid. So one of the hotspares was used to do the restore.
After that, i ran mdadm --readwrite /dev/md0, because the raid was set to read only. The problem with mounting the btrfs filesys still occurred.
> But at this point we need to hear back from Patrick.
Sorry for taking so long, long day at work :(
> You should not assemble the original RAID anymore anyhow, anything
> you write to the array at this point will likely only increase
> damages. The overlay allows you to experiment in read-"write"
> mode without actually changing anything on your disks.
> Agreed. If it turns out there are some repairs needed within Btrfs
> it's better with the overlay because it's unclear based on the errors
> thus far what repair step to use next, and some of these repair
> attempts can still sometimes make things worse (which are of course
> bugs, but nevertheless...)
I'll look into overlays and try that tomorrow, it's too late and i don't want to further screw this up by doing this half asleep :/
Based on your advice, I'll use an overlay on the array and then try to fix the btrfs filesystem. If I understand correctly, the dm overlay file would enable me to revert to the current state, in case the btrfs repair goes wrong?
Regards,
Patrick
^ permalink raw reply
* Re: problems with dm-raid 6
From: Patrick Tschackert @ 2016-03-21 22:19 UTC (permalink / raw)
To: philip; +Cc: linux-raid
Hi Philip, thanks for answering!
> Your smartctl output shows pending sector problems with sdf, sdh, and
> sdj. The latter are WD Reds that won't keep those problems through a
> scrub, so I guess the smartctl report was from before that?
The smartctl results are "fresh", i ran the commands just before sending my last eMail.
>> mdadm --examine output for your disks?
>Yes, we want these.
Here: http://pastebin.com/JW8rbJYY
> Your mdadm -D output clearly shows a 2014 creation date,
> so you definitely hadn't done --create --assume-clean at that point.
> (Don't.)
I didn't do that, I used mdadm --run /dev/md0 to start the rebuild/restore
> Something else is wrong, quite possibly hardware. You don't get a
> mismatch count like that without it showing up in smartctl too, unless
> corrupt data was being written to one or more disks for a long time.
As I said in my initial eMail, I got
$ cat /sys/block/md0/md/mismatch_cnt
0
directly after the rebuild/restore. I then ran
$ for i in /sys/class/scsi_generic/*/device/timeout; do echo 120 > "$i"; done
to correct disk timeouts (got that advice from irc) and
$ echo check > /sys/block/md0/md/sync_action
to start a check on the raid. After the check was completed i got
$ cat /sys/block/md0/md/mismatch_cnt
311936608
> If you used ddrescue to replace sdf instead of letting mdadm reconstruct
> it, that would have introduced zero sectors that would scramble your
> encrypted filesystem. Please let us know that you didn't use ddrescue.
I didn't do that, I just ran mdadm --run /dev/md0, which started the rebuild, nothing else.
> The encryption inside your array will frustrate any attempt to do
> per-member analysis. I don't think there's anything still wrong with
> the array (anything fixable, that is).
> If an array error stomped on the key area of your dm-crypt layer, you
> are totally destroyed, unless you happen to have a key backup you can
> restore.
> Otherwise you are at the mercy of fsck to try to fix your volume. I
> would use an overlay for that.
Well, the key area seems alright, i can open the volume using "cryptsetup luksOpen /dev/md0 storage", it asks for my passphrase and then opens the volume.
I can even read the BTRFS superblock (of the filesys on my luks volume), so the whole thing doesn't seem to be completely borked.
I'll read up on overlays and try them maybe.
Kind regards
^ permalink raw reply
* Re: problems with dm-raid 6
From: Patrick Tschackert @ 2016-03-21 22:06 UTC (permalink / raw)
To: Andreas.Klauer; +Cc: linux-raid
Thank you for answering!
>> After rebooting the system, one of the harddisks was missing from my md raid 6 (the drive was /dev/sdf), so i rebuilt it with a hotspare that was already present in the system.
>> I physically removed the "missing" /dev/sdf drive after the restore and replaced it with a new drive.
>Exact commands involved for those steps?
Well since the /dev/sdf disk was missing from the array after the reboot, i didn't use any command to remove it. I just used
$ mdadm --run /dev/md0
to trigger the rebuild/restore. As i had two spare drives present in the array anyway, i thought that was the smartest thing to do.
After the restore was done, i shut down the system and swapped the missing disk (/dev/sdf) with a new one.
I then added the new disk to the array as a spare (mdadm --add /dev/md0 /dev/sdf)
> mdadm --examine output for your disks?
Here is the output for every disk in the array: http://pastebin.com/JW8rbJYY
> This is what you get when you use --create --assume-clean on disks
> that are not actually clean... or if you somehow convince md to
> integrate a disk that does not have valid data on, for example
> because you copied partition table and md metadata - but not
> everything else - using dd.
I didn't use that command or anything like that, i just triggered the rebuild with mdadm --run. It then started the restore (i monitored the progress by looking at /proc/mdstat), and it seemed to complete successfully.
> Your best bet is that the data is valid on n-2 disks.
> Use overlay https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
> Assemble the overlay RAID with any 2 disks missing (try all combinations) and see if you get valid data.
Thanks, I will definitely try that!
^ permalink raw reply
* Re: problems with dm-raid 6
From: Chris Murphy @ 2016-03-21 21:46 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Chris Murphy, Phil Turmel, Patrick Tschackert, linux-raid
In-Reply-To: <20160321213829.GA16677@EIS>
On Mon, Mar 21, 2016 at 3:38 PM, Andreas Klauer
<Andreas.Klauer@metamorpher.de> wrote:
> On Mon, Mar 21, 2016 at 03:26:30PM -0600, Chris Murphy wrote:
>> At this point I'm skeptical this will work. Also, I'm not familiar
>> with this overlay technique. I did look at the URL provided by
>> Andreas, my concern is whether it's possible for the volume UUID to
>> appear more than once to the kernel?
>
> Thanks for pointing out this issue, I'm not familiar with btrfs at all.
>
> Normally this would be a problem and you'd have to find some way
> to change the UUID in the overlay or whatever, but if it's LUKS
> encrypted that should solve the problem nicely, you just have to
> stop the original RAID and close the original LUKS container
> and then only open the one of the overlay.
Yes hiding it with LUKS is the way to go here. There's no practical
way to change the volume UUID as it's pervasive throughout the fs
metadata, it's used everywhere. Changing it with btrfstune means
completely rewriting out the entire fs metadata. It's the same now for
XFS v5 which similarly uses fs volume UUID everywhere as a part of
consistency checking, along with metadata checksumming. I'm not sure
about the optional ext4 checksumming but I think it too is making wide
use of fs volume UUID throughout metadata.
So I think we have to be careful with multiple instances of UUID
appearing and probably there need to be better checks in kernel code
to avoid confusion. Btrfs is harder because it explicitly supports
multiple devices, and the volume UUID correctly appears on all devices
(as well as a unique device specific UUID). So it's used to seeing
multiple instances of a volume UUID. But anyway, this is a side track
topic.
>
> You should not assemble the original RAID anymore anyhow, anything
> you write to the array at this point will likely only increase
> damages. The overlay allows you to experiment in read-"write"
> mode without actually changing anything on your disks.
Agreed. If it turns out there are some repairs needed within Btrfs
it's better with the overlay because it's unclear based on the errors
thus far what repair step to use next, and some of these repair
attempts can still sometimes make things worse (which are of course
bugs, but nevertheless...)
--
Chris Murphy
^ permalink raw reply
* Re: problems with dm-raid 6
From: Andreas Klauer @ 2016-03-21 21:38 UTC (permalink / raw)
To: Chris Murphy; +Cc: Phil Turmel, Patrick Tschackert, linux-raid
In-Reply-To: <CAJCQCtQDWdZPBvB1Q2Mh_AOBZpWSPoqK42meWvARhQi3hE5uzw@mail.gmail.com>
On Mon, Mar 21, 2016 at 03:26:30PM -0600, Chris Murphy wrote:
> At this point I'm skeptical this will work. Also, I'm not familiar
> with this overlay technique. I did look at the URL provided by
> Andreas, my concern is whether it's possible for the volume UUID to
> appear more than once to the kernel?
Thanks for pointing out this issue, I'm not familiar with btrfs at all.
Normally this would be a problem and you'd have to find some way
to change the UUID in the overlay or whatever, but if it's LUKS
encrypted that should solve the problem nicely, you just have to
stop the original RAID and close the original LUKS container
and then only open the one of the overlay.
You should not assemble the original RAID anymore anyhow, anything
you write to the array at this point will likely only increase
damages. The overlay allows you to experiment in read-"write"
mode without actually changing anything on your disks.
> The issue is raid6 assembly somehow being wonky.
> But at this point we need to hear back from Patrick.
Yes, it's quite a mystery, I hope the OP can shed some light on
what might have caused this.
Regards
Andreas Klauer
^ permalink raw reply
* Re: problems with dm-raid 6
From: Chris Murphy @ 2016-03-21 21:26 UTC (permalink / raw)
To: Phil Turmel; +Cc: Andreas Klauer, Patrick Tschackert, linux-raid
In-Reply-To: <56EFEC28.7020207@turmel.org>
Original thread on btrfs list (the OP links here didn't work for me):
http://www.spinics.net/lists/linux-btrfs/msg53143.html
On Mon, Mar 21, 2016 at 6:42 AM, Phil Turmel <philip@turmel.org> wrote:
> Hi Patrick,
>
> On 03/20/2016 06:37 PM, Andreas Klauer wrote:
>> On Sun, Mar 20, 2016 at 10:44:57PM +0100, Patrick Tschackert wrote:
>>> After rebooting the system, one of the harddisks was missing from my md raid 6 (the drive was /dev/sdf), so i rebuilt it with a hotspare that was already present in the system.
>>> I physically removed the "missing" /dev/sdf drive after the restore and replaced it with a new drive.
>
> Your smartctl output shows pending sector problems with sdf, sdh, and
> sdj. The latter are WD Reds that won't keep those problems through a
> scrub, so I guess the smartctl report was from before that?
From what I understand, no, the smartctl are after the scrub check.
The dmesg shows read errors but no md attempt to fix up those errors,
which I thought was strange but might be a good thing if the raid is
not assembled correctly.
>> Your best bet is that the data is valid on n-2 disks.
>>
>> Use overlay https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>>
>> Assemble the overlay RAID with any 2 disks missing (try all combinations) and see if you get valid data.
>
> No. Something else is wrong, quite possibly hardware. You don't get a
> mismatch count like that without it showing up in smartctl too, unless
> corrupt data was being written to one or more disks for a long time.
>
> It's unclear from your dmesg what might have happened. Probably bad
> stuff going back years.
Seems unlikely because this was a functioning raid6 with Btrfs on top.
So there'd have been a ton of Btrfs complaints.
I think something wrong happened with the device replace procedure, I
just can't tell what because all the devices are present and working
according to the -D output.
In that first message on the btrfs list you can see what things work
and don't work in more detail. The summary is, all three Btrfs super
blocks are found. This wouldn't be possible if the array weren't at
least partially correct, and also the LUKS volume were being unlocked
correctly. Unless there's something very nuanced and detailed we're
not understanding yet.
But as soon as commands are used to look for other things, there are
immediate failures, lots of metadata checksum errors, an inability to
read the chunk and root trees. So it's like there's a hole in the file
system. I just can't tell if it's a small one like the size of a drive
or a big one.
> Otherwise you are at the mercy of fsck to try to fix your volume. I
> would use an overlay for that.
At this point I'm skeptical this will work. Also, I'm not familiar
with this overlay technique. I did look at the URL provided by
Andreas, my concern is whether it's possible for the volume UUID to
appear more than once to the kernel? There are some very tricky things
about Btrfs's dependency on volume UUID that can make it get confused
where it should be writing when it sees more than one device with the
same UUID. This is a problem with for example Btrfs on LVM, taking a
snapshot of an LV, and both LV's being active means in effect two
Btrfs instances with the same UUID and Btrfs can clobber them both in
a bad way.
https://btrfs.wiki.kernel.org/index.php/Gotchas
I really think the Btrfs file system, based on the OP's description on
the Btrfs list, is probably OK. The issue is raid6 assembly somehow
being wonky. Even if it were double degraded by pulling any two
suspect drives, I'd expect things to immediately get better and a
nomodify 'btrfs check' will then come up clean. The OP had a clean
shutdown. But it's an open question how long after device failure he
actually noticed it before doing the rebuild and how he did that
rebuild; and whether there's missing critical data on any of the other
bad sectors on the three remaining drives. Chances are, those sectors
don't overlap though.
But at this point we need to hear back from Patrick.
--
Chris Murphy
^ permalink raw reply
* Re: [PATCH] md:raid1: fix a dead loop when read from a WriteMostly disk
From: Shaohua Li @ 2016-03-21 17:29 UTC (permalink / raw)
To: Wei Fang; +Cc: linux-raid
In-Reply-To: <1458559112-27683-1-git-send-email-fangwei1@huawei.com>
On Mon, Mar 21, 2016 at 07:18:32PM +0800, Wei Fang wrote:
> If first_bad == this_sector when we get the WriteMostly disk
> in read_balance(), valid disk will be returned with zero
> max_sectors. It'll lead to a dead loop in make_request(), and
> OOM will happen because of endless allocation of struct bio.
>
> Since we can't get data from this disk in this case, so
> continue for another disk.
Good catch, applied.
Thanks,
Shaohua
^ permalink raw reply
* Re: problems with dm-raid 6
From: Andreas Klauer @ 2016-03-21 13:27 UTC (permalink / raw)
To: Phil Turmel; +Cc: Patrick Tschackert, linux-raid
In-Reply-To: <56EFEC28.7020207@turmel.org>
On Mon, Mar 21, 2016 at 08:42:16AM -0400, Phil Turmel wrote:
> On 03/20/2016 06:37 PM, Andreas Klauer wrote:
> > This is what you get when you use --create --assume-clean on disks
> > that are not actually clean... or if you somehow convince md to
> > integrate a disk that does not have valid data on, for example
> > because you copied partition table and md metadata - but not
> > everything else - using dd.
> >
> > Something really bad happened here and the only person who
> > can explain it, is probably yourself.
>
> This is wrong. Your mdadm -D output clearly shows a 2014 creation date,
> so you definitely hadn't done --create --assume-clean at that point.
> (Don't.)
It was just an example. You get a mismatch if (at least) one disk has
wrong data on it; this many mismatches means there is a disk full of
wrong data for some reason we do not know, hence I suggested trying
to assemble it with disks missing, in the hope the data will start
making sense once the bad disk(s) are gone.
> Something else is wrong, quite possibly hardware.
Nothing is impossible, but for a hardware error this is very unusual.
> I would use an overlay for that.
Use overlays, and for disks with bad sectors, make good copies of them
(make sure to remember which of them were bad and where the holes in
the copies are, i.e. keep the ddrescue logs). If you play around with
disks that already have bad sectors, they might die completely on you.
Regards
Andreas Klauer
^ permalink raw reply
* Re: problems with dm-raid 6
From: Phil Turmel @ 2016-03-21 12:42 UTC (permalink / raw)
To: Andreas Klauer, Patrick Tschackert; +Cc: linux-raid
In-Reply-To: <20160320223727.GA29895@EIS>
Hi Patrick,
On 03/20/2016 06:37 PM, Andreas Klauer wrote:
> On Sun, Mar 20, 2016 at 10:44:57PM +0100, Patrick Tschackert wrote:
>> After rebooting the system, one of the harddisks was missing from my md raid 6 (the drive was /dev/sdf), so i rebuilt it with a hotspare that was already present in the system.
>> I physically removed the "missing" /dev/sdf drive after the restore and replaced it with a new drive.
Your smartctl output shows pending sector problems with sdf, sdh, and
sdj. The latter are WD Reds that won't keep those problems through a
scrub, so I guess the smartctl report was from before that?
> Exact commands involved for those steps?
>
> mdadm --examine output for your disks?
Yes, we want these.
>> $ cat /sys/block/md0/md/mismatch_cnt
>> 311936608
>
> Basically the whole array out of whack.
Wow.
> This is what you get when you use --create --assume-clean on disks
> that are not actually clean... or if you somehow convince md to
> integrate a disk that does not have valid data on, for example
> because you copied partition table and md metadata - but not
> everything else - using dd.
>
> Something really bad happened here and the only person who
> can explain it, is probably yourself.
This is wrong. Your mdadm -D output clearly shows a 2014 creation date,
so you definitely hadn't done --create --assume-clean at that point.
(Don't.)
> Your best bet is that the data is valid on n-2 disks.
>
> Use overlay https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
>
> Assemble the overlay RAID with any 2 disks missing (try all combinations) and see if you get valid data.
No. Something else is wrong, quite possibly hardware. You don't get a
mismatch count like that without it showing up in smartctl too, unless
corrupt data was being written to one or more disks for a long time.
It's unclear from your dmesg what might have happened. Probably bad
stuff going back years.
If you used ddrescue to replace sdf instead of letting mdadm reconstruct
it, that would have introduced zero sectors that would scramble your
encrypted filesystem. Please let us know that you didn't use ddrescue.
The encryption inside your array will frustrate any attempt to do
per-member analysis. I don't think there's anything still wrong with
the array (anything fixable, that is).
If an array error stomped on the key area of your dm-crypt layer, you
are totally destroyed, unless you happen to have a key backup you can
restore.
Otherwise you are at the mercy of fsck to try to fix your volume. I
would use an overlay for that.
Phil
^ permalink raw reply
* [PATCH] md: fix a trivial typo in comments
From: Wei Fang @ 2016-03-21 11:19 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Wei Fang
Fix a trivial typo in md_ioctl().
Signed-off-by: Wei Fang <fangwei1@huawei.com>
---
drivers/md/md.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index e55e6cf..9804355 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -6883,7 +6883,7 @@ static int md_ioctl(struct block_device *bdev, fmode_t mode,
case ADD_NEW_DISK:
/* We can support ADD_NEW_DISK on read-only arrays
- * on if we are re-adding a preexisting device.
+ * only if we are re-adding a preexisting device.
* So require mddev->pers and MD_DISK_SYNC.
*/
if (mddev->pers) {
--
1.7.1
^ permalink raw reply related
* [PATCH] md:raid1: fix a dead loop when read from a WriteMostly disk
From: Wei Fang @ 2016-03-21 11:18 UTC (permalink / raw)
To: shli; +Cc: linux-raid, Wei Fang
If first_bad == this_sector when we get the WriteMostly disk
in read_balance(), valid disk will be returned with zero
max_sectors. It'll lead to a dead loop in make_request(), and
OOM will happen because of endless allocation of struct bio.
Since we can't get data from this disk in this case, so
continue for another disk.
Signed-off-by: Wei Fang <fangwei1@huawei.com>
---
drivers/md/raid1.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 4e3843f..74d42c0 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -570,7 +570,7 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect
if (best_dist_disk < 0) {
if (is_badblock(rdev, this_sector, sectors,
&first_bad, &bad_sectors)) {
- if (first_bad < this_sector)
+ if (first_bad <= this_sector)
/* Cannot use this */
continue;
best_good_sectors = first_bad - this_sector;
--
1.7.1
^ permalink raw reply related
* [GIT PULL] MD for 4.6
From: Shaohua Li @ 2016-03-21 5:10 UTC (permalink / raw)
To: torvalds; +Cc: linux-kernel, linux-raid, neilb
Hi Linus,
Could you please pull the MD update for 4.6? This update mainly fixes bugs.
- A raid5 discard related fix from Jes
- A MD multipath bio clone fix from Ming
- raid1 error handling deadlock fix from Nate and corresponding raid10 fix from
myself
- A raid5 stripe batch fix from Neil
- A patch from Sebastian to avoid unnecessary uevent
- Several cleanup/debug patches
Thanks,
Shaohua
The following changes since commit 6dc390ad61ac8dfca5fa9b0823981fb6f7ec17a0:
Merge tag 'arc-4.5-rc6-fixes-upd' of git://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arc (2016-02-24 14:06:17 -0800)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/shli/md.git tags/md/4.6-rc1
for you to fetch changes up to 1d034e68e2c256640eb1f44bd7dcd89f90806ccf:
md/raid5: Cleanup cpu hotplug notifier (2016-03-17 14:30:15 -0700)
----------------------------------------------------------------
Anna-Maria Gleixner (1):
md/raid5: Cleanup cpu hotplug notifier
Eric Engestrom (1):
md/bitmap: remove redundant check
Guoqing Jiang (3):
md/raid1: remove unnecessary BUG_ON
md/bitmap: remove redundant return in bitmap_checkpage
md: fix typos for stipe
Jes Sorensen (1):
md/raid5: Compare apples to apples (or sectors to sectors)
Ming Lei (1):
md: multipath: don't hardcopy bio in .make_request path
Nate Dailey (1):
raid1: include bio_end_io_list in nr_queued to prevent freeze_array hang
NeilBrown (1):
md/raid5: preserve STRIPE_PREREAD_ACTIVE in break_stripe_batch_list
Sebastian Parschauer (1):
md: Drop sending a change uevent when stopping
Shaohua Li (6):
RAID5: check_reshape() shouldn't call mddev_suspend
RAID5: revert e9e4c377e2f563 to fix a livelock
MD: warn for potential deadlock
Update MD git tree URL
md/raid5: output stripe state for debug
raid10: include bio_end_io_list in nr_queued to prevent freeze_array hang
MAINTAINERS | 2 +-
drivers/md/bitmap.c | 4 +---
drivers/md/bitmap.h | 4 ++--
drivers/md/md.c | 2 +-
drivers/md/multipath.c | 4 +++-
drivers/md/raid1.c | 8 ++++---
drivers/md/raid10.c | 7 ++++--
drivers/md/raid5.c | 63 +++++++++++++++++++++++++++++---------------------
drivers/md/raid5.h | 4 +++-
9 files changed, 58 insertions(+), 40 deletions(-)
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox