* Re: mdadm Consistency Policy initialization
From: Artur Paszkiewicz @ 2017-04-24 10:07 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <2ca923fa-c147-f8c9-2399-f3a0e0d1721a@gmail.com>
On 04/20/2017 06:08 PM, Jes Sorensen wrote:
> On 04/19/2017 06:29 AM, Artur Paszkiewicz wrote:
>> On 04/18/2017 06:50 PM, Jes Sorensen wrote:
>>> Hi Artur,
>>>
>>> In 5308f11727b889965efe5ac0e854d197c2b51f6d you introduced struct mdinfo: enum consistency_policy, but in mdadm.c you initialize it to UnSet which isn't part of the enum.
>>>
>>> Is there any actual difference between CONSISTENCY_POLICY_UNKNOWN and UnSet? It seems suboptimal to mix and match within the enum like this, and if CONSISTENCY_POLICY_UNKNOWN does the job, couldn't we just initialize with that?
>>
>> Hi Jes,
>>
>> The "enum consistency_policy" and "mapping_t consistency_policies[]"
>> represent values that can appear in sysfs. md/consistency_policy can be
>> "unknown" when the array is inactive. On the other hand, UnSet just
>> means that the --consistency-policy= parameter was not provided by the
>> user. I wanted to differentiate between these two cases. If you think
>> this is redundant I can change it and use CONSISTENCY_POLICY_UNKNOWN
>> instead, this should be straightforward.
>
> Hi Artur,
>
> I made some changes to map_name() and noticed that you already there default to CONSISTENCY_POLICY_UNKNOWN if it returns UnSet in sysfs_read(). However given that you do a lot of checks manually outside of sysfs.c, I dind't change the code there for now. I think we the change I made to map_name() it should be possible to simply a bunch of that checking code.
Hi Jes,
It does seem to be better this way. I'll send a patch that removes using
UnSet from all consistency policy related places soon.
Thanks,
Artur
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-24 11:04 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: Brad Campbell, linux-raid
In-Reply-To: <CAHKno4AZukKvOEkVXPYMtnu1nypXLi95HaiFw+WNhAKP4KoLkg@mail.gmail.com>
On Mon, Apr 24, 2017 at 09:34:04AM +0200, Patrik Dahlström wrote:
> I've let a program compare both raid sets (5 and 6 disk) overnight. So
> far it has gone from 128 MB to 14 TB without finding common data. Does
> that tell us anything?
Are both RAID sets created correctly?
On the 6 disk one, `file -s /dev/mdX` should say ext filesystem.
If that's not there it's certainly incorrect. (The reverse isn't true though.)
I experiment a little:
# truncate -s 100M a b c d e f
# for f in ?; do losetup --find --show "$f"; done
# mdadm --create /dev/md42 --level=5 --raid-devices=5 /dev/loop{0,1,2,3,4}
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md42 started.
# i=0; while printf "%015x\n" $i; do let i+=16; done > /dev/md42
# hexdump -C -n 64 -s 808080 /dev/md42
000c5490 30 30 30 30 30 30 30 30 30 30 63 35 34 39 30 0a |0000000000c5490.|
000c54a0 30 30 30 30 30 30 30 30 30 30 63 35 34 61 30 0a |0000000000c54a0.|
000c54b0 30 30 30 30 30 30 30 30 30 30 63 35 34 62 30 0a |0000000000c54b0.|
000c54c0 30 30 30 30 30 30 30 30 30 30 63 35 34 63 30 0a |0000000000c54c0.|
000c54d0
So in this sample array the data itself represents the offset it should be at.
This is just so we can verify later.
Now grow.
# echo 1 > /sys/block/md42/md/sync_speed_min
# echo 256 > /sys/block/md42/md/sync_speed_max
# mdadm --grow /dev/md42 --raid-devices=6 --add /dev/loop5
mdadm: added /dev/loop5
mdadm: Need to backup 10240K of critical section..
# watch grep -A3 md42 /proc/mdstat
... wait for it to reach around 50% or whatever ...
# mdadm --stop /dev/md42
mdadm: stopped /dev/md42
# mdadm --examine /dev/loop1
[...]
Reshape pos'n : 296960 (290.00 MiB 304.09 MB)
Delta Devices : 1 (5->6)
[...]
Now create two RAID sets:
# losetup -D
# for f in ? ; do cp "$f" "$f".a ; done;
# for f in ? ; do cp "$f" "$f".b ; done;
# for a in *.a ; do losetup --find --show "$a" ; done
# for b in *.b ; do losetup --find --show "$b" ; done
# mdadm --create /dev/md42 --assume-clean --level=5 --raid-devices=5 /dev/loop{0,1,2,3,4}
# mdadm --create /dev/md42 --assume-clean --level=5 --raid-devices=6 /dev/loop{5,6,7,8,9,10}
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md42 : active raid5 loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
405504 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
md43 : active raid5 loop10[5] loop9[4] loop8[3] loop7[2] loop6[1] loop5[0]
506880 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
And compare:
# hexdump -C -n 64 /dev/md42
00000000 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0a |000000000000000.|
00000010 30 30 30 30 30 30 30 30 30 30 30 30 30 31 30 0a |000000000000010.|
00000020 30 30 30 30 30 30 30 30 30 30 30 30 30 32 30 0a |000000000000020.|
00000030 30 30 30 30 30 30 30 30 30 30 30 30 30 33 30 0a |000000000000030.|
00000040
# hexdump -C -n 64 /dev/md43
00000000 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0a |000000000000000.|
00000010 30 30 30 30 30 30 30 30 30 30 30 30 30 31 30 0a |000000000000010.|
00000020 30 30 30 30 30 30 30 30 30 30 30 30 30 32 30 0a |000000000000020.|
00000030 30 30 30 30 30 30 30 30 30 30 30 30 30 33 30 0a |000000000000030.|
00000040
This is identical because in this example, the offset didn't change.
# hexdump -C -n 64 -s 80808080 /dev/md42
04d10890 30 30 30 30 30 30 30 30 35 66 31 30 38 39 30 0a |000000005f10890.|
04d108a0 30 30 30 30 30 30 30 30 35 66 31 30 38 61 30 0a |000000005f108a0.|
04d108b0 30 30 30 30 30 30 30 30 35 66 31 30 38 62 30 0a |000000005f108b0.|
04d108c0 30 30 30 30 30 30 30 30 35 66 31 30 38 63 30 0a |000000005f108c0.|
04d108d0
# hexdump -C -n 64 -s 80808080 /dev/md43
04d10890 30 30 30 30 30 30 30 30 34 64 31 30 38 39 30 0a |000000004d10890.|
04d108a0 30 30 30 30 30 30 30 30 34 64 31 30 38 61 30 0a |000000004d108a0.|
04d108b0 30 30 30 30 30 30 30 30 34 64 31 30 38 62 30 0a |000000004d108b0.|
04d108c0 30 30 30 30 30 30 30 30 34 64 31 30 38 63 30 0a |000000004d108c0.|
04d108d0
For this offset, md42 was wrong, md43 is correct.
# hexdump -C -n 64 -s 300808080 /dev/md42
11edf790 30 30 30 30 30 30 30 31 31 65 64 66 37 39 30 0a |000000011edf790.|
11edf7a0 30 30 30 30 30 30 30 31 31 65 64 66 37 61 30 0a |000000011edf7a0.|
11edf7b0 30 30 30 30 30 30 30 31 31 65 64 66 37 62 30 0a |000000011edf7b0.|
11edf7c0 30 30 30 30 30 30 30 31 31 65 64 66 37 63 30 0a |000000011edf7c0.|
11edf7d0
# hexdump -C -n 64 -s 300808080 /dev/md43
11edf790 30 30 30 30 30 30 30 31 31 65 64 66 37 39 30 0a |000000011edf790.|
11edf7a0 30 30 30 30 30 30 30 31 31 65 64 66 37 61 30 0a |000000011edf7a0.|
11edf7b0 30 30 30 30 30 30 30 31 31 65 64 66 37 62 30 0a |000000011edf7b0.|
11edf7c0 30 30 30 30 30 30 30 31 31 65 64 66 37 63 30 0a |000000011edf7c0.|
11edf7d0
For this offset, md42 and md43 overlapped. Grow progressed that far yet
without writing into the original data of the 5disk raid5. This could be
a suitable merge point for a linear device mapping.
# hexdump -C -n 64 -s 400008080 /dev/md42
17d7a390 30 30 30 30 30 30 30 31 37 64 37 61 33 39 30 0a |000000017d7a390.|
17d7a3a0 30 30 30 30 30 30 30 31 37 64 37 61 33 61 30 0a |000000017d7a3a0.|
17d7a3b0 30 30 30 30 30 30 30 31 37 64 37 61 33 62 30 0a |000000017d7a3b0.|
17d7a3c0 30 30 30 30 30 30 30 31 37 64 37 61 33 63 30 0a |000000017d7a3c0.|
17d7a3d0
# hexdump -C -n 64 -s 400008080 /dev/md43
17d7a390 30 30 30 30 30 30 30 31 33 31 37 61 33 39 30 0a |00000001317a390.|
17d7a3a0 30 30 30 30 30 30 30 31 33 31 37 61 33 61 30 0a |00000001317a3a0.|
17d7a3b0 30 30 30 30 30 30 30 31 33 31 37 61 33 62 30 0a |00000001317a3b0.|
17d7a3c0 30 30 30 30 30 30 30 31 33 31 37 61 33 63 30 0a |00000001317a3c0.|
17d7a3d0
For this offset, md42 is correct and md43 is wrong.
Grow did not progress that far.
That's the general outline of the idea.
The problem in your case is of course, your data is not that easy to verify.
( You can't even easily verify your disk order, offsets, et cetera.
These are things you have to figure out by yourself,
not sure how else to help you. Best of luck. )
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-24 12:13 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <20170424110455.GA4949@metamorpher.de>
2017-04-24 13:04 GMT+02:00, Andreas Klauer <Andreas.Klauer@metamorpher.de>:
> On Mon, Apr 24, 2017 at 09:34:04AM +0200, Patrik Dahlström wrote:
>> I've let a program compare both raid sets (5 and 6 disk) overnight. So
>> far it has gone from 128 MB to 14 TB without finding common data. Does
>> that tell us anything?
>
> Are both RAID sets created correctly?
>
> On the 6 disk one, `file -s /dev/mdX` should say ext filesystem.
>
> If that's not there it's certainly incorrect. (The reverse isn't true
> though.)
I'm afraid it doesn't say that. I can get the exact command I used
when I get home. I do know that both raids contains only zeros for
many MB before any data appears.
> That's the general outline of the idea.
> The problem in your case is of course, your data is not that easy to
> verify.
My raid contains many large files (8-12 GB each). If I can get
reference data, I should be able to locate where on the disks the file
is split up. Would that help? I imagine file system fragmentation
could become an issue.
>
> ( You can't even easily verify your disk order, offsets, et cetera.
> These are things you have to figure out by yourself,
> not sure how else to help you. Best of luck. )
From old kernel log, we know that the disk order was /dev/sd[abdcef],
given that the drives were always discovered in that order. Could the
offsets be verified with data from reference files as discussed above?
>
> Regards
> Andreas Klauer
>
Best regards
Patrik Dahlström
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-24 12:37 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: Brad Campbell, linux-raid
In-Reply-To: <CAHKno4A12N6UGGtJQJRkY0aPbFBBf0Rek91th9Cnx6XWJGouWg@mail.gmail.com>
On Mon, Apr 24, 2017 at 02:13:24PM +0200, Patrik Dahlström wrote:
> I'm afraid it doesn't say that.
You said you found an ext header in the raw data.
If that exists then only thing I can think of is that you ended
up picking the wrong offset (or disk order) after all.
> I do know that both raids contains only zeros for
> many MB before any data appears.
This could be normal for the 5disk array since that part already
reshaped and the offset changed and there just could happen to
be zeroes somewhere in the beginning of a filesystem after the
first block of metadata.
Basically the 5disk array is supposed to have bogus data at
the start in your case. But it should turn into valid data
at whatever point the reshape did not yet reach.
This bogus data makes it hard to determine the correct offset
but according to the output you showed before the offset should
be 128M here.
For the 6disk array you should see valid data (starting with
the filesystem header) for however far the reshape was already done.
Depending on how the filesystem works you might even be able to
mount it but everything that is located behind the progress point
would appear corrupted.
Again if one of the disks actually was kicked from the array
while the grow was going on, you should leave that disk out as missing
as otherwise it will just appear as wrong data in both arrays.
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-24 12:54 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <20170424123706.GA6798@metamorpher.de>
2017-04-24 14:37 GMT+02:00, Andreas Klauer <Andreas.Klauer@metamorpher.de>:
> On Mon, Apr 24, 2017 at 02:13:24PM +0200, Patrik Dahlström wrote:
>> I'm afraid it doesn't say that.
>
> You said you found an ext header in the raw data.
Yes, I found ext headers in both /dev/sda and /dev/sdf, but it doesn't
show up in the 6 disk raid (/dev/md1).
>
> If that exists then only thing I can think of is that you ended
> up picking the wrong offset (or disk order) after all.
What offset are you referring to here? The --data-offset to the mdadm
--create command?
>
>> I do know that both raids contains only zeros for
>> many MB before any data appears.
>
> This could be normal for the 5disk array since that part already
> reshaped and the offset changed and there just could happen to
> be zeroes somewhere in the beginning of a filesystem after the
> first block of metadata.
Makes sense
>
> Basically the 5disk array is supposed to have bogus data at
> the start in your case. But it should turn into valid data
> at whatever point the reshape did not yet reach.
Should I then be able to find a copy of ext4 superblock in the 5 disk
array once valid data start to appear?
>
> This bogus data makes it hard to determine the correct offset
> but according to the output you showed before the offset should
> be 128M here.
I found out that there existed a ext4 file system at offset 0x7B80000
(123,5 MB) on both /dev/sda and /dev/sdb. I will adjust my mdadm
--create commands to this offset when I get home and try again.
>
> For the 6disk array you should see valid data (starting with
> the filesystem header) for however far the reshape was already done.
> Depending on how the filesystem works you might even be able to
> mount it but everything that is located behind the progress point
> would appear corrupted.
Interesting. Like I said above, I will retry the create commands with
123.5 MB --data-offset.
>
> Again if one of the disks actually was kicked from the array
> while the grow was going on, you should leave that disk out as missing
> as otherwise it will just appear as wrong data in both arrays.
I don't think the disk was ever kicked out. The kernel reset the link
and continued, I believe.
>
> Regards
> Andreas Klauer
>
Best regards
Patrik Dahlström
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-24 13:39 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: Brad Campbell, linux-raid
In-Reply-To: <CAHKno4BR3kdsLU9AnNHvarc-e6UibNr7WQks2ssPX4pE1ZoUpA@mail.gmail.com>
On Mon, Apr 24, 2017 at 02:54:50PM +0200, Patrik Dahlström wrote:
> What offset are you referring to here? The --data-offset to the mdadm
> --create command?
Yes, data offset...
> I don't think the disk was ever kicked out. The kernel reset the link
> and continued, I believe.
Okay.
Another thing you can check is, pick an arbitrary offset that does not
have zeroes on the disks, and see if the XOR matches for 5, or 6 disks.
It should match for 6 for however far the grow progressed,
and match for 5 afterwards.
Again with my example from before,
# for hexdump -C -n 16 -s 10000000 "$f"
00989680 30 30 30 30 30 30 30 30 32 61 38 39 36 38 30 0a |000000002a89680.|
00989690 ^^
00989680 30 30 30 30 30 30 30 30 32 61 38 39 36 38 30 0a |000000002a89680.|
00989690 ^^
00989680 30 30 30 30 30 30 30 30 32 62 30 39 36 38 30 0a |000000002b09680.|
00989690 ^^
00989680 30 30 30 30 30 30 30 30 32 62 38 39 36 38 30 0a |000000002b89680.|
00989690 ^^
00989680 30 30 30 30 30 30 30 30 32 63 30 39 36 38 30 0a |000000002c09680.|
00989690 ^^
00989680 30 30 30 30 30 30 30 30 32 63 38 39 36 38 30 0a |000000002c89680.|
00989690 ^^
2a ^ 2a ^ 2b ^ 2b ^ 2c = 2c OK for 6 disk area
# for hexdump -C -n 16 -s 90000000 "$f"
055d4a80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
055d4a90 ^^
055d4a80 30 30 30 30 30 30 30 31 35 32 35 34 61 38 30 0a |000000015254a80.|
055d4a90 ^^
055d4a80 30 30 30 30 30 30 30 31 35 32 64 34 61 38 30 0a |0000000152d4a80.|
055d4a90 ^^
055d4a80 30 30 30 30 30 30 30 31 35 33 35 34 61 38 30 0a |000000015354a80.|
055d4a90 ^^
055d4a80 30 30 30 30 30 30 30 31 35 33 64 34 61 38 30 0a |0000000153d4a80.|
055d4a90 ^^
25 ^ 2d ^ 35 ^ 3d = \0\0 OK for 5 disk area
(I did not intend for the parity to be zero in this example,
but that's just how it turned out to be...)
If you have a 5disk parity match in the zone that should be 6disks,
then maybe one of your earlier mdadm create experiments started to
sync the raid.
This would be yet another nail in your coffin.
Regards
Andreas Klauer
^ permalink raw reply
* Re: mdadm Consistency Policy initialization
From: Jes Sorensen @ 2017-04-24 13:45 UTC (permalink / raw)
To: Artur Paszkiewicz; +Cc: linux-raid
In-Reply-To: <ac885b46-4f5f-c18c-ee45-a5d1886466e9@intel.com>
On 04/24/2017 06:07 AM, Artur Paszkiewicz wrote:
> On 04/20/2017 06:08 PM, Jes Sorensen wrote:
>> On 04/19/2017 06:29 AM, Artur Paszkiewicz wrote:
>>> Hi Jes,
>>>
>>> The "enum consistency_policy" and "mapping_t consistency_policies[]"
>>> represent values that can appear in sysfs. md/consistency_policy can be
>>> "unknown" when the array is inactive. On the other hand, UnSet just
>>> means that the --consistency-policy= parameter was not provided by the
>>> user. I wanted to differentiate between these two cases. If you think
>>> this is redundant I can change it and use CONSISTENCY_POLICY_UNKNOWN
>>> instead, this should be straightforward.
>>
>> Hi Artur,
>>
>> I made some changes to map_name() and noticed that you already there default to CONSISTENCY_POLICY_UNKNOWN if it returns UnSet in sysfs_read(). However given that you do a lot of checks manually outside of sysfs.c, I dind't change the code there for now. I think we the change I made to map_name() it should be possible to simply a bunch of that checking code.
>
> Hi Jes,
>
> It does seem to be better this way. I'll send a patch that removes using
> UnSet from all consistency policy related places soon.
Awesome!
I am glad you like it.
Jes
^ permalink raw reply
* [PATCH] Don't use UnSet with consistency_policy
From: Artur Paszkiewicz @ 2017-04-24 14:03 UTC (permalink / raw)
To: jes.sorensen; +Cc: linux-raid, Artur Paszkiewicz
Use CONSISTENCY_POLICY_UNKNOWN instead. Simplify some checks because
since 5e8e35fb7e17 ("maps: Use keyvalue for null terminator to indicate
'unset' value") map_name() can return this default directly.
Suggested-by: Jes Sorensen <Jes.Sorensen@gmail.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
---
maps.c | 2 +-
mdadm.c | 12 ++++++------
super-intel.c | 4 +---
sysfs.c | 10 ++++------
4 files changed, 12 insertions(+), 16 deletions(-)
diff --git a/maps.c b/maps.c
index 11dd3d20..8fe0a6df 100644
--- a/maps.c
+++ b/maps.c
@@ -137,7 +137,7 @@ mapping_t consistency_policies[] = {
{ "bitmap", CONSISTENCY_POLICY_BITMAP},
{ "journal", CONSISTENCY_POLICY_JOURNAL},
{ "ppl", CONSISTENCY_POLICY_PPL},
- { NULL, UnSet }
+ { NULL, CONSISTENCY_POLICY_UNKNOWN }
};
mapping_t sysfs_array_states[] = {
diff --git a/mdadm.c b/mdadm.c
index 41dae1db..b689e328 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -78,7 +78,7 @@ int main(int argc, char *argv[])
.level = UnSet,
.layout = UnSet,
.bitmap_chunk = UnSet,
- .consistency_policy = UnSet,
+ .consistency_policy = CONSISTENCY_POLICY_UNKNOWN,
};
char sys_hostname[256];
@@ -1228,8 +1228,7 @@ int main(int argc, char *argv[])
case O(GROW, 'k'):
s.consistency_policy = map_name(consistency_policies,
optarg);
- if (s.consistency_policy == UnSet ||
- s.consistency_policy < CONSISTENCY_POLICY_RESYNC) {
+ if (s.consistency_policy < CONSISTENCY_POLICY_RESYNC) {
pr_err("Invalid consistency policy: %s\n",
optarg);
exit(2);
@@ -1267,7 +1266,7 @@ int main(int argc, char *argv[])
pr_err("--write-journal is only supported for RAID level 4/5/6.\n");
exit(2);
}
- if (s.consistency_policy != UnSet &&
+ if (s.consistency_policy != CONSISTENCY_POLICY_UNKNOWN &&
s.consistency_policy != CONSISTENCY_POLICY_JOURNAL) {
pr_err("--write-journal is not supported with consistency policy: %s\n",
map_num(consistency_policies, s.consistency_policy));
@@ -1275,7 +1274,8 @@ int main(int argc, char *argv[])
}
}
- if (mode == CREATE && s.consistency_policy != UnSet) {
+ if (mode == CREATE &&
+ s.consistency_policy != CONSISTENCY_POLICY_UNKNOWN) {
if (s.level <= 0) {
pr_err("--consistency-policy not meaningful with level %s.\n",
map_num(pers, s.level));
@@ -1687,7 +1687,7 @@ int main(int argc, char *argv[])
rv = Grow_reshape(devlist->devname, mdfd,
devlist->next,
data_offset, &c, &s);
- } else if (s.consistency_policy != UnSet) {
+ } else if (s.consistency_policy != CONSISTENCY_POLICY_UNKNOWN) {
rv = Grow_consistency_policy(devlist->devname, mdfd, &c, &s);
} else if (array_size == 0)
pr_err("no changes to --grow\n");
diff --git a/super-intel.c b/super-intel.c
index 0aed57c8..fbff215c 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -5369,9 +5369,7 @@ static int init_super_imsm_volume(struct supertype *st, mdu_array_info_t *info,
}
mpb->num_raid_devs++;
- if (s->consistency_policy == UnSet ||
- s->consistency_policy == CONSISTENCY_POLICY_RESYNC ||
- s->consistency_policy == CONSISTENCY_POLICY_NONE) {
+ if (s->consistency_policy <= CONSISTENCY_POLICY_RESYNC) {
dev->rwh_policy = RWH_OFF;
} else if (s->consistency_policy == CONSISTENCY_POLICY_PPL) {
dev->rwh_policy = RWH_DISTRIBUTED;
diff --git a/sysfs.c b/sysfs.c
index 712f8b3e..aa30de5c 100644
--- a/sysfs.c
+++ b/sysfs.c
@@ -254,13 +254,11 @@ struct mdinfo *sysfs_read(int fd, char *devnm, unsigned long options)
if (options & GET_CONSISTENCY_POLICY) {
strcpy(base, "consistency_policy");
- if (load_sys(fname, buf, sizeof(buf))) {
+ if (load_sys(fname, buf, sizeof(buf)))
sra->consistency_policy = CONSISTENCY_POLICY_UNKNOWN;
- } else {
- sra->consistency_policy = map_name(consistency_policies, buf);
- if (sra->consistency_policy == UnSet)
- sra->consistency_policy = CONSISTENCY_POLICY_UNKNOWN;
- }
+ else
+ sra->consistency_policy = map_name(consistency_policies,
+ buf);
}
if (! (options & GET_DEVS))
--
2.11.0
^ permalink raw reply related
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-24 14:05 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <20170424133946.GA7057@metamorpher.de>
2017-04-24 15:39 GMT+02:00, Andreas Klauer <Andreas.Klauer@metamorpher.de>:
> On Mon, Apr 24, 2017 at 02:54:50PM +0200, Patrik Dahlström wrote:
>> What offset are you referring to here? The --data-offset to the mdadm
>> --create command?
>
> Yes, data offset...
It all makes sense to me now! I believe I was still in sort of a
panicked mode yesterday. The pieces are starting to make more sense
today.
> Another thing you can check is, pick an arbitrary offset that does not
> have zeroes on the disks, and see if the XOR matches for 5, or 6 disks.
I will try that when I get home.
>
> If you have a 5disk parity match in the zone that should be 6disks,
> then maybe one of your earlier mdadm create experiments started to
> sync the raid.
This is definitely a possibility.
>
> This would be yet another nail in your coffin.
Could you define how big nail? I can live with some data loss, but
preferably not 20 TB.
Best regards
Patrik Dahlström
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-24 14:21 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: Brad Campbell, linux-raid
In-Reply-To: <CAHKno4BwvVTtwqrHjTF8seSgR-QArmvATwuNy_21ctUPdh0biA@mail.gmail.com>
On Mon, Apr 24, 2017 at 04:05:39PM +0200, Patrik Dahlström wrote:
> > This would be yet another nail in your coffin.
> Could you define how big nail?
Lost another drive ...
^ permalink raw reply
* Re: [PATCH] dm verity: deferred hash checking for restart/logging mode
From: Sami Tolvanen @ 2017-04-24 15:54 UTC (permalink / raw)
To: Milan Broz
Cc: Bongkyu Kim, agk, snitzer, dm-devel, shli, linux-raid,
linux-kernel, Will Drewry
In-Reply-To: <76b0b4ab-f811-7c8e-c88c-1210a0e67119@gmail.com>
On Mon, Apr 24, 2017 at 10:05:03AM +0200, Milan Broz wrote:
> Isn't one of the goal of integrity checking to PREVENT that userspace
> accesses tampered data?
Absolutely. It's not acceptable for dm-verity to return unverified data
to userspace in restart mode.
Sami
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-24 16:00 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <CAHKno4BwvVTtwqrHjTF8seSgR-QArmvATwuNy_21ctUPdh0biA@mail.gmail.com>
On 04/24/2017 04:05 PM, Patrik Dahlström wrote:
> 2017-04-24 15:39 GMT+02:00, Andreas Klauer <Andreas.Klauer@metamorpher.de>:
>> On Mon, Apr 24, 2017 at 02:54:50PM +0200, Patrik Dahlström wrote:
>>> What offset are you referring to here? The --data-offset to the mdadm
>>> --create command?
>>
>> Yes, data offset...
> It all makes sense to me now! I believe I was still in sort of a
> panicked mode yesterday. The pieces are starting to make more sense
> today.
Picking the offset I found out yesterday, I did indeed get a ext4 file
system on the 6 disk raid. I could even mount it at one point. Strangely
enough, I also got a file system on the 5 disk raid. Running a quick
test showed that offset 0.5 MB to 1.5 MB (3 chunks) had identical data,
and also one chunk at 32 GB. I'm gonna leave the program running for a
while now and see what I can figure out. The first chunk is most
probably identical too, but my program ignores chunks that are all zeros.
>
>> Another thing you can check is, pick an arbitrary offset that does not
>> have zeroes on the disks, and see if the XOR matches for 5, or 6 disks.
> I will try that when I get home.
Still haven't tried this. I have some life stuff to do as well.
Best regards
// Patrik
^ permalink raw reply
* EXTREMELY IMPORTANT
From: Ms. Katherine @ 2017-04-24 18:52 UTC (permalink / raw)
To: Recipients
Dear Beloved, Sorry for the inconvenience, I am getting in touch with you regarding an extremely important and urgent matter, Please I need your urgent assistance and idea to finish up a project (Orphanage Home) Due to my health condition, Everything is available to finish up the project, All I need is your idea and trust.
Sehr geehrte Geliebte, traurig für die Unannehmlichkeiten, ich bin in Kontakt mit Ihnen über eine äußerst wichtige und dringende Angelegenheit, Bitte brauche ich Ihre dringende Hilfe und Idee, um ein Projekt zu beenden (Orphanage Home) Wegen meines Gesundheitszustandes ist alles verfügbar Beenden Sie das Projekt, Alles was ich brauche ist Ihre Idee und Vertrauen.
Please contact me for more details.
Contact Email: kathrynnmison@gmail.com
Thanks
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
^ permalink raw reply
* Re: Recover array after I panicked
From: Phil Turmel @ 2017-04-24 21:04 UTC (permalink / raw)
To: Patrik Dahlström, Wols Lists, Roman Mamedov
Cc: Andreas Klauer, linux-raid
In-Reply-To: <e4c2acc6-420a-e454-fe74-c82737a1bbf0@powerlamerz.org>
On 04/23/2017 08:15 AM, Patrik Dahlström wrote:
>
>
> On 04/23/2017 02:11 PM, Wols Lists wrote:
>> On 23/04/17 12:58, Roman Mamedov wrote:
>>> On Sun, 23 Apr 2017 12:36:24 +0100
>>> Wols Lists <antlists@youngman.org.uk> wrote:
>>>
>>>> And, as the raid wiki tells you, download lspci and run that
>>>
>>> Maybe you meant lsdrv. https://github.com/pturmel/lsdrv
>>>
>> Sorry, yes I did ... (too many ls_xxx commands :-)
> Ok, I had to patch lsdrv a bit to make it run. Diff:
Thanks for the patch. Could you elaborate a bit on the errors you
received so I can reproduce and document this fully?
Also, do you have some large files (media files, perhaps) that you know
are in your array but you have a copy in hand? If so, you could use the
findHash script in my github account to map how that file is laid out on
your array's devices. Since large media files tend to be contiguous,
such a map would definitively show your chunk size and device order.
It would also show if your data offsets are consistent among the member
drives (but not the absolute value of the offset).
Phil
^ permalink raw reply
* Re: [PATCH] Don't use UnSet with consistency_policy
From: Jes Sorensen @ 2017-04-24 21:11 UTC (permalink / raw)
To: Artur Paszkiewicz; +Cc: linux-raid
In-Reply-To: <20170424140326.2707-1-artur.paszkiewicz@intel.com>
On 04/24/2017 10:03 AM, Artur Paszkiewicz wrote:
> Use CONSISTENCY_POLICY_UNKNOWN instead. Simplify some checks because
> since 5e8e35fb7e17 ("maps: Use keyvalue for null terminator to indicate
> 'unset' value") map_name() can return this default directly.
>
> Suggested-by: Jes Sorensen <Jes.Sorensen@gmail.com>
> Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
> ---
> maps.c | 2 +-
> mdadm.c | 12 ++++++------
> super-intel.c | 4 +---
> sysfs.c | 10 ++++------
> 4 files changed, 12 insertions(+), 16 deletions(-)
Applied!
Thanks,
Jes
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-24 21:56 UTC (permalink / raw)
To: Phil Turmel, Wols Lists, Roman Mamedov; +Cc: Andreas Klauer, linux-raid
In-Reply-To: <09feff90-bb2b-7e27-599e-40284ca004ae@turmel.org>
On 04/24/2017 11:04 PM, Phil Turmel wrote:
> On 04/23/2017 08:15 AM, Patrik Dahlström wrote:
>>
>>
>> On 04/23/2017 02:11 PM, Wols Lists wrote:
>>> On 23/04/17 12:58, Roman Mamedov wrote:
>>>> On Sun, 23 Apr 2017 12:36:24 +0100
>>>> Wols Lists <antlists@youngman.org.uk> wrote:
>>>>
>>>>> And, as the raid wiki tells you, download lspci and run that
>>>>
>>>> Maybe you meant lsdrv. https://github.com/pturmel/lsdrv
>>>>
>>> Sorry, yes I did ... (too many ls_xxx commands :-)
>> Ok, I had to patch lsdrv a bit to make it run. Diff:
>
> Thanks for the patch. Could you elaborate a bit on the errors you
> received so I can reproduce and document this fully?
Sure. It started out with this error:
$ ./lsdrv
Traceback (most recent call last):
File "./lsdrv", line 413, in <module>
probe_block('/sys/block/'+x)
File "./lsdrv", line 389, in probe_block
blk.FS = "MD %s (%s/%s)%s %s" % (blk.array.md.LEVEL, blk.slave.slot,
blk.array.md.raid_disks, peers, blk.slave.state)
AttributeError: 'NoneType' object has no attribute 'LEVEL'
So I added an if statement for blk.array.md.
Next, I got this error:
$ ./lsdrv
Traceback (most recent call last):
File "./lsdrv", line 414, in <module>
probe_block('/sys/block/'+x)
File "./lsdrv", line 406, in probe_block
blk.FS += " '%s'" % blk.ID_FS_LABEL
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'
That's what the other 2 if statements are for. I don't claim to know the
root cause of the errors, I've simply worked around them.
>
> Also, do you have some large files (media files, perhaps) that you know
> are in your array but you have a copy in hand? If so, you could use the
> findHash script in my github account to map how that file is laid out on
> your array's devices. Since large media files tend to be contiguous,
> such a map would definitively show your chunk size and device order.
I'll take a look. I definitely have some large continuous files on this
array.
>
> It would also show if your data offsets are consistent among the member
> drives (but not the absolute value of the offset).
>
> Phil
>
>
^ permalink raw reply
* mismatch_cnt > 0 during initial sync?
From: Stephane Thiell @ 2017-04-24 22:01 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi all,
Is it normal to get any mismatch_cnt > 0 during the initial sync? I just created a few new new raid6 volumes, and one of them is showing an increasing number of mismatch_cnt during its initial resync. Nothing is visible in the console nor after a smart short test (I also started a long test just in case).
$ mdadm --detail /dev/md7
/dev/md7:
Version : 1.2
Creation Time : Mon Apr 24 10:07:18 2017
Raid Level : raid6
Array Size : 62511163904 (59615.29 GiB 64011.43 GB)
Used Dev Size : 7813895488 (7451.91 GiB 8001.43 GB)
Raid Devices : 10
Total Devices : 10
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Apr 24 14:52:12 2017
State : clean, resyncing
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Resync Status : 24% complete
Name : oak-io1-s2:7 (local to host oak-io1-s2)
UUID : 6ece9d71:a97f9497:4612ed73:44dfdc0a
Events : 3308
Number Major Minor RaidDevice State
0 253 62 0 active sync /dev/dm-62
1 253 63 1 active sync /dev/dm-63
2 253 72 2 active sync /dev/dm-72
3 253 75 3 active sync /dev/dm-75
4 253 89 4 active sync /dev/dm-89
5 253 87 5 active sync /dev/dm-87
6 253 119 6 active sync /dev/dm-119
7 253 97 7 active sync /dev/dm-97
8 253 103 8 active sync /dev/dm-103
9 253 107 9 active sync /dev/dm-107
$ cat /sys/block/md7/md/sync_completed
3795284824 / 15627790976
$ cat /sys/block/md7/md/mismatch_cnt
3424
versions:
mdadm-3.4-14.el7_3.1.x86_64
CentOS7 (3.10.0-514.10.2.el7_lustre.x86_64)
Thanks!
Stephan
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-24 23:00 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <20170424133946.GA7057@metamorpher.de>
On 04/24/2017 03:39 PM, Andreas Klauer wrote:
> On Mon, Apr 24, 2017 at 02:54:50PM +0200, Patrik Dahlström wrote:
> Another thing you can check is, pick an arbitrary offset that does not
> have zeroes on the disks, and see if the XOR matches for 5, or 6 disks.
>
> It should match for 6 for however far the grow progressed,
> and match for 5 afterwards.
I got some success!
I started experimenting with this and realised that about 1 TB into
/dev/sdf, I started getting all zeros from the disk. Incidentally this
will also make the xor pass for both 5 and 6 disk arrays. Here's some
examples:
# for f in /dev/sd{a,b,d,c,e,f}; do hexdump -C -n 16 -s 0x8000000000 "$f"; done
8000000000 96 6c 5d 2c a4 03 7a 62 9c 7d 67 b9 55 24 aa 84 |.l],..zb.}g.U$..|
8000000010
8000000000 59 10 a8 e9 a7 5e fa e9 cd 8c 16 a2 7c 06 60 f6 |Y....^......|.`.|
8000000010
8000000000 7d ca 8e ea cc fe 2a 36 be b8 a8 b6 77 f9 fa 87 |}.....*6....w...|
8000000010
8000000000 4d 42 49 1e 0b ae a7 3b 42 50 68 bb c1 d5 89 96 |MBI....;BPh.....|
8000000010
8000000000 fc 3d c7 b7 82 39 06 a7 ad fc 00 81 05 5b 52 e1 |.=...9.......[R.|
8000000010
8000000000 03 c9 f5 86 46 34 0b 21 00 e5 b1 97 9a 55 eb 82 |....F4.!.....U..|
8000000010
5 disk raid xor check: 84 ^ f6 ^ 87 ^ 96 ^ e1 == 82, NOK
6 disk raid xor check: 84 ^ f6 ^ 87 ^ 96 ^ e1 ^ 82 == 0, OK
# for f in /dev/sd{a,b,d,c,e,f}; do hexdump -C -n 16 -s 0x10000000000 "$f"; done
10000000000 03 33 0a 04 d9 7a 44 4b dc 8f be 58 0b 80 8f 46 |.3...zDK...X...F|
10000000010
10000000000 79 5f 18 51 f7 44 03 59 7f aa ce 9d f1 a9 3d 73 |y_.Q.D.Y......=s|
10000000010
10000000000 67 f4 9e b2 34 b6 c2 43 b5 8d 2f 0d 5f 80 a4 6d |g...4..C../._..m|
10000000010
10000000000 4f ec f4 c9 da 04 64 db dc e9 e1 72 7f e4 74 06 |O.....d....r..t.|
10000000010
10000000000 52 74 78 2e c0 8c e1 8a ca 41 be ba da 4d 62 5e |Rtx......A...Mb^|
10000000010
10000000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
10000000010
5 disk raid xor check: 46 ^ 73 ^ 6d ^ 06 ^ 5e == 0, OK
6 disk raid xor check: 46 ^ 73 ^ 6d ^ 06 ^ 5e ^ 00 == 0, OK
So I did a binary search from 0x8000000000 to 0x10000000000 to see
where I start getting all zeros from the newest drive. I ended up at
offset 0xfaa2880000 (~0.98 TB or ~18 % into the disk). After that, it's
all zeros.
# for f in /dev/sd{a,b,d,c,e,f}; do hexdump -C -n 16 -s 0xfaa287fff8 "$f"; done
faa287fff8 e5 4d 6b e6 ef 7b 1f b0 2e 30 82 8e 5b 4b e0 30 |.Mk..{...0..[K.0|
faa2880008
faa287fff8 dd f7 ab 02 cf e8 a2 6d 93 a8 08 a7 d8 9e c7 b4 |.......m........|
faa2880008
faa287fff8 31 f8 7b d9 41 c7 72 13 f1 37 b6 4a 51 fc 46 74 |1.{.A.r..7.JQ.Ft|
faa2880008
faa287fff8 87 29 59 58 97 05 87 1b 1a 8d 83 84 83 b0 21 4a |.)YX..........!J|
faa2880008
faa287fff8 04 2b 32 6d 0f ab fb b7 56 22 bf e7 51 99 40 ba |.+2m....V"..Q.@.|
faa2880008
faa287fff8 f4 06 c8 81 00 00 03 ae 00 00 00 00 00 00 00 00 |................|
faa2880008
Now, at offset 0xfaa2880000 the 5 disk raid xor sums are OK:
0xfaa2880000: 2e ^ 93 ^ f1 ^ 1a ^ 56 = 0, OK
But immediately before that, I can't get the xor sums to line up:
0xfaa287ffff: b0 ^ 6d ^ 13 ^ 1b ^ b7 != ae (62 actually), NOK
This would mean that it's incorrect for both 5 and 6 disk raids.
So I did a binary search backwards to find out where the checksums match
again and came to offset 0xed90332000 (~0.93 TB or ~17 % into the disk).
# for f in /dev/sd{a,b,d,c,e,f}; do hexdump -C -n 16 -s 0xed90331ff8 "$f"; done
ed90331ff8 69 1a b1 fb 9f 80 cf fa 6f 97 dc b7 26 40 38 6f |i.......o...&@8o|
ed90332008
ed90331ff8 e3 58 c4 e7 7a 9a b5 a4 74 37 5a de d8 24 a6 e5 |.X..z...t7Z..$..|
ed90332008
ed90331ff8 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
ed90332008
ed90331ff8 53 0b 7c bb 49 d6 63 55 ff fb b2 40 35 00 03 39 |S.|.I.cU...@5..9|
ed90332008
ed90331ff8 9b 49 25 23 ec ad 89 0c ce a0 29 29 a2 86 bc c2 |.I%#......))....|
ed90332008
ed90331ff8 bd ff d3 7b bf 9e 6f f8 17 bc f3 8d e7 9b d0 65 |...{..o........e|
ed90332008
0xed90331fff: fa ^ a4 ^ ff ^ 55 ^ 0c == f8, OK for 6 disk raid
0xed90332000: 6f ^ 74 ^ ff ^ ff ^ ce != 17 (d5 actually), NOK
That is a span of ~52 GB where I presumably can't get the checksums
right. What does all this mean? What am I missing?
Best regards
// Patrik
^ permalink raw reply
* Re: Recover array after I panicked
From: Phil Turmel @ 2017-04-24 23:35 UTC (permalink / raw)
To: Patrik Dahlström, Wols Lists, Roman Mamedov
Cc: Andreas Klauer, linux-raid
In-Reply-To: <a88537bc-a133-482a-e726-7d3680727af6@powerlamerz.org>
On 04/24/2017 05:56 PM, Patrik Dahlström wrote:
> On 04/24/2017 11:04 PM, Phil Turmel wrote:
>> Thanks for the patch. Could you elaborate a bit on the errors you
>> received so I can reproduce and document this fully?
> Sure. It started out with this error:
> $ ./lsdrv
> Traceback (most recent call last):
> File "./lsdrv", line 413, in <module>
> probe_block('/sys/block/'+x)
> File "./lsdrv", line 389, in probe_block
> blk.FS = "MD %s (%s/%s)%s %s" % (blk.array.md.LEVEL, blk.slave.slot,
> blk.array.md.raid_disks, peers, blk.slave.state)
> AttributeError: 'NoneType' object has no attribute 'LEVEL'
Ok. I'll spin up an Ubuntu 16.04 VM to play with this. Thanks.
>> Also, do you have some large files (media files, perhaps) that you know
>> are in your array but you have a copy in hand? If so, you could use the
>> findHash script in my github account to map how that file is laid out on
>> your array's devices. Since large media files tend to be contiguous,
>> such a map would definitively show your chunk size and device order.
> I'll take a look. I definitely have some large continuous files on this
> array.
The first draft of this script was written for a fellow in a situation
very similar to yours. The results were miraculous. Not to get your
hopes up too high, though -- my first impression of this thread is that
you're screwed. /-:
Phil
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-25 0:16 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: Brad Campbell, linux-raid
In-Reply-To: <727eeaf5-b856-305d-99b6-e42f3383b699@powerlamerz.org>
On Tue, Apr 25, 2017 at 01:00:47AM +0200, Patrik Dahlström wrote:
> 6 disk raid xor check: 84 ^ f6 ^ 87 ^ 96 ^ e1 ^ 82 == 0, OK
This should be the 6 disk raid area.
> 5 disk raid xor check: 46 ^ 73 ^ 6d ^ 06 ^ 5e == 0, OK
This should be the 5 disk raid area.
> 6 disk raid xor check: 46 ^ 73 ^ 6d ^ 06 ^ 5e ^ 00 == 0, OK
Still 5 disks... grow did not progess until here,
and the 6th disk is likely zero because it's new.
> But immediately before that, I can't get the xor sums to line up:
> 0xfaa287ffff: b0 ^ 6d ^ 13 ^ 1b ^ b7 != ae (62 actually), NOK
> This would mean that it's incorrect for both 5 and 6 disk raids.
Not too sure about this point.
If it up and died in mid-grow there might be a chunk that's wrong.
But that's a few kilobytes, not...
> That is a span of ~52 GB where I presumably can't get the checksums
> right. What does all this mean? What am I missing?
...well, it would make sense if a disk got kicked / went missing
and it progressed the reshape for another ~52GB afterwards.
If you still had your original md metadata the --examine would clear
that point up but unfortunately...
In a /dev/md that doesn't have that same disk as missing, this would
result in roughly ~260-320Gs of data that is garbage (because one drive
was not reshaped but the others were so every nth chunk is wrong).
You might still be able to survive that (if the raid6 <-> raid5 overlap
zone is larger than that - I didn't do the math, but at a progress of
17% of your 6T disks you've added about 1T? Might just work out).
So you might have these zones on your RAIDs
6DISK: ?G VALID-DATA : ~320G of GARBAGE : ?G 5DISK-WRONGOFFSET-NONSENSE
5DISK: ?G 6DISK-WRONGOFFSET-NONSENSE : ~260G of GARBAGE : ?G VALID DATA
And you're hoping the VALID DATA areas will overlap. They would if it
progressed far enough with all disks and not too far with one missing.
Or you just have to identify the questionable drive and kick it out.
You have some experimenteering to do :-|
(
Not sure if I'm still making sense at this point. Sorry.
)
Regards
Andreas Klauer
^ permalink raw reply
* Re: mismatch_cnt > 0 during initial sync?
From: Stephane Thiell @ 2017-04-25 5:14 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <9A8A3C34-EED0-42FF-B631-7AD1D6EAF9A2@stanford.edu>
I just did some tests and this seems normal if one of the raid disk previously has some non-zero data. Perhaps there should be a note about that in md(4) or elsewhere? (sorry if I missed it)
Thanks,
Stephan
> On Apr 24, 2017, at 3:01 PM, Stephane Thiell <sthiell@stanford.edu> wrote:
>
>
> Hi all,
>
> Is it normal to get any mismatch_cnt > 0 during the initial sync? I just created a few new new raid6 volumes, and one of them is showing an increasing number of mismatch_cnt during its initial resync. Nothing is visible in the console nor after a smart short test (I also started a long test just in case).
>
>
> $ mdadm --detail /dev/md7
> /dev/md7:
> Version : 1.2
> Creation Time : Mon Apr 24 10:07:18 2017
> Raid Level : raid6
> Array Size : 62511163904 (59615.29 GiB 64011.43 GB)
> Used Dev Size : 7813895488 (7451.91 GiB 8001.43 GB)
> Raid Devices : 10
> Total Devices : 10
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Mon Apr 24 14:52:12 2017
> State : clean, resyncing
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Resync Status : 24% complete
>
> Name : oak-io1-s2:7 (local to host oak-io1-s2)
> UUID : 6ece9d71:a97f9497:4612ed73:44dfdc0a
> Events : 3308
>
> Number Major Minor RaidDevice State
> 0 253 62 0 active sync /dev/dm-62
> 1 253 63 1 active sync /dev/dm-63
> 2 253 72 2 active sync /dev/dm-72
> 3 253 75 3 active sync /dev/dm-75
> 4 253 89 4 active sync /dev/dm-89
> 5 253 87 5 active sync /dev/dm-87
> 6 253 119 6 active sync /dev/dm-119
> 7 253 97 7 active sync /dev/dm-97
> 8 253 103 8 active sync /dev/dm-103
> 9 253 107 9 active sync /dev/dm-107
>
> $ cat /sys/block/md7/md/sync_completed
> 3795284824 / 15627790976
>
> $ cat /sys/block/md7/md/mismatch_cnt
> 3424
>
> versions:
> mdadm-3.4-14.el7_3.1.x86_64
> CentOS7 (3.10.0-514.10.2.el7_lustre.x86_64)
>
> Thanks!
>
> Stephan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-25 8:44 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <20170425001614.GA3936@metamorpher.de>
2017-04-25 2:16 GMT+02:00, Andreas Klauer <Andreas.Klauer@metamorpher.de>:
> On Tue, Apr 25, 2017 at 01:00:47AM +0200, Patrik Dahlström wrote:
>> 6 disk raid xor check: 84 ^ f6 ^ 87 ^ 96 ^ e1 ^ 82 == 0, OK
>
> This should be the 6 disk raid area.
Agreed
>
>> 5 disk raid xor check: 46 ^ 73 ^ 6d ^ 06 ^ 5e == 0, OK
>
> This should be the 5 disk raid area.
Agreed
>
>> 6 disk raid xor check: 46 ^ 73 ^ 6d ^ 06 ^ 5e ^ 00 == 0, OK
>
> Still 5 disks... grow did not progess until here,
> and the 6th disk is likely zero because it's new.
Agreed
>
>> But immediately before that, I can't get the xor sums to line up:
>> 0xfaa287ffff: b0 ^ 6d ^ 13 ^ 1b ^ b7 != ae (62 actually), NOK
>> This would mean that it's incorrect for both 5 and 6 disk raids.
>
> Not too sure about this point.
>
> If it up and died in mid-grow there might be a chunk that's wrong.
>
> But that's a few kilobytes, not...
>
>> That is a span of ~52 GB where I presumably can't get the checksums
>> right. What does all this mean? What am I missing?
>
> ...well, it would make sense if a disk got kicked / went missing
> and it progressed the reshape for another ~52GB afterwards.
Would that mean that I should be able to get the checksum to match if
I remove one of the values in a mismatching series. I have tried this,
but never got it correct.
> And you're hoping the VALID DATA areas will overlap. They would if it
> progressed far enough with all disks and not too far with one missing.
How do I test this? I haven't really used overlays before.
> Or you just have to identify the questionable drive and kick it out.
I'm still not sure if a drive was actually kicked out of the array. I
don't have any memory of discovering that a drive was suddenly kicked
out. Would a kicked out drive be automatically re-added if it came
back?
Shouldn't I find something about it in my syslog or kernel log?
I checked my command history and I didn't find any --fail commands
(except on /dev/mapper/sdf)
Should I post the command history here? It is quite long and doesn't
contain return codes or command output.
>
> You have some experimenteering to do :-|
>
> (
> Not sure if I'm still making sense at this point. Sorry.
> )
>
> Regards
> Andreas Klauer
>
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-25 9:01 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: Brad Campbell, linux-raid
In-Reply-To: <CAHKno4A7umuNBUPaHk0vmgj2fnjPgBi2oZJWidhKKaRX7ekgKA@mail.gmail.com>
On Tue, Apr 25, 2017 at 10:44:15AM +0200, Patrik Dahlström wrote:
> Would that mean that I should be able to get the checksum to match if
> I remove one of the values in a mismatching series. I have tried this,
> but never got it correct.
No, there is no parity left with a missing drive in a raid5.
But you might find some duplicate blocks of one drive still in 5disk
layout on the other drives where it would have ended up in 6disk layout.
You won't find all of them since there are parity blocks on both sides.
> Shouldn't I find something about it in my syslog or kernel log?
I suppose so, if you didn't boot a rescue/live system in between
(they assemble raid and grow continues and whatever happens there
is not in your logs).
Mind showing the --examine output of the two raid sets you're working
with, did you check that the offsets are correct for either one?
You found multiple filesystem headers, do you know the UUID it should
have so you're not working with some old remnant?
Not sure what to tell you really. :)
I hope things turn out to be simpler than they appear right now.
Good luck
Andreas Klauer
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-25 10:40 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <20170425090122.GA4488@metamorpher.de>
2017-04-25 11:01 GMT+02:00, Andreas Klauer <Andreas.Klauer@metamorpher.de>:
> On Tue, Apr 25, 2017 at 10:44:15AM +0200, Patrik Dahlström wrote:
>> Would that mean that I should be able to get the checksum to match if
>> I remove one of the values in a mismatching series. I have tried this,
>> but never got it correct.
>
> No, there is no parity left with a missing drive in a raid5.
I was thinking if it was in fact a 5 disk raid + garbage from 1 disk,
then the checksum would be correct if the garbage disk was filtered
out. Does that make sens?
>
> But you might find some duplicate blocks of one drive still in 5disk
> layout on the other drives where it would have ended up in 6disk layout.
> You won't find all of them since there are parity blocks on both sides.
Not sure if I follow
>
>> Shouldn't I find something about it in my syslog or kernel log?
>
> I suppose so, if you didn't boot a rescue/live system in between
> (they assemble raid and grow continues and whatever happens there
> is not in your logs).
I did boot 1 rescue system, but it was a raid rescue system that,
according to their docs, never write anything to the disks so we
should be fine. I did a grep on mdadm in both syslogs and kernel logs
and I didn't find anything about kicking a drive.
>
> Mind showing the --examine output of the two raid sets you're working
> with, did you check that the offsets are correct for either one?
Attached to the end of the email. AFAIK, the data offsets are pointing
to where the ext4 file system begins. Honestly, I don't know what to
look for in the output.
>
> You found multiple filesystem headers, do you know the UUID it should
> have so you're not working with some old remnant?
The file systems looks correct:
$ grep storage /etc/fstab
# commented out during recovery
#UUID=345ec7b8-b523-45d3-8c2e-35cda1ab62c1 /storage ext4
errors=remount-ro 0 1
$ file -s /dev/md0
/dev/md0: Linux rev 1.0 ext4 filesystem data,
UUID=345ec7b8-b523-45d3-8c2e-35cda1ab62c1 (extents) (64bit) (large
files) (huge files)
$ file -s /dev/md1
/dev/md1: Linux rev 1.0 ext4 filesystem data,
UUID=345ec7b8-b523-45d3-8c2e-35cda1ab62c1 (extents) (64bit) (large
files) (huge files)
You mentioned something about linear device mapping before. What is
that? Is it something I could experiment with? How do I do that?
If I can get copies of files I know should be present on the raid set,
would that help?
Output of examine calls:
$ mdadm --examine /dev/mapper/sd{a,b,d,c,e}
/dev/mapper/sda:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d7570c5:da2c78dd:eb7add42:73d27d8d
Name : rack-server-1:0 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:54 2017
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 23441584128 (22355.64 GiB 24004.18 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : c285c661:ae3a9df5:dadb24d8:ea226018
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:54 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 97093390 - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d7570c5:da2c78dd:eb7add42:73d27d8d
Name : rack-server-1:0 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:54 2017
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 12111493040 (5775.21 GiB 6201.08 GB)
Array Size : 23441584128 (22355.64 GiB 24004.18 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=390700976 sectors
State : clean
Device UUID : fe47f0cf:68894c26:9464bfc6:3dd1b973
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:54 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 97201a94 - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d7570c5:da2c78dd:eb7add42:73d27d8d
Name : rack-server-1:0 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:54 2017
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 23441584128 (22355.64 GiB 24004.18 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : 80965407:ecae968e:936beab7:0dedbc67
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:54 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 4b3126a - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d7570c5:da2c78dd:eb7add42:73d27d8d
Name : rack-server-1:0 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:54 2017
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 23441584128 (22355.64 GiB 24004.18 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : 121f2396:94a105f4:db8d9b34:8f950eb3
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:54 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c0f3586f - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 1d7570c5:da2c78dd:eb7add42:73d27d8d
Name : rack-server-1:0 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:54 2017
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 23441584128 (22355.64 GiB 24004.18 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : 973d238d:dd08107c:186646f0:2131ad56
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:54 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9f47520d - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
$ mdadm --examine /dev/mapper/sd{a,b,d,c,e,f}-2
/dev/mapper/sda-2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b83927d8:ec75eb8e:9d1d1016:4cce5dff
Name : rack-server-1:1 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:22 2017
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 29301980160 (27944.55 GiB 30005.23 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : 4536ac05:80fa79ce:fb940867:4dea2d14
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:22 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : a7c0d060 - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdb-2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b83927d8:ec75eb8e:9d1d1016:4cce5dff
Name : rack-server-1:1 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:22 2017
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 12111493040 (5775.21 GiB 6201.08 GB)
Array Size : 29301980160 (27944.55 GiB 30005.23 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=390700976 sectors
State : clean
Device UUID : 2990130a:7759c23c:fed7a8e8:1ea1542f
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:22 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : ce812210 - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdd-2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b83927d8:ec75eb8e:9d1d1016:4cce5dff
Name : rack-server-1:1 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:22 2017
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 29301980160 (27944.55 GiB 30005.23 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : ebd7e323:0480c5e8:f6362f38:0926ec10
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:22 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : ae28d543 - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdc-2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b83927d8:ec75eb8e:9d1d1016:4cce5dff
Name : rack-server-1:1 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:22 2017
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 29301980160 (27944.55 GiB 30005.23 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : 5bd0d200:3878824d:f1ad243f:130503c0
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:22 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : a5e11bed - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sde-2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b83927d8:ec75eb8e:9d1d1016:4cce5dff
Name : rack-server-1:1 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:22 2017
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 29301980160 (27944.55 GiB 30005.23 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : b859bdd4:b10a1761:a5803a47:76c9775b
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:22 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 30eacedc - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/mapper/sdf-2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : b83927d8:ec75eb8e:9d1d1016:4cce5dff
Name : rack-server-1:1 (local to host rack-server-1)
Creation Time : Mon Apr 24 21:54:22 2017
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 11720792240 (5588.91 GiB 6001.05 GB)
Array Size : 29301980160 (27944.55 GiB 30005.23 GB)
Used Dev Size : 11720792064 (5588.91 GiB 6001.05 GB)
Data Offset : 252928 sectors
Super Offset : 8 sectors
Unused Space : before=252840 sectors, after=176 sectors
State : clean
Device UUID : 1c2e4b33:e87a49fa:55e6f42e:bab18c10
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Apr 24 21:54:22 2017
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c57a616b - correct
Events : 0
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-25 10:51 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <CAHKno4AuKNxYiMazUy+hRJ2kJCdNNYw5L0uExvWc_52Ocn5V=g@mail.gmail.com>
2017-04-25 12:40 GMT+02:00, Patrik Dahlström <risca@powerlamerz.org>:
> 2017-04-25 11:01 GMT+02:00, Andreas Klauer <Andreas.Klauer@metamorpher.de>:
>> On Tue, Apr 25, 2017 at 10:44:15AM +0200, Patrik Dahlström wrote:
>> You found multiple filesystem headers, do you know the UUID it should
>> have so you're not working with some old remnant?
> The file systems looks correct:
> $ grep storage /etc/fstab
> # commented out during recovery
> #UUID=345ec7b8-b523-45d3-8c2e-35cda1ab62c1 /storage ext4
> errors=remount-ro 0 1
> $ file -s /dev/md0
> /dev/md0: Linux rev 1.0 ext4 filesystem data,
> UUID=345ec7b8-b523-45d3-8c2e-35cda1ab62c1 (extents) (64bit) (large
> files) (huge files)
> $ file -s /dev/md1
> /dev/md1: Linux rev 1.0 ext4 filesystem data,
> UUID=345ec7b8-b523-45d3-8c2e-35cda1ab62c1 (extents) (64bit) (large
> files) (huge files)
This actually got me thinking. During my destructive recovery
attempts, I would either don't specify a --data-offset or use
--data-offset=128M. Since the correct offsets are less than 128M
(123,5 MB actually), that data would be untouched in case a
reshape/rebuild was triggered by my attempts. That must explain why
the first 4 chunks of /dev/md{0,1} were identical when using correct
offset, right?
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox