* Re: mdadm Consistency Policy initialization
From: Artur Paszkiewicz @ 2017-04-24 10:07 UTC (permalink / raw)
To: Jes Sorensen; +Cc: linux-raid
In-Reply-To: <2ca923fa-c147-f8c9-2399-f3a0e0d1721a@gmail.com>
On 04/20/2017 06:08 PM, Jes Sorensen wrote:
> On 04/19/2017 06:29 AM, Artur Paszkiewicz wrote:
>> On 04/18/2017 06:50 PM, Jes Sorensen wrote:
>>> Hi Artur,
>>>
>>> In 5308f11727b889965efe5ac0e854d197c2b51f6d you introduced struct mdinfo: enum consistency_policy, but in mdadm.c you initialize it to UnSet which isn't part of the enum.
>>>
>>> Is there any actual difference between CONSISTENCY_POLICY_UNKNOWN and UnSet? It seems suboptimal to mix and match within the enum like this, and if CONSISTENCY_POLICY_UNKNOWN does the job, couldn't we just initialize with that?
>>
>> Hi Jes,
>>
>> The "enum consistency_policy" and "mapping_t consistency_policies[]"
>> represent values that can appear in sysfs. md/consistency_policy can be
>> "unknown" when the array is inactive. On the other hand, UnSet just
>> means that the --consistency-policy= parameter was not provided by the
>> user. I wanted to differentiate between these two cases. If you think
>> this is redundant I can change it and use CONSISTENCY_POLICY_UNKNOWN
>> instead, this should be straightforward.
>
> Hi Artur,
>
> I made some changes to map_name() and noticed that you already there default to CONSISTENCY_POLICY_UNKNOWN if it returns UnSet in sysfs_read(). However given that you do a lot of checks manually outside of sysfs.c, I dind't change the code there for now. I think we the change I made to map_name() it should be possible to simply a bunch of that checking code.
Hi Jes,
It does seem to be better this way. I'll send a patch that removes using
UnSet from all consistency policy related places soon.
Thanks,
Artur
^ permalink raw reply
* Re: [PATCH] dm verity: deferred hash checking for restart/logging mode
From: kbuild test robot @ 2017-04-24 9:58 UTC (permalink / raw)
Cc: kbuild-all, agk, snitzer, dm-devel, shli, linux-raid,
linux-kernel, Bongkyu Kim
In-Reply-To: <1493019260-43938-1-git-send-email-bongkyu.kim@lge.com>
[-- Attachment #1: Type: text/plain, Size: 871 bytes --]
Hi Bongkyu,
[auto build test ERROR on dm/for-next]
[also build test ERROR on v4.11-rc8 next-20170421]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Bongkyu-Kim/dm-verity-deferred-hash-checking-for-restart-logging-mode/20170424-154859
base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next
config: x86_64-rhel (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
All errors (new ones prefixed by >>):
>> ERROR: "free_io" [drivers/md/dm-verity.ko] undefined!
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 38849 bytes --]
^ permalink raw reply
* [MD PATCH 1/1] Don't add nr_pending for resync requests
From: Xiao Ni @ 2017-04-24 8:47 UTC (permalink / raw)
To: linux-raid; +Cc: shli, ncroxon
In newer barrier codes, raise_barrier waits if conf->nr_pending[idx] is not zero.
After all the conditions are true, the resync request can go on be handling. But
it adds conf->nr_pending[idx] again. The next resync request hit the same bucket
idx need to wait the resync request which is submitted before. The performance
of resync/recovery is degraded.
I did some tests:
1. Before removing the codes, the resync performance is:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 0.00 0.00 164.00 0.00 10.13 126.46 0.95 5.80 0.00 5.80 5.82 95.40
sdb 0.00 0.00 163.00 2.00 10.19 0.00 126.47 0.04 0.25 0.18 5.50 0.25 4.10
2. After removing the codes, the resync performance is:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 2164.00 0.00 751.00 0.00 182.06 496.49 7.49 9.95 0.00 9.95 1.33 100.00
sdb 2162.00 0.00 752.00 0.00 182.25 0.00 496.34 0.56 0.74 0.74 0.00 0.65 48.80
Signed-off-by: Xiao Ni <xni@redhat.com>
---
drivers/md/raid1.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index a34f587..48567ea 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -869,7 +869,6 @@ static void raise_barrier(struct r1conf *conf, sector_t sector_nr)
atomic_read(&conf->barrier[idx]) < RESYNC_DEPTH,
conf->resync_lock);
- atomic_inc(&conf->nr_pending[idx]);
spin_unlock_irq(&conf->resync_lock);
}
@@ -880,7 +879,6 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr)
BUG_ON(atomic_read(&conf->barrier[idx]) <= 0);
atomic_dec(&conf->barrier[idx]);
- atomic_dec(&conf->nr_pending[idx]);
wake_up(&conf->wait_barrier);
}
--
2.7.4
^ permalink raw reply related
* Re: [PATCH] dm verity: deferred hash checking for restart/logging mode
From: Milan Broz @ 2017-04-24 8:05 UTC (permalink / raw)
To: Bongkyu Kim, agk, snitzer
Cc: Will Drewry, linux-kernel, linux-raid, dm-devel, Sami Tolvanen,
shli
In-Reply-To: <1493019260-43938-1-git-send-email-bongkyu.kim@lge.com>
On 04/24/2017 09:34 AM, Bongkyu Kim wrote:
> This patch introduces deferred hash checking for dm-verity.
> In case of restart and logging mode, I/O return first and hash checking is
> deferred. It can improve dm-verity's performance greatly.
>
> This is my testing on qualcomm snapdragon 821 platform with UFS 2.0.
> dd if=/dev/block/dm-0 of=/dev/null bs=1024 count=1000000
> - vanilla: 238.14 MB/s
> - patched: 331.52 MB/s (+39.2%)
>
> fio --rw=randread --size=64M --bs=4k --filename=/dev/block/dm-0 --name=job1
> --name=job2 --name=job3 --name=job4
> - vanilla: 325.21 MB/s
> - patched: 356.42 MB/s (+9.6%)
>
> One major concoren is security risk.
> If data is tampered, this data will not return -EIO, and can be transferred
> to user process. But, device will be rebooted after hash verification.
Isn't one of the goal of integrity checking to PREVENT that userspace
accesses tampered data?
What happens if that corrupted part is malware that just send one packet
with confidential data in time window before it restarts?
Now I am not sure what is the Google Chromebook/Android threat model here.
If we accept patches that allows non-verified data to be returned to userspace,
dmverity just becomes way how to detect flaky hw, I cannot imagine it is acceptable
in "verified boot" chain.
Could someone from dm-verity authors comment this?
(I added Sami and Will to cc but not sure if there is anyone else)?
Milan
> After rebooting, device will work with EIO mode.
> In my opinion, this is enough for gurantee integrity by each power cycles.
>
> Signed-off-by: Bongkyu Kim <bongkyu.kim@lge.com>
> ---
> drivers/md/dm-verity-target.c | 58 ++++++++++++++++++++++++++++++++++++++++---
> drivers/md/dm.c | 17 +++++++++++--
> drivers/md/dm.h | 4 +++
> 3 files changed, 73 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
> index 7335d8a..37c4d9c 100644
> --- a/drivers/md/dm-verity-target.c
> +++ b/drivers/md/dm-verity-target.c
> @@ -16,6 +16,7 @@
>
> #include "dm-verity.h"
> #include "dm-verity-fec.h"
> +#include "dm.h"
>
> #include <linux/module.h>
> #include <linux/reboot.h>
> @@ -62,6 +63,9 @@ struct buffer_aux {
> int hash_verified;
> };
>
> +static void verity_submit_prefetch(struct dm_verity *v,
> + struct dm_verity_io *io);
> +
> /*
> * Initialize struct buffer_aux for a freshly created buffer.
> */
> @@ -462,12 +466,35 @@ static void verity_finish_io(struct dm_verity_io *io, int error)
> struct dm_verity *v = io->v;
> struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size);
>
> - bio->bi_end_io = io->orig_bi_end_io;
> - bio->bi_error = error;
> + if (v->mode == DM_VERITY_MODE_EIO) {
> + bio->bi_end_io = io->orig_bi_end_io;
> + bio->bi_error = error;
> + }
>
> verity_fec_finish_io(io);
>
> - bio_endio(bio);
> + if (v->mode == DM_VERITY_MODE_EIO) {
> + bio_endio(bio);
> + } else {
> + struct dm_target_io *tio = container_of(bio,
> + struct dm_target_io, clone);
> + struct dm_io *dmio = tio->io;
> + struct bio *ori_bio = dm_io_get_bio(dmio);
> + struct bio_vec *bv;
> + int i;
> +
> + bio_put(bio);
> + free_io(dm_io_get_md(dmio), dmio);
> +
> + bio_for_each_segment_all(bv, ori_bio, i) {
> + struct page *page = bv->bv_page;
> +
> + put_page(page);
> + }
> +
> + bio_put(ori_bio);
> +
> + }
> }
>
> static void verity_work(struct work_struct *w)
> @@ -486,6 +513,28 @@ static void verity_end_io(struct bio *bio)
> return;
> }
>
> + if (io->v->mode != DM_VERITY_MODE_EIO) {
> + struct dm_target_io *tio = container_of(bio,
> + struct dm_target_io, clone);
> + struct bio *ori_bio = dm_io_get_bio(tio->io);
> + struct bio_vec *bv;
> + int i;
> +
> + bio_for_each_segment_all(bv, ori_bio, i) {
> + struct page *page = bv->bv_page;
> +
> + get_page(page);
> + }
> +
> + bio_get(ori_bio);
> + bio_get(bio);
> +
> + bio->bi_end_io = io->orig_bi_end_io;
> + bio_endio(bio);
> +
> + verity_submit_prefetch(io->v, io);
> + }
> +
> INIT_WORK(&io->work, verity_work);
> queue_work(io->v->verify_wq, &io->work);
> }
> @@ -586,7 +635,8 @@ static int verity_map(struct dm_target *ti, struct bio *bio)
>
> verity_fec_init_io(io);
>
> - verity_submit_prefetch(v, io);
> + if (v->mode == DM_VERITY_MODE_EIO)
> + verity_submit_prefetch(v, io);
>
> generic_make_request(bio);
>
> diff --git a/drivers/md/dm.c b/drivers/md/dm.c
> index 8bf3977..9eca0a0 100644
> --- a/drivers/md/dm.c
> +++ b/drivers/md/dm.c
> @@ -164,6 +164,18 @@ static unsigned dm_get_numa_node(void)
> DM_NUMA_NODE, num_online_nodes() - 1);
> }
>
> +struct bio *dm_io_get_bio(struct dm_io *io)
> +{
> + return io->bio;
> +}
> +EXPORT_SYMBOL(dm_io_get_bio);
> +
> +struct mapped_device *dm_io_get_md(struct dm_io *io)
> +{
> + return io->md;
> +}
> +EXPORT_SYMBOL(dm_io_get_md);
> +
> static int __init local_init(void)
> {
> int r = -ENOMEM;
> @@ -489,7 +501,7 @@ static struct dm_io *alloc_io(struct mapped_device *md)
> return mempool_alloc(md->io_pool, GFP_NOIO);
> }
>
> -static void free_io(struct mapped_device *md, struct dm_io *io)
> +void free_io(struct mapped_device *md, struct dm_io *io)
> {
> mempool_free(io, md->io_pool);
> }
> @@ -796,7 +808,8 @@ static void dec_pending(struct dm_io *io, int error)
> io_error = io->error;
> bio = io->bio;
> end_io_acct(io);
> - free_io(md, io);
> + if (atomic_read(&bio->__bi_cnt) <= 1)
> + free_io(md, io);
>
> if (io_error == DM_ENDIO_REQUEUE)
> return;
> diff --git a/drivers/md/dm.h b/drivers/md/dm.h
> index f298b01..b026178 100644
> --- a/drivers/md/dm.h
> +++ b/drivers/md/dm.h
> @@ -213,4 +213,8 @@ void dm_free_md_mempools(struct dm_md_mempools *pools);
> */
> unsigned dm_get_reserved_bio_based_ios(void);
>
> +struct bio *dm_io_get_bio(struct dm_io *io);
> +struct mapped_device *dm_io_get_md(struct dm_io *io);
> +void free_io(struct mapped_device *md, struct dm_io *io);
> +
> #endif
>
^ permalink raw reply
* [PATCH] md: clear WantReplacement once disk is removed
From: Guoqing Jiang @ 2017-04-24 7:58 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, shli, Guoqing Jiang
We can clear 'WantReplacement' flag directly no
matter it's replacement existed or not since the
semantic is same as before.
Also since the disk is removed from array, then
it is straightforward to remove 'WantReplacement'
flag and the comments in raid10/5 can be removed
as well.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/raid1.c | 6 +++---
drivers/md/raid10.c | 8 ++------
drivers/md/raid5.c | 9 +++------
3 files changed, 8 insertions(+), 15 deletions(-)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 57611f4..e93dd50 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1831,9 +1831,9 @@ static int raid1_remove_disk(struct mddev *mddev, struct md_rdev *rdev)
p->rdev = repl;
conf->mirrors[conf->raid_disks + number].rdev = NULL;
unfreeze_array(conf);
- clear_bit(WantReplacement, &rdev->flags);
- } else
- clear_bit(WantReplacement, &rdev->flags);
+ }
+
+ clear_bit(WantReplacement, &rdev->flags);
err = md_integrity_register(mddev);
}
abort:
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index acb3f46..72b2a9c 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1874,13 +1874,9 @@ static int raid10_remove_disk(struct mddev *mddev, struct md_rdev *rdev)
* but will never see neither -- if they are careful.
*/
p->replacement = NULL;
- clear_bit(WantReplacement, &rdev->flags);
- } else
- /* We might have just remove the Replacement as faulty
- * Clear the flag just in case
- */
- clear_bit(WantReplacement, &rdev->flags);
+ }
+ clear_bit(WantReplacement, &rdev->flags);
err = md_integrity_register(mddev);
abort:
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 356cd9c..3d971e5 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7603,15 +7603,12 @@ static int raid5_remove_disk(struct mddev *mddev, struct md_rdev *rdev)
* but will never see neither - if they are careful
*/
p->replacement = NULL;
- clear_bit(WantReplacement, &rdev->flags);
if (!err)
err = log_modify(conf, p->rdev, true);
- } else
- /* We might have just removed the Replacement as faulty-
- * clear the bit just in case
- */
- clear_bit(WantReplacement, &rdev->flags);
+ }
+
+ clear_bit(WantReplacement, &rdev->flags);
abort:
print_raid5_conf(conf);
--
2.10.0
^ permalink raw reply related
* [PATCH] dm verity: deferred hash checking for restart/logging mode
From: Bongkyu Kim @ 2017-04-24 7:34 UTC (permalink / raw)
To: agk, snitzer; +Cc: dm-devel, shli, linux-raid, linux-kernel, Bongkyu Kim
This patch introduces deferred hash checking for dm-verity.
In case of restart and logging mode, I/O return first and hash checking is
deferred. It can improve dm-verity's performance greatly.
This is my testing on qualcomm snapdragon 821 platform with UFS 2.0.
dd if=/dev/block/dm-0 of=/dev/null bs=1024 count=1000000
- vanilla: 238.14 MB/s
- patched: 331.52 MB/s (+39.2%)
fio --rw=randread --size=64M --bs=4k --filename=/dev/block/dm-0 --name=job1
--name=job2 --name=job3 --name=job4
- vanilla: 325.21 MB/s
- patched: 356.42 MB/s (+9.6%)
One major concoren is security risk.
If data is tampered, this data will not return -EIO, and can be transferred
to user process. But, device will be rebooted after hash verification.
After rebooting, device will work with EIO mode.
In my opinion, this is enough for gurantee integrity by each power cycles.
Signed-off-by: Bongkyu Kim <bongkyu.kim@lge.com>
---
drivers/md/dm-verity-target.c | 58 ++++++++++++++++++++++++++++++++++++++++---
drivers/md/dm.c | 17 +++++++++++--
drivers/md/dm.h | 4 +++
3 files changed, 73 insertions(+), 6 deletions(-)
diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
index 7335d8a..37c4d9c 100644
--- a/drivers/md/dm-verity-target.c
+++ b/drivers/md/dm-verity-target.c
@@ -16,6 +16,7 @@
#include "dm-verity.h"
#include "dm-verity-fec.h"
+#include "dm.h"
#include <linux/module.h>
#include <linux/reboot.h>
@@ -62,6 +63,9 @@ struct buffer_aux {
int hash_verified;
};
+static void verity_submit_prefetch(struct dm_verity *v,
+ struct dm_verity_io *io);
+
/*
* Initialize struct buffer_aux for a freshly created buffer.
*/
@@ -462,12 +466,35 @@ static void verity_finish_io(struct dm_verity_io *io, int error)
struct dm_verity *v = io->v;
struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size);
- bio->bi_end_io = io->orig_bi_end_io;
- bio->bi_error = error;
+ if (v->mode == DM_VERITY_MODE_EIO) {
+ bio->bi_end_io = io->orig_bi_end_io;
+ bio->bi_error = error;
+ }
verity_fec_finish_io(io);
- bio_endio(bio);
+ if (v->mode == DM_VERITY_MODE_EIO) {
+ bio_endio(bio);
+ } else {
+ struct dm_target_io *tio = container_of(bio,
+ struct dm_target_io, clone);
+ struct dm_io *dmio = tio->io;
+ struct bio *ori_bio = dm_io_get_bio(dmio);
+ struct bio_vec *bv;
+ int i;
+
+ bio_put(bio);
+ free_io(dm_io_get_md(dmio), dmio);
+
+ bio_for_each_segment_all(bv, ori_bio, i) {
+ struct page *page = bv->bv_page;
+
+ put_page(page);
+ }
+
+ bio_put(ori_bio);
+
+ }
}
static void verity_work(struct work_struct *w)
@@ -486,6 +513,28 @@ static void verity_end_io(struct bio *bio)
return;
}
+ if (io->v->mode != DM_VERITY_MODE_EIO) {
+ struct dm_target_io *tio = container_of(bio,
+ struct dm_target_io, clone);
+ struct bio *ori_bio = dm_io_get_bio(tio->io);
+ struct bio_vec *bv;
+ int i;
+
+ bio_for_each_segment_all(bv, ori_bio, i) {
+ struct page *page = bv->bv_page;
+
+ get_page(page);
+ }
+
+ bio_get(ori_bio);
+ bio_get(bio);
+
+ bio->bi_end_io = io->orig_bi_end_io;
+ bio_endio(bio);
+
+ verity_submit_prefetch(io->v, io);
+ }
+
INIT_WORK(&io->work, verity_work);
queue_work(io->v->verify_wq, &io->work);
}
@@ -586,7 +635,8 @@ static int verity_map(struct dm_target *ti, struct bio *bio)
verity_fec_init_io(io);
- verity_submit_prefetch(v, io);
+ if (v->mode == DM_VERITY_MODE_EIO)
+ verity_submit_prefetch(v, io);
generic_make_request(bio);
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 8bf3977..9eca0a0 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -164,6 +164,18 @@ static unsigned dm_get_numa_node(void)
DM_NUMA_NODE, num_online_nodes() - 1);
}
+struct bio *dm_io_get_bio(struct dm_io *io)
+{
+ return io->bio;
+}
+EXPORT_SYMBOL(dm_io_get_bio);
+
+struct mapped_device *dm_io_get_md(struct dm_io *io)
+{
+ return io->md;
+}
+EXPORT_SYMBOL(dm_io_get_md);
+
static int __init local_init(void)
{
int r = -ENOMEM;
@@ -489,7 +501,7 @@ static struct dm_io *alloc_io(struct mapped_device *md)
return mempool_alloc(md->io_pool, GFP_NOIO);
}
-static void free_io(struct mapped_device *md, struct dm_io *io)
+void free_io(struct mapped_device *md, struct dm_io *io)
{
mempool_free(io, md->io_pool);
}
@@ -796,7 +808,8 @@ static void dec_pending(struct dm_io *io, int error)
io_error = io->error;
bio = io->bio;
end_io_acct(io);
- free_io(md, io);
+ if (atomic_read(&bio->__bi_cnt) <= 1)
+ free_io(md, io);
if (io_error == DM_ENDIO_REQUEUE)
return;
diff --git a/drivers/md/dm.h b/drivers/md/dm.h
index f298b01..b026178 100644
--- a/drivers/md/dm.h
+++ b/drivers/md/dm.h
@@ -213,4 +213,8 @@ void dm_free_md_mempools(struct dm_md_mempools *pools);
*/
unsigned dm_get_reserved_bio_based_ios(void);
+struct bio *dm_io_get_bio(struct dm_io *io);
+struct mapped_device *dm_io_get_md(struct dm_io *io);
+void free_io(struct mapped_device *md, struct dm_io *io);
+
#endif
--
2.6.2
^ permalink raw reply related
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-24 7:34 UTC (permalink / raw)
To: Brad Campbell; +Cc: Andreas Klauer, linux-raid
In-Reply-To: <63d6d0f5-d9b1-1218-b98e-48f9cf21a8bc@fnarfbargle.com>
2017-04-24 4:09 GMT+02:00, Brad Campbell <lists2009@fnarfbargle.com>:
> On 24/04/17 03:21, Patrik Dahlström wrote:
>
>> I just thought of something: since /dev/sdf is a completely new disk,
>> shouldn't I be able to locate the starting point by looking at how much
>> of the new disk is filled. It comes filled with zeros from factory,
>> right?
>
> Pretty close. You should also be able to see how the reshape worked
> (front to back, back to front) and you should get a pretty good idea of
> the data-offset by the gap between the superblock and the data.
I've let a program compare both raid sets (5 and 6 disk) overnight. So
far it has gone from 128 MB to 14 TB without finding common data. Does
that tell us anything?
Should I pause the comparison and look at the end of sdf to find the
last written offset? I currently only have 1 shell in Ubuntu
maintenance/rescue mode
>
> I must say I'm having trouble imagining a more difficult recovery
> scenario, so at least you present an interesting challenge.
I always like a challenge, but this one I believe I could've skipped :)
>
> There's a lot to be said for the old "Don't panic" mantra, but hindsight
> is always 20/20.
All too true
Best regards
// Patrik
^ permalink raw reply
* Re: Recover array after I panicked
From: Brad Campbell @ 2017-04-24 2:09 UTC (permalink / raw)
To: Patrik Dahlström, Andreas Klauer; +Cc: linux-raid
In-Reply-To: <718f75d1-b74f-f788-783a-6ffbdff275a8@powerlamerz.org>
On 24/04/17 03:21, Patrik Dahlström wrote:
> I just thought of something: since /dev/sdf is a completely new disk,
> shouldn't I be able to locate the starting point by looking at how much
> of the new disk is filled. It comes filled with zeros from factory, right?
Pretty close. You should also be able to see how the reshape worked
(front to back, back to front) and you should get a pretty good idea of
the data-offset by the gap between the superblock and the data.
I must say I'm having trouble imagining a more difficult recovery
scenario, so at least you present an interesting challenge.
There's a lot to be said for the old "Don't panic" mantra, but hindsight
is always 20/20.
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 19:21 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <20170423154240.GA3494@metamorpher.de>
On 04/23/2017 05:42 PM, Andreas Klauer wrote:
> On Sun, Apr 23, 2017 at 05:11:08PM +0200, Patrik Dahlström wrote:
>> * I create a 5 and a 6 drive raid set and try to find an offset where
>> they both carry the same raw data. With some overlays, I should be able
>> to create both these raids at the same time, correct?
>
> Yes, two sets of overlays.
>
> So overlay A is your 5 disk raid5, overlay B is your 6 disk raid5,
> and then you'll just have to take a stab at it with hexdump.
>
> So kind of like,
>
> hexdump --skip $((17*1024*1024*1024)) --length 4096 /dev/md42
> hexdump --skip $((17*1024*1024*1024)) --length 4096 /dev/md43
>
> If that produces the same random-looking data then your reshape
> might have progressed 17G-ish and you could try to use that as
> a starting point for a linear device mapping that uses the
> first 17G of the 6 disk raid and everything else from the 5disk.
>
> The further the grow processed the larger a zone of overlap there
> should be since more data ends up on the additional drive so
> the original representation of the same data isn't yet written
> into on the old drives.
>
> Does that make sense?
>
> 5 disks: a b c d e : f g h i j : k l m n o : p q r s t : ...
> 6 disks: a b c d e f : g h i j k l : m n o p q r : s t : ...
>
> If it stopped at "t", both have "r s t"... and that "r s t"
> would be what you have to find. You have to be wary of false
> matches though (such as zeroes or other common patterns of data
> you might find anywhere).
I just thought of something: since /dev/sdf is a completely new disk,
shouldn't I be able to locate the starting point by looking at how much
of the new disk is filled. It comes filled with zeros from factory, right?
Best regards
// Patrik
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 16:29 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Brad Campbell, linux-raid
In-Reply-To: <20170423154240.GA3494@metamorpher.de>
On 04/23/2017 05:42 PM, Andreas Klauer wrote:
> On Sun, Apr 23, 2017 at 05:11:08PM +0200, Patrik Dahlström wrote:
>> * I try to find 0x53EF (ext4 magic) within the first 128M of
>> /dev/sd[abcde]. Not after? This will be an indication of my "New
>> Offset". I need to adjust the offset a bit since the ext4 magic is
>> located at 0x438 offset.
>
> Yes, that should be about it.
>
>> * I create a 5 and a 6 drive raid set and try to find an offset where
>> they both carry the same raw data. With some overlays, I should be able
>> to create both these raids at the same time, correct?
>
> Yes, two sets of overlays.
>
> So overlay A is your 5 disk raid5, overlay B is your 6 disk raid5,
> and then you'll just have to take a stab at it with hexdump.
>
> So kind of like,
>
> hexdump --skip $((17*1024*1024*1024)) --length 4096 /dev/md42
> hexdump --skip $((17*1024*1024*1024)) --length 4096 /dev/md43
>
> If that produces the same random-looking data then your reshape
> might have progressed 17G-ish and you could try to use that as
> a starting point for a linear device mapping that uses the
> first 17G of the 6 disk raid and everything else from the 5disk.
>
> The further the grow processed the larger a zone of overlap there
> should be since more data ends up on the additional drive so
> the original representation of the same data isn't yet written
> into on the old drives.
>
> Does that make sense?
>
> 5 disks: a b c d e : f g h i j : k l m n o : p q r s t : ...
> 6 disks: a b c d e f : g h i j k l : m n o p q r : s t : ...
>
> If it stopped at "t", both have "r s t"... and that "r s t"
> would be what you have to find. You have to be wary of false
> matches though (such as zeroes or other common patterns of data
> you might find anywhere).
>
> And there's one more thing, you mentioned a disk had a bad sata cable.
> If that disk got kicked and reshape went on for a while afterwards,
> you should take that disk out of consideration. (Specify as "missing"
> when creating the arrays.) It will have outdated/unreshaped data on
> it that would be hard to incorporate into your recovery attempt...
The bad SATA cable has been there for a while and reported spurious I/O
errors on the old 5 disk raid. It really started to produce I/O errors
when I placed the new drive in that slot.
Is it OK to specify the new drive as missing? No special information
that should be located on that drive?
I wrote a small program to help me locate the correct offset:
https://gist.github.com/Risca/3eda5e7aba3dc6b72d61e79eaf7cc147
I am on the right track with that program?
>
> Regards
> Andreas Klauer
>
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-23 15:42 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: Brad Campbell, linux-raid
In-Reply-To: <f353b7fc-a63a-5503-5cda-6a62698f69ae@powerlamerz.org>
On Sun, Apr 23, 2017 at 05:11:08PM +0200, Patrik Dahlström wrote:
> * I try to find 0x53EF (ext4 magic) within the first 128M of
> /dev/sd[abcde]. Not after? This will be an indication of my "New
> Offset". I need to adjust the offset a bit since the ext4 magic is
> located at 0x438 offset.
Yes, that should be about it.
> * I create a 5 and a 6 drive raid set and try to find an offset where
> they both carry the same raw data. With some overlays, I should be able
> to create both these raids at the same time, correct?
Yes, two sets of overlays.
So overlay A is your 5 disk raid5, overlay B is your 6 disk raid5,
and then you'll just have to take a stab at it with hexdump.
So kind of like,
hexdump --skip $((17*1024*1024*1024)) --length 4096 /dev/md42
hexdump --skip $((17*1024*1024*1024)) --length 4096 /dev/md43
If that produces the same random-looking data then your reshape
might have progressed 17G-ish and you could try to use that as
a starting point for a linear device mapping that uses the
first 17G of the 6 disk raid and everything else from the 5disk.
The further the grow processed the larger a zone of overlap there
should be since more data ends up on the additional drive so
the original representation of the same data isn't yet written
into on the old drives.
Does that make sense?
5 disks: a b c d e : f g h i j : k l m n o : p q r s t : ...
6 disks: a b c d e f : g h i j k l : m n o p q r : s t : ...
If it stopped at "t", both have "r s t"... and that "r s t"
would be what you have to find. You have to be wary of false
matches though (such as zeroes or other common patterns of data
you might find anywhere).
And there's one more thing, you mentioned a disk had a bad sata cable.
If that disk got kicked and reshape went on for a while afterwards,
you should take that disk out of consideration. (Specify as "missing"
when creating the arrays.) It will have outdated/unreshaped data on
it that would be hard to incorporate into your recovery attempt...
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 15:24 UTC (permalink / raw)
To: Andreas Klauer, Brad Campbell; +Cc: linux-raid
In-Reply-To: <f353b7fc-a63a-5503-5cda-6a62698f69ae@powerlamerz.org>
On 04/23/2017 05:11 PM, Patrik Dahlström wrote:
>
>
> On 04/23/2017 04:48 PM, Andreas Klauer wrote:
>> On Sun, Apr 23, 2017 at 10:06:15PM +0800, Brad Campbell wrote:
>>> Nobody seems to have mentioned the reshape issue.
>>
>> Good point.
>>
>> If it was mid-reshape you need two sets of overlays,
>> create two RAIDs (one for each configuration), and
>> then find the point where it converges.
>>
>>> If my reading of the code is correct (and my memory
>>> is any good), simply adding a disk to a raid5 on a
>>> recent enough kernel should make the resync go backwards.
>>
>> Doesn't it cut the offset by half and grow forwards...?
>>
>> With growing a disk that should give you a segment where
>> data is identical for both 5-disk and 6-disk RAID-5.
>> And that's where you join them using dmsetup linear.
>>
>> Before:
>>
>> /dev/loop0:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : 4611f41b:0464e815:8b6f9cfe:b29c56fd
>> Name : EIS:42 (local to host EIS)
>> Creation Time : Sun Apr 23 16:44:59 2017
>> Raid Level : raid5
>> Raid Devices : 5
>>
>> Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
>> Array Size : 23441565696 (22355.62 GiB 24004.16 GB)
>> Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262064 sectors, after=176 sectors
>> State : clean
>> Device UUID : acd8d9fd:7b7cf9a0:f63369d1:907ffa66
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Sun Apr 23 16:44:59 2017
>> Bad Block Log : 512 entries available at offset 32 sectors
>> Checksum : f89bdc5 - correct
>> Events : 2
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>>
>> After/During grow:
>>
>> /dev/loop0:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x45
>> Array UUID : 4611f41b:0464e815:8b6f9cfe:b29c56fd
>> Name : EIS:42 (local to host EIS)
>> Creation Time : Sun Apr 23 16:44:59 2017
>> Raid Level : raid5
>> Raid Devices : 6
>>
>> Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
>> Array Size : 29301957120 (27944.52 GiB 30005.20 GB)
>> Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
>> Data Offset : 262144 sectors
>> | New Offset : 257024 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : acd8d9fd:7b7cf9a0:f63369d1:907ffa66
>>
>> Internal Bitmap : 8 sectors from superblock
>> | Reshape pos'n : 1472000 (1437.50 MiB 1507.33 MB)
>> | Delta Devices : 1 (5->6)
>>
>> Update Time : Sun Apr 23 16:45:38 2017
>> Bad Block Log : 512 entries available at offset 32 sectors
>> Checksum : fbd9a55 - correct
>> Events : 30
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
>>
>> Basically you have to know the New Offset
>> (search first 128M of your drives for filesystem headers, that should be it)
> Let's see if I understand you correctly:
>
> * I try to find 0x53EF (ext4 magic) within the first 128M of
> /dev/sd[abcde]. Not after? This will be an indication of my "New
> Offset". I need to adjust the offset a bit since the ext4 magic is
> located at 0x438 offset.
>
Okay, I located what appears to be a ext4 file system header at
0x7B80000 in both /dev/sda and /dev/sdf. I used this command:
dd if=/dev/sda bs=524288 count=256 | ./ext2scan
where ext2scan comes from https://goo.gl/2TnZSR
>> and then guess the Reshape pos'n by comparing raw data at offset X
>> (find non-zero data at identical offsets for both raid sets)
>
> * I create a 5 and a 6 drive raid set and try to find an offset where
> they both carry the same raw data. With some overlays, I should be able
> to create both these raids at the same time, correct?
I'm still working on this one. Should I start looking at ~15% of the raid?
What is the next step after this?
>
>>
>> Regards
>> Andreas Klauer
>>
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 15:11 UTC (permalink / raw)
To: Andreas Klauer, Brad Campbell; +Cc: linux-raid
In-Reply-To: <20170423144835.GB12093@metamorpher.de>
On 04/23/2017 04:48 PM, Andreas Klauer wrote:
> On Sun, Apr 23, 2017 at 10:06:15PM +0800, Brad Campbell wrote:
>> Nobody seems to have mentioned the reshape issue.
>
> Good point.
>
> If it was mid-reshape you need two sets of overlays,
> create two RAIDs (one for each configuration), and
> then find the point where it converges.
>
>> If my reading of the code is correct (and my memory
>> is any good), simply adding a disk to a raid5 on a
>> recent enough kernel should make the resync go backwards.
>
> Doesn't it cut the offset by half and grow forwards...?
>
> With growing a disk that should give you a segment where
> data is identical for both 5-disk and 6-disk RAID-5.
> And that's where you join them using dmsetup linear.
>
> Before:
>
> /dev/loop0:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x1
> Array UUID : 4611f41b:0464e815:8b6f9cfe:b29c56fd
> Name : EIS:42 (local to host EIS)
> Creation Time : Sun Apr 23 16:44:59 2017
> Raid Level : raid5
> Raid Devices : 5
>
> Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
> Array Size : 23441565696 (22355.62 GiB 24004.16 GB)
> Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262064 sectors, after=176 sectors
> State : clean
> Device UUID : acd8d9fd:7b7cf9a0:f63369d1:907ffa66
>
> Internal Bitmap : 8 sectors from superblock
> Update Time : Sun Apr 23 16:44:59 2017
> Bad Block Log : 512 entries available at offset 32 sectors
> Checksum : f89bdc5 - correct
> Events : 2
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>
> After/During grow:
>
> /dev/loop0:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x45
> Array UUID : 4611f41b:0464e815:8b6f9cfe:b29c56fd
> Name : EIS:42 (local to host EIS)
> Creation Time : Sun Apr 23 16:44:59 2017
> Raid Level : raid5
> Raid Devices : 6
>
> Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
> Array Size : 29301957120 (27944.52 GiB 30005.20 GB)
> Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
> Data Offset : 262144 sectors
> | New Offset : 257024 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : acd8d9fd:7b7cf9a0:f63369d1:907ffa66
>
> Internal Bitmap : 8 sectors from superblock
> | Reshape pos'n : 1472000 (1437.50 MiB 1507.33 MB)
> | Delta Devices : 1 (5->6)
>
> Update Time : Sun Apr 23 16:45:38 2017
> Bad Block Log : 512 entries available at offset 32 sectors
> Checksum : fbd9a55 - correct
> Events : 30
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
>
> Basically you have to know the New Offset
> (search first 128M of your drives for filesystem headers, that should be it)
Let's see if I understand you correctly:
* I try to find 0x53EF (ext4 magic) within the first 128M of
/dev/sd[abcde]. Not after? This will be an indication of my "New
Offset". I need to adjust the offset a bit since the ext4 magic is
located at 0x438 offset.
> and then guess the Reshape pos'n by comparing raw data at offset X
> (find non-zero data at identical offsets for both raid sets)
* I create a 5 and a 6 drive raid set and try to find an offset where
they both carry the same raw data. With some overlays, I should be able
to create both these raids at the same time, correct?
>
> Regards
> Andreas Klauer
>
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-23 14:48 UTC (permalink / raw)
To: Brad Campbell; +Cc: Patrik Dahlström, linux-raid
In-Reply-To: <807de641-043c-41a0-cffe-e28710503aba@fnarfbargle.com>
On Sun, Apr 23, 2017 at 10:06:15PM +0800, Brad Campbell wrote:
> Nobody seems to have mentioned the reshape issue.
Good point.
If it was mid-reshape you need two sets of overlays,
create two RAIDs (one for each configuration), and
then find the point where it converges.
> If my reading of the code is correct (and my memory
> is any good), simply adding a disk to a raid5 on a
> recent enough kernel should make the resync go backwards.
Doesn't it cut the offset by half and grow forwards...?
With growing a disk that should give you a segment where
data is identical for both 5-disk and 6-disk RAID-5.
And that's where you join them using dmsetup linear.
Before:
/dev/loop0:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 4611f41b:0464e815:8b6f9cfe:b29c56fd
Name : EIS:42 (local to host EIS)
Creation Time : Sun Apr 23 16:44:59 2017
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
Array Size : 23441565696 (22355.62 GiB 24004.16 GB)
Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=176 sectors
State : clean
Device UUID : acd8d9fd:7b7cf9a0:f63369d1:907ffa66
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Apr 23 16:44:59 2017
Bad Block Log : 512 entries available at offset 32 sectors
Checksum : f89bdc5 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
After/During grow:
/dev/loop0:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 4611f41b:0464e815:8b6f9cfe:b29c56fd
Name : EIS:42 (local to host EIS)
Creation Time : Sun Apr 23 16:44:59 2017
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 11720783024 (5588.90 GiB 6001.04 GB)
Array Size : 29301957120 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720782848 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
| New Offset : 257024 sectors
Super Offset : 8 sectors
State : clean
Device UUID : acd8d9fd:7b7cf9a0:f63369d1:907ffa66
Internal Bitmap : 8 sectors from superblock
| Reshape pos'n : 1472000 (1437.50 MiB 1507.33 MB)
| Delta Devices : 1 (5->6)
Update Time : Sun Apr 23 16:45:38 2017
Bad Block Log : 512 entries available at offset 32 sectors
Checksum : fbd9a55 - correct
Events : 30
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
Basically you have to know the New Offset
(search first 128M of your drives for filesystem headers, that should be it)
and then guess the Reshape pos'n by comparing raw data at offset X
(find non-zero data at identical offsets for both raid sets)
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 14:45 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20170423143640.GA12093@metamorpher.de>
On 04/23/2017 04:36 PM, Andreas Klauer wrote:
> On Sun, Apr 23, 2017 at 03:49:16PM +0200, Patrik Dahlström wrote:
>>> Again, you'd do this with overlays only...
>> I did
>> $ mdadm --create /dev/md1 --assume-clean --data-offset=128M --level=5 --raid-devices=5 /dev/mapper/sd[abdce]
>> $ dmesg | tail
>
> Hi,
>
> the shell globbing style [abdce] actually expands alphabetically,
> you probably have to write {a,b,d,c,e} instead.
I spelled it out for every device but no change in the result I'm
afraid. It still won't mount it, nor will fsck recognize it.
Doesn't the triggered reshape pose an issue here? It went ~16 % through.
// Patrik
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-23 14:36 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: linux-raid
In-Reply-To: <9fcdece2-68b7-8e5b-5995-caf15af18bf3@powerlamerz.org>
On Sun, Apr 23, 2017 at 03:49:16PM +0200, Patrik Dahlström wrote:
> > Again, you'd do this with overlays only...
> I did
> $ mdadm --create /dev/md1 --assume-clean --data-offset=128M --level=5 --raid-devices=5 /dev/mapper/sd[abdce]
> $ dmesg | tail
Hi,
the shell globbing style [abdce] actually expands alphabetically,
you probably have to write {a,b,d,c,e} instead.
$ echo [edcba]
a b c d e
$ echo {e,d,c,b,a}
e d c b a
Use 'dmsetup ls' to make sense of the dm-XX (253:XX) numbers.
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recover array after I panicked
From: Brad Campbell @ 2017-04-23 14:25 UTC (permalink / raw)
To: Patrik Dahlström, linux-raid
In-Reply-To: <760766b7-1801-0b2a-6ef1-2da910d976f0@powerlamerz.org>
On 23/04/17 22:09, Patrik Dahlström wrote:
>
> On 04/23/2017 04:06 PM, Brad Campbell wrote:
>> On 23/04/17 17:47, Patrik Dahlström wrote:
>>> Hello,
>>>
>>> Here's the story:
>>>
>>> I started with a 5x6 TB raid5 array. I added another 6 TB drive and
>>> started to grow the array. However, one of my SATA cables were bad and
>>> the reshape gave me lots of I/O errors.
>>>
>>> Instead of fixing the SATA cable issue directly, I shutdown the server
>>> and swapped places of 2 drives. My reasoning was that putting the new
>>> drive in a good slot would reduce the I/O errors. Bad move, I know. I
>>> tried a few commands but was not able to continue the reshape.
>>>
>> Nobody seems to have mentioned the reshape issue. What sort of reshape
>> were you running? How far into the reshape did it get? Do you have any
>> logs of the errors (which might at least indicate whereabouts in the
>> array things were before you pushed it over the edge)?
> These were the grow commands I ran:
> mdadm --add /dev/md1 /dev/sdf
> mdadm --grow --raid-devices=6 /dev/md1
>
> It got to roughly 15-17 % before I decided that the I/O errors were more
> scary than stopping the reshape.
>
You might be very lucky. If my reading of the code is correct (and my
memory is any good), simply adding a disk to a raid5 on a recent enough
kernel should make the resync go backwards. So it should have started at
the end and worked towards the start. This would mean the majority of
your data should be on your 5 original disks.
If this is the case, then permuting the array with 6 disks is going to
fail always as 1/5th of every stripe will be bogus. Doing it with the
original 5 disks may ultimately yield something in the order of 85% of
your data if your estimate of 15-17% is correct.
No harm testing it with some overlays anyway. Someone more familiar with
the code will correct me if I'm wrong.
Regards,
Brad
--
Dolphins are so intelligent that within a few weeks they can
train Americans to stand at the edge of the pool and throw them
fish.
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 14:20 UTC (permalink / raw)
To: Brad Campbell, linux-raid
In-Reply-To: <760766b7-1801-0b2a-6ef1-2da910d976f0@powerlamerz.org>
On 04/23/2017 04:09 PM, Patrik Dahlström wrote:
>
>
> On 04/23/2017 04:06 PM, Brad Campbell wrote:
>> On 23/04/17 17:47, Patrik Dahlström wrote:
>>> Hello,
>>>
>>> Here's the story:
>>>
>>> I started with a 5x6 TB raid5 array. I added another 6 TB drive and
>>> started to grow the array. However, one of my SATA cables were bad and
>>> the reshape gave me lots of I/O errors.
>>>
>>> Instead of fixing the SATA cable issue directly, I shutdown the server
>>> and swapped places of 2 drives. My reasoning was that putting the new
>>> drive in a good slot would reduce the I/O errors. Bad move, I know. I
>>> tried a few commands but was not able to continue the reshape.
>>>
>>
>> Nobody seems to have mentioned the reshape issue. What sort of reshape
>> were you running? How far into the reshape did it get? Do you have any
>> logs of the errors (which might at least indicate whereabouts in the
>> array things were before you pushed it over the edge)?
> These were the grow commands I ran:
> mdadm --add /dev/md1 /dev/sdf
> mdadm --grow --raid-devices=6 /dev/md1
>
I found the kernel log output from when I ran the command:
[ 1912.303661] md: bind<sdf>
[ 1912.355423] RAID conf printout:
[ 1912.355426] --- level:5 rd:5 wd:5
[ 1912.355428] disk 0, o:1, dev:sda
[ 1912.355429] disk 1, o:1, dev:sdb
[ 1912.355430] disk 2, o:1, dev:sdd
[ 1912.355431] disk 3, o:1, dev:sdc
[ 1912.355432] disk 4, o:1, dev:sde
[ 1937.287333] RAID conf printout:
[ 1937.287341] --- level:5 rd:6 wd:6
[ 1937.287347] disk 0, o:1, dev:sda
[ 1937.287351] disk 1, o:1, dev:sdb
[ 1937.287355] disk 2, o:1, dev:sdd
[ 1937.287358] disk 3, o:1, dev:sdc
[ 1937.287361] disk 4, o:1, dev:sde
[ 1937.287365] disk 5, o:1, dev:sdf
[ 1937.287469] md: reshape of RAID array md1
[ 1937.287475] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 1937.287478] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
[ 1937.287487] md: using 128k window, over a total of 5860391424k.
[ 1937.424014] ata6.00: exception Emask 0x10 SAct 0x20000 SErr 0x480100 action 0x6 frozen
[ 1937.424086] ata6.00: irq_stat 0x08000000, interface fatal error
[ 1937.424134] ata6: SError: { UnrecovData 10B8B Handshk }
[ 1937.424179] ata6.00: failed command: WRITE FPDMA QUEUED
[ 1937.424227] ata6.00: cmd 61/40:88:00:dc:03/01:00:00:00:00/40 tag 17 ncq 163840 out
[ 1937.424227] res 40/00:88:00:dc:03/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
[ 1937.424341] ata6.00: status: { DRDY }
[ 1937.424375] ata6: hard resetting link
[ 1937.743934] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 1937.745491] ata6.00: configured for UDMA/133
[ 1937.745498] ata6: EH complete
[ 1937.751920] ata6.00: exception Emask 0x10 SAct 0xc00000 SErr 0x400100 action 0x6 frozen
[ 1937.751948] ata6.00: irq_stat 0x08000000, interface fatal error
[ 1937.751966] ata6: SError: { UnrecovData Handshk }
[ 1937.751982] ata6.00: failed command: WRITE FPDMA QUEUED
[ 1937.751999] ata6.00: cmd 61/b8:b0:80:e2:03/02:00:00:00:00/40 tag 22 ncq 356352 out
[ 1937.751999] res 40/00:b8:40:dd:03/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
[ 1937.752042] ata6.00: status: { DRDY }
[ 1937.752053] ata6.00: failed command: WRITE FPDMA QUEUED
[ 1937.752070] ata6.00: cmd 61/40:b8:40:dd:03/05:00:00:00:00/40 tag 23 ncq 688128 out
[ 1937.752070] res 40/00:b8:40:dd:03/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
[ 1937.752113] ata6.00: status: { DRDY }
[ 1937.752125] ata6: hard resetting link
[ 1938.072176] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 1938.074013] ata6.00: configured for UDMA/133
[ 1938.074036] ata6: EH complete
etc.
The rest is lots and lots of I/O errors due to bad SATA cable.
> It got to roughly 15-17 % before I decided that the I/O errors were more
> scary than stopping the reshape.
>>
>>
>> What you'll have is one part of the array in one configuration, the
>> remaining part in another and no record of where that split begins.
> Like I said, ~15-17 % into the reshape.
>>
>> Regards,
>> Brad
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 14:09 UTC (permalink / raw)
To: Brad Campbell, linux-raid
In-Reply-To: <807de641-043c-41a0-cffe-e28710503aba@fnarfbargle.com>
On 04/23/2017 04:06 PM, Brad Campbell wrote:
> On 23/04/17 17:47, Patrik Dahlström wrote:
>> Hello,
>>
>> Here's the story:
>>
>> I started with a 5x6 TB raid5 array. I added another 6 TB drive and
>> started to grow the array. However, one of my SATA cables were bad and
>> the reshape gave me lots of I/O errors.
>>
>> Instead of fixing the SATA cable issue directly, I shutdown the server
>> and swapped places of 2 drives. My reasoning was that putting the new
>> drive in a good slot would reduce the I/O errors. Bad move, I know. I
>> tried a few commands but was not able to continue the reshape.
>>
>
> Nobody seems to have mentioned the reshape issue. What sort of reshape
> were you running? How far into the reshape did it get? Do you have any
> logs of the errors (which might at least indicate whereabouts in the
> array things were before you pushed it over the edge)?
These were the grow commands I ran:
mdadm --add /dev/md1 /dev/sdf
mdadm --grow --raid-devices=6 /dev/md1
It got to roughly 15-17 % before I decided that the I/O errors were more
scary than stopping the reshape.
>
>
> What you'll have is one part of the array in one configuration, the
> remaining part in another and no record of where that split begins.
Like I said, ~15-17 % into the reshape.
>
> Regards,
> Brad
^ permalink raw reply
* Re: Recover array after I panicked
From: Brad Campbell @ 2017-04-23 14:06 UTC (permalink / raw)
To: Patrik Dahlström, linux-raid
In-Reply-To: <3957da08-6ff4-3c15-e499-157244a767aa@powerlamerz.org>
On 23/04/17 17:47, Patrik Dahlström wrote:
> Hello,
>
> Here's the story:
>
> I started with a 5x6 TB raid5 array. I added another 6 TB drive and
> started to grow the array. However, one of my SATA cables were bad and
> the reshape gave me lots of I/O errors.
>
> Instead of fixing the SATA cable issue directly, I shutdown the server
> and swapped places of 2 drives. My reasoning was that putting the new
> drive in a good slot would reduce the I/O errors. Bad move, I know. I
> tried a few commands but was not able to continue the reshape.
>
Nobody seems to have mentioned the reshape issue. What sort of reshape
were you running? How far into the reshape did it get? Do you have any
logs of the errors (which might at least indicate whereabouts in the
array things were before you pushed it over the edge)?
What you'll have is one part of the array in one configuration, the
remaining part in another and no record of where that split begins.
Regards,
Brad
--
Dolphins are so intelligent that within a few weeks they can
train Americans to stand at the edge of the pool and throw them
fish.
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 13:49 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20170423131605.GA9955@metamorpher.de>
On 04/23/2017 03:16 PM, Andreas Klauer wrote:
> On Sun, Apr 23, 2017 at 01:12:54PM +0200, Patrik Dahlström wrote:
>> I got some of that!
>
>> [ 3.100700] RAID conf printout:
>> [ 3.100700] --- level:5 rd:5 wd:5
>> [ 3.100700] disk 0, o:1, dev:sda
>> [ 3.100700] disk 1, o:1, dev:sdb
>> [ 3.100701] disk 2, o:1, dev:sdd
>> [ 3.100701] disk 3, o:1, dev:sdc
>> [ 3.100701] disk 4, o:1, dev:sde
>> [ 3.101006] created bitmap (44 pages) for device md1
>> [ 3.102245] md1: bitmap initialized from disk: read 3 pages, set 0 of
>> 89423 bits
>> [ 3.159019] md1: detected capacity change from 0 to 24004163272704
>
> Fairly standard, RAID5, presumably 1.2 metadata with 128M data offset,
> which is the default mdadm uses lately. Older RAIDs would have smaller
> data offsets.
>
> So... ...the output above really is from before any of your accidents?
Yes, it is from before adding /dev/sdf and starting a reshape
> How old is your raid ...?
The raid is roughly 1 year old. It started as a combination of raids:
md0: 4x2TB raid5
md1: 2x6TB + md0 raid5
A few months after that, md0 was replaced with a 6 TB drive (/dev/sdd).
Last august I added /dev/sde and this january I added /dev/sde.
Yesterday I tried to add /dev/sdf.
>
> Tested with loop devices:
>
> # truncate -s 6001175126016 0 1 2 3 4
> # losetup --find --show
> # mdadm --create /dev/md42 --assume-clean --data-offset=128M --level=5 --raid-devices=5 /dev/loop[01234]
>
> | [14580.373999] md/raid:md42: device loop4 operational as raid disk 4
> | [14580.373999] md/raid:md42: device loop3 operational as raid disk 3
> | [14580.374000] md/raid:md42: device loop2 operational as raid disk 2
> | [14580.374000] md/raid:md42: device loop1 operational as raid disk 1
> | [14580.374001] md/raid:md42: device loop0 operational as raid disk 0
> | [14580.374308] md/raid:md42: raid level 5 active with 5 out of 5 devices, algorithm 2
> | [14580.377043] md42: detected capacity change from 0 to 24004163272704
>
> (Results in identical capacity as yours so it's the most likely match.)
>
> Again, you'd do this with overlays only...
I did
$ mdadm --create /dev/md1 --assume-clean --data-offset=128M --level=5 --raid-devices=5 /dev/mapper/sd[abdce]
$ dmesg | tail
[10079.442770] md: bind<dm-2>
[10079.442835] md: bind<dm-5>
[10079.442889] md: bind<dm-1>
[10079.442954] md: bind<dm-3>
[10079.443015] md: bind<dm-4>
[10079.443814] md/raid:md1: device dm-4 operational as raid disk 4
[10079.443815] md/raid:md1: device dm-3 operational as raid disk 3
[10079.443816] md/raid:md1: device dm-1 operational as raid disk 2
[10079.443830] md/raid:md1: device dm-5 operational as raid disk 1
[10079.443830] md/raid:md1: device dm-2 operational as raid disk 0
[10079.444123] md/raid:md1: allocated 5432kB
[10079.444168] md/raid:md1: raid level 5 active with 5 out of 5 devices, algorithm 2
[10079.444169] RAID conf printout:
[10079.444170] --- level:5 rd:5 wd:5
[10079.444171] disk 0, o:1, dev:dm-2
[10079.444171] disk 1, o:1, dev:dm-5
[10079.444172] disk 2, o:1, dev:dm-1
[10079.444173] disk 3, o:1, dev:dm-3
[10079.444173] disk 4, o:1, dev:dm-4
[10079.444237] created bitmap (44 pages) for device md1
[10079.446272] md1: bitmap initialized from disk: read 3 pages, set 89423 of 89423 bits
[10079.451821] md1: detected capacity change from 0 to 24004163272704
$ mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Apr 23 15:40:15 2017
Raid Level : raid5
Array Size : 23441565696 (22355.62 GiB 24004.16 GB)
Used Dev Size : 5860391424 (5588.90 GiB 6001.04 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Apr 23 15:40:15 2017
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : rack-server-1:1 (local to host rack-server-1)
UUID : 6beee843:59371bd6:c9278c83:1eb89111
Events : 0
Number Major Minor RaidDevice State
0 252 2 0 active sync /dev/dm-2
1 252 5 1 active sync /dev/dm-5
2 252 1 2 active sync /dev/dm-1
3 252 3 3 active sync /dev/dm-3
4 252 4 4 active sync /dev/dm-4
$ mount /dev/md1 /storage
mount: wrong fs type, bad option, bad superblock on /dev/md1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
Still no luck. Was the drives added in the wrong order?
>
> Regards
> Andreas Klauer
>
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-23 13:16 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: linux-raid
In-Reply-To: <754209bb-9990-2a36-2233-d7d1273c8e37@powerlamerz.org>
On Sun, Apr 23, 2017 at 01:12:54PM +0200, Patrik Dahlström wrote:
> I got some of that!
> [ 3.100700] RAID conf printout:
> [ 3.100700] --- level:5 rd:5 wd:5
> [ 3.100700] disk 0, o:1, dev:sda
> [ 3.100700] disk 1, o:1, dev:sdb
> [ 3.100701] disk 2, o:1, dev:sdd
> [ 3.100701] disk 3, o:1, dev:sdc
> [ 3.100701] disk 4, o:1, dev:sde
> [ 3.101006] created bitmap (44 pages) for device md1
> [ 3.102245] md1: bitmap initialized from disk: read 3 pages, set 0 of
> 89423 bits
> [ 3.159019] md1: detected capacity change from 0 to 24004163272704
Fairly standard, RAID5, presumably 1.2 metadata with 128M data offset,
which is the default mdadm uses lately. Older RAIDs would have smaller
data offsets.
So... ...the output above really is from before any of your accidents?
How old is your raid ...?
Tested with loop devices:
# truncate -s 6001175126016 0 1 2 3 4
# losetup --find --show
# mdadm --create /dev/md42 --assume-clean --data-offset=128M --level=5 --raid-devices=5 /dev/loop[01234]
| [14580.373999] md/raid:md42: device loop4 operational as raid disk 4
| [14580.373999] md/raid:md42: device loop3 operational as raid disk 3
| [14580.374000] md/raid:md42: device loop2 operational as raid disk 2
| [14580.374000] md/raid:md42: device loop1 operational as raid disk 1
| [14580.374001] md/raid:md42: device loop0 operational as raid disk 0
| [14580.374308] md/raid:md42: raid level 5 active with 5 out of 5 devices, algorithm 2
| [14580.377043] md42: detected capacity change from 0 to 24004163272704
(Results in identical capacity as yours so it's the most likely match.)
Again, you'd do this with overlays only...
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 12:57 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <20170423124514.GA7049@metamorpher.de>
On 04/23/2017 02:45 PM, Andreas Klauer wrote:
> On Sun, Apr 23, 2017 at 02:32:07PM +0200, Patrik Dahlström wrote:
>> Any thoughts?
>
> What's the exact size of your drives?
>
> blockdev --getsize64 /dev/sd[abcdefg]
>
$ blockdev --getsize64 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg
6001175126016
6201213935616
6001175126016
6001175126016
6001175126016
6001175126016
2000398934016
/dev/sdg is the OS disk running from USB cabinet.
I have a faint memory of not using the full drives when creating the
array. Maybe save ~%5 in case I get a replacement that is slightly
smaller than the drives I have today. I could probably check for zeros
at the end of a drive
^ permalink raw reply
* Re: Recover array after I panicked
From: Andreas Klauer @ 2017-04-23 12:45 UTC (permalink / raw)
To: Patrik Dahlström; +Cc: linux-raid
In-Reply-To: <645f80c8-9abe-e34e-91f0-950b9e852068@powerlamerz.org>
On Sun, Apr 23, 2017 at 02:32:07PM +0200, Patrik Dahlström wrote:
> Any thoughts?
What's the exact size of your drives?
blockdev --getsize64 /dev/sd[abcdefg]
^ permalink raw reply
* Re: Recover array after I panicked
From: Patrik Dahlström @ 2017-04-23 12:32 UTC (permalink / raw)
To: Andreas Klauer; +Cc: linux-raid
In-Reply-To: <37269c2b-1788-a0b6-6d91-84c6b6bdd16c@powerlamerz.org>
On 04/23/2017 12:23 PM, Patrik Dahlström wrote:
> On 04/23/2017 12:16 PM, Andreas Klauer wrote:
>> On Sun, Apr 23, 2017 at 11:47:34AM +0200, Patrik Dahlström wrote:
>>> Is there any help you can offer?
>>
>> Is there any mdadm --examine output?
> At this point, it is incorrect. I've lost the output from the working
> raid too, unless it's located in any log in /var/log/.
> I have /etc/mdadm/mdadm.conf, but don't know if it's updated.
>
>>
>> What was on the array? Regular filesystem, unencrypted, or LVM, LUKS, ...?
> Regular filesystem, unencrypted ext4.
>
>>
>> If it's LUKS encrypted and you had RAID metadata at the end, yet
>> mdadm --create'd new metadata at the start, that would likely have
>> damaged your LUKS header beyond repair (and regular filesystems
>> don't like it, either).
> No file system encryption.
>
>>
>> If it's unencrypted data, as a last resort you can always go and find
>> the header of a large file of known type... for example if you find
>> a megapixel JPEG image and the first 512K of it are part of that then
>> your chunksize would be 512K and then you can go looking for the
>> next chunk on the other disks... and that should give you some notion
>> of the RAID layout and offset.
> That's not a bad idea. Will hopefully narrow down my unknown variables.
Okay, I extracted parts of an mkv file and this is what I found out:
* playing 512 kb of data is OK
* playing 1024 kb of data will give me the following error (from mpv):
[mkv] Invalid EBML length at position 539473
[mkv] Corrupt file detected. Trying to resync starting from position
539473...
"position 539473" is at ~527 kb, which leads me to suspect that the
correct chunk size is 512 kb.
Any thoughts?
>
>>
>> Regards
>> Andreas Klauer
>>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox