* Re: block: fix blk_queue_split() resource exhaustion
From: Mike Snitzer @ 2016-06-24 15:15 UTC (permalink / raw)
To: Lars Ellenberg
Cc: Ming Lei, linux-block, Roland Kammerer, Jens Axboe, NeilBrown,
Kent Overstreet, Shaohua Li, Alasdair Kergon,
open list:DEVICE-MAPPER (LVM), Ingo Molnar, Peter Zijlstra,
Takashi Iwai, Jiri Kosina, Zheng Liu, Keith Busch,
Martin K. Petersen, Kirill A. Shutemov, Linux Kernel Mailing List,
open list:BCACHE (BLOCK LAYER CACHE),
open list:SOFTWARE RAID (Multiple Disks) SUPPORT
In-Reply-To: <20160624142711.GF3239@soda.linbit>
On Fri, Jun 24 2016 at 10:27am -0400,
Lars Ellenberg <lars.ellenberg@linbit.com> wrote:
> On Fri, Jun 24, 2016 at 07:36:57PM +0800, Ming Lei wrote:
> > >
> > > This is not a theoretical problem.
> > > At least int DRBD, and an unfortunately high IO concurrency wrt. the
> > > "max-buffers" setting, without this patch we have a reproducible deadlock.
> >
> > Is there any log about the deadlock? And is there any lockdep warning
> > if it is enabled?
>
> In DRBD, to avoid potentially very long internal queues as we wait for
> our replication peer device and local backend, we limit the number of
> in-flight bios we accept, and block in our ->make_request_fn() if that
> number exceeds a configured watermark ("max-buffers").
>
> Works fine, as long as we could assume that once our make_request_fn()
> returns, any bios we "recursively" submitted against the local backend
> would be dispatched. Which used to be the case.
It'd be useful to know whether this patch fixes your issue:
https://patchwork.kernel.org/patch/7398411/
Ming Lei didn't like it due to concerns about I contexts changing
(whereby breaking merging that occurs via plugging).
But if it _does_ fix your issue then the case for the change is
increased; and we just need to focus on addressing Ming's concerns
(Mikulas has some ideas).
Conversely, and in parallel, Mikulas can look to see if your approach
fixes the observed dm-snapshot deadlock that he set out to fix.
^ permalink raw reply
* [PATCH] dm stripe: add DAX support
From: Toshi Kani @ 2016-06-24 18:23 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, Alasdair Kergon
Change dm-stripe to implement direct_access function,
stripe_direct_access(), which maps bdev and sector and
calls direct_access function of its physical target device.
Signed-off-by: Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org>
Cc: Alasdair Kergon <agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: Dan Williams <dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Cc: Ross Zwisler <ross.zwisler-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
---
drivers/md/dm-stripe.c | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/drivers/md/dm-stripe.c b/drivers/md/dm-stripe.c
index 48f1c01..8925f6a 100644
--- a/drivers/md/dm-stripe.c
+++ b/drivers/md/dm-stripe.c
@@ -308,6 +308,30 @@ static int stripe_map(struct dm_target *ti, struct bio *bio)
return DM_MAPIO_REMAPPED;
}
+static long stripe_direct_access(struct dm_target *ti, sector_t sector,
+ void __pmem **kaddr, pfn_t *pfn, long size)
+{
+ struct stripe_c *sc;
+ struct block_device *bdev;
+ uint32_t stripe;
+ struct blk_dax_ctl dax = {
+ .size = size,
+ };
+ long ret;
+
+ sc = ti->private;
+ stripe_map_sector(sc, sector, &stripe, &dax.sector);
+
+ dax.sector += sc->stripe[stripe].physical_start;
+ bdev = sc->stripe[stripe].dev->bdev;
+
+ ret = bdev_direct_access(bdev, &dax);
+ *kaddr = dax.addr;
+ *pfn = dax.pfn;
+
+ return ret;
+}
+
/*
* Stripe status:
*
@@ -425,6 +449,7 @@ static struct target_type stripe_target = {
.status = stripe_status,
.iterate_devices = stripe_iterate_devices,
.io_hints = stripe_io_hints,
+ .direct_access = stripe_direct_access,
};
int __init dm_stripe_init(void)
^ permalink raw reply related
* Re: dm stripe: add DAX support
From: Mike Snitzer @ 2016-06-24 18:29 UTC (permalink / raw)
To: Toshi Kani
Cc: linux-nvdimm, linux-kernel, linux-raid, dm-devel, Dan Williams,
Ross Zwisler, Alasdair Kergon
In-Reply-To: <1466792610-30369-1-git-send-email-toshi.kani@hpe.com>
On Fri, Jun 24 2016 at 2:23pm -0400,
Toshi Kani <toshi.kani@hpe.com> wrote:
> Change dm-stripe to implement direct_access function,
> stripe_direct_access(), which maps bdev and sector and
> calls direct_access function of its physical target device.
>
> Signed-off-by: Toshi Kani <toshi.kani@hpe.com>
> Cc: Alasdair Kergon <agk@redhat.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
> ---
> drivers/md/dm-stripe.c | 25 +++++++++++++++++++++++++
> 1 file changed, 25 insertions(+)
>
> diff --git a/drivers/md/dm-stripe.c b/drivers/md/dm-stripe.c
> index 48f1c01..8925f6a 100644
> --- a/drivers/md/dm-stripe.c
> +++ b/drivers/md/dm-stripe.c
> @@ -308,6 +308,30 @@ static int stripe_map(struct dm_target *ti, struct bio *bio)
> return DM_MAPIO_REMAPPED;
> }
>
> +static long stripe_direct_access(struct dm_target *ti, sector_t sector,
> + void __pmem **kaddr, pfn_t *pfn, long size)
> +{
> + struct stripe_c *sc;
> + struct block_device *bdev;
> + uint32_t stripe;
> + struct blk_dax_ctl dax = {
> + .size = size,
> + };
> + long ret;
> +
> + sc = ti->private;
> + stripe_map_sector(sc, sector, &stripe, &dax.sector);
> +
> + dax.sector += sc->stripe[stripe].physical_start;
> + bdev = sc->stripe[stripe].dev->bdev;
> +
> + ret = bdev_direct_access(bdev, &dax);
> + *kaddr = dax.addr;
> + *pfn = dax.pfn;
> +
> + return ret;
> +}
> +
> /*
> * Stripe status:
> *
> @@ -425,6 +449,7 @@ static struct target_type stripe_target = {
> .status = stripe_status,
> .iterate_devices = stripe_iterate_devices,
> .io_hints = stripe_io_hints,
> + .direct_access = stripe_direct_access,
> };
>
> int __init dm_stripe_init(void)
Thanks, once jens queues the block changes I'll be able to pull this in
to linux-dm.git
BTW, if in your testing you could evaluate/quantify any extra overhead
from DM that'd be useful to share. It could be there are bottlenecks
that need to be fixed, etc.
Mike
^ permalink raw reply
* Re: dm stripe: add DAX support
From: Kani, Toshimitsu @ 2016-06-24 18:42 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160624182859.GD13898-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
> On Fri, Jun 24 2016 at 2:23pm -0400,
> Toshi Kani <toshi.kani@hpe.com> wrote:
:
> Thanks, once jens queues the block changes I'll be able to pull this in
> to linux-dm.git
>
> BTW, if in your testing you could evaluate/quantify any extra overhead
> from DM that'd be useful to share. It could be there are bottlenecks
> that need to be fixed, etc.
I ran some performance tests as sanity checks (results are in ballpark), but
have not verified them carefully. Yes, I will check to see if there are any
bottlenecks.
Thanks!
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Recommendations needed for RAID5 recovery
From: Peter Gebhard @ 2016-06-24 19:55 UTC (permalink / raw)
To: linux-raid
Hello,
I have been asked to attempt data recovery on a RAID5 array which appears to have had two disk failures (in an array of four disks). I am gratefully hoping that some on this list could offer recommendations for my next steps. I have provided below the current state of the array per https://raid.wiki.kernel.org/index.php/RAID_Recovery.
It appears from the output below that one of the disks (sdd1) failed last year and the admin did not notice this. Now, it appears a second disk (sdg1) has recently had read errors and was kicked out of the array.
Should I try to restore the array using the recreate_array.pl script provided on the RAID_Recovery site? Should I then try to recreate the array and/or perform ‘fsck’?
Thank you greatly in advance!
raid.status:
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
Name : <->:0
Creation Time : Tue Nov 29 17:33:39 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : ec1a6336:0a991298:5b409bf1:4585ccbe
Update Time : Sun Jun 7 02:28:00 2015
Checksum : f9323080 - correct
Events : 96203
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
Name : <->:0
Creation Time : Tue Nov 29 17:33:39 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : ed13b045:ca75ab96:83045f97:e4fd62cb
Update Time : Sun Jun 19 19:43:31 2016
Checksum : bb6a905f - correct
Events : 344993
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : .A.A ('A' == active, '.' == missing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
Name : <->:0
Creation Time : Tue Nov 29 17:33:39 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a1ea11e0:6465fe26:483f133d:680014b3
Update Time : Sun Jun 19 19:43:31 2016
Checksum : 738493f3 - correct
Events : 344993
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : .A.A ('A' == active, '.' == missing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
Name : <->:0
Creation Time : Tue Nov 29 17:33:39 2011
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : active
Device UUID : a9737439:17f81210:484d4f4c:c3d34a8a
Update Time : Sun Jun 19 12:18:49 2016
Checksum : 9c6d24bf - correct
Events : 343949
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : .AAA ('A' == active, '.' == missing)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] dm stripe: add DAX support
From: kbuild test robot @ 2016-06-24 19:58 UTC (permalink / raw)
To: Toshi Kani
Cc: snitzer-H+wXaHxf7aLQT0dZR+AlfA,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, kbuild-all-JC7UmRfGjtg,
Alasdair Kergon
In-Reply-To: <1466792610-30369-1-git-send-email-toshi.kani-ZPxbGqLxI0U@public.gmane.org>
Hi,
[auto build test WARNING on dm/for-next]
[also build test WARNING on v4.7-rc4 next-20160624]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Toshi-Kani/dm-stripe-add-DAX-support/20160625-022600
base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next
config: m68k-sun3_defconfig (attached as .config)
compiler: m68k-linux-gcc (GCC) 4.9.0
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=m68k
All warnings (new ones prefixed by >>):
drivers/md/dm-stripe.c:452:2: error: unknown field 'direct_access' specified in initializer
.direct_access = stripe_direct_access,
^
drivers/md/dm-stripe.c:452:2: warning: missing braces around initializer [-Wmissing-braces]
drivers/md/dm-stripe.c:452:2: warning: (near initialization for 'stripe_target.list') [-Wmissing-braces]
>> drivers/md/dm-stripe.c:452:2: warning: initialization from incompatible pointer type
drivers/md/dm-stripe.c:452:2: warning: (near initialization for 'stripe_target.list.next')
vim +452 drivers/md/dm-stripe.c
436
437 blk_limits_io_min(limits, chunk_size);
438 blk_limits_io_opt(limits, chunk_size * sc->stripes);
439 }
440
441 static struct target_type stripe_target = {
442 .name = "striped",
443 .version = {1, 5, 1},
444 .module = THIS_MODULE,
445 .ctr = stripe_ctr,
446 .dtr = stripe_dtr,
447 .map = stripe_map,
448 .end_io = stripe_end_io,
449 .status = stripe_status,
450 .iterate_devices = stripe_iterate_devices,
451 .io_hints = stripe_io_hints,
> 452 .direct_access = stripe_direct_access,
453 };
454
455 int __init dm_stripe_init(void)
456 {
457 int r;
458
459 r = dm_register_target(&stripe_target);
460 if (r < 0)
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
^ permalink raw reply
* Re: [PATCH] dm stripe: add DAX support
From: kbuild test robot @ 2016-06-24 19:59 UTC (permalink / raw)
To: Toshi Kani
Cc: snitzer-H+wXaHxf7aLQT0dZR+AlfA,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, kbuild-all-JC7UmRfGjtg,
Alasdair Kergon
In-Reply-To: <1466792610-30369-1-git-send-email-toshi.kani-ZPxbGqLxI0U@public.gmane.org>
Hi,
[auto build test ERROR on dm/for-next]
[also build test ERROR on v4.7-rc4 next-20160624]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Toshi-Kani/dm-stripe-add-DAX-support/20160625-022600
base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next
config: x86_64-randconfig-s5-06250328 (attached as .config)
compiler: gcc-6 (Debian 6.1.1-1) 6.1.1 20160430
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
All error/warnings (new ones prefixed by >>):
>> drivers/md/dm-stripe.c:452:2: error: unknown field 'direct_access' specified in initializer
.direct_access = stripe_direct_access,
^
>> drivers/md/dm-stripe.c:452:19: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
.direct_access = stripe_direct_access,
^~~~~~~~~~~~~~~~~~~~
drivers/md/dm-stripe.c:452:19: note: (near initialization for 'stripe_target.list.next')
>> drivers/md/dm-stripe.c:441:43: warning: missing braces around initializer [-Wmissing-braces]
static struct target_type stripe_target = {
^
drivers/md/dm-stripe.c:441:43: note: (near initialization for 'stripe_target')
cc1: some warnings being treated as errors
vim +/direct_access +452 drivers/md/dm-stripe.c
435 unsigned chunk_size = sc->chunk_size << SECTOR_SHIFT;
436
437 blk_limits_io_min(limits, chunk_size);
438 blk_limits_io_opt(limits, chunk_size * sc->stripes);
439 }
440
> 441 static struct target_type stripe_target = {
442 .name = "striped",
443 .version = {1, 5, 1},
444 .module = THIS_MODULE,
445 .ctr = stripe_ctr,
446 .dtr = stripe_dtr,
447 .map = stripe_map,
448 .end_io = stripe_end_io,
449 .status = stripe_status,
450 .iterate_devices = stripe_iterate_devices,
451 .io_hints = stripe_io_hints,
> 452 .direct_access = stripe_direct_access,
453 };
454
455 int __init dm_stripe_init(void)
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
^ permalink raw reply
* Re: dm stripe: add DAX support
From: Mike Snitzer @ 2016-06-24 20:02 UTC (permalink / raw)
To: kbuild test robot
Cc: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, kbuild-all-JC7UmRfGjtg,
Alasdair Kergon
In-Reply-To: <201606250312.OzHHesq4%fengguang.wu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
On Fri, Jun 24 2016 at 3:58pm -0400,
kbuild test robot <lkp-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> Hi,
>
> [auto build test WARNING on dm/for-next]
> [also build test WARNING on v4.7-rc4 next-20160624]
> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
>
> url: https://github.com/0day-ci/linux/commits/Toshi-Kani/dm-stripe-add-DAX-support/20160625-022600
> base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next
> config: m68k-sun3_defconfig (attached as .config)
> compiler: m68k-linux-gcc (GCC) 4.9.0
> reproduce:
> wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
> chmod +x ~/bin/make.cross
> # save the attached .config to linux build tree
> make.cross ARCH=m68k
>
> All warnings (new ones prefixed by >>):
>
> drivers/md/dm-stripe.c:452:2: error: unknown field 'direct_access' specified in initializer
> .direct_access = stripe_direct_access,
> ^
> drivers/md/dm-stripe.c:452:2: warning: missing braces around initializer [-Wmissing-braces]
> drivers/md/dm-stripe.c:452:2: warning: (near initialization for 'stripe_target.list') [-Wmissing-braces]
> >> drivers/md/dm-stripe.c:452:2: warning: initialization from incompatible pointer type
> drivers/md/dm-stripe.c:452:2: warning: (near initialization for 'stripe_target.list.next')
FYI, overzealous kbuild robot build failure here.. I haven't staged the
prereq DM core changes yet because I'm waiting for Jens to take the 2
block changes those depend on.
Mike
^ permalink raw reply
* Re: Recommendations needed for RAID5 recovery
From: Another Sillyname @ 2016-06-24 20:44 UTC (permalink / raw)
To: Linux-RAID
In-Reply-To: <B1389BE8-B7B5-4AB4-BE12-1636D4B91BB5@seas.upenn.edu>
Peter
Before attempting any recovery can I suggest that you get 4 x 2TB
drives and dd the current drives so you have a backup.
Then you can begin to think about performing the raid recovery in the
knowledge you have a fallback position if it blows up.
Regards
Tony
On 24 June 2016 at 20:55, Peter Gebhard <pgeb@seas.upenn.edu> wrote:
> Hello,
>
> I have been asked to attempt data recovery on a RAID5 array which appears to have had two disk failures (in an array of four disks). I am gratefully hoping that some on this list could offer recommendations for my next steps. I have provided below the current state of the array per https://raid.wiki.kernel.org/index.php/RAID_Recovery.
>
> It appears from the output below that one of the disks (sdd1) failed last year and the admin did not notice this. Now, it appears a second disk (sdg1) has recently had read errors and was kicked out of the array.
>
> Should I try to restore the array using the recreate_array.pl script provided on the RAID_Recovery site? Should I then try to recreate the array and/or perform ‘fsck’?
>
> Thank you greatly in advance!
>
> raid.status:
>
> /dev/sdd1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
> Name : <->:0
> Creation Time : Tue Nov 29 17:33:39 2011
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : ec1a6336:0a991298:5b409bf1:4585ccbe
>
> Update Time : Sun Jun 7 02:28:00 2015
> Checksum : f9323080 - correct
> Events : 96203
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAAA ('A' == active, '.' == missing)
>
> /dev/sde1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
> Name : <->:0
> Creation Time : Tue Nov 29 17:33:39 2011
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : ed13b045:ca75ab96:83045f97:e4fd62cb
>
> Update Time : Sun Jun 19 19:43:31 2016
> Checksum : bb6a905f - correct
> Events : 344993
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : .A.A ('A' == active, '.' == missing)
>
> /dev/sdf1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
> Name : <->:0
> Creation Time : Tue Nov 29 17:33:39 2011
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : a1ea11e0:6465fe26:483f133d:680014b3
>
> Update Time : Sun Jun 19 19:43:31 2016
> Checksum : 738493f3 - correct
> Events : 344993
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : .A.A ('A' == active, '.' == missing)
>
> /dev/sdg1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
> Name : <->:0
> Creation Time : Tue Nov 29 17:33:39 2011
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : a9737439:17f81210:484d4f4c:c3d34a8a
>
> Update Time : Sun Jun 19 12:18:49 2016
> Checksum : 9c6d24bf - correct
> Events : 343949
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : .AAA ('A' == active, '.' == missing)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: dm stripe: add DAX support
From: Kani, Toshimitsu @ 2016-06-24 20:46 UTC (permalink / raw)
To: lkp-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org,
snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
kbuild-all-JC7UmRfGjtg@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160624200239.GA15587-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Fri, 2016-06-24 at 16:02 -0400, Mike Snitzer wrote:
> On Fri, Jun 24 2016 at 3:58pm -0400,
> kbuild test robot <lkp@intel.com> wrote:
>
> >
> > Hi,
> >
> > [auto build test WARNING on dm/for-next]
> > [also build test WARNING on v4.7-rc4 next-20160624]
> > [if your patch is applied to the wrong git tree, please drop us a note to
> > help improve the system]
> >
> > url: https://github.com/0day-ci/linux/commits/Toshi-Kani/dm-stripe-add-
> > DAX-support/20160625-022600
> > base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linu
> > x-dm.git for-next
> > config: m68k-sun3_defconfig (attached as .config)
> > compiler: m68k-linux-gcc (GCC) 4.9.0
> > reproduce:
> > wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.gi
> > t/plain/sbin/make.cross -O ~/bin/make.cross
> > chmod +x ~/bin/make.cross
> > # save the attached .config to linux build tree
> > make.cross ARCH=m68k
> >
> > All warnings (new ones prefixed by >>):
> >
> > drivers/md/dm-stripe.c:452:2: error: unknown field 'direct_access'
> > specified in initializer
> > .direct_access = stripe_direct_access,
> > ^
> > drivers/md/dm-stripe.c:452:2: warning: missing braces around
> > initializer [-Wmissing-braces]
> > drivers/md/dm-stripe.c:452:2: warning: (near initialization for
> > 'stripe_target.list') [-Wmissing-braces]
> > >
> > > >
> > > > drivers/md/dm-stripe.c:452:2: warning: initialization from
> > > > incompatible pointer type
> > drivers/md/dm-stripe.c:452:2: warning: (near initialization for
> > 'stripe_target.list.next')
>
> FYI, overzealous kbuild robot build failure here.. I haven't staged the
> prereq DM core changes yet because I'm waiting for Jens to take the 2
> block changes those depend on.
This patch applies on top of wip. Could this be because 'for-next' does not
have the previous dm/dm-linear series?
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: Recommendations needed for RAID5 recovery
From: John Stoffel @ 2016-06-24 21:37 UTC (permalink / raw)
To: Another Sillyname; +Cc: Linux-RAID
In-Reply-To: <CAOS+5GGxVOWw1nEfEgpVcLY0+tHPVV3d48BzyRATjpApgvSkxg@mail.gmail.com>
Another> Before attempting any recovery can I suggest that you get 4 x
Another> 2TB drives and dd the current drives so you have a backup.
Not dd, dd_rescue instead. But yes, try to get new hardware and clone
all the suspect drives before you do anything else. Even just cloning
the most recently bad drive might be enough to get you going again.
John
Another> Then you can begin to think about performing the raid recovery in the
Another> knowledge you have a fallback position if it blows up.
Another> Regards
Another> Tony
Another> On 24 June 2016 at 20:55, Peter Gebhard <pgeb@seas.upenn.edu> wrote:
>> Hello,
>>
>> I have been asked to attempt data recovery on a RAID5 array which appears to have had two disk failures (in an array of four disks). I am gratefully hoping that some on this list could offer recommendations for my next steps. I have provided below the current state of the array per https://raid.wiki.kernel.org/index.php/RAID_Recovery.
>>
>> It appears from the output below that one of the disks (sdd1) failed last year and the admin did not notice this. Now, it appears a second disk (sdg1) has recently had read errors and was kicked out of the array.
>>
>> Should I try to restore the array using the recreate_array.pl script provided on the RAID_Recovery site? Should I then try to recreate the array and/or perform ‘fsck’?
>>
>> Thank you greatly in advance!
>>
>> raid.status:
>>
>> /dev/sdd1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
>> Name : <->:0
>> Creation Time : Tue Nov 29 17:33:39 2011
>> Raid Level : raid5
>> Raid Devices : 4
>>
>> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
>> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
>> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : ec1a6336:0a991298:5b409bf1:4585ccbe
>>
>> Update Time : Sun Jun 7 02:28:00 2015
>> Checksum : f9323080 - correct
>> Events : 96203
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 0
>> Array State : AAAA ('A' == active, '.' == missing)
>>
>> /dev/sde1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
>> Name : <->:0
>> Creation Time : Tue Nov 29 17:33:39 2011
>> Raid Level : raid5
>> Raid Devices : 4
>>
>> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
>> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
>> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : ed13b045:ca75ab96:83045f97:e4fd62cb
>>
>> Update Time : Sun Jun 19 19:43:31 2016
>> Checksum : bb6a905f - correct
>> Events : 344993
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 3
>> Array State : .A.A ('A' == active, '.' == missing)
>>
>> /dev/sdf1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
>> Name : <->:0
>> Creation Time : Tue Nov 29 17:33:39 2011
>> Raid Level : raid5
>> Raid Devices : 4
>>
>> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
>> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
>> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : clean
>> Device UUID : a1ea11e0:6465fe26:483f133d:680014b3
>>
>> Update Time : Sun Jun 19 19:43:31 2016
>> Checksum : 738493f3 - correct
>> Events : 344993
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : .A.A ('A' == active, '.' == missing)
>>
>> /dev/sdg1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
>> Name : <->:0
>> Creation Time : Tue Nov 29 17:33:39 2011
>> Raid Level : raid5
>> Raid Devices : 4
>>
>> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
>> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
>> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> State : active
>> Device UUID : a9737439:17f81210:484d4f4c:c3d34a8a
>>
>> Update Time : Sun Jun 19 12:18:49 2016
>> Checksum : 9c6d24bf - correct
>> Events : 343949
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Device Role : Active device 2
>> Array State : .AAA ('A' == active, '.' == missing)
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
Another> --
Another> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
Another> the body of a message to majordomo@vger.kernel.org
Another> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [RFC] block: fix blk_queue_split() resource exhaustion
From: Ming Lei @ 2016-06-25 9:30 UTC (permalink / raw)
To: Lars Ellenberg
Cc: linux-block, Roland Kammerer, Jens Axboe, NeilBrown,
Kent Overstreet, Shaohua Li, Alasdair Kergon, Mike Snitzer,
open list:DEVICE-MAPPER (LVM), Ingo Molnar, Peter Zijlstra,
Takashi Iwai, Jiri Kosina, Zheng Liu, Keith Busch,
Martin K. Petersen, Kirill A. Shutemov, Linux Kernel Mailing List,
open list:BCACHE (BLOCK LAYER CACHE),
open list:SOFTWARE RAID (Multiple Disks) SUPPORT
In-Reply-To: <20160624142711.GF3239@soda.linbit>
On Fri, Jun 24, 2016 at 10:27 PM, Lars Ellenberg
<lars.ellenberg@linbit.com> wrote:
> On Fri, Jun 24, 2016 at 07:36:57PM +0800, Ming Lei wrote:
>> >
>> > This is not a theoretical problem.
>> > At least int DRBD, and an unfortunately high IO concurrency wrt. the
>> > "max-buffers" setting, without this patch we have a reproducible deadlock.
>>
>> Is there any log about the deadlock? And is there any lockdep warning
>> if it is enabled?
>
> In DRBD, to avoid potentially very long internal queues as we wait for
> our replication peer device and local backend, we limit the number of
> in-flight bios we accept, and block in our ->make_request_fn() if that
> number exceeds a configured watermark ("max-buffers").
>
> Works fine, as long as we could assume that once our make_request_fn()
> returns, any bios we "recursively" submitted against the local backend
> would be dispatched. Which used to be the case.
>
> commit 54efd50 block: make generic_make_request handle arbitrarily sized bios
> introduced blk_queue_split(), which will split any bio that is
> violating the queue limits into a smaller piece to be processed
> right away, and queue the "rest" on current->bio_list to be
> processed by the iteration in generic_make_request() once the
> current ->make_request_fn() returns.
>
> Before that, any user was supposed to go through bio_add_page(),
> which would call our merge bvec function, and that should already
> be sufficient to not violate queue limits.
>
> Previously, typically the next in line bio to be processed would
> be the cloned one we passed down to our backend device, which in
> case of further stacking devices (e.g. logical volume, raid1 or
> similar) may again push further bios on that list.
> All of which would simply be processed in turn.
Could you clarify if the issue is triggered in drbd without dm/md involved?
Or the issue is always triggered with dm/md over drbd?
As Mike mentioned, there is one known issue with dm-snapshot.
>
> Now, with blk_queue_split(), the next-in-line bio would be the
> next piece of the "too big" original bio, so we end up calling
> several times into our ->make_request_fn() without even
> dispatching one bio to the backend.
If your ->make_request_fn() is called several times, each time
at least you should dispatch one bio returnd from blk_queue_split(),
so I don't understand why even one bio isn't dispatched out.
>
> With highly concurrent IO and/or very small "max-buffers" setting,
> this can deadlock, where the current submit path would wait for
> the completion of the bios supposedly already passed to the
> backend, but which actually are not even dispatched yet, rotting
> away on current->bio_list.
Suppose your ->make_request_fn only handles the bio returned
from blk_queue_split(), I don't understand why your driver waits
for the 'rest' bios from the blk_queue_split().
Maybe drbd driver calls generic_make_request() in
->make_request_fn() and supposes the bio is always dispatched
out once generic_make_request() returns, is it right?
>
> I could code around that in various ways inside the DRBD make_request_fn,
> or always dispatch the backend bios to a different (thread or work_queue)
> context, but I'd rather have the distinction between "recursively
> submitted" bios and "pushed back part of originally passed in bio" as
> implemented in this patch.
>
> Thanks,
>
> Lars
>
>> > I'm unsure if it could cause problems in md raid in some corner cases
>> > or when a rebuild/scrub/... starts in a "bad" moment.
>> >
>> > It also may increase "stress" on any involved bio_set,
>> > because it may increase the number of bios that are allocated
>> > before the first of them is actually processed, causing more
>> > frequent calls of punt_bios_to_rescuer().
>> >
>> > Alternatively,
>> > a simple one-line change to generic_make_request() would also "fix" it,
>> > by processing all recursive bios in LIFO order.
>> > But that may change the order in which bios reach the "physical" layer,
>> > in case they are in fact split into many smaller pieces, for example in
>> > a striping setup, which may have other performance implications.
>> >
>> > | diff --git a/block/blk-core.c b/block/blk-core.c
>> > | index 2475b1c7..a5623f6 100644
>> > | --- a/block/blk-core.c
>> > | +++ b/block/blk-core.c
>> > | @@ -2048,7 +2048,7 @@ blk_qc_t generic_make_request(struct bio *bio)
>> > | * should be added at the tail
>> > | */
>> > | if (current->bio_list) {
>> > | - bio_list_add(current->bio_list, bio);
>> > | + bio_list_add_head(current->bio_list, bio);
>> > | goto out;
>> > | }
>
>> > diff --git a/block/blk-core.c b/block/blk-core.c
>> > index 2475b1c7..f03ff4c 100644
>> > --- a/block/blk-core.c
>> > +++ b/block/blk-core.c
>> > @@ -2031,7 +2031,7 @@ end_io:
>> > */
>> > blk_qc_t generic_make_request(struct bio *bio)
>> > {
>> > - struct bio_list bio_list_on_stack;
>> > + struct recursion_to_iteration_bio_lists bio_lists_on_stack;
>> > blk_qc_t ret = BLK_QC_T_NONE;
>> >
>> > if (!generic_make_request_checks(bio))
>> > @@ -2040,15 +2040,18 @@ blk_qc_t generic_make_request(struct bio *bio)
>
>> > - if (current->bio_list) {
>> > - bio_list_add(current->bio_list, bio);
>> > + if (current->bio_lists) {
>> > + bio_list_add(¤t->bio_lists->recursion, bio);
>> > goto out;
>> > }
>> >
>
>> > @@ -2067,8 +2070,9 @@ blk_qc_t generic_make_request(struct bio *bio)
>> > * bio_list, and call into ->make_request() again.
>> > */
>> > BUG_ON(bio->bi_next);
>> > - bio_list_init(&bio_list_on_stack);
>> > - current->bio_list = &bio_list_on_stack;
>> > + bio_list_init(&bio_lists_on_stack.recursion);
>> > + bio_list_init(&bio_lists_on_stack.remainder);
>> > + current->bio_lists = &bio_lists_on_stack;
>> > do {
>> > struct request_queue *q = bdev_get_queue(bio->bi_bdev);
>> >
>> > @@ -2076,16 +2080,14 @@ blk_qc_t generic_make_request(struct bio *bio)
>> > ret = q->make_request_fn(q, bio);
>> >
>> > blk_queue_exit(q);
>> > -
>> > - bio = bio_list_pop(current->bio_list);
>> > } else {
>> > - struct bio *bio_next = bio_list_pop(current->bio_list);
>> > -
>> > bio_io_error(bio);
>> > - bio = bio_next;
>> > }
>> > + bio = bio_list_pop(¤t->bio_lists->recursion);
>> > + if (!bio)
>> > + bio = bio_list_pop(¤t->bio_lists->remainder);
>> > } while (bio);
>> > - current->bio_list = NULL; /* deactivate */
>> > + current->bio_lists = NULL; /* deactivate */
>> >
>> > out:
>> > return ret;
>> > diff --git a/block/blk-merge.c b/block/blk-merge.c
>> > index 2613531..8da0c22 100644
>> > --- a/block/blk-merge.c
>> > +++ b/block/blk-merge.c
>> > @@ -172,6 +172,7 @@ void blk_queue_split(struct request_queue *q, struct bio **bio,
>> > struct bio *split, *res;
>> > unsigned nsegs;
>> >
>> > + BUG_ON(!current->bio_lists);
>> > if ((*bio)->bi_rw & REQ_DISCARD)
>> > split = blk_bio_discard_split(q, *bio, bs, &nsegs);
>> > else if ((*bio)->bi_rw & REQ_WRITE_SAME)
>> > @@ -190,7 +191,7 @@ void blk_queue_split(struct request_queue *q, struct bio **bio,
>> >
>> > bio_chain(split, *bio);
>> > trace_block_split(q, split, (*bio)->bi_iter.bi_sector);
>> > - generic_make_request(*bio);
>> > + bio_list_add_head(¤t->bio_lists->remainder, *bio);
>> > *bio = split;
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-block" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Recommendations needed for RAID5 recovery
From: Wols Lists @ 2016-06-25 11:43 UTC (permalink / raw)
Cc: Linux-RAID
In-Reply-To: <22381.43040.31844.544728@quad.stoffel.home>
On 24/06/16 22:37, John Stoffel wrote:
>
> Another> Before attempting any recovery can I suggest that you get 4 x
> Another> 2TB drives and dd the current drives so you have a backup.
>
> Not dd, dd_rescue instead. But yes, try to get new hardware and clone
> all the suspect drives before you do anything else. Even just cloning
> the most recently bad drive might be enough to get you going again.
As I got told rather sharply :-) there's a big difference between dd and
ddrescue.
IFF dd completes successfully (it'll bomb on an error) then you have a
"known good" copy. In other words the problem with dd is it won't work
on a bad drive, but if it does work you're home and dry.
ddrescue will ALWAYS work - but if it can't read a block it will leave
an empty block in the copy! This is a bomb waiting to go off! In other
words, ddrescue is great at recovering what you can from a damaged
filesystem - less so at recovering a disk with a complicated setup on top.
I know you're getting conflicting advice, but I'd try to get a good dd
backup first. I don't know of any utility that will do an md integrity
check on a ddrescue'd disk :-( so you'd need to do a fsck and hope ...
Oh - and make sure you new disks are proper raid - eg WD Red or Seagate
NAS. And are your current disks proper raid? If not, fix the timeout
problem and your life *may* be made a lot simpler ...
Have you got spare SATA ports? If not, go out and get an add-in card! If
you can force the array to assemble, and create a temporary six-drive
array (the two dud ones being assembled with the --replace option to
move them to two new ones) that may be your best bet at recovery. If md
can get at a clean read from three drives for each block, then it'll be
able to rebuild the missing block.
Cheers,
Wol
>
> John
>
>
>
> Another> Then you can begin to think about performing the raid recovery in the
> Another> knowledge you have a fallback position if it blows up.
>
> Another> Regards
>
> Another> Tony
>
> Another> On 24 June 2016 at 20:55, Peter Gebhard <pgeb@seas.upenn.edu> wrote:
>>> Hello,
>>>
>>> I have been asked to attempt data recovery on a RAID5 array which appears to have had two disk failures (in an array of four disks). I am gratefully hoping that some on this list could offer recommendations for my next steps. I have provided below the current state of the array per https://raid.wiki.kernel.org/index.php/RAID_Recovery.
>>>
>>> It appears from the output below that one of the disks (sdd1) failed last year and the admin did not notice this. Now, it appears a second disk (sdg1) has recently had read errors and was kicked out of the array.
>>>
>>> Should I try to restore the array using the recreate_array.pl script provided on the RAID_Recovery site? Should I then try to recreate the array and/or perform ‘fsck’?
>>>
>>> Thank you greatly in advance!
>>>
>>> raid.status:
>>>
>>> /dev/sdd1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
>>> Name : <->:0
>>> Creation Time : Tue Nov 29 17:33:39 2011
>>> Raid Level : raid5
>>> Raid Devices : 4
>>>
>>> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
>>> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> State : clean
>>> Device UUID : ec1a6336:0a991298:5b409bf1:4585ccbe
>>>
>>> Update Time : Sun Jun 7 02:28:00 2015
>>> Checksum : f9323080 - correct
>>> Events : 96203
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 0
>>> Array State : AAAA ('A' == active, '.' == missing)
>>>
>>> /dev/sde1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
>>> Name : <->:0
>>> Creation Time : Tue Nov 29 17:33:39 2011
>>> Raid Level : raid5
>>> Raid Devices : 4
>>>
>>> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
>>> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> State : clean
>>> Device UUID : ed13b045:ca75ab96:83045f97:e4fd62cb
>>>
>>> Update Time : Sun Jun 19 19:43:31 2016
>>> Checksum : bb6a905f - correct
>>> Events : 344993
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 3
>>> Array State : .A.A ('A' == active, '.' == missing)
>>>
>>> /dev/sdf1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
>>> Name : <->:0
>>> Creation Time : Tue Nov 29 17:33:39 2011
>>> Raid Level : raid5
>>> Raid Devices : 4
>>>
>>> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
>>> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> State : clean
>>> Device UUID : a1ea11e0:6465fe26:483f133d:680014b3
>>>
>>> Update Time : Sun Jun 19 19:43:31 2016
>>> Checksum : 738493f3 - correct
>>> Events : 344993
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 1
>>> Array State : .A.A ('A' == active, '.' == missing)
>>>
>>> /dev/sdg1:
>>> Magic : a92b4efc
>>> Version : 1.2
>>> Feature Map : 0x0
>>> Array UUID : bfb03f95:834b520d:60f773d8:ecb6b9e3
>>> Name : <->:0
>>> Creation Time : Tue Nov 29 17:33:39 2011
>>> Raid Level : raid5
>>> Raid Devices : 4
>>>
>>> Avail Dev Size : 3907024896 (1863.01 GiB 2000.40 GB)
>>> Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
>>> Used Dev Size : 3907023872 (1863.01 GiB 2000.40 GB)
>>> Data Offset : 2048 sectors
>>> Super Offset : 8 sectors
>>> State : active
>>> Device UUID : a9737439:17f81210:484d4f4c:c3d34a8a
>>>
>>> Update Time : Sun Jun 19 12:18:49 2016
>>> Checksum : 9c6d24bf - correct
>>> Events : 343949
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Device Role : Active device 2
>>> Array State : .AAA ('A' == active, '.' == missing)
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Another> --
> Another> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> Another> the body of a message to majordomo@vger.kernel.org
> Another> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Recommendations needed for RAID5 recovery
From: Phil Turmel @ 2016-06-25 16:49 UTC (permalink / raw)
To: Wols Lists, Peter Gebhard; +Cc: Linux-RAID, Another Sillyname, John Stoffel
In-Reply-To: <576E6E68.9070209@youngman.org.uk>
Hi Wol, Peter,
{ Convention on kernel.org is to reply-to-all, bottom or interleave
replies, and trim unnecessary context. CC list fixed up accordingly. }
On 06/25/2016 07:43 AM, Wols Lists wrote:
> I know you're getting conflicting advice, but I'd try to get a good dd
> backup first. I don't know of any utility that will do an md integrity
> check on a ddrescue'd disk :-( so you'd need to do a fsck and hope ...
Conflicting advice indeed. More conflict ahead:
dd is totally useless for raid recovery in all cases. ddrescue may be
of use in this case:
If there is redundancy available for proper MD rewrite of UREs, you want
to run the original devices with the UREs, so they'll get fixed. No
need for dd. If there's no redundancy available, then you have to fix
the UREs without knowing the correct content, and ddrescue will do that
(putting zeroes in the copy).
> Oh - and make sure you new disks are proper raid - eg WD Red or Seagate
> NAS. And are your current disks proper raid? If not, fix the timeout
> problem and your life *may* be made a lot simpler ...
Yes, timeout mismatch is a common problem and absolutely *must* be
addressed if you run a raid array. Some older posts of mine that help
explain the issue are linked below.
If you'd like advice on the status of your drives, post the output of:
for x in /dev/sd[defg] ; do echo $x ; smartctl -iA -l scterc $x ; done
> Have you got spare SATA ports? If not, go out and get an add-in card! If
> you can force the array to assemble, and create a temporary six-drive
> array (the two dud ones being assembled with the --replace option to
> move them to two new ones) that may be your best bet at recovery. If md
> can get at a clean read from three drives for each block, then it'll be
> able to rebuild the missing block.
No. The first drive that dropped out did so more than a year ago --
it's content is totally untrustworthy. It is only suitable for wipe and
re-use if it is physically still OK.
Which means that the the balance of the drives have no redundancy
available to reconstruct data for any UREs remaining in the array. If
there were, forced assembly of originals after any timeout mismatch
fixes would be the correct solution. That would let remaining
redundancy fix UREs while adding more redundancy (the #1 reason for
choosing raid6 over raid5).
Peter, I strongly recommend that you perform a forced assembly on the
three drives, omitting the unit kicked out last year. (After fixing any
timeout issue, if any. Very likely, btw.) Mount the filesystem
read-only and backup the absolutely critical items. Do not use fsck
yet. You may encounter UREs that causes some of these copies to fail,
letting you know which files to not trust later. If you encounter
enough failures to drop the array again, simply repeat the forced
assembly and readonly mount and carry on.
When you've gotten all you can that way, shut down the array and use
ddrescue to duplicate all three drives. Take the originals out of the
box, and force assemble the new drives. Run fsck to fix any remaining
errors from zeroed blocks, then mount and backup anything else you need.
If you need to keep costs down, it would be fairly low risk to just
ddrescue the most recent failure onto the oldest (which will write over
any UREs it currently has). Then forced assemble with it instead.
And add a drive to the array to get back to a redundant operation.
Consider adding another drive after that and reshaping to raid6. If
your drives really are ok (timeout issue, not physical), then you could
re-use one or more of the originals to get back to full operation. Use
--zero-superblock on them to allow MD to use them again.
Phil
Readings for timeout mismatch: (whole threads if possible)
http://marc.info/?l=linux-raid&m=139050322510249&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=132477199207506
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142487508806844&w=3
http://marc.info/?l=linux-raid&m=144535576302583&w=2
^ permalink raw reply
* RAID 10 / 2 Devices Layout question
From: Paul Roland @ 2016-06-26 17:20 UTC (permalink / raw)
To: linux-raid
Hello, quick one I guess sorry if I am messing this email with such a
newbie question.
I have two SSDs, and I would like to use md/raid10 (2devices) for
performance reasons.
The question since I was only able to find straight answers on more
than 2 devices:
Is using near instead of N2 somehow less secure than F2?
I know there is a mirrors and then stripe but with N2 I suspect a very
tiny chance of data to be written on same physical disc. In a 4disk
that is easy to understand or anticipate but here I am not sure which
to use.
Thank you
^ permalink raw reply
* Re: RAID 10 / 2 Devices Layout question
From: Phil Turmel @ 2016-06-26 17:34 UTC (permalink / raw)
To: Paul Roland, linux-raid
In-Reply-To: <CA+3OdkMoF7jZNwsNR282Hvq4qwsKUuPtXaXrKXNUPq_4YD6BDw@mail.gmail.com>
On 06/26/2016 01:20 PM, Paul Roland wrote:
> I have two SSDs, and I would like to use md/raid10 (2devices) for
> performance reasons.
Ok, that's reasonable.
> Is using near instead of N2 somehow less secure than F2?
{ First, "near" == "near2" == "n2". Linux raid10 defaults to two copies
of each data block. "near3" == "n3" would be three copies of each data
block. Et cetera. }
No. The far layouts have exactly the same redundancy as the
corresponding near or offset layout. The far layouts are intended to
provide a slight performance boost based on the difference in linear
head velocity on spinning disks between the outer edge and the hub. It
has no advantage on SSDs.
The near layouts are equivalent to mirrors when using the same number of
disks as number of copies. The near layouts are equivalent to mirrors
of raid0 stripes when the number of disks is a multiple of the number of
copies.
HTH,
Phil
^ permalink raw reply
* [PATCH RFC] md/raid1: fix deadlock between freeze_array() and wait_barrier().
From: Alexander Lyakas @ 2016-06-26 19:18 UTC (permalink / raw)
To: NeilBrown, 马建朋, linux-raid; +Cc: Jes Sorensen
When we call wait_barrier, we might have some bios waiting
in current->bio_list, which prevents the array_freeze call to
complete. Those can only be internal READs, which have already
passed the wait_barrier call (thus incrementing nr_pending), but
still were not submitted to the lower level, due to generic_make_request
logic to avoid recursive calls. In such case, we have a deadlock:
- array_frozen is already set to 1, so wait_barrier unconditionally waits, so
- internal READ bios will not be submitted, thus freeze_array will
never completes
This problem was originally fixed in commit:
d6b42dc md/raid1,raid10: avoid deadlock during resync/recovery.
But then it was broken in commit:
b364e3d raid1: Add a field array_frozen to indicate whether raid in
freeze state.
Notes:
- the problem cannot happen for WRITEs, because when a WRITE is
submitted to MD, there are two cases:
- calling thread has a plug: in that case, when this thread will
go to sleep in wait_barrier (or anywhere else), all the plugged IOs
will be flushed via:
schedule()=>sched_submit_work()=>blk_schedule_flush_plug()
- calling thread is not plugged: in that case internal WRITEs are
handed off to raid1d, which doesn't have current->bio_list issues
- current->bio_list may contains an internal READ bio from a different
MD! This can happen, for example, when two or more MDs are under
dm-stripe.
- the commit causing the issue mentioned above, also made
freeze_array() call non-reentrant, because array_frozen can be 0 or 1
only. Thus if two threads call freeze_array() in parallel (and it may
happen), the first thread that calls unfreeze_array() sets
array_frozen=0, but the second thread still thinks array is frozen.
This patch does not address this problem, it only adds a warning.
This patch demonstrates a rudimentary fix, which fixes the issue for me.
The patch is based on kernel 3.18.
---
drivers/md/raid1.c | 128 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
drivers/md/raid1.h | 1 +
2 files changed, 127 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 3c8ada4..92541b4 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -59,20 +59,27 @@
#define IO_MADE_GOOD ((struct bio *)2)
#define BIO_SPECIAL(bio) ((unsigned long)bio <= 2)
/* When there are this many requests queue to be written by
* the raid1 thread, we become 'congested' to provide back-pressure
* for writeback.
*/
static int max_queued_requests = 1024;
+/* see wait_barrier_rescue */
+struct workqueue_struct *g_freeze_array_rescue_wq;
+struct work_struct g_freeze_array_rescue_work;
+struct bio_list g_bios_to_rescue;
+spinlock_t g_rescue_lock;
+
+
static void allow_barrier(struct r1conf *conf, sector_t start_next_window,
sector_t bi_sector);
static void lower_barrier(struct r1conf *conf);
static void * r1bio_pool_alloc(gfp_t gfp_flags, void *data)
{
struct pool_info *pi = data;
int size = offsetof(struct r1bio, bios[pi->raid_disks]);
/* allocate a r1bio with room for raid_disks entries in the bios array */
@@ -883,26 +890,112 @@ static bool need_to_wait_for_sync(struct r1conf
*conf, struct bio *bio)
(conf->next_resync + NEXT_NORMALIO_DISTANCE
<= bio->bi_iter.bi_sector))
wait = false;
else
wait = true;
}
return wait;
}
+/*
+ * When we call wait_barrier, we might have some bios waiting
+ * in current->bio_list, which prevents the array_freeze call to
+ * complete. Those can only be internal READs, which have already
+ * passed the wait_barrier call (thus incrementing nr_pending), but
+ * still were not submitted to the lower level, due to generic_make_request
+ * logic to avoid recursive calls. In such case, we have a deadlock:
+ * - array_frozen is already set to 1, so wait_barrier
unconditionally waits, so
+ * - internal READ bios will not be submitted, thus freeze_array will
never completes
+ *
+ * This problem was originally fixed in commit:
+ * d6b42dc md/raid1,raid10: avoid deadlock during resync/recovery.
+ *
+ * But then was broken in commit:
+ * b364e3d raid1: Add a field array_frozen to indicate whether raid
in freeze state.
+ *
+ * Notes:
+ * - the problem cannot happen for WRITEs, because when a WRITE is
submitted to MD, there are two cases:
+ * - calling thread has a plug: in that case, when this thread will
go to sleep in wait_barrier (or anywhere elese),
+ * all the plugged IOs will be flushed via:
schedule()=>sched_submit_work()=>blk_schedule_flush_plug()
+ * - calling thread is not plugged: in that case internal WRITEs
are handed off to raid1d, which
+ * doesn't have current->bio_list issues
+ * - note that current->bio_list may contains an internal READ bio
from a different MD! This can
+ * happen, for example, when two or more MDs are under dm-stripe.
+ */
+static void wait_barrier_rescue(struct r1conf *conf)
+{
+ struct bio_list bios_to_rescue;
+ struct bio *bio = NULL;
+
+ assert_spin_locked(&conf->resync_lock);
+
+ if (current->bio_list == NULL)
+ return;
+ if (!(conf->array_frozen && !conf->array_frozen_done))
+ return;
+
+ bio_list_init(&bios_to_rescue);
+
+ /* grab the whole current->bio_list locally */
+ bio = bio_list_get(current->bio_list);
+
+ /* iterate over the local list */
+ while (bio) {
+ struct bio *next = bio->bi_next;
+ bio->bi_next = NULL;
+
+ if (bio->bi_end_io == raid1_end_read_request)
+ /* this is a problematic bio, set it aside */
+ bio_list_add(&bios_to_rescue, bio);
+ else
+ /* return this bio back to current->bio_list */
+ bio_list_add(current->bio_list, bio);
+
+ bio = next;
+ }
+
+ /* do we have any bios to rescue? */
+ if (bios_to_rescue.head != NULL) {
+ spin_lock(&g_rescue_lock);
+ bio_list_merge(&g_bios_to_rescue, &bios_to_rescue);
+ spin_unlock(&g_rescue_lock);
+ queue_work(g_freeze_array_rescue_wq, &g_freeze_array_rescue_work);
+ }
+}
+
+static void raid1_rescue_work(struct work_struct *work)
+{
+ struct bio *bio = NULL;
+
+ spin_lock(&g_rescue_lock);
+ bio = bio_list_get(&g_bios_to_rescue);
+ spin_unlock(&g_rescue_lock);
+
+ while (bio) {
+ struct bio *next = bio->bi_next;
+
+ bio->bi_next = NULL;
+
+ generic_make_request(bio);
+
+ bio = next;
+ }
+}
+
static sector_t wait_barrier(struct r1conf *conf, struct bio *bio)
{
sector_t sector = 0;
spin_lock_irq(&conf->resync_lock);
if (need_to_wait_for_sync(conf, bio)) {
+ wait_barrier_rescue(conf);
conf->nr_waiting++;
/* Wait for the barrier to drop.
* However if there are already pending
* requests (preventing the barrier from
* rising completely), and the
* per-process bio queue isn't empty,
* then don't wait, as we need to empty
* that queue to allow conf->start_next_window
* to increase.
*/
@@ -978,32 +1071,35 @@ static void freeze_array(struct r1conf *conf, int extra)
* We wait until nr_pending match nr_queued+extra
* This is called in the context of one normal IO request
* that has failed. Thus any sync request that might be pending
* will be blocked by nr_pending, and we need to wait for
* pending IO requests to complete or be queued for re-try.
* Thus the number queued (nr_queued) plus this request (extra)
* must match the number of pending IOs (nr_pending) before
* we continue.
*/
spin_lock_irq(&conf->resync_lock);
+ WARN(conf->array_frozen, "%s-R1 re-enterting freeze_array!!!",
mdname(conf->mddev));
conf->array_frozen = 1;
wait_event_lock_irq_cmd(conf->wait_barrier,
conf->nr_pending == conf->nr_queued+extra,
conf->resync_lock,
flush_pending_writes(conf));
+ conf->array_frozen_done = 1;
spin_unlock_irq(&conf->resync_lock);
}
static void unfreeze_array(struct r1conf *conf)
{
/* reverse the effect of the freeze */
spin_lock_irq(&conf->resync_lock);
conf->array_frozen = 0;
+ conf->array_frozen_done = 0;
wake_up(&conf->wait_barrier);
spin_unlock_irq(&conf->resync_lock);
}
/* duplicate the data pages for behind I/O
*/
static void alloc_behind_pages(struct bio *bio, struct r1bio *r1_bio)
{
int i;
struct bio_vec *bvec;
@@ -3162,23 +3258,25 @@ static void *raid1_takeover(struct mddev *mddev)
{
/* raid1 can take over:
* raid5 with 2 devices, any layout or chunk size
*/
if (mddev->level == 5 && mddev->raid_disks == 2) {
struct r1conf *conf;
mddev->new_level = 1;
mddev->new_layout = 0;
mddev->new_chunk_sectors = 0;
conf = setup_conf(mddev);
- if (!IS_ERR(conf))
+ if (!IS_ERR(conf)) {
/* Array must appear to be quiesced */
conf->array_frozen = 1;
+ conf->array_frozen_done = 1;
+ }
return conf;
}
return ERR_PTR(-EINVAL);
}
static struct md_personality raid1_personality =
{
.name = "raid1",
.level = 1,
.owner = THIS_MODULE,
@@ -3193,25 +3291,51 @@ static struct md_personality raid1_personality =
.sync_request = sync_request,
.resize = raid1_resize,
.size = raid1_size,
.check_reshape = raid1_reshape,
.quiesce = raid1_quiesce,
.takeover = raid1_takeover,
};
static int __init raid_init(void)
{
- return register_md_personality(&raid1_personality);
+ int ret = 0;
+
+ bio_list_init(&g_bios_to_rescue);
+ spin_lock_init(&g_rescue_lock);
+
+ INIT_WORK(&g_freeze_array_rescue_work, raid1_rescue_work);
+
+ /* we have only one work item, so we need only one active work at a time */
+ g_freeze_array_rescue_wq = alloc_workqueue("raid1_rescue_wq",
+ WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_SYSFS,
+ 1/*max_active*/);
+ if (g_freeze_array_rescue_wq == NULL) {
+ pr_err("alloc_workqueue(zraid1_rescue_wq) failed");
+ ret =-ENOMEM;
+ goto out;
+ }
+
+ ret = register_md_personality(&raid1_personality);
+ if (ret != 0)
+ goto free_wq;
+
+free_wq:
+ destroy_workqueue(g_freeze_array_rescue_wq);
+out:
+ return ret;
}
static void raid_exit(void)
{
+ flush_workqueue(g_freeze_array_rescue_wq);
+ destroy_workqueue(g_freeze_array_rescue_wq);
unregister_md_personality(&raid1_personality);
}
module_init(raid_init);
module_exit(raid_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("RAID1 (mirroring) personality for MD");
MODULE_ALIAS("md-personality-3"); /* RAID1 */
MODULE_ALIAS("md-raid1");
MODULE_ALIAS("md-level-1");
diff --git a/drivers/md/raid1.h b/drivers/md/raid1.h
index 33bda55..a7d0bd4 100644
--- a/drivers/md/raid1.h
+++ b/drivers/md/raid1.h
@@ -72,20 +72,21 @@ struct r1conf {
* is no other IO. So when either is active, the other has to wait.
* See more details description in raid1.c near raise_barrier().
*/
wait_queue_head_t wait_barrier;
spinlock_t resync_lock;
int nr_pending;
int nr_waiting;
int nr_queued;
int barrier;
int array_frozen;
+ int array_frozen_done;
/* Set to 1 if a full sync is needed, (fresh device added).
* Cleared when a sync completes.
*/
int fullsync;
/* When the same as mddev->recovery_disabled we don't allow
* recovery to be attempted as we expect a read error.
*/
int recovery_disabled;
--
1.9.1
^ permalink raw reply related
* Re: Recommendations needed for RAID5 recovery
From: Wols Lists @ 2016-06-26 21:12 UTC (permalink / raw)
To: Phil Turmel, Peter Gebhard; +Cc: Linux-RAID, Another Sillyname, John Stoffel
In-Reply-To: <576EB5FD.1000309@turmel.org>
On 25/06/16 17:49, Phil Turmel wrote:
> Hi Wol, Peter,
>
> { Convention on kernel.org is to reply-to-all, bottom or interleave
> replies, and trim unnecessary context. CC list fixed up accordingly. }
Sorry, but the OP had already been trimmed, I trimmed a bit further...
>
> On 06/25/2016 07:43 AM, Wols Lists wrote:
>
>> I know you're getting conflicting advice, but I'd try to get a good dd
>> backup first. I don't know of any utility that will do an md integrity
>> check on a ddrescue'd disk :-( so you'd need to do a fsck and hope ...
>
> Conflicting advice indeed. More conflict ahead:
>
> dd is totally useless for raid recovery in all cases. ddrescue may be
> of use in this case:
And if dd gets a copy without errors, what's the difference between that
and a ddrescue? Surely they're identical?
That said, it struck me you're probably better off using ddrescue,
because ddrescue could get that copy in one. So if you can get it in
one, it doesn't matter which you use, so you should use ddrescue because
it saves a wasted attempt with dd. (I've just read the ddrescue man
page. Recommended reading ... :-)
>
> Which means that the the balance of the drives have no redundancy
> available to reconstruct data for any UREs remaining in the array. If
> there were, forced assembly of originals after any timeout mismatch
> fixes would be the correct solution. That would let remaining
> redundancy fix UREs while adding more redundancy (the #1 reason for
> choosing raid6 over raid5).
>
> Peter, I strongly recommend that you perform a forced assembly on the
> three drives, omitting the unit kicked out last year. (After fixing any
> timeout issue, if any. Very likely, btw.) Mount the filesystem
> read-only and backup the absolutely critical items. Do not use fsck
> yet. You may encounter UREs that causes some of these copies to fail,
> letting you know which files to not trust later. If you encounter
> enough failures to drop the array again, simply repeat the forced
> assembly and readonly mount and carry on.
>
> When you've gotten all you can that way, shut down the array and use
> ddrescue to duplicate all three drives. Take the originals out of the
> box, and force assemble the new drives. Run fsck to fix any remaining
> errors from zeroed blocks, then mount and backup anything else you need.
>
> If you need to keep costs down, it would be fairly low risk to just
> ddrescue the most recent failure onto the oldest (which will write over
> any UREs it currently has). Then forced assemble with it instead.
>
> And add a drive to the array to get back to a redundant operation.
> Consider adding another drive after that and reshaping to raid6. If
> your drives really are ok (timeout issue, not physical), then you could
> re-use one or more of the originals to get back to full operation. Use
> --zero-superblock on them to allow MD to use them again.
>
Hmm...
Would it be an idea to get 4 by 3TB drives? That way he can do the
backup straight on to a RAID6 array, and IF he gets a successful backup
then the old drives are now redundant, for backups or whatever (3TB Reds
or NASs are about £100 each...)
Cheers,
Wol
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Recommendations needed for RAID5 recovery
From: Phil Turmel @ 2016-06-26 22:18 UTC (permalink / raw)
To: Wols Lists, Peter Gebhard; +Cc: Linux-RAID, Another Sillyname, John Stoffel
In-Reply-To: <5770452F.8000109@youngman.org.uk>
On 06/26/2016 05:12 PM, Wols Lists wrote:
> On 25/06/16 17:49, Phil Turmel wrote:
>> dd is totally useless for raid recovery in all cases. ddrescue may be
>> of use in this case:
>
> And if dd gets a copy without errors, what's the difference between that
> and a ddrescue? Surely they're identical?
>
> That said, it struck me you're probably better off using ddrescue,
> because ddrescue could get that copy in one. So if you can get it in
> one, it doesn't matter which you use, so you should use ddrescue because
> it saves a wasted attempt with dd. (I've just read the ddrescue man
> page. Recommended reading ... :-)
dd will only copy a device that has no UREs. If it has no UREs, it'll
work in the raid array even if there's no redundancy. So duplicating
that device is pointless for pure recovery purposes.
If you already know you have one or more UREs on a device and no other
redundancy to reconstruct with, you go straight to ddrescue -- you know
that dd won't work, so why bother? And in this case we know -- the
drive was kicked out of the array. The OP hasn't provided any further
information, so we don't know if this is the typical timeout mismatch
calamity, or if there's more serious problems, but that doesn't change
the advice.
Finally, if you know you have UREs *and* you still have redundancy
(raid6 or raid10,n3 degraded by one, f.e.), you want to keep the drive
with the UREs in place until replaced so MD can perform reconstruction
when it hits that spot. Again, no role for dd.
Now, when the problem isn't just an array that needs to be reassembled
but rather a true reconstruction from unknown parameters, the duplicate
devices are needed because the experiments to discover the parameters
can be destructive. In this case we need complete copies for the
experiments, whether UREs are present or not. ddrescue again. (Knowing
there are UREs makes it important to use the original devices during
recreation after the experiments are done.)
I don't know *any* scenario where dd is useful for raid recovery.
> Hmm...
>
> Would it be an idea to get 4 by 3TB drives? That way he can do the
> backup straight on to a RAID6 array, and IF he gets a successful backup
> then the old drives are now redundant, for backups or whatever (3TB Reds
> or NASs are about £100 each...)
Many hobbyists and small business people have limited budgets. (Been
there!) I try to avoid recommending expensive replacements unless the
situation clearly calls for it. I specifically omitted any suggestion
of where to perform backups since it may make sense for the OP to just
copy important stuff onto an external USB device.
It's entirely possible that the OP is researching timeout mismatch and
patching up some green drives. It's not ideal, but such an array can
operate with the work-arounds for years if it's regularly scrubbed. And
it's cheap, which sometimes matters.
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Recommendations needed for RAID5 recovery
From: Andreas Klauer @ 2016-06-27 9:06 UTC (permalink / raw)
To: Phil Turmel
Cc: Wols Lists, Peter Gebhard, Linux-RAID, Another Sillyname,
John Stoffel
In-Reply-To: <5770549C.4060301@turmel.org>
On Sun, Jun 26, 2016 at 06:18:04PM -0400, Phil Turmel wrote:
> I don't know *any* scenario where dd is useful for raid recovery.
Basically you use dd to make a copy to play around with but
with overlays (as described in the RAID wiki) you don't need
those copies anymore. [as long as the device is intact]
There is conv=noerror,sync however I found recently that it
actually produces corrupt copies (changes offsets), still not
sure what the conv=noerror,sync is actually supposed to be for.
( http://superuser.com/a/1075837/195171 )
So you shouldn't use dd on bad disks...
I've had my fair share of issues with ddrescue as well, it likes
to get stuck forever and not continue. --min-read-rate=<bytes>
might help in such a case... (better to copy regions that still
work quickly rather than spending hours on error correction ...).
ddrescue produces a log (or a map) that records bad regions,
so if you somehow need a device that still produces read errors
for those, it can probably be magicked with the device mapper...
Regards
Andreas Klauer
^ permalink raw reply
* Re: Recommendations needed for RAID5 recovery
From: Mikael Abrahamsson @ 2016-06-27 10:05 UTC (permalink / raw)
To: Andreas Klauer; +Cc: Linux-RAID
In-Reply-To: <20160627090645.GA3255@metamorpher.de>
On Mon, 27 Jun 2016, Andreas Klauer wrote:
> So you shouldn't use dd on bad disks...
There is no downside to using ddrescue that I can see. If everything works
ok, then you get a 0 error copy, and you got the exact same result as with
dd.
If you did have errors, then ddrescue will try to do the best of the
situation and try to copy as much as possible from the drive You also get
a list of the errored sectors that you can use for future reference.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* We Apologize For Any Inconvenience
From: Webmaster @ 2016-06-27 11:04 UTC (permalink / raw)
Your mailbox has exceeded the storage limit 2.GB Founded by the administrator is currently 2.30GB, can not send or receive new messages until you re-validate your email.
Click the link below to Update your email
https://www.formcrafts.com/a/20433
Thank you!
Customer Care
Case number: 8941624
Copyright @ 2016 all rights reserved
^ permalink raw reply
* Reshapeing after 4 Days with 0.0% progress
From: iggy @ 2016-06-27 11:21 UTC (permalink / raw)
To: linux-raid
Hello Guys,
i wanted to grow my raid 5 with 3 3tb devices to 4 of 3tb devices.
1) i add a new device
2) i grow (please ask me not why i do this) with the help of the button in
webmin instead of the cli
cat /proc/mdstat:....
After 3 Days of 0.0% of reshaping progress i decided to abort this procedure
- because for me it seems like an error!
Now my raid disapears and i'm not able to mount and backup my data, because
of degraded. So i have no backup file during reshaping.
is there any way to mount my md0 to backup a fews of data before i complete
recreate my raid array?
i hofe for answers..
THANKS!
^ permalink raw reply
* Re: Reshapeing after 4 Days with 0.0% progress
From: Mikael Abrahamsson @ 2016-06-27 11:52 UTC (permalink / raw)
To: iggy; +Cc: linux-raid
In-Reply-To: <loom.20160627T131058-485@post.gmane.org>
On Mon, 27 Jun 2016, iggy wrote:
> Hello Guys,
>
> i wanted to grow my raid 5 with 3 3tb devices to 4 of 3tb devices.
>
> 1) i add a new device
> 2) i grow (please ask me not why i do this) with the help of the button in
> webmin instead of the cli
>
>
> cat /proc/mdstat:....
>
> After 3 Days of 0.0% of reshaping progress i decided to abort this procedure
> - because for me it seems like an error!
>
>
> Now my raid disapears and i'm not able to mount and backup my data, because
> of degraded. So i have no backup file during reshaping.
>
> is there any way to mount my md0 to backup a fews of data before i complete
> recreate my raid array?
>
> i hofe for answers..
> THANKS!
There are multiple threads here about this problem. What you should have
done is read them before you "aborted".
http://www.spinics.net/lists/raid/msg49846.html is one example of such a
thread. There you can also see examples of information needed to help you.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Reshapeing after 4 Days with 0.0% progress
From: iggy @ 2016-06-27 12:20 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <alpine.DEB.2.02.1606271349280.28955@uplift.swm.pp.se>
Thanks for your reply,..
i tried all of recommendations that i found - but nothing pushed me to a
working solution...
it seems ....the fast solution is recreating... :(
next time i'll do a full backup before manage the array
greetings!
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox