* Re: [PATCH 1/3] badblocks: Add core badblock management code
From: Jens Axboe @ 2015-11-24 19:19 UTC (permalink / raw)
To: Vishal Verma, linux-nvdimm
Cc: linux-block, linux-raid, linux-scsi, NeilBrown, Jeff Moyer
In-Reply-To: <1448066960-20119-2-git-send-email-vishal.l.verma@intel.com>
On 11/20/2015 05:49 PM, Vishal Verma wrote:
> Take the core badblocks implementation from md, and make it generally
> available. This follows the same style as kernel implementations of
> linked lists, rb-trees etc, where you can have a structure that can be
> embedded anywhere, and accessor functions to manipulate the data.
>
> The only changes in this copy of the code are ones to generalize
> function/variable names from md-specific ones. Also add init and free
> functions.
Split this into a .c file with the code.
--
Jens Axboe
^ permalink raw reply
* Re: [PATCH 2/3] block: Add badblock management for gendisks
From: Jeff Moyer @ 2015-11-24 19:14 UTC (permalink / raw)
To: Verma, Vishal L
Cc: linux-raid@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-block@vger.kernel.org, neilb@suse.com, axboe@fb.com,
linux-nvdimm@ml01.01.org
In-Reply-To: <1448391837.27481.16.camel@intel.com>
"Verma, Vishal L" <vishal.l.verma@intel.com> writes:
> On Tue, 2015-11-24 at 10:34 -0500, Jeff Moyer wrote:
>> Vishal Verma <vishal.l.verma@intel.com> writes:
>>
>> > NVDIMM devices, which can behave more like DRAM rather than block
>> > devices, may develop bad cache lines, or 'poison'. A block device
>> > exposed by the pmem driver can then consume poison via a read (or
>> > write), and cause a machine check. On platforms without machine
>> > check recovery features, this would mean a crash.
>> >
>> > The block device maintaining a runtime list of all known sectors
>> > that
>> > have poison can directly avoid this, and also provide a path forward
>> > to enable proper handling/recovery for DAX faults on such a device.
>> >
>> > Use the new badblock management interfaces to add a badblocks list
>> > to
>> > gendisks.
>>
>> Because disk_alloc_badblocks can fail, you need to check for a NULL
>> disk->bb in all of the utility functions you've defined.
>>
>
> Thanks, Jeff - I'll fix this. I have a handful of other fixes queued up
> too, will send out a v2 soon.
I'm not sure whether it makes sense to continue without badblock
management for the RAID code. I was hoping Neil would comment on that.
-Jeff
^ permalink raw reply
* Re: [PATCH 2/3] block: Add badblock management for gendisks
From: Verma, Vishal L @ 2015-11-24 19:03 UTC (permalink / raw)
To: jmoyer@redhat.com
Cc: linux-raid@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-block@vger.kernel.org, neilb@suse.com, axboe@fb.com,
linux-nvdimm@ml01.01.org
In-Reply-To: <x49a8q32yxy.fsf@segfault.boston.devel.redhat.com>
On Tue, 2015-11-24 at 10:34 -0500, Jeff Moyer wrote:
> Vishal Verma <vishal.l.verma@intel.com> writes:
>
> > NVDIMM devices, which can behave more like DRAM rather than block
> > devices, may develop bad cache lines, or 'poison'. A block device
> > exposed by the pmem driver can then consume poison via a read (or
> > write), and cause a machine check. On platforms without machine
> > check recovery features, this would mean a crash.
> >
> > The block device maintaining a runtime list of all known sectors
> > that
> > have poison can directly avoid this, and also provide a path forward
> > to enable proper handling/recovery for DAX faults on such a device.
> >
> > Use the new badblock management interfaces to add a badblocks list
> > to
> > gendisks.
>
> Because disk_alloc_badblocks can fail, you need to check for a NULL
> disk->bb in all of the utility functions you've defined.
>
Thanks, Jeff - I'll fix this. I have a handful of other fixes queued up
too, will send out a v2 soon.
-Vishal
^ permalink raw reply
* Re: [PATCH 2/3] block: Add badblock management for gendisks
From: Jeff Moyer @ 2015-11-24 15:34 UTC (permalink / raw)
To: Vishal Verma
Cc: linux-nvdimm, linux-block, linux-raid, linux-scsi, Jens Axboe,
NeilBrown
In-Reply-To: <1448066960-20119-3-git-send-email-vishal.l.verma@intel.com>
Vishal Verma <vishal.l.verma@intel.com> writes:
> NVDIMM devices, which can behave more like DRAM rather than block
> devices, may develop bad cache lines, or 'poison'. A block device
> exposed by the pmem driver can then consume poison via a read (or
> write), and cause a machine check. On platforms without machine
> check recovery features, this would mean a crash.
>
> The block device maintaining a runtime list of all known sectors that
> have poison can directly avoid this, and also provide a path forward
> to enable proper handling/recovery for DAX faults on such a device.
>
> Use the new badblock management interfaces to add a badblocks list to
> gendisks.
Because disk_alloc_badblocks can fail, you need to check for a NULL
disk->bb in all of the utility functions you've defined.
Cheers,
Jeff
>
> Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
> ---
> block/genhd.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++
> include/linux/genhd.h | 6 +++++
> 2 files changed, 70 insertions(+)
>
> diff --git a/block/genhd.c b/block/genhd.c
> index 0c706f3..4209c32 100644
> --- a/block/genhd.c
> +++ b/block/genhd.c
> @@ -20,6 +20,7 @@
> #include <linux/idr.h>
> #include <linux/log2.h>
> #include <linux/pm_runtime.h>
> +#include <linux/badblocks.h>
>
> #include "blk.h"
>
> @@ -505,6 +506,20 @@ static int exact_lock(dev_t devt, void *data)
> return 0;
> }
>
> +static void disk_alloc_badblocks(struct gendisk *disk)
> +{
> + disk->bb = kzalloc(sizeof(disk->bb), GFP_KERNEL);
> + if (!disk->bb) {
> + pr_warn("%s: failed to allocate space for badblocks\n",
> + disk->disk_name);
> + return;
> + }
> +
> + if (badblocks_init(disk->bb, 1))
> + pr_warn("%s: failed to initialize badblocks\n",
> + disk->disk_name);
> +}
> +
> static void register_disk(struct gendisk *disk)
> {
> struct device *ddev = disk_to_dev(disk);
> @@ -609,6 +624,7 @@ void add_disk(struct gendisk *disk)
> disk->first_minor = MINOR(devt);
>
> disk_alloc_events(disk);
> + disk_alloc_badblocks(disk);
>
> /* Register BDI before referencing it from bdev */
> bdi = &disk->queue->backing_dev_info;
> @@ -657,6 +673,9 @@ void del_gendisk(struct gendisk *disk)
> blk_unregister_queue(disk);
> blk_unregister_region(disk_devt(disk), disk->minors);
>
> + badblocks_free(disk->bb);
> + kfree(disk->bb);
> +
> part_stat_set_all(&disk->part0, 0);
> disk->part0.stamp = 0;
>
> @@ -670,6 +689,48 @@ void del_gendisk(struct gendisk *disk)
> }
> EXPORT_SYMBOL(del_gendisk);
>
> +/*
> + * The gendisk usage of badblocks does not track acknowledgements for
> + * badblocks. We always assume they are acknowledged.
> + */
> +int disk_check_badblocks(struct gendisk *disk, sector_t s, int sectors,
> + sector_t *first_bad, int *bad_sectors)
> +{
> + return badblocks_check(disk->bb, s, sectors, first_bad, bad_sectors);
> +}
> +EXPORT_SYMBOL(disk_check_badblocks);
> +
> +int disk_set_badblocks(struct gendisk *disk, sector_t s, int sectors)
> +{
> + return badblocks_set(disk->bb, s, sectors, 1);
> +}
> +EXPORT_SYMBOL(disk_set_badblocks);
> +
> +int disk_clear_badblocks(struct gendisk *disk, sector_t s, int sectors)
> +{
> + return badblocks_clear(disk->bb, s, sectors);
> +}
> +EXPORT_SYMBOL(disk_clear_badblocks);
> +
> +/* sysfs access to bad-blocks list. */
> +static ssize_t disk_badblocks_show(struct device *dev,
> + struct device_attribute *attr,
> + char *page)
> +{
> + struct gendisk *disk = dev_to_disk(dev);
> +
> + return badblocks_show(disk->bb, page, 0);
> +}
> +
> +static ssize_t disk_badblocks_store(struct device *dev,
> + struct device_attribute *attr,
> + const char *page, size_t len)
> +{
> + struct gendisk *disk = dev_to_disk(dev);
> +
> + return badblocks_store(disk->bb, page, len, 0);
> +}
> +
> /**
> * get_gendisk - get partitioning information for a given device
> * @devt: device to get partitioning information for
> @@ -988,6 +1049,8 @@ static DEVICE_ATTR(discard_alignment, S_IRUGO, disk_discard_alignment_show,
> static DEVICE_ATTR(capability, S_IRUGO, disk_capability_show, NULL);
> static DEVICE_ATTR(stat, S_IRUGO, part_stat_show, NULL);
> static DEVICE_ATTR(inflight, S_IRUGO, part_inflight_show, NULL);
> +static DEVICE_ATTR(badblocks, S_IRUGO | S_IWUSR, disk_badblocks_show,
> + disk_badblocks_store);
> #ifdef CONFIG_FAIL_MAKE_REQUEST
> static struct device_attribute dev_attr_fail =
> __ATTR(make-it-fail, S_IRUGO|S_IWUSR, part_fail_show, part_fail_store);
> @@ -1009,6 +1072,7 @@ static struct attribute *disk_attrs[] = {
> &dev_attr_capability.attr,
> &dev_attr_stat.attr,
> &dev_attr_inflight.attr,
> + &dev_attr_badblocks.attr,
> #ifdef CONFIG_FAIL_MAKE_REQUEST
> &dev_attr_fail.attr,
> #endif
> diff --git a/include/linux/genhd.h b/include/linux/genhd.h
> index 2adbfa6..5563bde 100644
> --- a/include/linux/genhd.h
> +++ b/include/linux/genhd.h
> @@ -162,6 +162,7 @@ struct disk_part_tbl {
> };
>
> struct disk_events;
> +struct badblocks;
>
> struct gendisk {
> /* major, first_minor and minors are input parameters only,
> @@ -201,6 +202,7 @@ struct gendisk {
> struct blk_integrity *integrity;
> #endif
> int node_id;
> + struct badblocks *bb;
> };
>
> static inline struct gendisk *part_to_disk(struct hd_struct *part)
> @@ -421,6 +423,10 @@ extern void add_disk(struct gendisk *disk);
> extern void del_gendisk(struct gendisk *gp);
> extern struct gendisk *get_gendisk(dev_t dev, int *partno);
> extern struct block_device *bdget_disk(struct gendisk *disk, int partno);
> +extern int disk_check_badblocks(struct gendisk *disk, sector_t s, int sectors,
> + sector_t *first_bad, int *bad_sectors);
> +extern int disk_set_badblocks(struct gendisk *disk, sector_t s, int sectors);
> +extern int disk_clear_badblocks(struct gendisk *disk, sector_t s, int sectors);
>
> extern void set_device_ro(struct block_device *bdev, int flag);
> extern void set_disk_ro(struct gendisk *disk, int flag);
^ permalink raw reply
* mdadm and size differences
From: Marco De Vitis @ 2015-11-24 10:32 UTC (permalink / raw)
To: linux-raid
Hi,
I cannot understand a difference in size I can see on an old Debian 5.0
machine, maybe I'm just missing something obvious.
/dev/sdd is partitioned as follows, and /dev/sdf is exactly the same:
> #>fdisk -l /dev/sdd
>
> Disk /dev/sdd: 500.1 GB, 500107862016 bytes
> 255 heads, 63 sectors/track, 60801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x000901d8
>
> Device Boot Start End Blocks Id System
> /dev/sdd1 1 24 192748+ 83 Linux
> /dev/sdd2 25 60801 488191252+ 5 Extended
> /dev/sdd5 25 632 4883728+ 83 Linux
> /dev/sdd6 633 2982 18876343+ 83 Linux
> /dev/sdd7 2983 4199 9775521 83 Linux
> /dev/sdd8 4200 4807 4883728+ 83 Linux
> /dev/sdd9 4808 5173 2939863+ 82 Linux swap / Solaris
> /dev/sdd10 5174 31281 209712478+ 83 Linux
You can see that sdd6 is twice the size of sdd7.
cfdisk reports sizes of around 20 GB and 10 GB:
sdd6 -> 19329.41
sdd7 -> 10010.17
All partitions in these two identical drives are configured as RAID 1
using mdadm, but the md devices associated to sdd6/sdf6 and sdd7/sdf7
are both reported to be 10 GB in size, as can be seen from the mdadm
output below.
Why?
Thanks for any info.
> #>mdadm -vv --detail /dev/md2
> /dev/md2:
> Version : 00.90
> Creation Time : Thu Feb 22 16:18:11 2007
> Raid Level : raid1
> Array Size : 9767424 (9.31 GiB 10.00 GB)
> Used Dev Size : 9767424 (9.31 GiB 10.00 GB)
> Raid Devices : 2
> Total Devices : 2
> Preferred Minor : 2
> Persistence : Superblock is persistent
>
> Update Time : Mon Nov 23 14:19:23 2015
> State : clean
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> UUID : 5582dad7:7ac89883:0c2b9e1d:5009dd35
> Events : 0.69228
>
> Number Major Minor RaidDevice State
> 0 8 54 0 active sync /dev/sdd6
> 1 8 86 1 active sync /dev/sdf6
>
> #>mdadm -vv --detail /dev/md3
> /dev/md3:
> Version : 00.90
> Creation Time : Thu Feb 22 16:18:54 2007
> Raid Level : raid1
> Array Size : 9767424 (9.31 GiB 10.00 GB)
> Used Dev Size : 9767424 (9.31 GiB 10.00 GB)
> Raid Devices : 2
> Total Devices : 2
> Preferred Minor : 3
> Persistence : Superblock is persistent
>
> Update Time : Mon Nov 23 14:19:27 2015
> State : clean
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 0
> Spare Devices : 0
>
> UUID : e44836cf:6340f7fa:52fb3562:a78d4dbb
> Events : 0.187630
>
> Number Major Minor RaidDevice State
> 0 8 55 0 active sync /dev/sdd7
> 1 8 87 1 active sync /dev/sdf7
--
Ciao,
Marco.
^ permalink raw reply
* (unknown),
From: Jaime M Towns-18128 @ 2015-11-24 7:23 UTC (permalink / raw)
Your RSA Email has won R1.5M in the 2015 Rugby World Championship Games with E-Ticket ENG/RWC/SA/SBK/09/2015. Contact; phnath.rwc2015@gmail.com<mailto:phnath.rwc2015@gmail.com> or 00447937387888. T's and C's apply!
^ permalink raw reply
* Re: raid10 resync hangs in 4.2.6, 4.3
From: Andre Tomt @ 2015-11-23 19:48 UTC (permalink / raw)
To: Artur Paszkiewicz, linux-raid, John Stoffel
In-Reply-To: <5652E1CA.3050907@tomt.net>
On 23. nov. 2015 10:52, Andre Tomt wrote:
> On 23. nov. 2015 10:20, Artur Paszkiewicz wrote:
>> Hi Andre,
>>
>> I've recently sent a patch for a similar problem, I suspect the root
>> cause is the same here. Please check out this patch:
>>
>> http://marc.info/?l=linux-raid&m=144665464232126&w=2
>
> Indeed this is where the bisect just landed me. Going to test the fix
> out later today.
>
> Thanks!
>
> And also thanks to Roy Sigurd Karlsbakk for confirming I had not gone
> completely off the rails :-)
Artur Paszkiewicz's patch fixes the crashes. I'm going to do some data
integrity checking later today/tomorrow.
It probably should be promoted to stable as both 4.2.x and 4.3.x is
currently very broken.
^ permalink raw reply
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
From: Phil Turmel @ 2015-11-23 18:24 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid
In-Reply-To: <189791448292518@web28m.yandex.ru>
Hi Semyon,
On 11/23/2015 10:28 AM, Semyon Enskiy wrote:
> Hi all,
>
> I am sorry for bad English, it is not my primary language.
No worries. Good report.
> Does solution, described here -
> "http://marc.info/?l=linux-raid&m=144659416216285&w=2", matches for mine
> issue?
No, not related. You have v1.2 metadata.
> Please, read issue description below, I am ready to provide additional info.
>
> RAID1 (md1; boot), RAID10 (md2; swap, root) and
> RAID5 (md3; LVM -> ext4 -> data)
> are placed on 10 disks, each of them is 4TB WD Red with 3 partitions on it.
> All disks model is WDC WD40EFRX-68WT0N0.
This is very good. You don't have a timeout mismatch problem.
> sd?1 partitions belongs to md1, sd?2 belongs to md2, sd?3 belongs to md3.
>
> sdj caught IO error and was reinitialized under new minor and in-/dev/ IDs,
> in fact partitions on it was dropped from arrays.
This bothers me. With no timeout issue, this must be a real problem.
Possibilities:
1) Drive, cable, or controller hardware flaw.
2) Transient power supply problem.
3) Kernel bug.
4) Memory bit flip if no ECC Ram.
I've looked at your dmesg, and I'm concerned about the errors on ata12
and ata8. They report the error source as the ata bus. And they are
happening during your check scrubs. That suggests the sustained load
across the entire array during the scrub is stressing some component.
Insufficient power has been the most popular reason for this in the
past. Since it seems to be happening to ata12 the most, with some ata8,
it might just be loose connectors or bad cables.
> sdk1 and sdk2, previous sdj1 and sdj2 accordingly, was readded to their
> arrays by me, but md3 stay leave sdj3 as member of array, I can't do
> anything with it, only receiving errors like that:
> # mdadm /dev/md3 -r detached
> mdadm: Cannot find 8:147: No such file or directory
> so sdk3 can't be readded.
This is likely a kernel bug. A reboot at this point would probably have
cleared it and made --re-add possible.
> So I've managed to grow md3 array with command like following, do not
> remember why and strictly command:
> # mdadm --grow --raid-devices=11 -add /dev/md3 /dev/sdk3
> array start to reshaping, but caught second error and reshaping process
> stalled with 1536 bytes transferred. Then was executed near two useless
> commands, like making sdk3 spare device.
Yeah, growing the array was a mistake. (Where'd that idea come from?)
With mdadm v3.3 or later, you can use --assemble --update=revert-reshape
to cancel this. It is at position 1536, so hasn't actually done any
significant reshaping yet.
--re-add was the correct operation for you to perform. When it failed
due to a busy device, fixing the busy device issue is the correct next
step. But this is no longer possible, as some new metadata has been
written to the device as part of the --grow operation.
The grow operation made no progress due to a kernel bug, I believe. You
have stuck task backtraces in your dmesg for what I believe are
operations directed at the missing but not removed sdj. Not making
progress is a lucky chance, as reshape is not the right answer anyways.
> Now md3 consists 11 devices, was 10, has 2 faulty devices, actually 1 - old
> sdj3 and sdj3 under new ID - sdk3, which is slightly corrupted by writing
> 1536 bytes.
> Seems, that I should and write data back to source block device to do it
> consistent and able to reboot, recreate array with one corrupted device and
> resync it.
No, I think you may really have bad hardware. Check all of your power
and sata cables -- replace the cables to sdj/sdk. Check the voltage on
the drive power rails.
When you are sure power is OK, and all the sata connectors are seated
tightly, reboot into an environment that has mdadm v3.3, and revert the
reshape.
> After this all happened, host was not rebooted, stay online, FS on md3 array
> unavailable, smartctl reports (-H argument), that all disks are healthy.
smartctl's health report is not sufficient info. Please supply the
output of:
for x in /dev/sd[a-z] ; do echo $x ; smartctl -i -A $x ; done
Paste this inline in your next reply, with word-wrap turned off.
Then we might be able to say what's next.
> # dmesg
> "http://paste.debian.net/335374/"
External resources generally don't last as long as this list's archives.
In the future, please paste inline anything important. This list
accepts messages up to ~100k.
Phil
^ permalink raw reply
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
From: Wols Lists @ 2015-11-23 16:08 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid@vger.kernel.org
In-Reply-To: <3290501448294245@web16j.yandex.ru>
On 23/11/15 15:57, Semyon Enskiy wrote:
>> Something I couldn't work out from your post - what are your drives? Are
>> they proper raid drives, or desktop drives? Is SCTERC (think that's what
>> it's called) enabled and functioning?
>>
>> I won't be able to help beyond that, but the experts will want that info...
>>
>> smartctl -x
>>
>> Cheers,
>> Wol
>
> Thanks for your reply, Wol.
>
> As written in first message of this thread, all disks are 4TB WD Red,
> "WDC WD40EFRX-68WT0N0", in-kernel timeout for them is 30s.
>
Sorry, missed that. That's good.
Cheers,
Wol
^ permalink raw reply
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
From: Semyon Enskiy @ 2015-11-23 15:57 UTC (permalink / raw)
To: Wols Lists, linux-raid@vger.kernel.org
In-Reply-To: <5653351C.7070302@youngman.org.uk>
> Something I couldn't work out from your post - what are your drives? Are
> they proper raid drives, or desktop drives? Is SCTERC (think that's what
> it's called) enabled and functioning?
>
> I won't be able to help beyond that, but the experts will want that info...
>
> smartctl -x
>
> Cheers,
> Wol
Thanks for your reply, Wol.
As written in first message of this thread, all disks are 4TB WD Red,
"WDC WD40EFRX-68WT0N0", in-kernel timeout for them is 30s.
^ permalink raw reply
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
From: Wols Lists @ 2015-11-23 15:47 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid@vger.kernel.org
In-Reply-To: <2542241448292944@web16h.yandex.ru>
On 23/11/15 15:35, Semyon Enskiy wrote:
> /
> Seems, that I should write data back to source block device to do it
> consistent and able to recreate array with one corrupted device, reboot,
> recreate array, resync faulty device.
> /
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Something I couldn't work out from your post - what are your drives? Are
they proper raid drives, or desktop drives? Is SCTERC (think that's what
it's called) enabled and functioning?
I won't be able to help beyond that, but the experts will want that info...
smartctl -x
Cheers,
Wol
^ permalink raw reply
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
From: Semyon Enskiy @ 2015-11-23 15:35 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <189791448292518@web28m.yandex.ru>
/
Seems, that I should write data back to source block device to do it
consistent and able to recreate array with one corrupted device, reboot,
recreate array, resync faulty device.
/
^ permalink raw reply
* Recovering RAID5 with 2, actually 1, faulty disks.
From: Semyon Enskiy @ 2015-11-23 15:28 UTC (permalink / raw)
To: linux-raid
Hi all,
I am sorry for bad English, it is not my primary language.
Does solution, described here -
"http://marc.info/?l=linux-raid&m=144659416216285&w=2", matches for mine
issue?
Please, read issue description below, I am ready to provide additional info.
RAID1 (md1; boot), RAID10 (md2; swap, root) and
RAID5 (md3; LVM -> ext4 -> data)
are placed on 10 disks, each of them is 4TB WD Red with 3 partitions on it.
All disks model is WDC WD40EFRX-68WT0N0.
sd?1 partitions belongs to md1, sd?2 belongs to md2, sd?3 belongs to md3.
sdj caught IO error and was reinitialized under new minor and in-/dev/ IDs,
in fact partitions on it was dropped from arrays.
sdk1 and sdk2, previous sdj1 and sdj2 accordingly, was readded to their
arrays by me, but md3 stay leave sdj3 as member of array, I can't do
anything with it, only receiving errors like that:
# mdadm /dev/md3 -r detached
mdadm: Cannot find 8:147: No such file or directory
so sdk3 can't be readded.
So I've managed to grow md3 array with command like following, do not
remember why and strictly command:
# mdadm --grow --raid-devices=11 -add /dev/md3 /dev/sdk3
array start to reshaping, but caught second error and reshaping process
stalled with 1536 bytes transferred. Then was executed near two useless
commands, like making sdk3 spare device.
Now md3 consists 11 devices, was 10, has 2 faulty devices, actually 1 - old
sdj3 and sdj3 under new ID - sdk3, which is slightly corrupted by writing
1536 bytes.
Seems, that I should and write data back to source block device to do it
consistent and able to reboot, recreate array with one corrupted device and
resync it.
Which data was transferred at reshaping - raw data from first block device
in the array, including mdadm metadata (/dev/sd?3), or data from array's
summary block device (/dev/md3)?
After this all happened, host was not rebooted, stay online, FS on md3 array
unavailable, smartctl reports (-H argument), that all disks are healthy.
# uname -sr
Linux 3.13.0-52-generic
# strace -p `pgrep -o mdadm` -f
Process 1250 attached
select(5, NULL, NULL, [4], {181, 601546}
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid10 sdk2[12] sdg2[1] sdh2[10] sdi2[5] sdf2[9] sde2[8] sdd2[6] sdc2[4] sdb2[11] sda2[0]
35653120 blocks super 1.2 512K chunks 2 near-copies [10/10] [UUUUUUUUUU]
bitmap: 1/1 pages [4KB], 65536KB chunk
md1 : active raid1 sdk1[9] sdg1[6] sdh1[7] sdi1[8] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
203712 blocks [10/10] [UUUUUUUUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md3 : active raid5 sdk3[14](S) sdg3[6] sdh3[11] sdj3[13](F) sdi3[10] sde3[4] sdf3[5] sdd3[3] sdc3[2] sdb3[12] sda3[0]
35095924224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [11/10] [UUUUUUUU_U_]
[>....................] reshape = 0.0% (1536/3899547136) finish=10215829552.5min speed=0K/sec
bitmap: 0/15 pages [0KB], 131072KB chunk
unused devices: <none>
# blkid /dev/sda3 /dev/sdk3
/dev/sda3: UUID="9bdc0939-4838-89af-b67c-199b2aaecb4e" UUID_SUB="fa7c541d-a25b-abd8-14ff-e1c0c6c1578c" LABEL="ubuntu:3" TYPE="linux_raid_member"
/dev/sdk3: UUID="9bdc0939-4838-89af-b67c-199b2aaecb4e" UUID_SUB="c0155214-a9c8-e378-d368-b6f6735048a8" LABEL="ubuntu:3" TYPE="linux_raid_member"
# mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 3899547136 (3718.90 GiB 3993.14 GB)
Raid Devices : 11
Total Devices : 11
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Nov 6 16:53:06 2015
State : active, FAILED, reshaping
Active Devices : 9
Working Devices : 10
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Reshape Status : 0% complete
Delta Devices : 1, (10->11)
Name : ubuntu:3
UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Events : 21586
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
12 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
3 8 51 3 active sync /dev/sdd3
4 8 67 4 active sync /dev/sde3
5 8 83 5 active sync /dev/sdf3
6 8 99 6 active sync /dev/sdg3
11 8 115 7 active sync /dev/sdh3
16 0 0 16 removed
10 8 131 9 active sync /dev/sdi3
20 0 0 20 removed
13 8 147 - faulty
14 8 163 - spare /dev/sdk3
# "mdadm --examine" output
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fa7c541d:a25babd8:14ffe1c0:c6c1578c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : ac275763 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 70afdaad:b0979fb1:1b99ef18:dd58cc3a
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 1c2bf759 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 345b0212:6c3a0ae8:d8ed9273:ecc53573
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 49c30753 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a4b8b425:b0307fe5:9eb8e3ac:ae985311
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 3258f890 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4490843e:d2c5c2e7:d222fa0a:f98b277c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 1656c2d2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1d41056f:88149b1a:48e939e5:8c775e23
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : fb26746a - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 375d85e2:9077dd68:bc1327af:8fee4112
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 75b99505 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 67cd2bfa:affa578c:e0c1286a:bd4d2627
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 80c895f3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a5f2cea8:a89c88a0:0dd3900c:624f0491
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 4fda6fb3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952a55f2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952a55f2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
# grep'ed smartctl output #1,
/dev/sda
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdb
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdc
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdd
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sde
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdf
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdg
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdh
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdi
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdk
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
# grep 'sd.3' /proc/partitions
8 3 3899678535 sda3
8 19 3899678535 sdb3
8 35 3899678535 sdc3
8 51 3899678535 sdd3
8 67 3899678535 sde3
8 83 3899678535 sdf3
8 99 3899678535 sdg3
8 115 3899678535 sdh3
8 131 3899678535 sdi3
8 163 3899678535 sdk3
# grep'ed smartctl output #2
/dev/sda
9 Power_On_Hours -O--CK 082 082 000 - 13233
/dev/sdb
9 Power_On_Hours -O--CK 085 085 000 - 11051
/dev/sdc
9 Power_On_Hours -O--CK 085 085 000 - 11379
/dev/sdd
9 Power_On_Hours -O--CK 085 085 000 - 11403
/dev/sde
9 Power_On_Hours -O--CK 081 081 000 - 14233
/dev/sdf
9 Power_On_Hours -O--CK 085 085 000 - 11357
/dev/sdg
9 Power_On_Hours -O--CK 085 085 000 - 11382
/dev/sdh
9 Power_On_Hours -O--CK 085 085 000 - 11097
/dev/sdi
9 Power_On_Hours -O--CK 082 082 000 - 13256
/dev/sdk
9 Power_On_Hours -O--CK 089 089 000 - 8413
# dmesg
"http://paste.debian.net/335374/"
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] md: avoid warning for 32-bit sector_t
From: Arnd Bergmann @ 2015-11-23 13:33 UTC (permalink / raw)
To: linux-raid, Neil Brown; +Cc: linux-kernel, linux-arm-kernel
When CONFIG_LBDAF is not set, sector_t is only 32-bits wide, which
means we cannot have devices with more than 2TB, and the code that
is trying to handle compatibility support for large devices in
md version 0.90 is meaningless but also causes a compile-time warning:
drivers/md/md.c: In function 'super_90_load':
drivers/md/md.c:1029:19: warning: large integer implicitly truncated to unsigned type [-Woverflow]
drivers/md/md.c: In function 'super_90_rdev_size_change':
drivers/md/md.c:1323:17: warning: large integer implicitly truncated to unsigned type [-Woverflow]
This adds a check for CONFIG_LBDAF to avoid even getting into this
code path, and also adds an explicit cast to let the compiler know
it doesn't have to warn about the truncation.
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
---
Noticed on ARM randconfig builds with recent gcc versions.
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 0b48c5d7c489..ee9a7ab44f32 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -1025,8 +1025,9 @@ static int super_90_load(struct md_rdev *rdev, struct md_rdev *refdev, int minor
* (not needed for Linear and RAID0 as metadata doesn't
* record this size)
*/
- if (rdev->sectors >= (2ULL << 32) && sb->level >= 1)
- rdev->sectors = (2ULL << 32) - 2;
+ if (IS_ENABLED(CONFIG_LBDAF) && (u64)rdev->sectors >= (2ULL << 32) &&
+ sb->level >= 1)
+ rdev->sectors = (sector_t)(2ULL << 32) - 2;
if (rdev->sectors < ((sector_t)sb->size) * 2 && sb->level >= 1)
/* "this cannot possibly happen" ... */
@@ -1319,8 +1320,9 @@ super_90_rdev_size_change(struct md_rdev *rdev, sector_t num_sectors)
/* Limit to 4TB as metadata cannot record more than that.
* 4TB == 2^32 KB, or 2*2^32 sectors.
*/
- if (num_sectors >= (2ULL << 32) && rdev->mddev->level >= 1)
- num_sectors = (2ULL << 32) - 2;
+ if (IS_ENABLED(CONFIG_LBDAF) && (u64)num_sectors >= (2ULL << 32) &&
+ rdev->mddev->level >= 1)
+ num_sectors = (sector_t)(2ULL << 32) - 2;
md_super_write(rdev->mddev, rdev, rdev->sb_start, rdev->sb_size,
rdev->sb_page);
md_super_wait(rdev->mddev);
^ permalink raw reply related
* Re: raid10 resync hangs in 4.2.6, 4.3
From: Andre Tomt @ 2015-11-23 9:52 UTC (permalink / raw)
To: Artur Paszkiewicz, linux-raid, John Stoffel
In-Reply-To: <5652DA63.9080309@intel.com>
On 23. nov. 2015 10:20, Artur Paszkiewicz wrote:
> Hi Andre,
>
> I've recently sent a patch for a similar problem, I suspect the root
> cause is the same here. Please check out this patch:
>
> http://marc.info/?l=linux-raid&m=144665464232126&w=2
Indeed this is where the bisect just landed me. Going to test the fix
out later today.
Thanks!
And also thanks to Roy Sigurd Karlsbakk for confirming I had not gone
completely off the rails :-)
> atomt@mental:~/build/linux-upstream$ git bisect bad
> c31df25f20e35add6a453328c61eca15434fae18 is the first bad commit
> commit c31df25f20e35add6a453328c61eca15434fae18
> Author: Kent Overstreet <kent.overstreet@gmail.com>
> Date: Wed May 6 23:34:20 2015 -0700
>
> md/raid10: make sync_request_write() call bio_copy_data()
>
> Refactor sync_request_write() of md/raid10 to use bio_copy_data()
> instead of open coding bio_vec iterations.
>
> Cc: Christoph Hellwig <hch@infradead.org>
> Cc: Neil Brown <neilb@suse.de>
> Cc: linux-raid@vger.kernel.org
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Acked-by: NeilBrown <neilb@suse.de>
> Signed-off-by: Kent Overstreet <kent.overstreet@gmail.com>
> [dpark: add more description in commit message]
> Signed-off-by: Dongsu Park <dpark@posteo.net>
> Signed-off-by: Ming Lin <mlin@kernel.org>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> :040000 040000 de7fe22262d763cd544d0dbc53039926e5c9a6f4 8cf70fec46bcc652fd3756eb6edd1a746b41a4cd M drivers
^ permalink raw reply
* Re: raid10 resync hangs in 4.2.6, 4.3
From: Artur Paszkiewicz @ 2015-11-23 9:20 UTC (permalink / raw)
To: Andre Tomt, linux-raid
In-Reply-To: <564F6304.3040502@tomt.net>
On 11/20/2015 07:14 PM, Andre Tomt wrote:
> [resend with compressed attachments, first may have gotten eaten by a grue]
>
> Hi
>
> I'm seeing hangs with RAID10 resyncs on my system. RAID5/6 recovery on the same drive set works without any problems. BTRFS RAID6 is problem free on a different set of (very busy) drives on the same controllers as well.
>
> It happens shortly after array creation, from seconds to a couple minutes in.
>
> wchan for md0_resync kernel thread shows it sitting in raise_barrier() forever, while md0_raid10 keeps a CPU core 100% busy (but shows no wchan), and no resyncing or I/O to the array is getting done anymore.
>
> After a short while kernel starts spitting out rcu_sched self-detected stall on CPU warnings, and other rcu use starts getting iffy (I think).
>
> I/O directly to the RAID member disk (below md layer, eg /dev/sdX directly) continues to work after the hang, and there are no I/O errors in the kernel log.
>
> The array is a 8 drive array spread over 3 HBAs, created with:
> mdadm --create /dev/md0 --level=10 --chunk=128 --bitmap=none --raid-devices=8 /dev/sda1 /dev/sdg1 /dev/sdl1 /dev/sdm1 /dev/sdi1 /dev/sdj1 /dev/sdp1 /dev/sds1
>
> The HBAs are LSI SAS2008 in IT mode (mpt2sas driver), oldish 2TB SATA drives. Dual socket Xeon E5 v3 system with both sockets populated.
>
> This happens on at least 4.2.6 and 4.3
> I'm going to test some earlier kernels.
>
> attached some more info.
>
> md0_resync stack:
>> root@mental:~# cat /proc/1663/stack
>> [<ffffffffc042dd02>] raise_barrier+0x11b/0x14d [raid10]
>> [<ffffffffc0432830>] sync_request+0x193/0x14fc [raid10]
>> [<ffffffffc0559ac6>] md_do_sync+0x7d2/0xd78 [md_mod]
>> [<ffffffffc0556df9>] md_thread+0x12f/0x145 [md_mod]
>> [<ffffffff9d061db2>] kthread+0xcd/0xd5
>> [<ffffffff9d3b7a8f>] ret_from_fork+0x3f/0x70
>> [<ffffffffffffffff>] 0xffffffffffffffff
>
> md0_raid10 stack:
>> root@mental:~# cat /proc/1662/stack
>> [<ffffffffffffffff>] 0xffffffffffffffff
>
> cat stack trying to read /dev/md0 after hang:
>> root@mental:~# cat /proc/1737/stack
>> [<ffffffffc042de2b>] wait_barrier+0xd8/0x118 [raid10]
>> [<ffffffffc042f83c>] __make_request+0x3e/0xb17 [raid10]
>> [<ffffffffc0430399>] make_request+0x84/0xdc [raid10]
>> [<ffffffffc0557aab>] md_make_request+0xf6/0x1cc [md_mod]
>> [<ffffffff9d1a369a>] generic_make_request+0x97/0xd6
>> [<ffffffff9d1a37d1>] submit_bio+0xf8/0x140
>> [<ffffffff9d140e71>] mpage_bio_submit+0x25/0x2c
>> [<ffffffff9d141499>] mpage_readpages+0x10e/0x11f
>> [<ffffffff9d13c76c>] blkdev_readpages+0x18/0x1a
>> [<ffffffff9d0d09e4>] __do_page_cache_readahead+0x13c/0x1e0
>> [<ffffffff9d0d0c67>] ondemand_readahead+0x1df/0x1f2
>> [<ffffffff9d0d0da0>] page_cache_sync_readahead+0x38/0x3a
>> [<ffffffff9d0c70b3>] generic_file_read_iter+0x184/0x50b
>> [<ffffffff9d13c8f7>] blkdev_read_iter+0x33/0x38
>> [<ffffffff9d1132fd>] __vfs_read+0x8d/0xb1
>> [<ffffffff9d113820>] vfs_read+0x95/0x120
>> [<ffffffff9d11402d>] SyS_read+0x49/0x84
>> [<ffffffff9d3b772e>] entry_SYSCALL_64_fastpath+0x12/0x71
>> [<ffffffffffffffff>] 0xffffffffffffffff
>
>
> First OOPS (more in dmesg.txt)
>> [ 150.183473] md0: detected capacity change from 0 to 8001054310400
>> [ 150.183647] md: resync of RAID array md0
>> [ 150.183652] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
>> [ 150.183654] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
>> [ 150.183678] md: using 128k window, over a total of 7813529600k.
>> [ 233.068271] INFO: rcu_sched self-detected stall on CPU
>> [ 233.068308] 5: (17999 ticks this GP) idle=695/140000000000001/0 softirq=1235/1235 fqs=8999
>> [ 233.068335] (t=18000 jiffies g=935 c=934 q=37)
>> [ 233.068354] Task dump for CPU 5:
>> [ 233.068356] md0_raid10 R running task 0 1662 2 0x00000008
>> [ 233.068358] 0000000000000000 ffff88103fca3de0 ffffffff9d069f07 0000000000000005
>> [ 233.068360] ffffffff9d63d0c0 ffff88103fca3df8 ffffffff9d06beb9 ffffffff9d63d0c0
>> [ 233.068361] ffff88103fca3e28 ffffffff9d08c836 ffffffff9d63d0c0 ffff88103fcb4e80
>> [ 233.068363] Call Trace:
>> [ 233.068364] <IRQ> [<ffffffff9d069f07>] sched_show_task+0xb9/0xbe
>> [ 233.068372] [<ffffffff9d06beb9>] dump_cpu_task+0x32/0x35
>> [ 233.068375] [<ffffffff9d08c836>] rcu_dump_cpu_stacks+0x71/0x8c
>> [ 233.068378] [<ffffffff9d08f32c>] rcu_check_callbacks+0x20f/0x5a3
>> [ 233.068382] [<ffffffff9d0b72da>] ? acct_account_cputime+0x17/0x19
>> [ 233.068384] [<ffffffff9d0911a2>] update_process_times+0x2a/0x4f
>> [ 233.068387] [<ffffffff9d09cd55>] tick_sched_handle.isra.5+0x31/0x33
>> [ 233.068388] [<ffffffff9d09cd8f>] tick_sched_timer+0x38/0x60
>> [ 233.068390] [<ffffffff9d0917e1>] __hrtimer_run_queues+0xa1/0x10c
>> [ 233.068392] [<ffffffff9d091c52>] hrtimer_interrupt+0xa0/0x172
>> [ 233.068395] [<ffffffff9d0367a4>] smp_trace_apic_timer_interrupt+0x76/0x88
>> [ 233.068397] [<ffffffff9d0367bf>] smp_apic_timer_interrupt+0x9/0xb
>> [ 233.068400] [<ffffffff9d3b8402>] apic_timer_interrupt+0x82/0x90
>> [ 233.068401] <EOI> [<ffffffff9d19e9aa>] ? bio_copy_data+0xce/0x2af
>> [ 233.068410] [<ffffffffc04320e5>] raid10d+0x974/0xf2c [raid10]
>> [ 233.068417] [<ffffffffc0556df9>] md_thread+0x12f/0x145 [md_mod]
>> [ 233.068421] [<ffffffffc0556df9>] ? md_thread+0x12f/0x145 [md_mod]
>> [ 233.068424] [<ffffffff9d07ad2e>] ? wait_woken+0x6d/0x6d
>> [ 233.068428] [<ffffffffc0556cca>] ? md_wait_for_blocked_rdev+0x102/0x102 [md_mod]
>> [ 233.068431] [<ffffffff9d061db2>] kthread+0xcd/0xd5
>> [ 233.068434] [<ffffffff9d061ce5>] ? kthread_worker_fn+0x13f/0x13f
>> [ 233.068436] [<ffffffff9d3b7a8f>] ret_from_fork+0x3f/0x70
>> [ 233.068438] [<ffffffff9d061ce5>] ? kthread_worker_fn+0x13f/0x13f
>
Hi Andre,
I've recently sent a patch for a similar problem, I suspect the root
cause is the same here. Please check out this patch:
http://marc.info/?l=linux-raid&m=144665464232126&w=2
Artur
^ permalink raw reply
* Re: raid10 resync hangs in 4.2.6, 4.3
From: John Stoffel @ 2015-11-22 19:35 UTC (permalink / raw)
To: Andre Tomt; +Cc: John Stoffel, linux-raid
In-Reply-To: <564FC890.3040507@tomt.net>
>>>>> "Andre" == Andre Tomt <andre@tomt.net> writes:
Andre> On 20. nov. 2015 21:30, John Stoffel wrote:
>>
>> Andre,
>> I don't have time myself to test this, but since I'm running 4.2.6
>> myself with triple mirror RAID1 arrays, I'm slightly worried now. But
>> can you make this failure happen with loop-back mounts by any chance?
>> That might help narrow down if it's an MD layer, block layer or
>> controller issue.
Andre> I've failed to reproduce it using loop devices, but I've found
Andre> out the problem appeared somewhere between 4.1 and 4.2.
Andre> 4.1.13 is stable, 4.2.0 to 4.2.6 and 4.3 is not.
Andre> Guess I'm starting a bisect now ;-)
Good luck! I just got back from a weekend of camping and digging out
from email and tent cleanup and other tasks.
^ permalink raw reply
* Re: [PATCH] mdadm: check bitmap first when reshape raid1 to raid0
From: Xiao Ni @ 2015-11-21 1:47 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1446510328-22917-1-git-send-email-xni@redhat.com>
Hi Neil
Could you review this patch? For the bitmap part, I don't remove it in
Grow_reshape.
I described the reason in the patch.
Regards
Xiao
On 11/03/2015 08:25 AM, Xiao Ni wrote:
> One raid1 with bitmap is composed by 4 disks. It'll fail when rashape
> to raid0 and lose 3 disks. It should check bitmap first when reshape
> raid1 to raid0.
>
> And need to free subarray, close cfd in Grow_reshape.
>
> It can't remove bitmap in Grow_reshape, because it's already frozen. I
> think it should not unfreeze in the process of the function. So I just
> test the bitmap.
>
> Signed-off-by: Xiao Ni <xni@redhat.com>
> ---
> Grow.c | 45 ++++++++++++++++++++++++++++-----------------
> mdadm.c | 16 ++++++++++++----
> 2 files changed, 40 insertions(+), 21 deletions(-)
>
> diff --git a/Grow.c b/Grow.c
> index 80d7b22..c4ea2a5 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -1600,15 +1600,14 @@ int Grow_reshape(char *devname, int fd,
> cfd = open_dev_excl(st->container_devnm);
> } else {
> container = st->devnm;
> - close(fd);
> cfd = open_dev_excl(st->devnm);
> fd = cfd;
> }
> if (cfd < 0) {
> pr_err("Unable to open container for %s\n",
> devname);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> rv = st->ss->load_container(st, cfd, NULL);
> @@ -1616,8 +1615,8 @@ int Grow_reshape(char *devname, int fd,
> if (rv) {
> pr_err("Cannot read superblock for %s\n",
> devname);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> /* check if operation is supported for metadata handler */
> @@ -1641,8 +1640,8 @@ int Grow_reshape(char *devname, int fd,
> pr_err("cannot reshape arrays in container with unsupported metadata: %s(%s)\n",
> devname, container);
> sysfs_free(cc);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
> }
> sysfs_free(cc);
> @@ -1662,7 +1661,8 @@ int Grow_reshape(char *devname, int fd,
> s->raiddisks - array.raid_disks,
> s->raiddisks - array.raid_disks == 1 ? "" : "s",
> array.spare_disks + added_disks);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> sra = sysfs_read(fd, NULL, GET_LEVEL | GET_DISKS | GET_DEVS
> @@ -1675,17 +1675,18 @@ int Grow_reshape(char *devname, int fd,
> } else {
> pr_err("failed to read sysfs parameters for %s\n",
> devname);
> - return 1;
> + rv = 1;
> + goto release;
> }
> frozen = freeze(st);
> if (frozen < -1) {
> /* freeze() already spewed the reason */
> - sysfs_free(sra);
> - return 1;
> + rv = 1;
> + goto release;
> } else if (frozen < 0) {
> pr_err("%s is performing resync/recovery and cannot be reshaped\n", devname);
> - sysfs_free(sra);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> /* ========= set size =============== */
> @@ -1898,11 +1899,16 @@ size_change_error:
> array.layout == ((1 << 8) + 2) && !(array.raid_disks & 1)) ||
> (s->level == 0 && array.level == 1 && sra)) {
> int err;
> +
> + if (array.state & (1<<MD_SB_BITMAP_PRESENT)) {
> + pr_err("Bitmap must be removed before level can be changed\n");
> + rv = 1;
> + goto release;
> + }
> +
> err = remove_disks_for_takeover(st, sra, array.layout);
> if (err) {
> - dprintf("Array cannot be reshaped\n");
> - if (cfd > -1)
> - close(cfd);
> + pr_err("Array cannot be reshaped\n");
> rv = 1;
> goto release;
> }
> @@ -2078,9 +2084,14 @@ size_change_error:
> frozen = 0;
> }
> release:
> - sysfs_free(sra);
> if (frozen > 0)
> unfreeze(st);
> + if (sra)
> + sysfs_free(sra);
> + if (cfd > -1)
> + close(cfd);
> + if (subarray)
> + free(subarray);
> return rv;
> }
>
> diff --git a/mdadm.c b/mdadm.c
> index f56a8cf..95db2a0 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -1299,7 +1299,8 @@ int main(int argc, char *argv[])
> pr_err("'1' is an unusual number of drives for an array, so it is probably\n"
> " a mistake. If you really mean it you will need to specify --force before\n"
> " setting the number of drives.\n");
> - exit(2);
> + rv = 2;
> + goto release;
> }
> }
>
> @@ -1326,13 +1327,15 @@ int main(int argc, char *argv[])
> rv = get_cluster_name(&c.homecluster);
> if (rv) {
> pr_err("The md can't get cluster name\n");
> - exit(1);
> + rv = 1;
> + goto release;
> }
> }
>
> if (c.backup_file && data_offset != INVALID_SECTORS) {
> pr_err("--backup-file and --data-offset are incompatible\n");
> - exit(2);
> + rv = 2;
> + goto release;
> }
>
> if ((mode == MISC && devmode == 'E')
> @@ -1340,7 +1343,8 @@ int main(int argc, char *argv[])
> /* Anyone may try this */;
> else if (geteuid() != 0) {
> pr_err("must be super-user to perform this action\n");
> - exit(1);
> + rv = 1;
> + goto release;
> }
>
> ident.autof = c.autof;
> @@ -1640,6 +1644,10 @@ int main(int argc, char *argv[])
> autodetect();
> break;
> }
> +
> +release:
> + if (mdfd > -1)
> + close(mdfd);
> exit(rv);
> }
>
^ permalink raw reply
* Re: raid10 resync hangs in 4.2.6, 4.3
From: Andre Tomt @ 2015-11-21 1:27 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <22095.33515.989734.113452@quad.stoffel.home>
On 20. nov. 2015 21:30, John Stoffel wrote:
>
> Andre,
> I don't have time myself to test this, but since I'm running 4.2.6
> myself with triple mirror RAID1 arrays, I'm slightly worried now. But
> can you make this failure happen with loop-back mounts by any chance?
> That might help narrow down if it's an MD layer, block layer or
> controller issue.
I've failed to reproduce it using loop devices, but I've found out the
problem appeared somewhere between 4.1 and 4.2.
4.1.13 is stable, 4.2.0 to 4.2.6 and 4.3 is not.
Guess I'm starting a bisect now ;-)
^ permalink raw reply
* [PATCH 3/3] md: convert to use the generic badblocks code
From: Vishal Verma @ 2015-11-21 0:49 UTC (permalink / raw)
To: linux-nvdimm
Cc: Vishal Verma, linux-block, linux-raid, linux-scsi, Jens Axboe,
NeilBrown, Jeff Moyer
In-Reply-To: <1448066960-20119-1-git-send-email-vishal.l.verma@intel.com>
Retain badblocks as part of rdev, but use the accessor functions from
include/linux/badblocks for all manipulation.
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
---
drivers/md/md.c | 495 +++-----------------------------------------------------
drivers/md/md.h | 31 +---
2 files changed, 21 insertions(+), 505 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index c702de1..82994d7 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -34,6 +34,7 @@
#include <linux/kthread.h>
#include <linux/blkdev.h>
+#include <linux/badblocks.h>
#include <linux/sysctl.h>
#include <linux/seq_file.h>
#include <linux/fs.h>
@@ -1358,8 +1359,6 @@ static __le32 calc_sb_1_csum(struct mdp_superblock_1 *sb)
return cpu_to_le32(csum);
}
-static int md_set_badblocks(struct badblocks *bb, sector_t s, int sectors,
- int acknowledged);
static int super_1_load(struct md_rdev *rdev, struct md_rdev *refdev, int minor_version)
{
struct mdp_superblock_1 *sb;
@@ -1484,7 +1483,7 @@ static int super_1_load(struct md_rdev *rdev, struct md_rdev *refdev, int minor_
count <<= sb->bblog_shift;
if (bb + 1 == 0)
break;
- if (md_set_badblocks(&rdev->badblocks,
+ if (badblocks_set(&rdev->badblocks,
sector, count, 1) == 0)
return -EINVAL;
}
@@ -2226,7 +2225,7 @@ repeat:
rdev_for_each(rdev, mddev) {
if (rdev->badblocks.changed) {
rdev->badblocks.changed = 0;
- md_ack_all_badblocks(&rdev->badblocks);
+ ack_all_badblocks(&rdev->badblocks);
md_error(mddev, rdev);
}
clear_bit(Blocked, &rdev->flags);
@@ -2352,7 +2351,7 @@ repeat:
clear_bit(Blocked, &rdev->flags);
if (any_badblocks_changed)
- md_ack_all_badblocks(&rdev->badblocks);
+ ack_all_badblocks(&rdev->badblocks);
clear_bit(BlockedBadBlocks, &rdev->flags);
wake_up(&rdev->blocked_wait);
}
@@ -2944,11 +2943,17 @@ static ssize_t recovery_start_store(struct md_rdev *rdev, const char *buf, size_
static struct rdev_sysfs_entry rdev_recovery_start =
__ATTR(recovery_start, S_IRUGO|S_IWUSR, recovery_start_show, recovery_start_store);
-static ssize_t
-badblocks_show(struct badblocks *bb, char *page, int unack);
-static ssize_t
-badblocks_store(struct badblocks *bb, const char *page, size_t len, int unack);
-
+/* sysfs access to bad-blocks list.
+ * We present two files.
+ * 'bad-blocks' lists sector numbers and lengths of ranges that
+ * are recorded as bad. The list is truncated to fit within
+ * the one-page limit of sysfs.
+ * Writing "sector length" to this file adds an acknowledged
+ * bad block list.
+ * 'unacknowledged-bad-blocks' lists bad blocks that have not yet
+ * been acknowledged. Writing to this file adds bad blocks
+ * without acknowledging them. This is largely for testing.
+ */
static ssize_t bb_show(struct md_rdev *rdev, char *page)
{
return badblocks_show(&rdev->badblocks, page, 0);
@@ -8348,253 +8353,7 @@ void md_finish_reshape(struct mddev *mddev)
}
EXPORT_SYMBOL(md_finish_reshape);
-/* Bad block management.
- * We can record which blocks on each device are 'bad' and so just
- * fail those blocks, or that stripe, rather than the whole device.
- * Entries in the bad-block table are 64bits wide. This comprises:
- * Length of bad-range, in sectors: 0-511 for lengths 1-512
- * Start of bad-range, sector offset, 54 bits (allows 8 exbibytes)
- * A 'shift' can be set so that larger blocks are tracked and
- * consequently larger devices can be covered.
- * 'Acknowledged' flag - 1 bit. - the most significant bit.
- *
- * Locking of the bad-block table uses a seqlock so md_is_badblock
- * might need to retry if it is very unlucky.
- * We will sometimes want to check for bad blocks in a bi_end_io function,
- * so we use the write_seqlock_irq variant.
- *
- * When looking for a bad block we specify a range and want to
- * know if any block in the range is bad. So we binary-search
- * to the last range that starts at-or-before the given endpoint,
- * (or "before the sector after the target range")
- * then see if it ends after the given start.
- * We return
- * 0 if there are no known bad blocks in the range
- * 1 if there are known bad block which are all acknowledged
- * -1 if there are bad blocks which have not yet been acknowledged in metadata.
- * plus the start/length of the first bad section we overlap.
- */
-int md_is_badblock(struct badblocks *bb, sector_t s, int sectors,
- sector_t *first_bad, int *bad_sectors)
-{
- int hi;
- int lo;
- u64 *p = bb->page;
- int rv;
- sector_t target = s + sectors;
- unsigned seq;
-
- if (bb->shift > 0) {
- /* round the start down, and the end up */
- s >>= bb->shift;
- target += (1<<bb->shift) - 1;
- target >>= bb->shift;
- sectors = target - s;
- }
- /* 'target' is now the first block after the bad range */
-
-retry:
- seq = read_seqbegin(&bb->lock);
- lo = 0;
- rv = 0;
- hi = bb->count;
-
- /* Binary search between lo and hi for 'target'
- * i.e. for the last range that starts before 'target'
- */
- /* INVARIANT: ranges before 'lo' and at-or-after 'hi'
- * are known not to be the last range before target.
- * VARIANT: hi-lo is the number of possible
- * ranges, and decreases until it reaches 1
- */
- while (hi - lo > 1) {
- int mid = (lo + hi) / 2;
- sector_t a = BB_OFFSET(p[mid]);
- if (a < target)
- /* This could still be the one, earlier ranges
- * could not. */
- lo = mid;
- else
- /* This and later ranges are definitely out. */
- hi = mid;
- }
- /* 'lo' might be the last that started before target, but 'hi' isn't */
- if (hi > lo) {
- /* need to check all range that end after 's' to see if
- * any are unacknowledged.
- */
- while (lo >= 0 &&
- BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > s) {
- if (BB_OFFSET(p[lo]) < target) {
- /* starts before the end, and finishes after
- * the start, so they must overlap
- */
- if (rv != -1 && BB_ACK(p[lo]))
- rv = 1;
- else
- rv = -1;
- *first_bad = BB_OFFSET(p[lo]);
- *bad_sectors = BB_LEN(p[lo]);
- }
- lo--;
- }
- }
-
- if (read_seqretry(&bb->lock, seq))
- goto retry;
-
- return rv;
-}
-EXPORT_SYMBOL_GPL(md_is_badblock);
-
-/*
- * Add a range of bad blocks to the table.
- * This might extend the table, or might contract it
- * if two adjacent ranges can be merged.
- * We binary-search to find the 'insertion' point, then
- * decide how best to handle it.
- */
-static int md_set_badblocks(struct badblocks *bb, sector_t s, int sectors,
- int acknowledged)
-{
- u64 *p;
- int lo, hi;
- int rv = 1;
- unsigned long flags;
-
- if (bb->shift < 0)
- /* badblocks are disabled */
- return 0;
-
- if (bb->shift) {
- /* round the start down, and the end up */
- sector_t next = s + sectors;
- s >>= bb->shift;
- next += (1<<bb->shift) - 1;
- next >>= bb->shift;
- sectors = next - s;
- }
-
- write_seqlock_irqsave(&bb->lock, flags);
-
- p = bb->page;
- lo = 0;
- hi = bb->count;
- /* Find the last range that starts at-or-before 's' */
- while (hi - lo > 1) {
- int mid = (lo + hi) / 2;
- sector_t a = BB_OFFSET(p[mid]);
- if (a <= s)
- lo = mid;
- else
- hi = mid;
- }
- if (hi > lo && BB_OFFSET(p[lo]) > s)
- hi = lo;
-
- if (hi > lo) {
- /* we found a range that might merge with the start
- * of our new range
- */
- sector_t a = BB_OFFSET(p[lo]);
- sector_t e = a + BB_LEN(p[lo]);
- int ack = BB_ACK(p[lo]);
- if (e >= s) {
- /* Yes, we can merge with a previous range */
- if (s == a && s + sectors >= e)
- /* new range covers old */
- ack = acknowledged;
- else
- ack = ack && acknowledged;
-
- if (e < s + sectors)
- e = s + sectors;
- if (e - a <= BB_MAX_LEN) {
- p[lo] = BB_MAKE(a, e-a, ack);
- s = e;
- } else {
- /* does not all fit in one range,
- * make p[lo] maximal
- */
- if (BB_LEN(p[lo]) != BB_MAX_LEN)
- p[lo] = BB_MAKE(a, BB_MAX_LEN, ack);
- s = a + BB_MAX_LEN;
- }
- sectors = e - s;
- }
- }
- if (sectors && hi < bb->count) {
- /* 'hi' points to the first range that starts after 's'.
- * Maybe we can merge with the start of that range */
- sector_t a = BB_OFFSET(p[hi]);
- sector_t e = a + BB_LEN(p[hi]);
- int ack = BB_ACK(p[hi]);
- if (a <= s + sectors) {
- /* merging is possible */
- if (e <= s + sectors) {
- /* full overlap */
- e = s + sectors;
- ack = acknowledged;
- } else
- ack = ack && acknowledged;
-
- a = s;
- if (e - a <= BB_MAX_LEN) {
- p[hi] = BB_MAKE(a, e-a, ack);
- s = e;
- } else {
- p[hi] = BB_MAKE(a, BB_MAX_LEN, ack);
- s = a + BB_MAX_LEN;
- }
- sectors = e - s;
- lo = hi;
- hi++;
- }
- }
- if (sectors == 0 && hi < bb->count) {
- /* we might be able to combine lo and hi */
- /* Note: 's' is at the end of 'lo' */
- sector_t a = BB_OFFSET(p[hi]);
- int lolen = BB_LEN(p[lo]);
- int hilen = BB_LEN(p[hi]);
- int newlen = lolen + hilen - (s - a);
- if (s >= a && newlen < BB_MAX_LEN) {
- /* yes, we can combine them */
- int ack = BB_ACK(p[lo]) && BB_ACK(p[hi]);
- p[lo] = BB_MAKE(BB_OFFSET(p[lo]), newlen, ack);
- memmove(p + hi, p + hi + 1,
- (bb->count - hi - 1) * 8);
- bb->count--;
- }
- }
- while (sectors) {
- /* didn't merge (it all).
- * Need to add a range just before 'hi' */
- if (bb->count >= MD_MAX_BADBLOCKS) {
- /* No room for more */
- rv = 0;
- break;
- } else {
- int this_sectors = sectors;
- memmove(p + hi + 1, p + hi,
- (bb->count - hi) * 8);
- bb->count++;
-
- if (this_sectors > BB_MAX_LEN)
- this_sectors = BB_MAX_LEN;
- p[hi] = BB_MAKE(s, this_sectors, acknowledged);
- sectors -= this_sectors;
- s += this_sectors;
- }
- }
-
- bb->changed = 1;
- if (!acknowledged)
- bb->unacked_exist = 1;
- write_sequnlock_irqrestore(&bb->lock, flags);
-
- return rv;
-}
+/* Bad block management */
int rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
int is_new)
@@ -8604,8 +8363,7 @@ int rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
s += rdev->new_data_offset;
else
s += rdev->data_offset;
- rv = md_set_badblocks(&rdev->badblocks,
- s, sectors, 0);
+ rv = badblocks_set(&rdev->badblocks, s, sectors, 0);
if (rv) {
/* Make sure they get written out promptly */
sysfs_notify_dirent_safe(rdev->sysfs_state);
@@ -8617,101 +8375,6 @@ int rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
}
EXPORT_SYMBOL_GPL(rdev_set_badblocks);
-/*
- * Remove a range of bad blocks from the table.
- * This may involve extending the table if we spilt a region,
- * but it must not fail. So if the table becomes full, we just
- * drop the remove request.
- */
-static int md_clear_badblocks(struct badblocks *bb, sector_t s, int sectors)
-{
- u64 *p;
- int lo, hi;
- sector_t target = s + sectors;
- int rv = 0;
-
- if (bb->shift > 0) {
- /* When clearing we round the start up and the end down.
- * This should not matter as the shift should align with
- * the block size and no rounding should ever be needed.
- * However it is better the think a block is bad when it
- * isn't than to think a block is not bad when it is.
- */
- s += (1<<bb->shift) - 1;
- s >>= bb->shift;
- target >>= bb->shift;
- sectors = target - s;
- }
-
- write_seqlock_irq(&bb->lock);
-
- p = bb->page;
- lo = 0;
- hi = bb->count;
- /* Find the last range that starts before 'target' */
- while (hi - lo > 1) {
- int mid = (lo + hi) / 2;
- sector_t a = BB_OFFSET(p[mid]);
- if (a < target)
- lo = mid;
- else
- hi = mid;
- }
- if (hi > lo) {
- /* p[lo] is the last range that could overlap the
- * current range. Earlier ranges could also overlap,
- * but only this one can overlap the end of the range.
- */
- if (BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > target) {
- /* Partial overlap, leave the tail of this range */
- int ack = BB_ACK(p[lo]);
- sector_t a = BB_OFFSET(p[lo]);
- sector_t end = a + BB_LEN(p[lo]);
-
- if (a < s) {
- /* we need to split this range */
- if (bb->count >= MD_MAX_BADBLOCKS) {
- rv = -ENOSPC;
- goto out;
- }
- memmove(p+lo+1, p+lo, (bb->count - lo) * 8);
- bb->count++;
- p[lo] = BB_MAKE(a, s-a, ack);
- lo++;
- }
- p[lo] = BB_MAKE(target, end - target, ack);
- /* there is no longer an overlap */
- hi = lo;
- lo--;
- }
- while (lo >= 0 &&
- BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > s) {
- /* This range does overlap */
- if (BB_OFFSET(p[lo]) < s) {
- /* Keep the early parts of this range. */
- int ack = BB_ACK(p[lo]);
- sector_t start = BB_OFFSET(p[lo]);
- p[lo] = BB_MAKE(start, s - start, ack);
- /* now low doesn't overlap, so.. */
- break;
- }
- lo--;
- }
- /* 'lo' is strictly before, 'hi' is strictly after,
- * anything between needs to be discarded
- */
- if (hi - lo > 1) {
- memmove(p+lo+1, p+hi, (bb->count - hi) * 8);
- bb->count -= (hi - lo - 1);
- }
- }
-
- bb->changed = 1;
-out:
- write_sequnlock_irq(&bb->lock);
- return rv;
-}
-
int rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
int is_new)
{
@@ -8719,133 +8382,11 @@ int rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
s += rdev->new_data_offset;
else
s += rdev->data_offset;
- return md_clear_badblocks(&rdev->badblocks,
+ return badblocks_clear(&rdev->badblocks,
s, sectors);
}
EXPORT_SYMBOL_GPL(rdev_clear_badblocks);
-/*
- * Acknowledge all bad blocks in a list.
- * This only succeeds if ->changed is clear. It is used by
- * in-kernel metadata updates
- */
-void md_ack_all_badblocks(struct badblocks *bb)
-{
- if (bb->page == NULL || bb->changed)
- /* no point even trying */
- return;
- write_seqlock_irq(&bb->lock);
-
- if (bb->changed == 0 && bb->unacked_exist) {
- u64 *p = bb->page;
- int i;
- for (i = 0; i < bb->count ; i++) {
- if (!BB_ACK(p[i])) {
- sector_t start = BB_OFFSET(p[i]);
- int len = BB_LEN(p[i]);
- p[i] = BB_MAKE(start, len, 1);
- }
- }
- bb->unacked_exist = 0;
- }
- write_sequnlock_irq(&bb->lock);
-}
-EXPORT_SYMBOL_GPL(md_ack_all_badblocks);
-
-/* sysfs access to bad-blocks list.
- * We present two files.
- * 'bad-blocks' lists sector numbers and lengths of ranges that
- * are recorded as bad. The list is truncated to fit within
- * the one-page limit of sysfs.
- * Writing "sector length" to this file adds an acknowledged
- * bad block list.
- * 'unacknowledged-bad-blocks' lists bad blocks that have not yet
- * been acknowledged. Writing to this file adds bad blocks
- * without acknowledging them. This is largely for testing.
- */
-
-static ssize_t
-badblocks_show(struct badblocks *bb, char *page, int unack)
-{
- size_t len;
- int i;
- u64 *p = bb->page;
- unsigned seq;
-
- if (bb->shift < 0)
- return 0;
-
-retry:
- seq = read_seqbegin(&bb->lock);
-
- len = 0;
- i = 0;
-
- while (len < PAGE_SIZE && i < bb->count) {
- sector_t s = BB_OFFSET(p[i]);
- unsigned int length = BB_LEN(p[i]);
- int ack = BB_ACK(p[i]);
- i++;
-
- if (unack && ack)
- continue;
-
- len += snprintf(page+len, PAGE_SIZE-len, "%llu %u\n",
- (unsigned long long)s << bb->shift,
- length << bb->shift);
- }
- if (unack && len == 0)
- bb->unacked_exist = 0;
-
- if (read_seqretry(&bb->lock, seq))
- goto retry;
-
- return len;
-}
-
-#define DO_DEBUG 1
-
-static ssize_t
-badblocks_store(struct badblocks *bb, const char *page, size_t len, int unack)
-{
- unsigned long long sector;
- int length;
- char newline;
-#ifdef DO_DEBUG
- /* Allow clearing via sysfs *only* for testing/debugging.
- * Normally only a successful write may clear a badblock
- */
- int clear = 0;
- if (page[0] == '-') {
- clear = 1;
- page++;
- }
-#endif /* DO_DEBUG */
-
- switch (sscanf(page, "%llu %d%c", §or, &length, &newline)) {
- case 3:
- if (newline != '\n')
- return -EINVAL;
- case 2:
- if (length <= 0)
- return -EINVAL;
- break;
- default:
- return -EINVAL;
- }
-
-#ifdef DO_DEBUG
- if (clear) {
- md_clear_badblocks(bb, sector, length);
- return len;
- }
-#endif /* DO_DEBUG */
- if (md_set_badblocks(bb, sector, length, !unack))
- return len;
- else
- return -ENOSPC;
-}
-
static int md_notify_reboot(struct notifier_block *this,
unsigned long code, void *x)
{
diff --git a/drivers/md/md.h b/drivers/md/md.h
index ab33957..dd9fab8 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -17,6 +17,7 @@
#include <linux/blkdev.h>
#include <linux/backing-dev.h>
+#include <linux/badblocks.h>
#include <linux/kobject.h>
#include <linux/list.h>
#include <linux/mm.h>
@@ -28,13 +29,6 @@
#define MaxSector (~(sector_t)0)
-/* Bad block numbers are stored sorted in a single page.
- * 64bits is used for each block or extent.
- * 54 bits are sector number, 9 bits are extent size,
- * 1 bit is an 'acknowledged' flag.
- */
-#define MD_MAX_BADBLOCKS (PAGE_SIZE/8)
-
/*
* MD's 'extended' device
*/
@@ -111,22 +105,7 @@ struct md_rdev {
struct kernfs_node *sysfs_state; /* handle for 'state'
* sysfs entry */
- struct badblocks {
- int count; /* count of bad blocks */
- int unacked_exist; /* there probably are unacknowledged
- * bad blocks. This is only cleared
- * when a read discovers none
- */
- int shift; /* shift from sectors to block size
- * a -ve shift means badblocks are
- * disabled.*/
- u64 *page; /* badblock list */
- int changed;
- seqlock_t lock;
-
- sector_t sector;
- sector_t size; /* in sectors */
- } badblocks;
+ struct badblocks badblocks;
};
enum flag_bits {
Faulty, /* device is known to have a fault */
@@ -183,13 +162,11 @@ enum flag_bits {
#define BB_ACK(x) (!!((x) & BB_ACK_MASK))
#define BB_MAKE(a, l, ack) (((a)<<9) | ((l)-1) | ((u64)(!!(ack)) << 63))
-extern int md_is_badblock(struct badblocks *bb, sector_t s, int sectors,
- sector_t *first_bad, int *bad_sectors);
static inline int is_badblock(struct md_rdev *rdev, sector_t s, int sectors,
sector_t *first_bad, int *bad_sectors)
{
if (unlikely(rdev->badblocks.count)) {
- int rv = md_is_badblock(&rdev->badblocks, rdev->data_offset + s,
+ int rv = badblocks_check(&rdev->badblocks, rdev->data_offset + s,
sectors,
first_bad, bad_sectors);
if (rv)
@@ -202,8 +179,6 @@ extern int rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
int is_new);
extern int rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
int is_new);
-extern void md_ack_all_badblocks(struct badblocks *bb);
-
struct md_cluster_info;
struct mddev {
--
2.5.0
^ permalink raw reply related
* [PATCH 2/3] block: Add badblock management for gendisks
From: Vishal Verma @ 2015-11-21 0:49 UTC (permalink / raw)
To: linux-nvdimm
Cc: Vishal Verma, linux-block, linux-raid, linux-scsi, Jens Axboe,
NeilBrown, Jeff Moyer
In-Reply-To: <1448066960-20119-1-git-send-email-vishal.l.verma@intel.com>
NVDIMM devices, which can behave more like DRAM rather than block
devices, may develop bad cache lines, or 'poison'. A block device
exposed by the pmem driver can then consume poison via a read (or
write), and cause a machine check. On platforms without machine
check recovery features, this would mean a crash.
The block device maintaining a runtime list of all known sectors that
have poison can directly avoid this, and also provide a path forward
to enable proper handling/recovery for DAX faults on such a device.
Use the new badblock management interfaces to add a badblocks list to
gendisks.
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
---
block/genhd.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/genhd.h | 6 +++++
2 files changed, 70 insertions(+)
diff --git a/block/genhd.c b/block/genhd.c
index 0c706f3..4209c32 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -20,6 +20,7 @@
#include <linux/idr.h>
#include <linux/log2.h>
#include <linux/pm_runtime.h>
+#include <linux/badblocks.h>
#include "blk.h"
@@ -505,6 +506,20 @@ static int exact_lock(dev_t devt, void *data)
return 0;
}
+static void disk_alloc_badblocks(struct gendisk *disk)
+{
+ disk->bb = kzalloc(sizeof(disk->bb), GFP_KERNEL);
+ if (!disk->bb) {
+ pr_warn("%s: failed to allocate space for badblocks\n",
+ disk->disk_name);
+ return;
+ }
+
+ if (badblocks_init(disk->bb, 1))
+ pr_warn("%s: failed to initialize badblocks\n",
+ disk->disk_name);
+}
+
static void register_disk(struct gendisk *disk)
{
struct device *ddev = disk_to_dev(disk);
@@ -609,6 +624,7 @@ void add_disk(struct gendisk *disk)
disk->first_minor = MINOR(devt);
disk_alloc_events(disk);
+ disk_alloc_badblocks(disk);
/* Register BDI before referencing it from bdev */
bdi = &disk->queue->backing_dev_info;
@@ -657,6 +673,9 @@ void del_gendisk(struct gendisk *disk)
blk_unregister_queue(disk);
blk_unregister_region(disk_devt(disk), disk->minors);
+ badblocks_free(disk->bb);
+ kfree(disk->bb);
+
part_stat_set_all(&disk->part0, 0);
disk->part0.stamp = 0;
@@ -670,6 +689,48 @@ void del_gendisk(struct gendisk *disk)
}
EXPORT_SYMBOL(del_gendisk);
+/*
+ * The gendisk usage of badblocks does not track acknowledgements for
+ * badblocks. We always assume they are acknowledged.
+ */
+int disk_check_badblocks(struct gendisk *disk, sector_t s, int sectors,
+ sector_t *first_bad, int *bad_sectors)
+{
+ return badblocks_check(disk->bb, s, sectors, first_bad, bad_sectors);
+}
+EXPORT_SYMBOL(disk_check_badblocks);
+
+int disk_set_badblocks(struct gendisk *disk, sector_t s, int sectors)
+{
+ return badblocks_set(disk->bb, s, sectors, 1);
+}
+EXPORT_SYMBOL(disk_set_badblocks);
+
+int disk_clear_badblocks(struct gendisk *disk, sector_t s, int sectors)
+{
+ return badblocks_clear(disk->bb, s, sectors);
+}
+EXPORT_SYMBOL(disk_clear_badblocks);
+
+/* sysfs access to bad-blocks list. */
+static ssize_t disk_badblocks_show(struct device *dev,
+ struct device_attribute *attr,
+ char *page)
+{
+ struct gendisk *disk = dev_to_disk(dev);
+
+ return badblocks_show(disk->bb, page, 0);
+}
+
+static ssize_t disk_badblocks_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *page, size_t len)
+{
+ struct gendisk *disk = dev_to_disk(dev);
+
+ return badblocks_store(disk->bb, page, len, 0);
+}
+
/**
* get_gendisk - get partitioning information for a given device
* @devt: device to get partitioning information for
@@ -988,6 +1049,8 @@ static DEVICE_ATTR(discard_alignment, S_IRUGO, disk_discard_alignment_show,
static DEVICE_ATTR(capability, S_IRUGO, disk_capability_show, NULL);
static DEVICE_ATTR(stat, S_IRUGO, part_stat_show, NULL);
static DEVICE_ATTR(inflight, S_IRUGO, part_inflight_show, NULL);
+static DEVICE_ATTR(badblocks, S_IRUGO | S_IWUSR, disk_badblocks_show,
+ disk_badblocks_store);
#ifdef CONFIG_FAIL_MAKE_REQUEST
static struct device_attribute dev_attr_fail =
__ATTR(make-it-fail, S_IRUGO|S_IWUSR, part_fail_show, part_fail_store);
@@ -1009,6 +1072,7 @@ static struct attribute *disk_attrs[] = {
&dev_attr_capability.attr,
&dev_attr_stat.attr,
&dev_attr_inflight.attr,
+ &dev_attr_badblocks.attr,
#ifdef CONFIG_FAIL_MAKE_REQUEST
&dev_attr_fail.attr,
#endif
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index 2adbfa6..5563bde 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -162,6 +162,7 @@ struct disk_part_tbl {
};
struct disk_events;
+struct badblocks;
struct gendisk {
/* major, first_minor and minors are input parameters only,
@@ -201,6 +202,7 @@ struct gendisk {
struct blk_integrity *integrity;
#endif
int node_id;
+ struct badblocks *bb;
};
static inline struct gendisk *part_to_disk(struct hd_struct *part)
@@ -421,6 +423,10 @@ extern void add_disk(struct gendisk *disk);
extern void del_gendisk(struct gendisk *gp);
extern struct gendisk *get_gendisk(dev_t dev, int *partno);
extern struct block_device *bdget_disk(struct gendisk *disk, int partno);
+extern int disk_check_badblocks(struct gendisk *disk, sector_t s, int sectors,
+ sector_t *first_bad, int *bad_sectors);
+extern int disk_set_badblocks(struct gendisk *disk, sector_t s, int sectors);
+extern int disk_clear_badblocks(struct gendisk *disk, sector_t s, int sectors);
extern void set_device_ro(struct block_device *bdev, int flag);
extern void set_disk_ro(struct gendisk *disk, int flag);
--
2.5.0
^ permalink raw reply related
* [PATCH 1/3] badblocks: Add core badblock management code
From: Vishal Verma @ 2015-11-21 0:49 UTC (permalink / raw)
To: linux-nvdimm
Cc: Vishal Verma, linux-block, linux-raid, linux-scsi, Jens Axboe,
NeilBrown, Jeff Moyer
In-Reply-To: <1448066960-20119-1-git-send-email-vishal.l.verma@intel.com>
Take the core badblocks implementation from md, and make it generally
available. This follows the same style as kernel implementations of
linked lists, rb-trees etc, where you can have a structure that can be
embedded anywhere, and accessor functions to manipulate the data.
The only changes in this copy of the code are ones to generalize
function/variable names from md-specific ones. Also add init and free
functions.
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
---
include/linux/badblocks.h | 512 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 512 insertions(+)
create mode 100644 include/linux/badblocks.h
diff --git a/include/linux/badblocks.h b/include/linux/badblocks.h
new file mode 100644
index 0000000..94fa348
--- /dev/null
+++ b/include/linux/badblocks.h
@@ -0,0 +1,512 @@
+#ifndef _LINUX_BADBLOCKS_H
+#define _LINUX_BADBLOCKS_H
+
+#include <linux/types.h>
+#include <linux/stddef.h>
+#include <linux/kernel.h>
+
+#define BB_LEN_MASK (0x00000000000001FFULL)
+#define BB_OFFSET_MASK (0x7FFFFFFFFFFFFE00ULL)
+#define BB_ACK_MASK (0x8000000000000000ULL)
+#define BB_MAX_LEN 512
+#define BB_OFFSET(x) (((x) & BB_OFFSET_MASK) >> 9)
+#define BB_LEN(x) (((x) & BB_LEN_MASK) + 1)
+#define BB_ACK(x) (!!((x) & BB_ACK_MASK))
+#define BB_MAKE(a, l, ack) (((a)<<9) | ((l)-1) | ((u64)(!!(ack)) << 63))
+
+/* Bad block numbers are stored sorted in a single page.
+ * 64bits is used for each block or extent.
+ * 54 bits are sector number, 9 bits are extent size,
+ * 1 bit is an 'acknowledged' flag.
+ */
+#define MAX_BADBLOCKS (PAGE_SIZE/8)
+
+struct badblocks {
+ int count; /* count of bad blocks */
+ int unacked_exist; /* there probably are unacknowledged
+ * bad blocks. This is only cleared
+ * when a read discovers none
+ */
+ int shift; /* shift from sectors to block size
+ * a -ve shift means badblocks are
+ * disabled.*/
+ u64 *page; /* badblock list */
+ int changed;
+ seqlock_t lock;
+ sector_t sector;
+ sector_t size; /* in sectors */
+};
+
+/* Bad block management.
+ * We can record which blocks on each device are 'bad' and so just
+ * fail those blocks, or that stripe, rather than the whole device.
+ * Entries in the bad-block table are 64bits wide. This comprises:
+ * Length of bad-range, in sectors: 0-511 for lengths 1-512
+ * Start of bad-range, sector offset, 54 bits (allows 8 exbibytes)
+ * A 'shift' can be set so that larger blocks are tracked and
+ * consequently larger devices can be covered.
+ * 'Acknowledged' flag - 1 bit. - the most significant bit.
+ *
+ * Locking of the bad-block table uses a seqlock so badblocks_check
+ * might need to retry if it is very unlucky.
+ * We will sometimes want to check for bad blocks in a bi_end_io function,
+ * so we use the write_seqlock_irq variant.
+ *
+ * When looking for a bad block we specify a range and want to
+ * know if any block in the range is bad. So we binary-search
+ * to the last range that starts at-or-before the given endpoint,
+ * (or "before the sector after the target range")
+ * then see if it ends after the given start.
+ * We return
+ * 0 if there are no known bad blocks in the range
+ * 1 if there are known bad block which are all acknowledged
+ * -1 if there are bad blocks which have not yet been acknowledged in metadata.
+ * plus the start/length of the first bad section we overlap.
+ */
+static inline int badblocks_check(struct badblocks *bb, sector_t s, int sectors,
+ sector_t *first_bad, int *bad_sectors)
+{
+ int hi;
+ int lo;
+ u64 *p = bb->page;
+ int rv;
+ sector_t target = s + sectors;
+ unsigned seq;
+
+ if (bb->shift > 0) {
+ /* round the start down, and the end up */
+ s >>= bb->shift;
+ target += (1<<bb->shift) - 1;
+ target >>= bb->shift;
+ sectors = target - s;
+ }
+ /* 'target' is now the first block after the bad range */
+
+retry:
+ seq = read_seqbegin(&bb->lock);
+ lo = 0;
+ rv = 0;
+ hi = bb->count;
+
+ /* Binary search between lo and hi for 'target'
+ * i.e. for the last range that starts before 'target'
+ */
+ /* INVARIANT: ranges before 'lo' and at-or-after 'hi'
+ * are known not to be the last range before target.
+ * VARIANT: hi-lo is the number of possible
+ * ranges, and decreases until it reaches 1
+ */
+ while (hi - lo > 1) {
+ int mid = (lo + hi) / 2;
+ sector_t a = BB_OFFSET(p[mid]);
+ if (a < target)
+ /* This could still be the one, earlier ranges
+ * could not. */
+ lo = mid;
+ else
+ /* This and later ranges are definitely out. */
+ hi = mid;
+ }
+ /* 'lo' might be the last that started before target, but 'hi' isn't */
+ if (hi > lo) {
+ /* need to check all range that end after 's' to see if
+ * any are unacknowledged.
+ */
+ while (lo >= 0 &&
+ BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > s) {
+ if (BB_OFFSET(p[lo]) < target) {
+ /* starts before the end, and finishes after
+ * the start, so they must overlap
+ */
+ if (rv != -1 && BB_ACK(p[lo]))
+ rv = 1;
+ else
+ rv = -1;
+ *first_bad = BB_OFFSET(p[lo]);
+ *bad_sectors = BB_LEN(p[lo]);
+ }
+ lo--;
+ }
+ }
+
+ if (read_seqretry(&bb->lock, seq))
+ goto retry;
+
+ return rv;
+}
+
+/*
+ * Add a range of bad blocks to the table.
+ * This might extend the table, or might contract it
+ * if two adjacent ranges can be merged.
+ * We binary-search to find the 'insertion' point, then
+ * decide how best to handle it.
+ */
+static inline int badblocks_set(struct badblocks *bb, sector_t s, int sectors,
+ int acknowledged)
+{
+ u64 *p;
+ int lo, hi;
+ int rv = 1;
+ unsigned long flags;
+
+ if (bb->shift < 0)
+ /* badblocks are disabled */
+ return 0;
+
+ if (bb->shift) {
+ /* round the start down, and the end up */
+ sector_t next = s + sectors;
+ s >>= bb->shift;
+ next += (1<<bb->shift) - 1;
+ next >>= bb->shift;
+ sectors = next - s;
+ }
+
+ write_seqlock_irqsave(&bb->lock, flags);
+
+ p = bb->page;
+ lo = 0;
+ hi = bb->count;
+ /* Find the last range that starts at-or-before 's' */
+ while (hi - lo > 1) {
+ int mid = (lo + hi) / 2;
+ sector_t a = BB_OFFSET(p[mid]);
+ if (a <= s)
+ lo = mid;
+ else
+ hi = mid;
+ }
+ if (hi > lo && BB_OFFSET(p[lo]) > s)
+ hi = lo;
+
+ if (hi > lo) {
+ /* we found a range that might merge with the start
+ * of our new range
+ */
+ sector_t a = BB_OFFSET(p[lo]);
+ sector_t e = a + BB_LEN(p[lo]);
+ int ack = BB_ACK(p[lo]);
+ if (e >= s) {
+ /* Yes, we can merge with a previous range */
+ if (s == a && s + sectors >= e)
+ /* new range covers old */
+ ack = acknowledged;
+ else
+ ack = ack && acknowledged;
+
+ if (e < s + sectors)
+ e = s + sectors;
+ if (e - a <= BB_MAX_LEN) {
+ p[lo] = BB_MAKE(a, e-a, ack);
+ s = e;
+ } else {
+ /* does not all fit in one range,
+ * make p[lo] maximal
+ */
+ if (BB_LEN(p[lo]) != BB_MAX_LEN)
+ p[lo] = BB_MAKE(a, BB_MAX_LEN, ack);
+ s = a + BB_MAX_LEN;
+ }
+ sectors = e - s;
+ }
+ }
+ if (sectors && hi < bb->count) {
+ /* 'hi' points to the first range that starts after 's'.
+ * Maybe we can merge with the start of that range */
+ sector_t a = BB_OFFSET(p[hi]);
+ sector_t e = a + BB_LEN(p[hi]);
+ int ack = BB_ACK(p[hi]);
+ if (a <= s + sectors) {
+ /* merging is possible */
+ if (e <= s + sectors) {
+ /* full overlap */
+ e = s + sectors;
+ ack = acknowledged;
+ } else
+ ack = ack && acknowledged;
+
+ a = s;
+ if (e - a <= BB_MAX_LEN) {
+ p[hi] = BB_MAKE(a, e-a, ack);
+ s = e;
+ } else {
+ p[hi] = BB_MAKE(a, BB_MAX_LEN, ack);
+ s = a + BB_MAX_LEN;
+ }
+ sectors = e - s;
+ lo = hi;
+ hi++;
+ }
+ }
+ if (sectors == 0 && hi < bb->count) {
+ /* we might be able to combine lo and hi */
+ /* Note: 's' is at the end of 'lo' */
+ sector_t a = BB_OFFSET(p[hi]);
+ int lolen = BB_LEN(p[lo]);
+ int hilen = BB_LEN(p[hi]);
+ int newlen = lolen + hilen - (s - a);
+ if (s >= a && newlen < BB_MAX_LEN) {
+ /* yes, we can combine them */
+ int ack = BB_ACK(p[lo]) && BB_ACK(p[hi]);
+ p[lo] = BB_MAKE(BB_OFFSET(p[lo]), newlen, ack);
+ memmove(p + hi, p + hi + 1,
+ (bb->count - hi - 1) * 8);
+ bb->count--;
+ }
+ }
+ while (sectors) {
+ /* didn't merge (it all).
+ * Need to add a range just before 'hi' */
+ if (bb->count >= MAX_BADBLOCKS) {
+ /* No room for more */
+ rv = 0;
+ break;
+ } else {
+ int this_sectors = sectors;
+ memmove(p + hi + 1, p + hi,
+ (bb->count - hi) * 8);
+ bb->count++;
+
+ if (this_sectors > BB_MAX_LEN)
+ this_sectors = BB_MAX_LEN;
+ p[hi] = BB_MAKE(s, this_sectors, acknowledged);
+ sectors -= this_sectors;
+ s += this_sectors;
+ }
+ }
+
+ bb->changed = 1;
+ if (!acknowledged)
+ bb->unacked_exist = 1;
+ write_sequnlock_irqrestore(&bb->lock, flags);
+
+ return rv;
+}
+
+/*
+ * Remove a range of bad blocks from the table.
+ * This may involve extending the table if we spilt a region,
+ * but it must not fail. So if the table becomes full, we just
+ * drop the remove request.
+ */
+static inline int badblocks_clear(struct badblocks *bb, sector_t s, int sectors)
+{
+ u64 *p;
+ int lo, hi;
+ sector_t target = s + sectors;
+ int rv = 0;
+
+ if (bb->shift > 0) {
+ /* When clearing we round the start up and the end down.
+ * This should not matter as the shift should align with
+ * the block size and no rounding should ever be needed.
+ * However it is better the think a block is bad when it
+ * isn't than to think a block is not bad when it is.
+ */
+ s += (1<<bb->shift) - 1;
+ s >>= bb->shift;
+ target >>= bb->shift;
+ sectors = target - s;
+ }
+
+ write_seqlock_irq(&bb->lock);
+
+ p = bb->page;
+ lo = 0;
+ hi = bb->count;
+ /* Find the last range that starts before 'target' */
+ while (hi - lo > 1) {
+ int mid = (lo + hi) / 2;
+ sector_t a = BB_OFFSET(p[mid]);
+ if (a < target)
+ lo = mid;
+ else
+ hi = mid;
+ }
+ if (hi > lo) {
+ /* p[lo] is the last range that could overlap the
+ * current range. Earlier ranges could also overlap,
+ * but only this one can overlap the end of the range.
+ */
+ if (BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > target) {
+ /* Partial overlap, leave the tail of this range */
+ int ack = BB_ACK(p[lo]);
+ sector_t a = BB_OFFSET(p[lo]);
+ sector_t end = a + BB_LEN(p[lo]);
+
+ if (a < s) {
+ /* we need to split this range */
+ if (bb->count >= MAX_BADBLOCKS) {
+ rv = -ENOSPC;
+ goto out;
+ }
+ memmove(p+lo+1, p+lo, (bb->count - lo) * 8);
+ bb->count++;
+ p[lo] = BB_MAKE(a, s-a, ack);
+ lo++;
+ }
+ p[lo] = BB_MAKE(target, end - target, ack);
+ /* there is no longer an overlap */
+ hi = lo;
+ lo--;
+ }
+ while (lo >= 0 &&
+ BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > s) {
+ /* This range does overlap */
+ if (BB_OFFSET(p[lo]) < s) {
+ /* Keep the early parts of this range. */
+ int ack = BB_ACK(p[lo]);
+ sector_t start = BB_OFFSET(p[lo]);
+ p[lo] = BB_MAKE(start, s - start, ack);
+ /* now low doesn't overlap, so.. */
+ break;
+ }
+ lo--;
+ }
+ /* 'lo' is strictly before, 'hi' is strictly after,
+ * anything between needs to be discarded
+ */
+ if (hi - lo > 1) {
+ memmove(p+lo+1, p+hi, (bb->count - hi) * 8);
+ bb->count -= (hi - lo - 1);
+ }
+ }
+
+ bb->changed = 1;
+out:
+ write_sequnlock_irq(&bb->lock);
+ return rv;
+}
+
+/*
+ * Acknowledge all bad blocks in a list.
+ * This only succeeds if ->changed is clear. It is used by
+ * in-kernel metadata updates
+ */
+static inline void ack_all_badblocks(struct badblocks *bb)
+{
+ if (bb->page == NULL || bb->changed)
+ /* no point even trying */
+ return;
+ write_seqlock_irq(&bb->lock);
+
+ if (bb->changed == 0 && bb->unacked_exist) {
+ u64 *p = bb->page;
+ int i;
+ for (i = 0; i < bb->count ; i++) {
+ if (!BB_ACK(p[i])) {
+ sector_t start = BB_OFFSET(p[i]);
+ int len = BB_LEN(p[i]);
+ p[i] = BB_MAKE(start, len, 1);
+ }
+ }
+ bb->unacked_exist = 0;
+ }
+ write_sequnlock_irq(&bb->lock);
+}
+
+/* sysfs access to bad-blocks list. */
+static inline ssize_t badblocks_show(struct badblocks *bb, char *page,
+ int unack)
+{
+ size_t len;
+ int i;
+ u64 *p = bb->page;
+ unsigned seq;
+
+ if (bb->shift < 0)
+ return 0;
+
+retry:
+ seq = read_seqbegin(&bb->lock);
+
+ len = 0;
+ i = 0;
+
+ while (len < PAGE_SIZE && i < bb->count) {
+ sector_t s = BB_OFFSET(p[i]);
+ unsigned int length = BB_LEN(p[i]);
+ int ack = BB_ACK(p[i]);
+ i++;
+
+ if (unack && ack)
+ continue;
+
+ len += snprintf(page+len, PAGE_SIZE-len, "%llu %u\n",
+ (unsigned long long)s << bb->shift,
+ length << bb->shift);
+ }
+ if (unack && len == 0)
+ bb->unacked_exist = 0;
+
+ if (read_seqretry(&bb->lock, seq))
+ goto retry;
+
+ return len;
+}
+
+#define DO_DEBUG 1
+
+static inline ssize_t badblocks_store(struct badblocks *bb, const char *page,
+ size_t len, int unack)
+{
+ unsigned long long sector;
+ int length;
+ char newline;
+#ifdef DO_DEBUG
+ /* Allow clearing via sysfs *only* for testing/debugging.
+ * Normally only a successful write may clear a badblock
+ */
+ int clear = 0;
+ if (page[0] == '-') {
+ clear = 1;
+ page++;
+ }
+#endif /* DO_DEBUG */
+
+ switch (sscanf(page, "%llu %d%c", §or, &length, &newline)) {
+ case 3:
+ if (newline != '\n')
+ return -EINVAL;
+ case 2:
+ if (length <= 0)
+ return -EINVAL;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+#ifdef DO_DEBUG
+ if (clear) {
+ badblocks_clear(bb, sector, length);
+ return len;
+ }
+#endif /* DO_DEBUG */
+ if (badblocks_set(bb, sector, length, !unack))
+ return len;
+ else
+ return -ENOSPC;
+}
+
+static inline int badblocks_init(struct badblocks *bb, int enable)
+{
+ bb->count = 0;
+ if (enable)
+ bb->shift = 0;
+ else
+ bb->shift = -1;
+ bb->page = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ if (bb->page == NULL)
+ return -ENOMEM;
+ seqlock_init(&bb->lock);
+
+ return 0;
+}
+
+static inline void badblocks_free(struct badblocks *bb)
+{
+ kfree(bb->page);
+}
+
+#endif
--
2.5.0
^ permalink raw reply related
* [PATCH 0/3] Badblock tracking for gendisks
From: Vishal Verma @ 2015-11-21 0:49 UTC (permalink / raw)
To: linux-nvdimm
Cc: Vishal Verma, linux-block, linux-raid, linux-scsi, Jens Axboe,
NeilBrown, Jeff Moyer
Patch 1 copies badblock management code into a header of its own,
making it generally available. It follows common libraries of code
such as linked lists, where anyone may embed a core data structure
in another place, and use the provided accessor functions to
manipulate the data.
Patch 2 adds badblock tracking to gendisks (in preparation for use
by NVDIMM devices). Right now, it is turned on unconditionally - I'd
appreciate comments on if that is the right path.
Patch 3 converts md over to use the new badblocks 'library'. I have
done some pretty simple testing on this - created a raid 1 device,
made sure the sysfs entries show up, and can be used to add and view
badblocks. A closer look by the md folks would be nice here.
Vishal Verma (3):
badblocks: Add core badblock management code
block: Add badblock management for gendisks
md: convert to use the generic badblocks code
block/genhd.c | 64 ++++++
drivers/md/md.c | 495 ++------------------------------------------
drivers/md/md.h | 31 +--
include/linux/badblocks.h | 512 ++++++++++++++++++++++++++++++++++++++++++++++
include/linux/genhd.h | 6 +
5 files changed, 603 insertions(+), 505 deletions(-)
create mode 100644 include/linux/badblocks.h
--
2.5.0
^ permalink raw reply
* Re: raid10 resync hangs in 4.2.6, 4.3
From: John Stoffel @ 2015-11-20 20:30 UTC (permalink / raw)
To: Andre Tomt; +Cc: linux-raid
In-Reply-To: <564F6304.3040502@tomt.net>
Andre,
I don't have time myself to test this, but since I'm running 4.2.6
myself with triple mirror RAID1 arrays, I'm slightly worried now. But
can you make this failure happen with loop-back mounts by any chance?
That might help narrow down if it's an MD layer, block layer or
controller issue.
Andre> [resend with compressed attachments, first may have gotten eaten by a grue]
Andre> Hi
Andre> I'm seeing hangs with RAID10 resyncs on my system. RAID5/6 recovery on
Andre> the same drive set works without any problems. BTRFS RAID6 is problem
Andre> free on a different set of (very busy) drives on the same controllers as
Andre> well.
Andre> It happens shortly after array creation, from seconds to a couple
Andre> minutes in.
Andre> wchan for md0_resync kernel thread shows it sitting in raise_barrier()
Andre> forever, while md0_raid10 keeps a CPU core 100% busy (but shows no
Andre> wchan), and no resyncing or I/O to the array is getting done anymore.
Andre> After a short while kernel starts spitting out rcu_sched self-detected
Andre> stall on CPU warnings, and other rcu use starts getting iffy (I think).
Andre> I/O directly to the RAID member disk (below md layer, eg /dev/sdX
Andre> directly) continues to work after the hang, and there are no I/O errors
Andre> in the kernel log.
Andre> The array is a 8 drive array spread over 3 HBAs, created with:
Andre> mdadm --create /dev/md0 --level=10 --chunk=128 --bitmap=none
Andre> --raid-devices=8 /dev/sda1 /dev/sdg1 /dev/sdl1 /dev/sdm1 /dev/sdi1
Andre> /dev/sdj1 /dev/sdp1 /dev/sds1
Andre> The HBAs are LSI SAS2008 in IT mode (mpt2sas driver), oldish 2TB SATA
Andre> drives. Dual socket Xeon E5 v3 system with both sockets populated.
Andre> This happens on at least 4.2.6 and 4.3
Andre> I'm going to test some earlier kernels.
Andre> attached some more info.
Andre> md0_resync stack:
>> root@mental:~# cat /proc/1663/stack
>> [<ffffffffc042dd02>] raise_barrier+0x11b/0x14d [raid10]
>> [<ffffffffc0432830>] sync_request+0x193/0x14fc [raid10]
>> [<ffffffffc0559ac6>] md_do_sync+0x7d2/0xd78 [md_mod]
>> [<ffffffffc0556df9>] md_thread+0x12f/0x145 [md_mod]
>> [<ffffffff9d061db2>] kthread+0xcd/0xd5
>> [<ffffffff9d3b7a8f>] ret_from_fork+0x3f/0x70
>> [<ffffffffffffffff>] 0xffffffffffffffff
Andre> md0_raid10 stack:
>> root@mental:~# cat /proc/1662/stack
>> [<ffffffffffffffff>] 0xffffffffffffffff
Andre> cat stack trying to read /dev/md0 after hang:
>> root@mental:~# cat /proc/1737/stack
>> [<ffffffffc042de2b>] wait_barrier+0xd8/0x118 [raid10]
>> [<ffffffffc042f83c>] __make_request+0x3e/0xb17 [raid10]
>> [<ffffffffc0430399>] make_request+0x84/0xdc [raid10]
>> [<ffffffffc0557aab>] md_make_request+0xf6/0x1cc [md_mod]
>> [<ffffffff9d1a369a>] generic_make_request+0x97/0xd6
>> [<ffffffff9d1a37d1>] submit_bio+0xf8/0x140
>> [<ffffffff9d140e71>] mpage_bio_submit+0x25/0x2c
>> [<ffffffff9d141499>] mpage_readpages+0x10e/0x11f
>> [<ffffffff9d13c76c>] blkdev_readpages+0x18/0x1a
>> [<ffffffff9d0d09e4>] __do_page_cache_readahead+0x13c/0x1e0
>> [<ffffffff9d0d0c67>] ondemand_readahead+0x1df/0x1f2
>> [<ffffffff9d0d0da0>] page_cache_sync_readahead+0x38/0x3a
>> [<ffffffff9d0c70b3>] generic_file_read_iter+0x184/0x50b
>> [<ffffffff9d13c8f7>] blkdev_read_iter+0x33/0x38
>> [<ffffffff9d1132fd>] __vfs_read+0x8d/0xb1
>> [<ffffffff9d113820>] vfs_read+0x95/0x120
>> [<ffffffff9d11402d>] SyS_read+0x49/0x84
>> [<ffffffff9d3b772e>] entry_SYSCALL_64_fastpath+0x12/0x71
>> [<ffffffffffffffff>] 0xffffffffffffffff
Andre> First OOPS (more in dmesg.txt)
>> [ 150.183473] md0: detected capacity change from 0 to 8001054310400
>> [ 150.183647] md: resync of RAID array md0
>> [ 150.183652] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
>> [ 150.183654] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
>> [ 150.183678] md: using 128k window, over a total of 7813529600k.
>> [ 233.068271] INFO: rcu_sched self-detected stall on CPU
>> [ 233.068308] 5: (17999 ticks this GP) idle=695/140000000000001/0 softirq=1235/1235 fqs=8999
>> [ 233.068335] (t=18000 jiffies g=935 c=934 q=37)
>> [ 233.068354] Task dump for CPU 5:
>> [ 233.068356] md0_raid10 R running task 0 1662 2 0x00000008
>> [ 233.068358] 0000000000000000 ffff88103fca3de0 ffffffff9d069f07 0000000000000005
>> [ 233.068360] ffffffff9d63d0c0 ffff88103fca3df8 ffffffff9d06beb9 ffffffff9d63d0c0
>> [ 233.068361] ffff88103fca3e28 ffffffff9d08c836 ffffffff9d63d0c0 ffff88103fcb4e80
>> [ 233.068363] Call Trace:
>> [ 233.068364] <IRQ> [<ffffffff9d069f07>] sched_show_task+0xb9/0xbe
>> [ 233.068372] [<ffffffff9d06beb9>] dump_cpu_task+0x32/0x35
>> [ 233.068375] [<ffffffff9d08c836>] rcu_dump_cpu_stacks+0x71/0x8c
>> [ 233.068378] [<ffffffff9d08f32c>] rcu_check_callbacks+0x20f/0x5a3
>> [ 233.068382] [<ffffffff9d0b72da>] ? acct_account_cputime+0x17/0x19
>> [ 233.068384] [<ffffffff9d0911a2>] update_process_times+0x2a/0x4f
>> [ 233.068387] [<ffffffff9d09cd55>] tick_sched_handle.isra.5+0x31/0x33
>> [ 233.068388] [<ffffffff9d09cd8f>] tick_sched_timer+0x38/0x60
>> [ 233.068390] [<ffffffff9d0917e1>] __hrtimer_run_queues+0xa1/0x10c
>> [ 233.068392] [<ffffffff9d091c52>] hrtimer_interrupt+0xa0/0x172
>> [ 233.068395] [<ffffffff9d0367a4>] smp_trace_apic_timer_interrupt+0x76/0x88
>> [ 233.068397] [<ffffffff9d0367bf>] smp_apic_timer_interrupt+0x9/0xb
>> [ 233.068400] [<ffffffff9d3b8402>] apic_timer_interrupt+0x82/0x90
>> [ 233.068401] <EOI> [<ffffffff9d19e9aa>] ? bio_copy_data+0xce/0x2af
>> [ 233.068410] [<ffffffffc04320e5>] raid10d+0x974/0xf2c [raid10]
>> [ 233.068417] [<ffffffffc0556df9>] md_thread+0x12f/0x145 [md_mod]
>> [ 233.068421] [<ffffffffc0556df9>] ? md_thread+0x12f/0x145 [md_mod]
>> [ 233.068424] [<ffffffff9d07ad2e>] ? wait_woken+0x6d/0x6d
>> [ 233.068428] [<ffffffffc0556cca>] ? md_wait_for_blocked_rdev+0x102/0x102 [md_mod]
>> [ 233.068431] [<ffffffff9d061db2>] kthread+0xcd/0xd5
>> [ 233.068434] [<ffffffff9d061ce5>] ? kthread_worker_fn+0x13f/0x13f
>> [ 233.068436] [<ffffffff9d3b7a8f>] ret_from_fork+0x3f/0x70
>> [ 233.068438] [<ffffffff9d061ce5>] ? kthread_worker_fn+0x13f/0x13f
Andre> [DELETED ATTACHMENT config.txt.gz, application/gzip]
Andre> [DELETED ATTACHMENT dmesg.txt.gz, application/gzip]
Andre> [DELETED ATTACHMENT wchan.txt.gz, application/gzip]
^ permalink raw reply
* raid10 resync hangs in 4.2.6, 4.3
From: Andre Tomt @ 2015-11-20 18:14 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 5582 bytes --]
[resend with compressed attachments, first may have gotten eaten by a grue]
Hi
I'm seeing hangs with RAID10 resyncs on my system. RAID5/6 recovery on
the same drive set works without any problems. BTRFS RAID6 is problem
free on a different set of (very busy) drives on the same controllers as
well.
It happens shortly after array creation, from seconds to a couple
minutes in.
wchan for md0_resync kernel thread shows it sitting in raise_barrier()
forever, while md0_raid10 keeps a CPU core 100% busy (but shows no
wchan), and no resyncing or I/O to the array is getting done anymore.
After a short while kernel starts spitting out rcu_sched self-detected
stall on CPU warnings, and other rcu use starts getting iffy (I think).
I/O directly to the RAID member disk (below md layer, eg /dev/sdX
directly) continues to work after the hang, and there are no I/O errors
in the kernel log.
The array is a 8 drive array spread over 3 HBAs, created with:
mdadm --create /dev/md0 --level=10 --chunk=128 --bitmap=none
--raid-devices=8 /dev/sda1 /dev/sdg1 /dev/sdl1 /dev/sdm1 /dev/sdi1
/dev/sdj1 /dev/sdp1 /dev/sds1
The HBAs are LSI SAS2008 in IT mode (mpt2sas driver), oldish 2TB SATA
drives. Dual socket Xeon E5 v3 system with both sockets populated.
This happens on at least 4.2.6 and 4.3
I'm going to test some earlier kernels.
attached some more info.
md0_resync stack:
> root@mental:~# cat /proc/1663/stack
> [<ffffffffc042dd02>] raise_barrier+0x11b/0x14d [raid10]
> [<ffffffffc0432830>] sync_request+0x193/0x14fc [raid10]
> [<ffffffffc0559ac6>] md_do_sync+0x7d2/0xd78 [md_mod]
> [<ffffffffc0556df9>] md_thread+0x12f/0x145 [md_mod]
> [<ffffffff9d061db2>] kthread+0xcd/0xd5
> [<ffffffff9d3b7a8f>] ret_from_fork+0x3f/0x70
> [<ffffffffffffffff>] 0xffffffffffffffff
md0_raid10 stack:
> root@mental:~# cat /proc/1662/stack
> [<ffffffffffffffff>] 0xffffffffffffffff
cat stack trying to read /dev/md0 after hang:
> root@mental:~# cat /proc/1737/stack
> [<ffffffffc042de2b>] wait_barrier+0xd8/0x118 [raid10]
> [<ffffffffc042f83c>] __make_request+0x3e/0xb17 [raid10]
> [<ffffffffc0430399>] make_request+0x84/0xdc [raid10]
> [<ffffffffc0557aab>] md_make_request+0xf6/0x1cc [md_mod]
> [<ffffffff9d1a369a>] generic_make_request+0x97/0xd6
> [<ffffffff9d1a37d1>] submit_bio+0xf8/0x140
> [<ffffffff9d140e71>] mpage_bio_submit+0x25/0x2c
> [<ffffffff9d141499>] mpage_readpages+0x10e/0x11f
> [<ffffffff9d13c76c>] blkdev_readpages+0x18/0x1a
> [<ffffffff9d0d09e4>] __do_page_cache_readahead+0x13c/0x1e0
> [<ffffffff9d0d0c67>] ondemand_readahead+0x1df/0x1f2
> [<ffffffff9d0d0da0>] page_cache_sync_readahead+0x38/0x3a
> [<ffffffff9d0c70b3>] generic_file_read_iter+0x184/0x50b
> [<ffffffff9d13c8f7>] blkdev_read_iter+0x33/0x38
> [<ffffffff9d1132fd>] __vfs_read+0x8d/0xb1
> [<ffffffff9d113820>] vfs_read+0x95/0x120
> [<ffffffff9d11402d>] SyS_read+0x49/0x84
> [<ffffffff9d3b772e>] entry_SYSCALL_64_fastpath+0x12/0x71
> [<ffffffffffffffff>] 0xffffffffffffffff
First OOPS (more in dmesg.txt)
> [ 150.183473] md0: detected capacity change from 0 to 8001054310400
> [ 150.183647] md: resync of RAID array md0
> [ 150.183652] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 150.183654] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
> [ 150.183678] md: using 128k window, over a total of 7813529600k.
> [ 233.068271] INFO: rcu_sched self-detected stall on CPU
> [ 233.068308] 5: (17999 ticks this GP) idle=695/140000000000001/0 softirq=1235/1235 fqs=8999
> [ 233.068335] (t=18000 jiffies g=935 c=934 q=37)
> [ 233.068354] Task dump for CPU 5:
> [ 233.068356] md0_raid10 R running task 0 1662 2 0x00000008
> [ 233.068358] 0000000000000000 ffff88103fca3de0 ffffffff9d069f07 0000000000000005
> [ 233.068360] ffffffff9d63d0c0 ffff88103fca3df8 ffffffff9d06beb9 ffffffff9d63d0c0
> [ 233.068361] ffff88103fca3e28 ffffffff9d08c836 ffffffff9d63d0c0 ffff88103fcb4e80
> [ 233.068363] Call Trace:
> [ 233.068364] <IRQ> [<ffffffff9d069f07>] sched_show_task+0xb9/0xbe
> [ 233.068372] [<ffffffff9d06beb9>] dump_cpu_task+0x32/0x35
> [ 233.068375] [<ffffffff9d08c836>] rcu_dump_cpu_stacks+0x71/0x8c
> [ 233.068378] [<ffffffff9d08f32c>] rcu_check_callbacks+0x20f/0x5a3
> [ 233.068382] [<ffffffff9d0b72da>] ? acct_account_cputime+0x17/0x19
> [ 233.068384] [<ffffffff9d0911a2>] update_process_times+0x2a/0x4f
> [ 233.068387] [<ffffffff9d09cd55>] tick_sched_handle.isra.5+0x31/0x33
> [ 233.068388] [<ffffffff9d09cd8f>] tick_sched_timer+0x38/0x60
> [ 233.068390] [<ffffffff9d0917e1>] __hrtimer_run_queues+0xa1/0x10c
> [ 233.068392] [<ffffffff9d091c52>] hrtimer_interrupt+0xa0/0x172
> [ 233.068395] [<ffffffff9d0367a4>] smp_trace_apic_timer_interrupt+0x76/0x88
> [ 233.068397] [<ffffffff9d0367bf>] smp_apic_timer_interrupt+0x9/0xb
> [ 233.068400] [<ffffffff9d3b8402>] apic_timer_interrupt+0x82/0x90
> [ 233.068401] <EOI> [<ffffffff9d19e9aa>] ? bio_copy_data+0xce/0x2af
> [ 233.068410] [<ffffffffc04320e5>] raid10d+0x974/0xf2c [raid10]
> [ 233.068417] [<ffffffffc0556df9>] md_thread+0x12f/0x145 [md_mod]
> [ 233.068421] [<ffffffffc0556df9>] ? md_thread+0x12f/0x145 [md_mod]
> [ 233.068424] [<ffffffff9d07ad2e>] ? wait_woken+0x6d/0x6d
> [ 233.068428] [<ffffffffc0556cca>] ? md_wait_for_blocked_rdev+0x102/0x102 [md_mod]
> [ 233.068431] [<ffffffff9d061db2>] kthread+0xcd/0xd5
> [ 233.068434] [<ffffffff9d061ce5>] ? kthread_worker_fn+0x13f/0x13f
> [ 233.068436] [<ffffffff9d3b7a8f>] ret_from_fork+0x3f/0x70
> [ 233.068438] [<ffffffff9d061ce5>] ? kthread_worker_fn+0x13f/0x13f
[-- Attachment #2: config.txt.gz --]
[-- Type: application/gzip, Size: 31864 bytes --]
[-- Attachment #3: dmesg.txt.gz --]
[-- Type: application/gzip, Size: 31060 bytes --]
[-- Attachment #4: wchan.txt.gz --]
[-- Type: application/gzip, Size: 3051 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox