* Re: How to recover after md crash during reshape?
From: Phil Turmel @ 2015-10-21 16:05 UTC (permalink / raw)
To: Wols Lists, andras, Linux-RAID
In-Reply-To: <5627BAB4.7010402@youngman.org.uk>
Hi Wols,
I glad you've got the big picture correct, but some details need to be
addressed:
On 10/21/2015 12:17 PM, Wols Lists wrote:
> tl;dr summary ...
>
> Desktop drives are spec'd as being okay with one soft error per 10TB
> read - that's where a read fails, you try again, and everything's okay.
No, this isn't correct.
That spec is for *unrecoverable* read errors. For desktop drives,
typically spec'd as one such error every 1e14 bits read, on average.
These are failures where you really have lost the sector contents. Such
sectors are marked as "Pending Relocations" in drive firmware. But the
recording surface might still be good, so the drive waits for a write to
that pending sector, which it then verifies, before deciding to relocate
or not.
When MD raid receives a read error, whether in normal operation or a
scrub, it will reconstruct the missing data and write it back, closing
this loop immediately. Where "normal operation" means "read errors are
reported by the drive before the driver times out".
> A resync will scan the array from start to finish - if you have 10TB's
> worth of disk, you MUST be prepared to handle these errors.
>
> By default, mdadm will assume a disk is faulty and kick it after about
> 10secs, but a desktop drive will hang for maybe several minutes before
> reporting a problem.
MD raid has no timeout, and does not kick drives out for occassional
read errors. The timeout is in the per-device drivers (SCSI, SATA,
whatever). Which defaults to 30 seconds. Desktop drives typically keep
trying to read a bad sector for 120 seconds or more, ignoring the world
while they do so. Drives with default SCTERC support typically report a
read error within four to seven seconds.
With a desktop drive, the linux device driver bails after 30 seconds and
resets the link to the drive -- which gets ignored. And keeps getting
ignored until the original read retry cycle finishes. During this time,
MD has reconstructed the data and told the driver to write the fixed
sector. That *write* also fails (because the driver is failing to
reset) and that *write error* kicks the drive out of the array.
Anyways, please consider reading the threads I pointed Andras at :-)
Phil
^ permalink raw reply
* Re: [dm-devel] Possible bug in DM-RAID.
From: Austin S Hemmelgarn @ 2015-10-21 15:08 UTC (permalink / raw)
To: Heinz Mauelshagen, device-mapper development,
Linux-Kernel mailing list, linux-raid
In-Reply-To: <56279D2A.7020605@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 5616 bytes --]
Thanks for the quick response. I've cloned Linux's master branch (which
has the commit), built it, tested it, and everything works, so it looks
like this was indeed the bug I was seeing (that, or something else
between 4.2.3 and what I tested fixed things).
On 2015-10-21 10:11, Heinz Mauelshagen wrote:
>
> Neil,
>
> this looks like an incarnation of the md bitmap flaw (the one with the bogus
> slot number) leading to the false bitmap header page index.
>
>
> Austin,
> this is the respective upstream commit you need to fix your problem:
>
> commit da6fb7a9e5bd6f04f7e15070f630bdf1ea502841
> Author: NeilBrown <neilb@suse.com>
> Date: Thu Oct 1 16:03:38 2015 +1000
>
> md/bitmap: don't pass -1 to bitmap_storage_alloc.
>
> Passing -1 to bitmap_storage_alloc() causes page->index to be set to
> -1, which is quite problematic.
>
> So only pass ->cluster_slot if mddev_is_clustered().
>
> Fixes: b97e92574c0b ("Use separate bitmaps for each nodes in the
> cluster")
> Cc: stable@vger.kernel.org (v4.1+)
> Signed-off-by: NeilBrown <neilb@suse.com>
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index e51de52..48b5890 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -1997,7 +1997,8 @@ int bitmap_resize(struct bitmap *bitmap, sector_t
> blocks,
> if (bitmap->mddev->bitmap_info.offset ||
> bitmap->mddev->bitmap_info.file)
> ret = bitmap_storage_alloc(&store, chunks,
> !bitmap->mddev->bitmap_info.external,
> - bitmap->cluster_slot);
> + mddev_is_clustered(bitmap->mddev)
> + ? bitmap->cluster_slot : 0);
> if (ret)
> goto err;
>
>
> On 10/21/2015 03:39 AM, Neil Brown wrote:
>> Added dm-devel, which is probably the more appropriate list for dm
>> things.
>>
>> NeilBrown
>>
>> Austin S Hemmelgarn<ahferroin7@gmail.com> writes:
>>
>>> I think I've stumbled upon a bug in DM-RAID. The primary symptom is that when
>>> creating a new DM-RAID based device (using either LVM or dmsetup) in a RAID1
>>> configuration, it very quickly claims one by one that all of the disks failed
>>> except the first, and goes degraded. When this happens on a given system, the
>>> disks always 'fail' in the reverse of the order of the mirror numbers. All of
>>> the other RAID profiles work just fine. Curiously, it also only seems to
>>> happen for 'big' devices (I haven't been able to determine exactly what the
>>> minimum size is, but I see it 100% of the time with 32G devices, never with 16G
>>> ones, and only intermittently with 24G).
>>>
>>> Here's what I got from dmesg when creating a 32G LVM volume that exhibited
>>> this issue:
>>> [66318.401295] device-mapper: raid: Superblocks created for new array
>>> [66318.450452] md/raid1:mdX: active with 2 out of 2 mirrors
>>> [66318.450467] Choosing daemon_sleep default (5 sec)
>>> [66318.450482] created bitmap (32 pages) for device mdX
>>> [66318.450495] attempt to access beyond end of device
>>> [66318.450501] dm-91: rw=13329, want=0, limit=8192
>>> [66318.450506] md: super_written gets error=-5, uptodate=0
>>> [66318.450513] md/raid1:mdX: Disk failure on dm-92, disabling device.
>>> md/raid1:mdX: Operation continuing on 1 devices.
>>> [66318.459815] attempt to access beyond end of device
>>> [66318.459819] dm-89: rw=13329, want=0, limit=8192
>>> [66318.459822] md: super_written gets error=-5, uptodate=0
>>> [66318.492852] attempt to access beyond end of device
>>> [66318.492862] dm-89: rw=13329, want=0, limit=8192
>>> [66318.492868] md: super_written gets error=-5, uptodate=0
>>> [66318.627183] mdX: bitmap file is out of date, doing full recovery
>>> [66318.714107] mdX: bitmap initialized from disk: read 3 pages, set 65536 of 65536 bits
>>> [66318.782045] RAID1 conf printout:
>>> [66318.782054] --- wd:1 rd:2
>>> [66318.782061] disk 0, wo:0, o:1, dev:dm-90
>>> [66318.782068] disk 1, wo:1, o:0, dev:dm-92
>>> [66318.836598] RAID1 conf printout:
>>> [66318.836607] --- wd:1 rd:2
>>> [66318.836614] disk 0, wo:0, o:1, dev:dm-90
>>>
>>> And here's output for a 24G LVM volume that didn't display the issue.
>>> [66343.407954] device-mapper: raid: Superblocks created for new array
>>> [66343.479065] md/raid1:mdX: active with 2 out of 2 mirrors
>>> [66343.479078] Choosing daemon_sleep default (5 sec)
>>> [66343.479101] created bitmap (24 pages) for device mdX
>>> [66343.629329] mdX: bitmap file is out of date, doing full recovery
>>> [66343.677374] mdX: bitmap initialized from disk: read 2 pages, set 49152 of 49152 bits
>>>
>>> I'm using a lightly patched version of 4.2.3
>>> (the source can be found athttps://github.com/ferroin/linux)
>>> but none of the patches I'm using come anywhere near anything in the block layer,
>>> let alone the DM/MD code.
>>>
>>> I've attempted to bisect this, although it got kind of complicated. So far I've
>>> determined that the first commit that I see this issue on is d3b178a: md: Skip cluster setup for dm-raid
>>> Prior to that commit, I can't initialize any dm-raid devices due to the bug it fixes.
>>> I have not tested anything prior to d51e4fe (the merge commit that pulled in the md-cluster code),
>>> but I do distinctly remember that I did not see this issue in 3.19.
>>>
>>> I'll be happy to provide more info if needed.
>>>
>>>
>>> --
>>> dm-devel mailing list
>>> dm-devel@redhat.com
>>> https://www.redhat.com/mailman/listinfo/dm-devel
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply
* Re: Possible bug in DM-RAID.
From: Heinz Mauelshagen @ 2015-10-21 14:11 UTC (permalink / raw)
To: device-mapper development, Austin S Hemmelgarn,
Linux-Kernel mailing list, linux-raid
In-Reply-To: <87y4ex0zl7.fsf@notabene.neil.brown.name>
[-- Attachment #1.1: Type: text/plain, Size: 5044 bytes --]
Neil,
this looks like an incarnation of the md bitmap flaw (the one with the bogus
slot number) leading to the false bitmap header page index.
Austin,
this is the respective upstream commit you need to fix your problem:
commit da6fb7a9e5bd6f04f7e15070f630bdf1ea502841
Author: NeilBrown <neilb@suse.com>
Date: Thu Oct 1 16:03:38 2015 +1000
md/bitmap: don't pass -1 to bitmap_storage_alloc.
Passing -1 to bitmap_storage_alloc() causes page->index to be set to
-1, which is quite problematic.
So only pass ->cluster_slot if mddev_is_clustered().
Fixes: b97e92574c0b ("Use separate bitmaps for each nodes in the
cluster")
Cc: stable@vger.kernel.org (v4.1+)
Signed-off-by: NeilBrown <neilb@suse.com>
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index e51de52..48b5890 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -1997,7 +1997,8 @@ int bitmap_resize(struct bitmap *bitmap, sector_t
blocks,
if (bitmap->mddev->bitmap_info.offset ||
bitmap->mddev->bitmap_info.file)
ret = bitmap_storage_alloc(&store, chunks,
!bitmap->mddev->bitmap_info.external,
- bitmap->cluster_slot);
+ mddev_is_clustered(bitmap->mddev)
+ ? bitmap->cluster_slot : 0);
if (ret)
goto err;
On 10/21/2015 03:39 AM, Neil Brown wrote:
> Added dm-devel, which is probably the more appropriate list for dm
> things.
>
> NeilBrown
>
> Austin S Hemmelgarn <ahferroin7@gmail.com> writes:
>
>> I think I've stumbled upon a bug in DM-RAID. The primary symptom is that when
>> creating a new DM-RAID based device (using either LVM or dmsetup) in a RAID1
>> configuration, it very quickly claims one by one that all of the disks failed
>> except the first, and goes degraded. When this happens on a given system, the
>> disks always 'fail' in the reverse of the order of the mirror numbers. All of
>> the other RAID profiles work just fine. Curiously, it also only seems to
>> happen for 'big' devices (I haven't been able to determine exactly what the
>> minimum size is, but I see it 100% of the time with 32G devices, never with 16G
>> ones, and only intermittently with 24G).
>>
>> Here's what I got from dmesg when creating a 32G LVM volume that exhibited
>> this issue:
>> [66318.401295] device-mapper: raid: Superblocks created for new array
>> [66318.450452] md/raid1:mdX: active with 2 out of 2 mirrors
>> [66318.450467] Choosing daemon_sleep default (5 sec)
>> [66318.450482] created bitmap (32 pages) for device mdX
>> [66318.450495] attempt to access beyond end of device
>> [66318.450501] dm-91: rw=13329, want=0, limit=8192
>> [66318.450506] md: super_written gets error=-5, uptodate=0
>> [66318.450513] md/raid1:mdX: Disk failure on dm-92, disabling device.
>> md/raid1:mdX: Operation continuing on 1 devices.
>> [66318.459815] attempt to access beyond end of device
>> [66318.459819] dm-89: rw=13329, want=0, limit=8192
>> [66318.459822] md: super_written gets error=-5, uptodate=0
>> [66318.492852] attempt to access beyond end of device
>> [66318.492862] dm-89: rw=13329, want=0, limit=8192
>> [66318.492868] md: super_written gets error=-5, uptodate=0
>> [66318.627183] mdX: bitmap file is out of date, doing full recovery
>> [66318.714107] mdX: bitmap initialized from disk: read 3 pages, set 65536 of 65536 bits
>> [66318.782045] RAID1 conf printout:
>> [66318.782054] --- wd:1 rd:2
>> [66318.782061] disk 0, wo:0, o:1, dev:dm-90
>> [66318.782068] disk 1, wo:1, o:0, dev:dm-92
>> [66318.836598] RAID1 conf printout:
>> [66318.836607] --- wd:1 rd:2
>> [66318.836614] disk 0, wo:0, o:1, dev:dm-90
>>
>> And here's output for a 24G LVM volume that didn't display the issue.
>> [66343.407954] device-mapper: raid: Superblocks created for new array
>> [66343.479065] md/raid1:mdX: active with 2 out of 2 mirrors
>> [66343.479078] Choosing daemon_sleep default (5 sec)
>> [66343.479101] created bitmap (24 pages) for device mdX
>> [66343.629329] mdX: bitmap file is out of date, doing full recovery
>> [66343.677374] mdX: bitmap initialized from disk: read 2 pages, set 49152 of 49152 bits
>>
>> I'm using a lightly patched version of 4.2.3
>> (the source can be found at https://github.com/ferroin/linux)
>> but none of the patches I'm using come anywhere near anything in the block layer,
>> let alone the DM/MD code.
>>
>> I've attempted to bisect this, although it got kind of complicated. So far I've
>> determined that the first commit that I see this issue on is d3b178a: md: Skip cluster setup for dm-raid
>> Prior to that commit, I can't initialize any dm-raid devices due to the bug it fixes.
>> I have not tested anything prior to d51e4fe (the merge commit that pulled in the md-cluster code),
>> but I do distinctly remember that I did not see this issue in 3.19.
>>
>> I'll be happy to provide more info if needed.
>>
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
[-- Attachment #1.2: Type: text/html, Size: 6556 bytes --]
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply related
* Re: Possible bug in DM-RAID.
From: Austin S Hemmelgarn @ 2015-10-21 13:24 UTC (permalink / raw)
To: Linux-Kernel mailing list, linux-raid, dm-devel, Neil Brown
In-Reply-To: <562790FF.1060401@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4419 bytes --]
On 2015-10-21 09:19, Austin S Hemmelgarn wrote:
Hmm, dm-devel@redhat.org seems to have bounced for me. Any ideas why
RedHat would be blocking inbound mail from Google's mail servers?
> On 2015-10-20 11:12, Austin S Hemmelgarn wrote:
>> I think I've stumbled upon a bug in DM-RAID. The primary symptom is
>> that when
>> creating a new DM-RAID based device (using either LVM or dmsetup) in a
>> RAID1
>> configuration, it very quickly claims one by one that all of the disks
>> failed
>> except the first, and goes degraded. When this happens on a given
>> system, the
>> disks always 'fail' in the reverse of the order of the mirror
>> numbers. All of
>> the other RAID profiles work just fine. Curiously, it also only seems to
>> happen for 'big' devices (I haven't been able to determine exactly
>> what the
>> minimum size is, but I see it 100% of the time with 32G devices, never
>> with 16G
>> ones, and only intermittently with 24G).
> OK, I've done some more experimentation, and have figured out that
> adjusting the sync region size from the default (and thus adjusting the
> bitmap size) can temporarily work around this. If I adjust things so
> that the bitmap is less than 32 pages, then everything works fine, until
> I try to reboot, at which point the device either (in order of
> decreasing probability):
> 1. Fails just like I've outlined above.
> 2. Refuses to activate at all (if using LVM, you get some complaint
> about 'expected raid1 segment type, but got NULL' or 'reload ioctl on
> failed')
> 3. It works for a while, and then one of the first two things happens
> the next time I reboot.
>>
>> Here's what I got from dmesg when creating a 32G LVM volume that
>> exhibited
>> this issue:
>> [66318.401295] device-mapper: raid: Superblocks created for new array
>> [66318.450452] md/raid1:mdX: active with 2 out of 2 mirrors
>> [66318.450467] Choosing daemon_sleep default (5 sec)
>> [66318.450482] created bitmap (32 pages) for device mdX
>> [66318.450495] attempt to access beyond end of device
>> [66318.450501] dm-91: rw=13329, want=0, limit=8192
>> [66318.450506] md: super_written gets error=-5, uptodate=0
>> [66318.450513] md/raid1:mdX: Disk failure on dm-92, disabling device.
>> md/raid1:mdX: Operation continuing on 1 devices.
>> [66318.459815] attempt to access beyond end of device
>> [66318.459819] dm-89: rw=13329, want=0, limit=8192
>> [66318.459822] md: super_written gets error=-5, uptodate=0
>> [66318.492852] attempt to access beyond end of device
>> [66318.492862] dm-89: rw=13329, want=0, limit=8192
>> [66318.492868] md: super_written gets error=-5, uptodate=0
>> [66318.627183] mdX: bitmap file is out of date, doing full recovery
>> [66318.714107] mdX: bitmap initialized from disk: read 3 pages, set
>> 65536 of 65536 bits
>> [66318.782045] RAID1 conf printout:
>> [66318.782054] --- wd:1 rd:2
>> [66318.782061] disk 0, wo:0, o:1, dev:dm-90
>> [66318.782068] disk 1, wo:1, o:0, dev:dm-92
>> [66318.836598] RAID1 conf printout:
>> [66318.836607] --- wd:1 rd:2
>> [66318.836614] disk 0, wo:0, o:1, dev:dm-90
>>
>> And here's output for a 24G LVM volume that didn't display the issue.
>> [66343.407954] device-mapper: raid: Superblocks created for new array
>> [66343.479065] md/raid1:mdX: active with 2 out of 2 mirrors
>> [66343.479078] Choosing daemon_sleep default (5 sec)
>> [66343.479101] created bitmap (24 pages) for device mdX
>> [66343.629329] mdX: bitmap file is out of date, doing full recovery
>> [66343.677374] mdX: bitmap initialized from disk: read 2 pages, set
>> 49152 of 49152 bits
>>
>> I'm using a lightly patched version of 4.2.3
>> (the source can be found at https://github.com/ferroin/linux)
>> but none of the patches I'm using come anywhere near anything in the
>> block layer,
>> let alone the DM/MD code.
>>
>> I've attempted to bisect this, although it got kind of complicated.
>> So far I've
>> determined that the first commit that I see this issue on is d3b178a:
>> md: Skip cluster setup for dm-raid
>> Prior to that commit, I can't initialize any dm-raid devices due to
>> the bug it fixes.
>> I have not tested anything prior to d51e4fe (the merge commit that
>> pulled in the md-cluster code),
>> but I do distinctly remember that I did not see this issue in 3.19.
>>
>> I'll be happy to provide more info if needed.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply
* Re: Possible bug in DM-RAID.
From: Austin S Hemmelgarn @ 2015-10-21 13:19 UTC (permalink / raw)
To: Linux-Kernel mailing list, linux-raid, dm-devel
In-Reply-To: <562659DD.1000505@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4099 bytes --]
On 2015-10-20 11:12, Austin S Hemmelgarn wrote:
> I think I've stumbled upon a bug in DM-RAID. The primary symptom is that when
> creating a new DM-RAID based device (using either LVM or dmsetup) in a RAID1
> configuration, it very quickly claims one by one that all of the disks failed
> except the first, and goes degraded. When this happens on a given system, the
> disks always 'fail' in the reverse of the order of the mirror numbers. All of
> the other RAID profiles work just fine. Curiously, it also only seems to
> happen for 'big' devices (I haven't been able to determine exactly what the
> minimum size is, but I see it 100% of the time with 32G devices, never with 16G
> ones, and only intermittently with 24G).
OK, I've done some more experimentation, and have figured out that
adjusting the sync region size from the default (and thus adjusting the
bitmap size) can temporarily work around this. If I adjust things so
that the bitmap is less than 32 pages, then everything works fine, until
I try to reboot, at which point the device either (in order of
decreasing probability):
1. Fails just like I've outlined above.
2. Refuses to activate at all (if using LVM, you get some complaint
about 'expected raid1 segment type, but got NULL' or 'reload ioctl on
failed')
3. It works for a while, and then one of the first two things happens
the next time I reboot.
>
> Here's what I got from dmesg when creating a 32G LVM volume that exhibited
> this issue:
> [66318.401295] device-mapper: raid: Superblocks created for new array
> [66318.450452] md/raid1:mdX: active with 2 out of 2 mirrors
> [66318.450467] Choosing daemon_sleep default (5 sec)
> [66318.450482] created bitmap (32 pages) for device mdX
> [66318.450495] attempt to access beyond end of device
> [66318.450501] dm-91: rw=13329, want=0, limit=8192
> [66318.450506] md: super_written gets error=-5, uptodate=0
> [66318.450513] md/raid1:mdX: Disk failure on dm-92, disabling device.
> md/raid1:mdX: Operation continuing on 1 devices.
> [66318.459815] attempt to access beyond end of device
> [66318.459819] dm-89: rw=13329, want=0, limit=8192
> [66318.459822] md: super_written gets error=-5, uptodate=0
> [66318.492852] attempt to access beyond end of device
> [66318.492862] dm-89: rw=13329, want=0, limit=8192
> [66318.492868] md: super_written gets error=-5, uptodate=0
> [66318.627183] mdX: bitmap file is out of date, doing full recovery
> [66318.714107] mdX: bitmap initialized from disk: read 3 pages, set 65536 of 65536 bits
> [66318.782045] RAID1 conf printout:
> [66318.782054] --- wd:1 rd:2
> [66318.782061] disk 0, wo:0, o:1, dev:dm-90
> [66318.782068] disk 1, wo:1, o:0, dev:dm-92
> [66318.836598] RAID1 conf printout:
> [66318.836607] --- wd:1 rd:2
> [66318.836614] disk 0, wo:0, o:1, dev:dm-90
>
> And here's output for a 24G LVM volume that didn't display the issue.
> [66343.407954] device-mapper: raid: Superblocks created for new array
> [66343.479065] md/raid1:mdX: active with 2 out of 2 mirrors
> [66343.479078] Choosing daemon_sleep default (5 sec)
> [66343.479101] created bitmap (24 pages) for device mdX
> [66343.629329] mdX: bitmap file is out of date, doing full recovery
> [66343.677374] mdX: bitmap initialized from disk: read 2 pages, set 49152 of 49152 bits
>
> I'm using a lightly patched version of 4.2.3
> (the source can be found at https://github.com/ferroin/linux)
> but none of the patches I'm using come anywhere near anything in the block layer,
> let alone the DM/MD code.
>
> I've attempted to bisect this, although it got kind of complicated. So far I've
> determined that the first commit that I see this issue on is d3b178a: md: Skip cluster setup for dm-raid
> Prior to that commit, I can't initialize any dm-raid devices due to the bug it fixes.
> I have not tested anything prior to d51e4fe (the merge commit that pulled in the md-cluster code),
> but I do distinctly remember that I did not see this issue in 3.19.
>
> I'll be happy to provide more info if needed.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Phil Turmel @ 2015-10-21 12:18 UTC (permalink / raw)
To: Neil Brown, andras, linux-raid
In-Reply-To: <874mhl2ed3.fsf@notabene.neil.brown.name>
Good morning Neil,
On 10/20/2015 09:35 PM, Neil Brown wrote:
> Nothing dumb about that - you don't need a --backup option.
> If you did, mdadm would have complained.
>
> You only need --backup when the size of the array is unchanged or
> decreasing.
> mdadm reads the first few stripes and stores them somewhere in each of
> the spares. md (in the kernel) then reads those stripes again and
> writes them out in the new configuration. It appears that one of the
> writes failed, others might have succeeded. This may not have corrupted
> anything (the first few blocks are in the same position for both the old
> and new layout) but it might have done.
> If you do want to look for the backup, it is around about the middle of
> the device and has some metadata which contains the string
> "md_backup_data-1". If you find that, you are close to getting the
> backup data back.
Hmmm. This feature has advanced beyond my last look at the code. I was
under the impression the backup option was only optional when mdadm
could move the data offset. Does this new algorithm apply to v0.90
metadata, a v3.2 kernel, and v3.2.5 mdadm?
Phil
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Phil Turmel @ 2015-10-21 12:01 UTC (permalink / raw)
To: andras; +Cc: Linux-RAID
In-Reply-To: <87f5f235fc9d29f9e146d5431d3642cb@tantosonline.com>
Good morning Andras,
On 10/20/2015 11:52 PM, andras@tantosonline.com wrote:
> Phil,
>
> Thank you so much for the detailed explanation and your patience with
> me! Sorry for not being more responsive - I don't have access to this
> mail account from work.
No worries.
>> for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
>>
>> (Arrange for this to happen on every boot, and keep doing it manually
>> until your boot scripts are fixed.)
>
> Yes, will do. In your links below it seems that you're half advocating
> for using desktop drives in RAID arrays, half advocating against. From
> what I can tell, it seems the recommendation might depend on the
> use-case. If one doesn't care too much about instant performance in case
> of errors, one might want to use desktop drivers (with the above fix).
> If one wants reliable performance, one probably wants NAS drives. Did I
> understand the basic trade-off correctly?
Times change. At the time some of those were written, desktop drives
with scterc support were still available, but default off. Those are ok
in a raid if you have the appropriate smartctl command in your boot scripts.
Long timeouts with non-scterc drives, in my opinion, create a user
impression that things are broken, even if the drive is fine (UREs are
natural and unavoidable in the life of a drive). Users are prone to
drastic measures when they think something is broken. Also,
*applications* might not wait that long for their read, either. So, I
only recommend the long timeout solution when an array is already in
trouble with such drives.
> It seems that people also think that green drives are a bad idea in
> RAIDs in general - mostly because the frequent parking of heads reduces
> life-time. Is that a correct statement?
I don't have enough experience with green drives to say. The few that I
have (bought before I discovered the dropped scterc support) became part
of my offsite backup rotation.
> Yes sir! I will go through the steps and report back. One question: the
> reason I shouldn't attempt to re-create the new 10-disk array is that it
> would wipe out the 7->10 grow progress, so MD would think that it's a
> fully grown 10-disk array, right?
Right. Your three extra drives never really were incorporated into the
array, so the data layout is still a 7-drive pattern.
Phil
^ permalink raw reply
* looking for good reference on storage configuration
From: Miles Fidelman @ 2015-10-21 12:00 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi Folks,
Perhaps someone here might be able to point me at a reference or two on
alternatives and best practices for configuring storage for a server -
using a mix of SSDs and conventional drives.
I'm in the process of upgrading a small cluster of 4 machines - each
containing RAID arrays, with DRBD across clusters to support failover.
Easy enough with a uniform collection of HDDs. But I can't get my head
around how to add SSDs into the mix - what with the mismatch in access
speeds and all, it would seem like one needs a more complex
configuration with a level of caching between the faster and slower media.
Seems like someone would have written a book, or tutorial, or case
study, or conference presentation - but damned if I can find anything.
Any pointers, or guidance, would be much appreciated.
Thanks,
Miles Fidelman
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: andras @ 2015-10-21 4:03 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
In-Reply-To: <874mhl2ed3.fsf@notabene.neil.brown.name>
Neil,
Thanks for helping me out!
>> Oct 17 18:59:48 bazsalikom kernel: [7875182.667831] md: using
>> 128k
>> window, over a total of 1465135936k.
>> --> Oct 17 18:59:50 bazsalikom kernel: [7875184.326245] md:
>> md_do_sync()
>> got signal ... exiting
>
> This is very strange ... maybe some messages missing?
> Probably an IO error while writing to a new device.
I'm not sure what have happened either. This is /var/log/messages. Maybe
those things go into a different log?
>> mdadm: WARNING /dev/sda and /dev/sda1 appear to have very similar
>> superblocks.
>> If they are really different, please --zero the superblock on
>> one
>> If they are the same or overlap, please remove one from the
>> DEVICE list in mdadm.conf.
>
> It's very hard to make messages like this clear without being
> incredibly
> verbose...
>
> In this case /dev/sda and /dev/sda1 obviously overlap (that is obvious,
> isn't it?).
> So in that case you need to remove one of them from the DEVICE list.
> You probably don't have a DEVICE list so it defaults to everything
> listed in
> /proc/partitions.
> The "correct" thing to do at this point would have been to add a DEVICE
> list to mdadm.conf which only listed the devices that might be part of
> an array. e.g.
>
> DEVICE /dev/sd[a-z][1-9]
Understood. My problem was that when I googled for the problem, people
agreed with the suggested solution of the zeroing the superblock. I
guess it tells you how much you should trust 'common wisdom'.
>
> Phil has given good advice on this point which is worth following.
> It is quite possible that there will still be corruption.
>
> mdadm reads the first few stripes and stores them somewhere in each of
> the spares. md (in the kernel) then reads those stripes again and
> writes them out in the new configuration. It appears that one of the
> writes failed, others might have succeeded. This may not have
> corrupted
> anything (the first few blocks are in the same position for both the
> old
> and new layout) but it might have done.
>
> So if the filesystem seems corrupt after the array is re-created, that
> is likely the reason.
> The data still exists in the backup on those new devices (if you
> haven't
> done anything to them) and could be restored.
>
> If you do want to look for the backup, it is around about the middle of
> the device and has some metadata which contains the string
> "md_backup_data-1". If you find that, you are close to getting the
> backup data back.
>
> NeilBrown
Oh, gosh, I hope I don't have to do that deep of a surgery. No, I
haven't touched the new HDDs other then zeroing the superblock. So
whatever was on them, is still there. I'll see how much damage there is
to the FS after I reconstruct the array.
Thanks for all the help!
Andras
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: andras @ 2015-10-21 3:52 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux-RAID
In-Reply-To: <562660EE.9020504@turmel.org>
Phil,
Thank you so much for the detailed explanation and your patience with
me! Sorry for not being more responsive - I don't have access to this
mail account from work.
>
>> Apparently my problems don't stop adding up: now SDD started
>> developing
>> problems, so my root partition (md0) is now degraded. I will attempt
>> to
>> dd out whatever I can from that drive and continue...
>
> Don't. You have another problem: green & desktop drives in a raid
> array. They aren't built for it and will give you grief of one form or
> another. Anyways, their problem with timeout mismatch can be worked
> around with long driver timeouts. Before you do anything else, you
> *MUST* run this command:
>
> for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
>
> (Arrange for this to happen on every boot, and keep doing it manually
> until your boot scripts are fixed.)
Yes, will do. In your links below it seems that you're half advocating
for using desktop drives in RAID arrays, half advocating against. From
what I can tell, it seems the recommendation might depend on the
use-case. If one doesn't care too much about instant performance in case
of errors, one might want to use desktop drivers (with the above fix).
If one wants reliable performance, one probably wants NAS drives. Did I
understand the basic trade-off correctly?
It seems that people also think that green drives are a bad idea in
RAIDs in general - mostly because the frequent parking of heads reduces
life-time. Is that a correct statement?
> Then you can add your missing mirror and let MD fix it:
>
> mdadm /dev/md0 --add /dev/sdd3
>
> After that's done syncing, you can have MD fix any remaining UREs in
> that raid1 with:
>
> echo check >/sys/block/md0/md/sync_action
>
> While that's in progress, take the time to read through the links in
> the
> postscript -- the timeout mismatch problem and its impact on
> unrecoverable read errors has been hashed out on this list many times.
>
> Now to your big array. It is vital that it also be cleaned of UREs
> after re-creation before you do anything else. Which means it must
> *not* be created degraded (the redundancy is needed to fix UREs).
>
> According to lsdrv and your "mdadm -E" reports, the creation order you
> need is:
>
> raid device 0 /dev/sdf2 {WD-WMAZA0209553}
> raid device 1 /dev/sdd2 {WD-WMAZA0348342}
> raid device 2 /dev/sdg1 {9VS1EFFD}
> raid device 3 /dev/sde1 {5XW05FFV}
> raid device 4 /dev/sdc1 {6XW0BQL0}
> raid device 5 /dev/sdh1 {ML2220F30TEBLE}
> raid device 6 /dev/sdi2 {WD-WMAY01975001}
>
> Chunk size is 64k.
>
> Make sure your partially assembled array is stopped:
>
> mdadm --stop /dev/md1
>
> Re-create your array as follows:
>
> mdadm --create --assume-clean --verbose \
> --metadata=1.0 --raid-devices=7 --chunk=64 --level=6 \
> /dev/md1 /dev/sd{f2,d2,g1,e1,c1,h1,i2}
>
> Use "fsck -n" to check your array's filesystem (expect some damage at
> the very begining). If it look reasonable, use fsck to fix any damage.
>
> Then clean up any lingering UREs:
>
> echo check > /sys/block/md1/md/sync_action
>
> Now you can mount it and catch any critical backups. (You do know that
> raid != backup, I hope.)
>
> Your array now has a new UUID, so you probably want to fix your
> mdadm.conf file and your initramfs.
Yes sir! I will go through the steps and report back. One question: the
reason I shouldn't attempt to re-create the new 10-disk array is that it
would wipe out the 7->10 grow progress, so MD would think that it's a
fully grown 10-disk array, right?
> Finaly, go back and do your --grow, with the --backup-file.
>
> In the future, buy drives with raid ratings like the WD Red family, and
> make sure you have a cron job that regularly kicks off array scrubs. I
> do mine weekly.
Thanks for the info. This is the first time someone mentions scrubbing
with regards to RAID to me, but it makes total sense. I will set it up.
Thanks again,
Andras
^ permalink raw reply
* Re: Possible bug in DM-RAID.
From: Neil Brown @ 2015-10-21 1:39 UTC (permalink / raw)
To: Austin S Hemmelgarn, Linux-Kernel mailing list, linux-raid,
device-mapper development
In-Reply-To: <562659DD.1000505@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3457 bytes --]
Added dm-devel, which is probably the more appropriate list for dm
things.
NeilBrown
Austin S Hemmelgarn <ahferroin7@gmail.com> writes:
> I think I've stumbled upon a bug in DM-RAID. The primary symptom is that when
> creating a new DM-RAID based device (using either LVM or dmsetup) in a RAID1
> configuration, it very quickly claims one by one that all of the disks failed
> except the first, and goes degraded. When this happens on a given system, the
> disks always 'fail' in the reverse of the order of the mirror numbers. All of
> the other RAID profiles work just fine. Curiously, it also only seems to
> happen for 'big' devices (I haven't been able to determine exactly what the
> minimum size is, but I see it 100% of the time with 32G devices, never with 16G
> ones, and only intermittently with 24G).
>
> Here's what I got from dmesg when creating a 32G LVM volume that exhibited
> this issue:
> [66318.401295] device-mapper: raid: Superblocks created for new array
> [66318.450452] md/raid1:mdX: active with 2 out of 2 mirrors
> [66318.450467] Choosing daemon_sleep default (5 sec)
> [66318.450482] created bitmap (32 pages) for device mdX
> [66318.450495] attempt to access beyond end of device
> [66318.450501] dm-91: rw=13329, want=0, limit=8192
> [66318.450506] md: super_written gets error=-5, uptodate=0
> [66318.450513] md/raid1:mdX: Disk failure on dm-92, disabling device.
> md/raid1:mdX: Operation continuing on 1 devices.
> [66318.459815] attempt to access beyond end of device
> [66318.459819] dm-89: rw=13329, want=0, limit=8192
> [66318.459822] md: super_written gets error=-5, uptodate=0
> [66318.492852] attempt to access beyond end of device
> [66318.492862] dm-89: rw=13329, want=0, limit=8192
> [66318.492868] md: super_written gets error=-5, uptodate=0
> [66318.627183] mdX: bitmap file is out of date, doing full recovery
> [66318.714107] mdX: bitmap initialized from disk: read 3 pages, set 65536 of 65536 bits
> [66318.782045] RAID1 conf printout:
> [66318.782054] --- wd:1 rd:2
> [66318.782061] disk 0, wo:0, o:1, dev:dm-90
> [66318.782068] disk 1, wo:1, o:0, dev:dm-92
> [66318.836598] RAID1 conf printout:
> [66318.836607] --- wd:1 rd:2
> [66318.836614] disk 0, wo:0, o:1, dev:dm-90
>
> And here's output for a 24G LVM volume that didn't display the issue.
> [66343.407954] device-mapper: raid: Superblocks created for new array
> [66343.479065] md/raid1:mdX: active with 2 out of 2 mirrors
> [66343.479078] Choosing daemon_sleep default (5 sec)
> [66343.479101] created bitmap (24 pages) for device mdX
> [66343.629329] mdX: bitmap file is out of date, doing full recovery
> [66343.677374] mdX: bitmap initialized from disk: read 2 pages, set 49152 of 49152 bits
>
> I'm using a lightly patched version of 4.2.3
> (the source can be found at https://github.com/ferroin/linux)
> but none of the patches I'm using come anywhere near anything in the block layer,
> let alone the DM/MD code.
>
> I've attempted to bisect this, although it got kind of complicated. So far I've
> determined that the first commit that I see this issue on is d3b178a: md: Skip cluster setup for dm-raid
> Prior to that commit, I can't initialize any dm-raid devices due to the bug it fixes.
> I have not tested anything prior to d51e4fe (the merge commit that pulled in the md-cluster code),
> but I do distinctly remember that I did not see this issue in 3.19.
>
> I'll be happy to provide more info if needed.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Neil Brown @ 2015-10-21 1:35 UTC (permalink / raw)
To: andras, linux-raid
In-Reply-To: <04cdcd6bd69b3aa1f8f24465f8485c90@tantosonline.com>
[-- Attachment #1: Type: text/plain, Size: 5493 bytes --]
andras@tantosonline.com writes:
Phil has provided lots of useful advice, I'll just add a couple of
clarifications;
>
> mdadm --grow --raid-devices=10 /dev/md1
>
> Yes, I was dumb enough to start the process without a backup option -
> (copy-paste error from https://raid.wiki.kernel.org/index.php/Growing).
Nothing dumb about that - you don't need a --backup option.
If you did, mdadm would have complained.
You only need --backup when the size of the array is unchanged or
decreasing.
(or when growing to a degraded array. e.g. you can reshape a 4-drive
raid5 to a degraded 5-drive raid5 without adding a spare. This will
required a --backup. I'm fairly sure it also requires --force because
it is a very strange thing to do).
When reshaping it a larger array, mdadm only requires a backup while
reshaping the first few stripes, and it uses some space in one of the
new (previously spare) devices to store that backup.
>
> This immediately (well, after 2 seconds) crashed the MD driver:
>
> Oct 17 17:30:27 bazsalikom kernel: [7869821.514718] sd 0:0:0:0:
> [sdj] Attached SCSI disk
> Oct 17 18:39:21 bazsalikom kernel: [7873955.418679] sdh: sdh1
> Oct 17 18:39:37 bazsalikom kernel: [7873972.155084] sdi: sdi1
> Oct 17 18:39:49 bazsalikom kernel: [7873983.916038] sdj: sdj1
> Oct 17 18:40:33 bazsalikom kernel: [7874027.963430] md: bind<sdh1>
> Oct 17 18:40:34 bazsalikom kernel: [7874028.263656] md: bind<sdi1>
> Oct 17 18:40:34 bazsalikom kernel: [7874028.361112] md: bind<sdj1>
> Oct 17 18:59:48 bazsalikom kernel: [7875182.667815] md: reshape of
> RAID array md1
> Oct 17 18:59:48 bazsalikom kernel: [7875182.667818] md: minimum
> _guaranteed_ speed: 1000 KB/sec/disk.
> Oct 17 18:59:48 bazsalikom kernel: [7875182.667821] md: using
> maximum available idle IO bandwidth (but not more than 200000 KB/sec)
> for reshape.
> Oct 17 18:59:48 bazsalikom kernel: [7875182.667831] md: using 128k
> window, over a total of 1465135936k.
> --> Oct 17 18:59:50 bazsalikom kernel: [7875184.326245] md: md_do_sync()
> got signal ... exiting
This is very strange ... maybe some messages missing?
Probably an IO error while writing to a new device.
>
> From here on, things went downhill pretty damn fast. I was not able to
> unmount the file-system, stop or re-start the array (/proc/mdstat went
> away), any process trying to touch /dev/md1 hung, so eventually, I run
> out of options and hit the reset button on the machine.
>
> Upon reboot, the array wouldn't assemble, it was complaining that SDA
> and SDA1 had the same superblock info on it.
>
> mdadm: WARNING /dev/sda and /dev/sda1 appear to have very similar
> superblocks.
> If they are really different, please --zero the superblock on one
> If they are the same or overlap, please remove one from the
> DEVICE list in mdadm.conf.
It's very hard to make messages like this clear without being incredibly
verbose...
In this case /dev/sda and /dev/sda1 obviously overlap (that is obvious,
isn't it?).
So in that case you need to remove one of them from the DEVICE list.
You probably don't have a DEVICE list so it defaults to everything listed in
/proc/partitions.
The "correct" thing to do at this point would have been to add a DEVICE
list to mdadm.conf which only listed the devices that might be part of
an array. e.g.
DEVICE /dev/sd[a-z][1-9]
> So, if I read this right, the superblock here states that the array is
> in the middle of a reshape from 7 to 10 devices, but it just started
> (4096 is the position).
> What's interesting is the device names listed here don't match the ones
> reported by /proc/mdstat, and are actually incorrect. The right
> partition numbers are in /proc/mdstat.
>
> The superblocks on the 6 other original disks match, except for of
> course which one they mark as 'this' and the checksum.
>
> I've read in here (http://ubuntuforums.org/showthread.php?t=2133576)
> among many other places that it might be possible to recover the data on
> the array by trying to re-create it to the state before the re-shape.
>
> I've also read that if I want to re-create an array in read-only mode, I
> should re-create it degraded.
>
> So, what I thought I would do is this:
>
> mdadm --create /dev/md1 --level=6 --raid-devices=7 /dev/sdh2
> /dev/sdf2 /dev/sdi1 /dev/sdg1 /dev/sde1 missing missing
Phil has given good advice on this point which is worth following.
It is quite possible that there will still be corruption.
mdadm reads the first few stripes and stores them somewhere in each of
the spares. md (in the kernel) then reads those stripes again and
writes them out in the new configuration. It appears that one of the
writes failed, others might have succeeded. This may not have corrupted
anything (the first few blocks are in the same position for both the old
and new layout) but it might have done.
So if the filesystem seems corrupt after the array is re-created, that
is likely the reason.
The data still exists in the backup on those new devices (if you haven't
done anything to them) and could be restored.
If you do want to look for the backup, it is around about the middle of
the device and has some metadata which contains the string
"md_backup_data-1". If you find that, you are close to getting the
backup data back.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 0/2] raid1/10: Handle write errors correctly in narrow_write_error()
From: Jes Sorensen @ 2015-10-20 23:12 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid, kent.overstreet, William.Kuzeja, xni
In-Reply-To: <87pp092sid.fsf@notabene.neil.brown.name>
Neil Brown <neilb@suse.de> writes:
> Jes.Sorensen@redhat.com writes:
>
>> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>>
>> Hi,
>>
>> Bill Kuzeja reported a problem to me about data corruption when
>> repeatedly removing and re-adding devices in raid1 arrays. It showed
>> up to be caused by the return value of submit_bio_wait() being handled
>> incorrectly. Tracking this down is credit of Bill!
>>
>> Looks like commit 9e882242c6193ae6f416f2d8d8db0d9126bd996b changed the
>> return of submit_bio_wait() to return != 0 on error, whereas before it
>> returned 0 on error.
>>
>> This fix should be suitable for -stable as far back as 3.9
>
> 3.10?
Yes, I guess there is no 3.9 stable, but certainly 3.10+
> Thanks to both of you!
>
> I took the liberty of changing the patches a little so they are now:
>
> - if (submit_bio_wait(WRITE, wbio) == 0)
> + if (submit_bio_wait(WRITE, wbio) < 0)
>
> because when there is no explicit test I tend to expect a Bool but these
> values are not Bool.
I based it on this from block/bio.c:
/**
* submit_bio_wait - submit a bio, and wait until it completes
* @rw: whether to %READ or %WRITE, or maybe to %READA (read ahead)
* @bio: The &struct bio which describes the I/O
*
* Simple wrapper around submit_bio(). Returns 0 on success, or the error from
* bio_endio() on failure.
*/
assuming anything but 0 is an error, but < 0 should be fine as well.
> Patches are in my for-linus branch and will be forwarded sometime this
> week.
Sounds great!
Thanks!
Jes
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Anugraha Sinha @ 2015-10-20 22:34 UTC (permalink / raw)
To: Phil Turmel; +Cc: Andras Tantos, Linux-RAID
In-Reply-To: <562660EE.9020504@turmel.org>
Hi Phil,
Thanks for all the information shared by you over this thread.
It is really informative.
Regards
Anugraha
On Wed, Oct 21, 2015 at 12:42 AM, Phil Turmel <philip@turmel.org> wrote:
> Hi Andras,
>
> { Added linux-raid back -- convention on kernel.org is to reply-to-all,
> trim replies, and either interleave or bottom post. I'm trimming less
> than normal this time so the list can see. }
>
> On 10/20/2015 10:48 AM, andras@tantosonline.com wrote:
>> On 2015-10-20 08:49, Phil Turmel wrote:
>
>>> Please supply all of you mdadm -E reports for the seven partitions and
>>> the lsdrv output I requests. Just post the text inline in your reply.
>>>
>>> Do *not* do anything else.
>>>
>>> Phil
>
>> Thanks for all the help!
>>
>> Here's the output of lsdrv:
>>
>> PCI [pata_marvell] 04:00.1 IDE interface: Marvell Technology Group Ltd.
>> 88SE9128 IDE Controller (rev 11)
>> ├scsi 0:x:x:x [Empty]
>> └scsi 2:x:x:x [Empty]
>> PCI [pata_jmicron] 05:00.1 IDE interface: JMicron Technology Corp.
>> JMB363 SATA/IDE Controller (rev 02)
>> ├scsi 1:x:x:x [Empty]
>> └scsi 3:x:x:x [Empty]
>> PCI [ahci] 04:00.0 SATA controller: Marvell Technology Group Ltd.
>> 88SE9123 PCIe SATA 6.0 Gb/s controller (rev 11)
>> ├scsi 4:0:0:0 ATA ST2000DM001-1ER1 {Z4Z1JDN8}
>> │└sda 1.82t [8:0] Partitioned (dos)
>> │ └sda1 1.82t [8:1] Empty/Unknown
>> └scsi 5:0:0:0 ATA ST2000DM001-1ER1 {Z4Z1H84Q}
>> └sdb 1.82t [8:16] Partitioned (dos)
>> └sdb1 1.82t [8:17] ext4 'data' {d1403616-a9c6-4cd9-8d92-1aabc81fe373}
>> PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10
>> Family) 4 port SATA IDE Controller #1
>> ├scsi 6:0:0:0 ATA ST31500541AS {6XW0BQL0}
>> │└sdc 1.36t [8:32] Partitioned (dos)
>> │ └sdc1 1.36t [8:33] MD raid6 (10) inactive
>> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
>> ├scsi 6:0:1:0 ATA WDC WD20EARS-00M {WD-WMAZA0348342}
>> │└sdd 1.82t [8:48] Partitioned (dos)
>> │ ├sdd1 525.53m [8:49] ext4 'boot1' {a3a1cedc-3866-4d80-af18-a7a4db99d880}
>> │ ├sdd2 1.36t [8:50] MD raid6 (10) inactive
>> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
>> │ └sdd3 465.24g [8:51] MD raid1 (3) inactive
>> {f89cbbf7-66e9-eb44-42ea-8b6c723593c7}
>> ├scsi 7:0:0:0 ATA ST31500541AS {5XW05FFV}
>> │└sde 1.36t [8:64] Partitioned (dos)
>> │ └sde1 1.36t [8:65] MD raid6 (10) inactive
>> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
>> └scsi 7:0:1:0 ATA WDC WD20EARS-00M {WD-WMAZA0209553}
>> └sdf 1.82t [8:80] Partitioned (dos)
>> ├sdf1 525.53m [8:81] ext4 'boot2' {9b0e1e49-c736-47c0-89a1-4cac07c1d5ef}
>> ├sdf2 1.36t [8:82] MD raid6 (10) inactive
>> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
>> └sdf3 465.24g [8:83] MD raid1 (1/3) (w/ sdi3) in_sync
>> {f89cbbf7-66e9-eb44-42ea-8b6c723593c7}
>> └md0 465.24g [9:0] MD v0.90 raid1 (3) clean DEGRADED
>> {f89cbbf7:66e9eb44:42ea8b6c:723593c7}
>> │ ext4 'root' {ceb15bfe-e082-484c-9015-1fcc8889b798}
>> └Mounted as /dev/disk/by-uuid/ceb15bfe-e082-484c-9015-1fcc8889b798 @ /
>> PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10
>> Family) 2 port SATA IDE Controller #2
>> ├scsi 8:0:0:0 ATA ST31500341AS {9VS1EFFD}
>> │└sdg 1.36t [8:96] Partitioned (dos)
>> │ └sdg1 1.36t [8:97] MD raid6 (10) inactive
>> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
>> └scsi 10:0:0:0 ATA Hitachi HDS5C302 {ML2220F30TEBLE}
>> └sdh 1.82t [8:112] Partitioned (dos)
>> └sdh1 1.82t [8:113] MD raid6 (10) inactive
>> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
>> PCI [ahci] 05:00.0 SATA controller: JMicron Technology Corp. JMB363
>> SATA/IDE Controller (rev 02)
>> ├scsi 9:0:0:0 ATA WDC WD2002FAEX-0 {WD-WMAY01975001}
>> │└sdi 1.82t [8:128] Partitioned (dos)
>> │ ├sdi1 525.53m [8:129] Empty/Unknown
>> │ ├sdi2 1.36t [8:130] MD raid6 (10) inactive
>> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
>> │ └sdi3 465.24g [8:131] MD raid1 (2/3) (w/ sdf3) in_sync
>> {f89cbbf7-66e9-eb44-42ea-8b6c723593c7}
>> │ └md0 465.24g [9:0] MD v0.90 raid1 (3) clean DEGRADED
>> {f89cbbf7:66e9eb44:42ea8b6c:723593c7}
>> │ ext4 'root' {ceb15bfe-e082-484c-9015-1fcc8889b798}
>> └scsi 11:0:0:0 ATA ST2000DM001-1ER1 {Z4Z1JCDE}
>> └sdj 1.82t [8:144] Partitioned (dos)
>> └sdj1 1.82t [8:145] Empty/Unknown
>> Other Block Devices
>> ├loop0 0.00k [7:0] Empty/Unknown
>> ├loop1 0.00k [7:1] Empty/Unknown
>> ├loop2 0.00k [7:2] Empty/Unknown
>> ├loop3 0.00k [7:3] Empty/Unknown
>> ├loop4 0.00k [7:4] Empty/Unknown
>> ├loop5 0.00k [7:5] Empty/Unknown
>> ├loop6 0.00k [7:6] Empty/Unknown
>> └loop7 0.00k [7:7] Empty/Unknown
>>
>>
>> mdadm output:
>>
>> mdadm -E /dev/sdb1 /dev/sda1 /dev/sdc1 /dev/sdd2 /dev/sde1 /dev/sdh1
>> /dev/sdg1 /dev/sdi2 /dev/sdj1 /dev/sdf2
>
>> mdadm: No md superblock detected on /dev/sdb1.
>
>> mdadm: No md superblock detected on /dev/sda1.
>
>> /dev/sdc1:
>> Magic : a92b4efc
>> Version : 0.91.00
>> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
>> Creation Time : Sat Oct 2 07:21:53 2010
>> Raid Level : raid6
>> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
>> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
>> Raid Devices : 10
>> Total Devices : 10
>> Preferred Minor : 1
>>
>> Reshape pos'n : 4096
>> Delta Devices : 3 (7->10)
>>
>> Update Time : Sat Oct 17 18:59:50 2015
>> State : active
>> Active Devices : 10
>> Working Devices : 10
>> Failed Devices : 0
>> Spare Devices : 0
>> Checksum : fad60723 - correct
>> Events : 2579239
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Number Major Minor RaidDevice State
>> this 4 8 1 4 active sync /dev/sda1
>>
>> 0 0 8 50 0 active sync /dev/sdd2
>> 1 1 8 18 1 active sync
>> 2 2 8 65 2 active sync /dev/sde1
>> 3 3 8 33 3 active sync /dev/sdc1
>> 4 4 8 1 4 active sync /dev/sda1
>> 5 5 8 81 5 active sync /dev/sdf1
>> 6 6 8 98 6 active sync
>> 7 7 8 145 7 active sync /dev/sdj1
>> 8 8 8 129 8 active sync /dev/sdi1
>> 9 9 8 113 9 active sync /dev/sdh1
>
>> /dev/sdd2:
>> Magic : a92b4efc
>> Version : 0.91.00
>> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
>> Creation Time : Sat Oct 2 07:21:53 2010
>> Raid Level : raid6
>> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
>> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
>> Raid Devices : 10
>> Total Devices : 10
>> Preferred Minor : 1
>>
>> Reshape pos'n : 4096
>> Delta Devices : 3 (7->10)
>>
>> Update Time : Sat Oct 17 18:59:50 2015
>> State : active
>> Active Devices : 10
>> Working Devices : 10
>> Failed Devices : 0
>> Spare Devices : 0
>> Checksum : fad6072e - correct
>> Events : 2579239
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Number Major Minor RaidDevice State
>> this 1 8 18 1 active sync
>>
>> 0 0 8 50 0 active sync /dev/sdd2
>> 1 1 8 18 1 active sync
>> 2 2 8 65 2 active sync /dev/sde1
>> 3 3 8 33 3 active sync /dev/sdc1
>> 4 4 8 1 4 active sync /dev/sda1
>> 5 5 8 81 5 active sync /dev/sdf1
>> 6 6 8 98 6 active sync
>> 7 7 8 145 7 active sync /dev/sdj1
>> 8 8 8 129 8 active sync /dev/sdi1
>> 9 9 8 113 9 active sync /dev/sdh1
>
>> /dev/sde1:
>> Magic : a92b4efc
>> Version : 0.91.00
>> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
>> Creation Time : Sat Oct 2 07:21:53 2010
>> Raid Level : raid6
>> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
>> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
>> Raid Devices : 10
>> Total Devices : 10
>> Preferred Minor : 1
>>
>> Reshape pos'n : 4096
>> Delta Devices : 3 (7->10)
>>
>> Update Time : Sat Oct 17 18:59:50 2015
>> State : active
>> Active Devices : 10
>> Working Devices : 10
>> Failed Devices : 0
>> Spare Devices : 0
>> Checksum : fad60741 - correct
>> Events : 2579239
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Number Major Minor RaidDevice State
>> this 3 8 33 3 active sync /dev/sdc1
>>
>> 0 0 8 50 0 active sync /dev/sdd2
>> 1 1 8 18 1 active sync
>> 2 2 8 65 2 active sync /dev/sde1
>> 3 3 8 33 3 active sync /dev/sdc1
>> 4 4 8 1 4 active sync /dev/sda1
>> 5 5 8 81 5 active sync /dev/sdf1
>> 6 6 8 98 6 active sync
>> 7 7 8 145 7 active sync /dev/sdj1
>> 8 8 8 129 8 active sync /dev/sdi1
>> 9 9 8 113 9 active sync /dev/sdh1
>
>> /dev/sdh1:
>> Magic : a92b4efc
>> Version : 0.91.00
>> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
>> Creation Time : Sat Oct 2 07:21:53 2010
>> Raid Level : raid6
>> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
>> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
>> Raid Devices : 10
>> Total Devices : 10
>> Preferred Minor : 1
>>
>> Reshape pos'n : 4096
>> Delta Devices : 3 (7->10)
>>
>> Update Time : Sat Oct 17 18:59:50 2015
>> State : active
>> Active Devices : 10
>> Working Devices : 10
>> Failed Devices : 0
>> Spare Devices : 0
>> Checksum : fad60775 - correct
>> Events : 2579239
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Number Major Minor RaidDevice State
>> this 5 8 81 5 active sync /dev/sdf1
>>
>> 0 0 8 50 0 active sync /dev/sdd2
>> 1 1 8 18 1 active sync
>> 2 2 8 65 2 active sync /dev/sde1
>> 3 3 8 33 3 active sync /dev/sdc1
>> 4 4 8 1 4 active sync /dev/sda1
>> 5 5 8 81 5 active sync /dev/sdf1
>> 6 6 8 98 6 active sync
>> 7 7 8 145 7 active sync /dev/sdj1
>> 8 8 8 129 8 active sync /dev/sdi1
>> 9 9 8 113 9 active sync /dev/sdh1
>
>> /dev/sdg1:
>> Magic : a92b4efc
>> Version : 0.91.00
>> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
>> Creation Time : Sat Oct 2 07:21:53 2010
>> Raid Level : raid6
>> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
>> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
>> Raid Devices : 10
>> Total Devices : 10
>> Preferred Minor : 1
>>
>> Reshape pos'n : 4096
>> Delta Devices : 3 (7->10)
>>
>> Update Time : Sat Oct 17 18:59:50 2015
>> State : active
>> Active Devices : 10
>> Working Devices : 10
>> Failed Devices : 0
>> Spare Devices : 0
>> Checksum : fad6075f - correct
>> Events : 2579239
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Number Major Minor RaidDevice State
>> this 2 8 65 2 active sync /dev/sde1
>>
>> 0 0 8 50 0 active sync /dev/sdd2
>> 1 1 8 18 1 active sync
>> 2 2 8 65 2 active sync /dev/sde1
>> 3 3 8 33 3 active sync /dev/sdc1
>> 4 4 8 1 4 active sync /dev/sda1
>> 5 5 8 81 5 active sync /dev/sdf1
>> 6 6 8 98 6 active sync
>> 7 7 8 145 7 active sync /dev/sdj1
>> 8 8 8 129 8 active sync /dev/sdi1
>> 9 9 8 113 9 active sync /dev/sdh1
>
>> /dev/sdi2:
>> Magic : a92b4efc
>> Version : 0.91.00
>> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
>> Creation Time : Sat Oct 2 07:21:53 2010
>> Raid Level : raid6
>> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
>> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
>> Raid Devices : 10
>> Total Devices : 10
>> Preferred Minor : 1
>>
>> Reshape pos'n : 4096
>> Delta Devices : 3 (7->10)
>>
>> Update Time : Sat Oct 17 18:59:50 2015
>> State : active
>> Active Devices : 10
>> Working Devices : 10
>> Failed Devices : 0
>> Spare Devices : 0
>> Checksum : fad60788 - correct
>> Events : 2579239
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Number Major Minor RaidDevice State
>> this 6 8 98 6 active sync
>>
>> 0 0 8 50 0 active sync /dev/sdd2
>> 1 1 8 18 1 active sync
>> 2 2 8 65 2 active sync /dev/sde1
>> 3 3 8 33 3 active sync /dev/sdc1
>> 4 4 8 1 4 active sync /dev/sda1
>> 5 5 8 81 5 active sync /dev/sdf1
>> 6 6 8 98 6 active sync
>> 7 7 8 145 7 active sync /dev/sdj1
>> 8 8 8 129 8 active sync /dev/sdi1
>> 9 9 8 113 9 active sync /dev/sdh1
>
>> mdadm: No md superblock detected on /dev/sdj1.
>
>> /dev/sdf2:
>> Magic : a92b4efc
>> Version : 0.91.00
>> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
>> Creation Time : Sat Oct 2 07:21:53 2010
>> Raid Level : raid6
>> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
>> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
>> Raid Devices : 10
>> Total Devices : 10
>> Preferred Minor : 1
>>
>> Reshape pos'n : 4096
>> Delta Devices : 3 (7->10)
>>
>> Update Time : Sat Oct 17 18:59:50 2015
>> State : active
>> Active Devices : 10
>> Working Devices : 10
>> Failed Devices : 0
>> Spare Devices : 0
>> Checksum : fad6074c - correct
>> Events : 2579239
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Number Major Minor RaidDevice State
>> this 0 8 50 0 active sync /dev/sdd2
>>
>> 0 0 8 50 0 active sync /dev/sdd2
>> 1 1 8 18 1 active sync
>> 2 2 8 65 2 active sync /dev/sde1
>> 3 3 8 33 3 active sync /dev/sdc1
>> 4 4 8 1 4 active sync /dev/sda1
>> 5 5 8 81 5 active sync /dev/sdf1
>> 6 6 8 98 6 active sync
>> 7 7 8 145 7 active sync /dev/sdj1
>> 8 8 8 129 8 active sync /dev/sdi1
>> 9 9 8 113 9 active sync /dev/sdh1
>
>> Apparently my problems don't stop adding up: now SDD started developing
>> problems, so my root partition (md0) is now degraded. I will attempt to
>> dd out whatever I can from that drive and continue...
>
> Don't. You have another problem: green & desktop drives in a raid
> array. They aren't built for it and will give you grief of one form or
> another. Anyways, their problem with timeout mismatch can be worked
> around with long driver timeouts. Before you do anything else, you
> *MUST* run this command:
>
> for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
>
> (Arrange for this to happen on every boot, and keep doing it manually
> until your boot scripts are fixed.)
>
> Then you can add your missing mirror and let MD fix it:
>
> mdadm /dev/md0 --add /dev/sdd3
>
> After that's done syncing, you can have MD fix any remaining UREs in
> that raid1 with:
>
> echo check >/sys/block/md0/md/sync_action
>
> While that's in progress, take the time to read through the links in the
> postscript -- the timeout mismatch problem and its impact on
> unrecoverable read errors has been hashed out on this list many times.
>
> Now to your big array. It is vital that it also be cleaned of UREs
> after re-creation before you do anything else. Which means it must
> *not* be created degraded (the redundancy is needed to fix UREs).
>
> According to lsdrv and your "mdadm -E" reports, the creation order you
> need is:
>
> raid device 0 /dev/sdf2 {WD-WMAZA0209553}
> raid device 1 /dev/sdd2 {WD-WMAZA0348342}
> raid device 2 /dev/sdg1 {9VS1EFFD}
> raid device 3 /dev/sde1 {5XW05FFV}
> raid device 4 /dev/sdc1 {6XW0BQL0}
> raid device 5 /dev/sdh1 {ML2220F30TEBLE}
> raid device 6 /dev/sdi2 {WD-WMAY01975001}
>
> Chunk size is 64k.
>
> Make sure your partially assembled array is stopped:
>
> mdadm --stop /dev/md1
>
> Re-create your array as follows:
>
> mdadm --create --assume-clean --verbose \
> --metadata=1.0 --raid-devices=7 --chunk=64 --level=6 \
> /dev/md1 /dev/sd{f2,d2,g1,e1,c1,h1,i2}
>
> Use "fsck -n" to check your array's filesystem (expect some damage at
> the very begining). If it look reasonable, use fsck to fix any damage.
>
> Then clean up any lingering UREs:
>
> echo check > /sys/block/md1/md/sync_action
>
> Now you can mount it and catch any critical backups. (You do know that
> raid != backup, I hope.)
>
> Your array now has a new UUID, so you probably want to fix your
> mdadm.conf file and your initramfs.
>
> Finaly, go back and do your --grow, with the --backup-file.
>
> In the future, buy drives with raid ratings like the WD Red family, and
> make sure you have a cron job that regularly kicks off array scrubs. I
> do mine weekly.
>
> HTH,
>
> Phil
>
> [1] http://marc.info/?l=linux-raid&m=139050322510249&w=2
> [2] http://marc.info/?l=linux-raid&m=135863964624202&w=2
> [3] http://marc.info/?l=linux-raid&m=135811522817345&w=1
> [4] http://marc.info/?l=linux-raid&m=133761065622164&w=2
> [5] http://marc.info/?l=linux-raid&m=132477199207506
> [6] http://marc.info/?l=linux-raid&m=133665797115876&w=2
> [7] https://www.marc.info/?l=linux-raid&m=142487508806844&w=3
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 0/2] raid1/10: Handle write errors correctly in narrow_write_error()
From: Neil Brown @ 2015-10-20 20:29 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid, kent.overstreet, William.Kuzeja, xni
In-Reply-To: <1445357353-19906-1-git-send-email-Jes.Sorensen@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]
Jes.Sorensen@redhat.com writes:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> Hi,
>
> Bill Kuzeja reported a problem to me about data corruption when
> repeatedly removing and re-adding devices in raid1 arrays. It showed
> up to be caused by the return value of submit_bio_wait() being handled
> incorrectly. Tracking this down is credit of Bill!
>
> Looks like commit 9e882242c6193ae6f416f2d8d8db0d9126bd996b changed the
> return of submit_bio_wait() to return != 0 on error, whereas before it
> returned 0 on error.
>
> This fix should be suitable for -stable as far back as 3.9
3.10?
Thanks to both of you!
I took the liberty of changing the patches a little so they are now:
- if (submit_bio_wait(WRITE, wbio) == 0)
+ if (submit_bio_wait(WRITE, wbio) < 0)
because when there is no explicit test I tend to expect a Bool but these
values are not Bool.
Patches are in my for-linus branch and will be forwarded sometime this
week.
This bug only causes a problem when bad-block logs are active, so
hopefully it won't have caused too much corruption yet -- you would need
to be using a newish mdadm.
Thanks,
NeilBrown
>
> Cheers,
> Jes
>
> Jes Sorensen (2):
> md/raid1: submit_bio_wait() returns 0 on success
> md/raid10: submit_bio_wait() returns 0 on success
>
> drivers/md/raid1.c | 2 +-
> drivers/md/raid10.c | 2 +-
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> --
> 2.4.3
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* [PATCH 2/2] md/raid10: submit_bio_wait() returns 0 on success
From: Jes.Sorensen @ 2015-10-20 16:09 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, kent.overstreet, William.Kuzeja, xni
In-Reply-To: <1445357353-19906-1-git-send-email-Jes.Sorensen@redhat.com>
From: Jes Sorensen <Jes.Sorensen@redhat.com>
This was introduced with 9e882242c6193ae6f416f2d8d8db0d9126bd996b
which changed the return value of submit_bio_wait() to return != 0 on
error, but didn't update the caller accordingly.
Fixes: 9e882242c6 ("block: Add submit_bio_wait(), remove from md")
Reported-by: Bill Kuzeja <William.Kuzeja@stratus.com>
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid10.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 9f69dc5..b3191c9 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -2467,7 +2467,7 @@ static int narrow_write_error(struct r10bio *r10_bio, int i)
choose_data_offset(r10_bio, rdev) +
(sector - r10_bio->sector));
wbio->bi_bdev = rdev->bdev;
- if (submit_bio_wait(WRITE, wbio) == 0)
+ if (submit_bio_wait(WRITE, wbio))
/* Failure! */
ok = rdev_set_badblocks(rdev, sector,
sectors, 0)
--
2.4.3
^ permalink raw reply related
* [PATCH 1/2] md/raid1: submit_bio_wait() returns 0 on success
From: Jes.Sorensen @ 2015-10-20 16:09 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, kent.overstreet, William.Kuzeja, xni
In-Reply-To: <1445357353-19906-1-git-send-email-Jes.Sorensen@redhat.com>
From: Jes Sorensen <Jes.Sorensen@redhat.com>
This was introduced with 9e882242c6193ae6f416f2d8d8db0d9126bd996b
which changed the return value of submit_bio_wait() to return != 0 on
error, but didn't update the caller accordingly.
Fixes: 9e882242c6 ("block: Add submit_bio_wait(), remove from md")
Reported-by: Bill Kuzeja <William.Kuzeja@stratus.com>
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid1.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index ddd8a5f..905f167 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -2195,7 +2195,7 @@ static int narrow_write_error(struct r1bio *r1_bio, int i)
bio_trim(wbio, sector - r1_bio->sector, sectors);
wbio->bi_iter.bi_sector += rdev->data_offset;
wbio->bi_bdev = rdev->bdev;
- if (submit_bio_wait(WRITE, wbio) == 0)
+ if (submit_bio_wait(WRITE, wbio))
/* failure! */
ok = rdev_set_badblocks(rdev, sector,
sectors, 0)
--
2.4.3
^ permalink raw reply related
* [PATCH 0/2] raid1/10: Handle write errors correctly in narrow_write_error()
From: Jes.Sorensen @ 2015-10-20 16:09 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, kent.overstreet, William.Kuzeja, xni
From: Jes Sorensen <Jes.Sorensen@redhat.com>
Hi,
Bill Kuzeja reported a problem to me about data corruption when
repeatedly removing and re-adding devices in raid1 arrays. It showed
up to be caused by the return value of submit_bio_wait() being handled
incorrectly. Tracking this down is credit of Bill!
Looks like commit 9e882242c6193ae6f416f2d8d8db0d9126bd996b changed the
return of submit_bio_wait() to return != 0 on error, whereas before it
returned 0 on error.
This fix should be suitable for -stable as far back as 3.9
Cheers,
Jes
Jes Sorensen (2):
md/raid1: submit_bio_wait() returns 0 on success
md/raid10: submit_bio_wait() returns 0 on success
drivers/md/raid1.c | 2 +-
drivers/md/raid10.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
2.4.3
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Phil Turmel @ 2015-10-20 15:42 UTC (permalink / raw)
To: andras, Linux-RAID
In-Reply-To: <3baf849321d819483c5d20c005a31844@tantosonline.com>
Hi Andras,
{ Added linux-raid back -- convention on kernel.org is to reply-to-all,
trim replies, and either interleave or bottom post. I'm trimming less
than normal this time so the list can see. }
On 10/20/2015 10:48 AM, andras@tantosonline.com wrote:
> On 2015-10-20 08:49, Phil Turmel wrote:
>> Please supply all of you mdadm -E reports for the seven partitions and
>> the lsdrv output I requests. Just post the text inline in your reply.
>>
>> Do *not* do anything else.
>>
>> Phil
> Thanks for all the help!
>
> Here's the output of lsdrv:
>
> PCI [pata_marvell] 04:00.1 IDE interface: Marvell Technology Group Ltd.
> 88SE9128 IDE Controller (rev 11)
> ├scsi 0:x:x:x [Empty]
> └scsi 2:x:x:x [Empty]
> PCI [pata_jmicron] 05:00.1 IDE interface: JMicron Technology Corp.
> JMB363 SATA/IDE Controller (rev 02)
> ├scsi 1:x:x:x [Empty]
> └scsi 3:x:x:x [Empty]
> PCI [ahci] 04:00.0 SATA controller: Marvell Technology Group Ltd.
> 88SE9123 PCIe SATA 6.0 Gb/s controller (rev 11)
> ├scsi 4:0:0:0 ATA ST2000DM001-1ER1 {Z4Z1JDN8}
> │└sda 1.82t [8:0] Partitioned (dos)
> │ └sda1 1.82t [8:1] Empty/Unknown
> └scsi 5:0:0:0 ATA ST2000DM001-1ER1 {Z4Z1H84Q}
> └sdb 1.82t [8:16] Partitioned (dos)
> └sdb1 1.82t [8:17] ext4 'data' {d1403616-a9c6-4cd9-8d92-1aabc81fe373}
> PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10
> Family) 4 port SATA IDE Controller #1
> ├scsi 6:0:0:0 ATA ST31500541AS {6XW0BQL0}
> │└sdc 1.36t [8:32] Partitioned (dos)
> │ └sdc1 1.36t [8:33] MD raid6 (10) inactive
> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
> ├scsi 6:0:1:0 ATA WDC WD20EARS-00M {WD-WMAZA0348342}
> │└sdd 1.82t [8:48] Partitioned (dos)
> │ ├sdd1 525.53m [8:49] ext4 'boot1' {a3a1cedc-3866-4d80-af18-a7a4db99d880}
> │ ├sdd2 1.36t [8:50] MD raid6 (10) inactive
> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
> │ └sdd3 465.24g [8:51] MD raid1 (3) inactive
> {f89cbbf7-66e9-eb44-42ea-8b6c723593c7}
> ├scsi 7:0:0:0 ATA ST31500541AS {5XW05FFV}
> │└sde 1.36t [8:64] Partitioned (dos)
> │ └sde1 1.36t [8:65] MD raid6 (10) inactive
> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
> └scsi 7:0:1:0 ATA WDC WD20EARS-00M {WD-WMAZA0209553}
> └sdf 1.82t [8:80] Partitioned (dos)
> ├sdf1 525.53m [8:81] ext4 'boot2' {9b0e1e49-c736-47c0-89a1-4cac07c1d5ef}
> ├sdf2 1.36t [8:82] MD raid6 (10) inactive
> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
> └sdf3 465.24g [8:83] MD raid1 (1/3) (w/ sdi3) in_sync
> {f89cbbf7-66e9-eb44-42ea-8b6c723593c7}
> └md0 465.24g [9:0] MD v0.90 raid1 (3) clean DEGRADED
> {f89cbbf7:66e9eb44:42ea8b6c:723593c7}
> │ ext4 'root' {ceb15bfe-e082-484c-9015-1fcc8889b798}
> └Mounted as /dev/disk/by-uuid/ceb15bfe-e082-484c-9015-1fcc8889b798 @ /
> PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10
> Family) 2 port SATA IDE Controller #2
> ├scsi 8:0:0:0 ATA ST31500341AS {9VS1EFFD}
> │└sdg 1.36t [8:96] Partitioned (dos)
> │ └sdg1 1.36t [8:97] MD raid6 (10) inactive
> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
> └scsi 10:0:0:0 ATA Hitachi HDS5C302 {ML2220F30TEBLE}
> └sdh 1.82t [8:112] Partitioned (dos)
> └sdh1 1.82t [8:113] MD raid6 (10) inactive
> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
> PCI [ahci] 05:00.0 SATA controller: JMicron Technology Corp. JMB363
> SATA/IDE Controller (rev 02)
> ├scsi 9:0:0:0 ATA WDC WD2002FAEX-0 {WD-WMAY01975001}
> │└sdi 1.82t [8:128] Partitioned (dos)
> │ ├sdi1 525.53m [8:129] Empty/Unknown
> │ ├sdi2 1.36t [8:130] MD raid6 (10) inactive
> {5e57a17d-43eb-0786-42ea-8b6c723593c7}
> │ └sdi3 465.24g [8:131] MD raid1 (2/3) (w/ sdf3) in_sync
> {f89cbbf7-66e9-eb44-42ea-8b6c723593c7}
> │ └md0 465.24g [9:0] MD v0.90 raid1 (3) clean DEGRADED
> {f89cbbf7:66e9eb44:42ea8b6c:723593c7}
> │ ext4 'root' {ceb15bfe-e082-484c-9015-1fcc8889b798}
> └scsi 11:0:0:0 ATA ST2000DM001-1ER1 {Z4Z1JCDE}
> └sdj 1.82t [8:144] Partitioned (dos)
> └sdj1 1.82t [8:145] Empty/Unknown
> Other Block Devices
> ├loop0 0.00k [7:0] Empty/Unknown
> ├loop1 0.00k [7:1] Empty/Unknown
> ├loop2 0.00k [7:2] Empty/Unknown
> ├loop3 0.00k [7:3] Empty/Unknown
> ├loop4 0.00k [7:4] Empty/Unknown
> ├loop5 0.00k [7:5] Empty/Unknown
> ├loop6 0.00k [7:6] Empty/Unknown
> └loop7 0.00k [7:7] Empty/Unknown
>
>
> mdadm output:
>
> mdadm -E /dev/sdb1 /dev/sda1 /dev/sdc1 /dev/sdd2 /dev/sde1 /dev/sdh1
> /dev/sdg1 /dev/sdi2 /dev/sdj1 /dev/sdf2
> mdadm: No md superblock detected on /dev/sdb1.
> mdadm: No md superblock detected on /dev/sda1.
> /dev/sdc1:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad60723 - correct
> Events : 2579239
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 4 8 1 4 active sync /dev/sda1
>
> 0 0 8 50 0 active sync /dev/sdd2
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
> /dev/sdd2:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad6072e - correct
> Events : 2579239
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 1 8 18 1 active sync
>
> 0 0 8 50 0 active sync /dev/sdd2
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
> /dev/sde1:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad60741 - correct
> Events : 2579239
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 3 8 33 3 active sync /dev/sdc1
>
> 0 0 8 50 0 active sync /dev/sdd2
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
> /dev/sdh1:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad60775 - correct
> Events : 2579239
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 5 8 81 5 active sync /dev/sdf1
>
> 0 0 8 50 0 active sync /dev/sdd2
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
> /dev/sdg1:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad6075f - correct
> Events : 2579239
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 2 8 65 2 active sync /dev/sde1
>
> 0 0 8 50 0 active sync /dev/sdd2
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
> /dev/sdi2:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad60788 - correct
> Events : 2579239
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 6 8 98 6 active sync
>
> 0 0 8 50 0 active sync /dev/sdd2
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
> mdadm: No md superblock detected on /dev/sdj1.
> /dev/sdf2:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad6074c - correct
> Events : 2579239
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 0 8 50 0 active sync /dev/sdd2
>
> 0 0 8 50 0 active sync /dev/sdd2
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
> Apparently my problems don't stop adding up: now SDD started developing
> problems, so my root partition (md0) is now degraded. I will attempt to
> dd out whatever I can from that drive and continue...
Don't. You have another problem: green & desktop drives in a raid
array. They aren't built for it and will give you grief of one form or
another. Anyways, their problem with timeout mismatch can be worked
around with long driver timeouts. Before you do anything else, you
*MUST* run this command:
for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
(Arrange for this to happen on every boot, and keep doing it manually
until your boot scripts are fixed.)
Then you can add your missing mirror and let MD fix it:
mdadm /dev/md0 --add /dev/sdd3
After that's done syncing, you can have MD fix any remaining UREs in
that raid1 with:
echo check >/sys/block/md0/md/sync_action
While that's in progress, take the time to read through the links in the
postscript -- the timeout mismatch problem and its impact on
unrecoverable read errors has been hashed out on this list many times.
Now to your big array. It is vital that it also be cleaned of UREs
after re-creation before you do anything else. Which means it must
*not* be created degraded (the redundancy is needed to fix UREs).
According to lsdrv and your "mdadm -E" reports, the creation order you
need is:
raid device 0 /dev/sdf2 {WD-WMAZA0209553}
raid device 1 /dev/sdd2 {WD-WMAZA0348342}
raid device 2 /dev/sdg1 {9VS1EFFD}
raid device 3 /dev/sde1 {5XW05FFV}
raid device 4 /dev/sdc1 {6XW0BQL0}
raid device 5 /dev/sdh1 {ML2220F30TEBLE}
raid device 6 /dev/sdi2 {WD-WMAY01975001}
Chunk size is 64k.
Make sure your partially assembled array is stopped:
mdadm --stop /dev/md1
Re-create your array as follows:
mdadm --create --assume-clean --verbose \
--metadata=1.0 --raid-devices=7 --chunk=64 --level=6 \
/dev/md1 /dev/sd{f2,d2,g1,e1,c1,h1,i2}
Use "fsck -n" to check your array's filesystem (expect some damage at
the very begining). If it look reasonable, use fsck to fix any damage.
Then clean up any lingering UREs:
echo check > /sys/block/md1/md/sync_action
Now you can mount it and catch any critical backups. (You do know that
raid != backup, I hope.)
Your array now has a new UUID, so you probably want to fix your
mdadm.conf file and your initramfs.
Finaly, go back and do your --grow, with the --backup-file.
In the future, buy drives with raid ratings like the WD Red family, and
make sure you have a cron job that regularly kicks off array scrubs. I
do mine weekly.
HTH,
Phil
[1] http://marc.info/?l=linux-raid&m=139050322510249&w=2
[2] http://marc.info/?l=linux-raid&m=135863964624202&w=2
[3] http://marc.info/?l=linux-raid&m=135811522817345&w=1
[4] http://marc.info/?l=linux-raid&m=133761065622164&w=2
[5] http://marc.info/?l=linux-raid&m=132477199207506
[6] http://marc.info/?l=linux-raid&m=133665797115876&w=2
[7] https://www.marc.info/?l=linux-raid&m=142487508806844&w=3
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Possible bug in DM-RAID.
From: Austin S Hemmelgarn @ 2015-10-20 15:12 UTC (permalink / raw)
To: Linux-Kernel mailing list, linux-raid
[-- Attachment #1: Type: text/plain, Size: 3267 bytes --]
I think I've stumbled upon a bug in DM-RAID. The primary symptom is that when
creating a new DM-RAID based device (using either LVM or dmsetup) in a RAID1
configuration, it very quickly claims one by one that all of the disks failed
except the first, and goes degraded. When this happens on a given system, the
disks always 'fail' in the reverse of the order of the mirror numbers. All of
the other RAID profiles work just fine. Curiously, it also only seems to
happen for 'big' devices (I haven't been able to determine exactly what the
minimum size is, but I see it 100% of the time with 32G devices, never with 16G
ones, and only intermittently with 24G).
Here's what I got from dmesg when creating a 32G LVM volume that exhibited
this issue:
[66318.401295] device-mapper: raid: Superblocks created for new array
[66318.450452] md/raid1:mdX: active with 2 out of 2 mirrors
[66318.450467] Choosing daemon_sleep default (5 sec)
[66318.450482] created bitmap (32 pages) for device mdX
[66318.450495] attempt to access beyond end of device
[66318.450501] dm-91: rw=13329, want=0, limit=8192
[66318.450506] md: super_written gets error=-5, uptodate=0
[66318.450513] md/raid1:mdX: Disk failure on dm-92, disabling device.
md/raid1:mdX: Operation continuing on 1 devices.
[66318.459815] attempt to access beyond end of device
[66318.459819] dm-89: rw=13329, want=0, limit=8192
[66318.459822] md: super_written gets error=-5, uptodate=0
[66318.492852] attempt to access beyond end of device
[66318.492862] dm-89: rw=13329, want=0, limit=8192
[66318.492868] md: super_written gets error=-5, uptodate=0
[66318.627183] mdX: bitmap file is out of date, doing full recovery
[66318.714107] mdX: bitmap initialized from disk: read 3 pages, set 65536 of 65536 bits
[66318.782045] RAID1 conf printout:
[66318.782054] --- wd:1 rd:2
[66318.782061] disk 0, wo:0, o:1, dev:dm-90
[66318.782068] disk 1, wo:1, o:0, dev:dm-92
[66318.836598] RAID1 conf printout:
[66318.836607] --- wd:1 rd:2
[66318.836614] disk 0, wo:0, o:1, dev:dm-90
And here's output for a 24G LVM volume that didn't display the issue.
[66343.407954] device-mapper: raid: Superblocks created for new array
[66343.479065] md/raid1:mdX: active with 2 out of 2 mirrors
[66343.479078] Choosing daemon_sleep default (5 sec)
[66343.479101] created bitmap (24 pages) for device mdX
[66343.629329] mdX: bitmap file is out of date, doing full recovery
[66343.677374] mdX: bitmap initialized from disk: read 2 pages, set 49152 of 49152 bits
I'm using a lightly patched version of 4.2.3
(the source can be found at https://github.com/ferroin/linux)
but none of the patches I'm using come anywhere near anything in the block layer,
let alone the DM/MD code.
I've attempted to bisect this, although it got kind of complicated. So far I've
determined that the first commit that I see this issue on is d3b178a: md: Skip cluster setup for dm-raid
Prior to that commit, I can't initialize any dm-raid devices due to the bug it fixes.
I have not tested anything prior to d51e4fe (the merge commit that pulled in the md-cluster code),
but I do distinctly remember that I did not see this issue in 3.19.
I'll be happy to provide more info if needed.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Phil Turmel @ 2015-10-20 13:49 UTC (permalink / raw)
To: andras, linux-raid
In-Reply-To: <04cdcd6bd69b3aa1f8f24465f8485c90@tantosonline.com>
Good morning Andras,
On 10/19/2015 10:35 PM, andras@tantosonline.com wrote:
> Dear all,
>
> I have a serious (to me) problem, and I'm seeking some pro advice in
> recovering a RAID6 volume after a crash at the beginning of a reshape.
> Thank you all in advance for any help!
>
> The details:
>
> I'm running Debian.
> uname -r says:
> kernel 3.2.0-4-amd64
> dmsg says:
> Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org)
> (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.68-1+deb7u3
> mdadm -v says:
> mdadm - v3.2.5 - 18th May 2012
>
> I used to have a RAID6 volume with 7 disks on it. I've recently bought
> another 3 new HDD-s and was trying to add them to the array.
> I've put them in the machine (hot-plug), partitioned them then did:
>
> mdadm --add /dev/md1 /dev/sdh1 /dev/sdi1 /dev/sdj1
>
> This worked fine, /proc/mdstat showed them as three spares. Then I did:
>
> mdadm --grow --raid-devices=10 /dev/md1
>
> Yes, I was dumb enough to start the process without a backup option -
> (copy-paste error from https://raid.wiki.kernel.org/index.php/Growing).
The normal way to recover from this mistake is to issue
mdadm --grow --continue /dev/md1 --backup-file .....
> This immediately (well, after 2 seconds) crashed the MD driver:
Crashing is a bug, of course, but you are using an old kernel. New
kernels *generally* have fewer bugs than old kernels :-) In newer
kernels it would have just held @ 0% progress while still otherwise running.
Same observation applies to the mdadm utility too. Consider using a
relatively new rescue CD for further operations.
[trim /]
> Upon reboot, the array wouldn't assemble, it was complaining that SDA
> and SDA1 had the same superblock info on it.
>
> mdadm: WARNING /dev/sda and /dev/sda1 appear to have very similar
> superblocks.
> If they are really different, please --zero the superblock on one
> If they are the same or overlap, please remove one from the
> DEVICE list in mdadm.conf.
This is a completely separate problem, and the warning is a bit
misleading. It is a side effect of version 0.90 metadata that could not
be solved in a backward compatible manner. Which is why v1.x metadata
was created and became the default years ago. Basically, v0.90
metadata, which is placed at the end of a device, when used on the last
partition of a disk, is ambiguous about whether it belongs to the last
partition or the disk as a whole.
Normally, you can update the metadata in place from v0.90 to v1.0 with
mdadm --assemble --update=metadata ....
> At this point, I looked at the drives and it appeared that the drive
> letters got re-arranged by the kernel. My three new HDD-s (which used to
> be SDH, SDI, SDJ) now appear as SDA, SDB and SDD.
This is common and often screws people up. The kernel assigns names
based on discovery order, which varies, especially with hotplugging.
You need a map of your array and its devices versus the underlying drive
serial numbers. This is so important I created a script years ago to
generate this information. Please download and run it, and post the
results here so we can precisely tailor the instructions we give.
https://github.com/pturmel/lsdrv
> I've read up on this a little and everyone seemed to suggest that you
> repair this super-block corruption by zeroing out the suport-block, so I
> did:
>
> mdadm --zero-superblock /dev/sda1
"Everyone" was wrong. Your drives only had the one superblock. It was
just misidentified in two contexts. You destroyed the only superblock
on those devices.
[trim /]
> After this, the array would assemble, but wouldn't start, stating that
> it doesn't have enough disks in it - which is correct for the new array:
> I just removed 3 drives from a RAID6.
>
> Right now, /proc/mdstat says:
>
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md1 : inactive sdh2[0](S) sdc2[6](S) sdj1[5](S) sde1[4](S)
> sdg1[3](S) sdi1[2](S) sdf2[1](S)
> 10744335040 blocks super 0.91
>
> mdadm -E /dev/sdc2 says:
> /dev/sdc2:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad60788 - correct
> Events : 2579239
>
>
> Layout : left-symmetric
> Chunk Size : 64K
>
>
> Number Major Minor RaidDevice State
> this 6 8 98 6 active sync
>
>
> 0 0 8 50 0 active sync
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
>
> So, if I read this right, the superblock here states that the array is
> in the middle of a reshape from 7 to 10 devices, but it just started
> (4096 is the position).
Yup, just a little ways in at the beginning. Probably where it tried to
write its first critical section to the backup file.
> What's interesting is the device names listed here don't match the ones
> reported by /proc/mdstat, and are actually incorrect. The right
> partition numbers are in /proc/mdstat.
Names in the superblock are recorded per the last successful assembly.
Which is why a map of actual roles vs. drive serial numbers is so important.
> I've read in here (http://ubuntuforums.org/showthread.php?t=2133576)
> among many other places that it might be possible to recover the data on
> the array by trying to re-create it to the state before the re-shape.
Yes, since you have destroyed those superblocks, and the reshape
position is so low. You might lose a little at the beginning of your
array. Or might not, if it crashed at the first critical section as I
suspect.
> I've also read that if I want to re-create an array in read-only mode, I
> should re-create it degraded.
Not necessary or recommended in this case.
> So, what I thought I would do is this:
>
> mdadm --create /dev/md1 --level=6 --raid-devices=7 /dev/sdh2
> /dev/sdf2 /dev/sdi1 /dev/sdg1 /dev/sde1 missing missing
>
> Obviously, at this point, I'm trying to be as cautious as possible in
> not causing any further damage, if that's at all possible.
Good, because the above would destroy your array. You'd get modern
defaults for metadata version, offset, and chunk size.
Please supply all of you mdadm -E reports for the seven partitions and
the lsdrv output I requests. Just post the text inline in your reply.
Do *not* do anything else.
Phil
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Wols Lists @ 2015-10-20 13:04 UTC (permalink / raw)
To: andras, linux-raid
In-Reply-To: <04cdcd6bd69b3aa1f8f24465f8485c90@tantosonline.com>
On 20/10/15 03:35, andras@tantosonline.com wrote:
> From here on, things went downhill pretty damn fast. I was not able to
> unmount the file-system, stop or re-start the array (/proc/mdstat went
> away), any process trying to touch /dev/md1 hung, so eventually, I run
> out of options and hit the reset button on the machine.
>
> Upon reboot, the array wouldn't assemble, it was complaining that SDA
> and SDA1 had the same superblock info on it.
>
> mdadm: WARNING /dev/sda and /dev/sda1 appear to have very similar
> superblocks.
> If they are really different, please --zero the superblock on one
> If they are the same or overlap, please remove one from the
> DEVICE list in mdadm.conf.
>
> At this point, I looked at the drives and it appeared that the drive
> letters got re-arranged by the kernel. My three new HDD-s (which used to
> be SDH, SDI, SDJ) now appear as SDA, SDB and SDD.
>
> I've read up on this a little and everyone seemed to suggest that you
> repair this super-block corruption by zeroing out the suport-block, so I
> did:
>
> mdadm --zero-superblock /dev/sda1
OUCH !!!
REALLY REALLY REALLY don't do anything now until the experts chime in !!!
It looks to me like you have a 0.9 superblock, and this error message is
both common and erroneous. There's only one superblock, but it looks to
mdadm like it's both a disk superblock and a partition superblock.
You've just wiped those drives, I think ...
The experts should be able to recover it for you (I hope), but your
array is now damaged - don't damage it any further !!!
Cheers,
Wol
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Anugraha Sinha @ 2015-10-20 12:50 UTC (permalink / raw)
To: andras, linux-raid
In-Reply-To: <04cdcd6bd69b3aa1f8f24465f8485c90@tantosonline.com>
Hi Andras,
> Upon reboot, the array wouldn't assemble, it was complaining that SDA
> and SDA1 had the same superblock info on it.
>
> mdadm: WARNING /dev/sda and /dev/sda1 appear to have very similar
> superblocks.
> If they are really different, please --zero the superblock on one
> If they are the same or overlap, please remove one from the
> DEVICE list in mdadm.conf.
>
> At this point, I looked at the drives and it appeared that the drive
> letters got re-arranged by the kernel. My three new HDD-s (which used to
> be SDH, SDI, SDJ) now appear as SDA, SDB and SDD.
>
> I've read up on this a little and everyone seemed to suggest that you
> repair this super-block corruption by zeroing out the suport-block, so I
> did:
>
> mdadm --zero-superblock /dev/sda1
>
> At this point mdadm started complaining about the super-block on SDB
> (and later SDD) so I ended up zeroing out the superblock on all three of
> the new hard-drives:
>
> mdadm --zero-superblock /dev/sdb1
> mdadm --zero-superblock /dev/sdd1
Before doing zero-superblock, you should have removed the drives from
the array first. Then you should have zero'd the superblock information.
This way array, would have got to know about removal of arrays, and it
would have reassembled and started again.
Anyways, I suggest, you should first remove the devices which mdadm is
expecting to be present.
In my opinion you should first execute
[Just as a safegaurd may do this as well]
mdadm --stop /dev/md1
[then]
mdadm /dev/md1 --fail /dev/sda1 --remove /dev/sda1
mdadm /dev/md1 --fail /dev/sdb1 --remove /dev/sdb1
mdadm /dev/md1 --fail /dev/sdd1 --remove /dev/sdd1
Then check what does /proc/mdstat says.
Check mdadm -D /dev/md1 says
If things are good and you are lucky, restart the array (mdadm --run)
Thereafter try and remove existing partitions on /dev/sda, /dev/sdb &
/dev/sdd. (Using GNU Parted)
Recreate partitions, and probably mkfs on newly created partitions as well.
The above will solve the issue that /dev/sda & /dev/sda1 have similar
superblock information.
Finally take a backup and then add and grow your array again.
I hope things work for you.
Regards
Anugraha
On 10/20/2015 11:35 AM, andras@tantosonline.com wrote:
> Dear all,
>
> I have a serious (to me) problem, and I'm seeking some pro advice in
> recovering a RAID6 volume after a crash at the beginning of a reshape.
> Thank you all in advance for any help!
>
> The details:
>
> I'm running Debian.
> uname -r says:
> kernel 3.2.0-4-amd64
> dmsg says:
> Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org)
> (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.68-1+deb7u3
> mdadm -v says:
> mdadm - v3.2.5 - 18th May 2012
>
> I used to have a RAID6 volume with 7 disks on it. I've recently bought
> another 3 new HDD-s and was trying to add them to the array.
> I've put them in the machine (hot-plug), partitioned them then did:
>
> mdadm --add /dev/md1 /dev/sdh1 /dev/sdi1 /dev/sdj1
>
> This worked fine, /proc/mdstat showed them as three spares. Then I did:
>
> mdadm --grow --raid-devices=10 /dev/md1
>
> Yes, I was dumb enough to start the process without a backup option -
> (copy-paste error from https://raid.wiki.kernel.org/index.php/Growing).
>
> This immediately (well, after 2 seconds) crashed the MD driver:
>
> Oct 17 17:30:27 bazsalikom kernel: [7869821.514718] sd 0:0:0:0:
> [sdj] Attached SCSI disk
> Oct 17 18:39:21 bazsalikom kernel: [7873955.418679] sdh: sdh1
> Oct 17 18:39:37 bazsalikom kernel: [7873972.155084] sdi: sdi1
> Oct 17 18:39:49 bazsalikom kernel: [7873983.916038] sdj: sdj1
> Oct 17 18:40:33 bazsalikom kernel: [7874027.963430] md: bind<sdh1>
> Oct 17 18:40:34 bazsalikom kernel: [7874028.263656] md: bind<sdi1>
> Oct 17 18:40:34 bazsalikom kernel: [7874028.361112] md: bind<sdj1>
> Oct 17 18:59:48 bazsalikom kernel: [7875182.667815] md: reshape of
> RAID array md1
> Oct 17 18:59:48 bazsalikom kernel: [7875182.667818] md: minimum
> _guaranteed_ speed: 1000 KB/sec/disk.
> Oct 17 18:59:48 bazsalikom kernel: [7875182.667821] md: using
> maximum available idle IO bandwidth (but not more than 200000 KB/sec)
> for reshape.
> Oct 17 18:59:48 bazsalikom kernel: [7875182.667831] md: using 128k
> window, over a total of 1465135936k.
> --> Oct 17 18:59:50 bazsalikom kernel: [7875184.326245] md: md_do_sync()
> got signal ... exiting
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928059] md1_raid6 D
> ffff88021fc12780 0 282 2 0x00000000
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928066]
> ffff880213fd9140 0000000000000046 ffff8800aa80c140 ffff880201fe08c0
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928073]
> 0000000000012780 ffff880211845fd8 ffff880211845fd8 ffff880213fd9140
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928079]
> ffff8800a77d8a40 ffffffff81071331 0000000000000046 ffff8802135a0c00
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928085] Call Trace:
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928095]
> [<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928111]
> [<ffffffffa0124c6c>] ? check_reshape+0x27b/0x51a [raid456]
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928128]
> [<ffffffffa013ade4>] ? scsi_request_fn+0x443/0x51e [scsi_mod]
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928134]
> [<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928144]
> [<ffffffffa00ef3b8>] ? md_check_recovery+0x2a5/0x514 [md_mod]
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928151]
> [<ffffffffa01286c7>] ? raid5d+0x1c/0x483 [raid456]
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928156]
> [<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928160]
> [<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928169]
> [<ffffffffa00e9256>] ? md_thread+0x114/0x132 [md_mod]
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928174]
> [<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928183]
> [<ffffffffa00e9142>] ? md_rdev_init+0xea/0xea [md_mod]
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928188]
> [<ffffffff8105f7a1>] ? kthread+0x76/0x7e
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928194]
> [<ffffffff81357ff4>] ? kernel_thread_helper+0x4/0x10
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928199]
> [<ffffffff8105f72b>] ? kthread_worker_fn+0x139/0x139
> Oct 17 19:02:46 bazsalikom kernel: [7875360.928204]
> [<ffffffff81357ff0>] ? gs_change+0x13/0x13
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928055] md1_raid6 D
> ffff88021fc12780 0 282 2 0x00000000
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928062]
> ffff880213fd9140 0000000000000046 ffff8800aa80c140 ffff880201fe08c0
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928069]
> 0000000000012780 ffff880211845fd8 ffff880211845fd8 ffff880213fd9140
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928075]
> ffff8800a77d8a40 ffffffff81071331 0000000000000046 ffff8802135a0c00
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928082] Call Trace:
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928091]
> [<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928108]
> [<ffffffffa0124c6c>] ? check_reshape+0x27b/0x51a [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928124]
> [<ffffffffa013ade4>] ? scsi_request_fn+0x443/0x51e [scsi_mod]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928130]
> [<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928141]
> [<ffffffffa00ef3b8>] ? md_check_recovery+0x2a5/0x514 [md_mod]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928148]
> [<ffffffffa01286c7>] ? raid5d+0x1c/0x483 [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928153]
> [<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928157]
> [<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928166]
> [<ffffffffa00e9256>] ? md_thread+0x114/0x132 [md_mod]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928171]
> [<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928180]
> [<ffffffffa00e9142>] ? md_rdev_init+0xea/0xea [md_mod]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928185]
> [<ffffffff8105f7a1>] ? kthread+0x76/0x7e
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928191]
> [<ffffffff81357ff4>] ? kernel_thread_helper+0x4/0x10
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928196]
> [<ffffffff8105f72b>] ? kthread_worker_fn+0x139/0x139
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928200]
> [<ffffffff81357ff0>] ? gs_change+0x13/0x13
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928212] jbd2/md1-8 D
> ffff88021fc92780 0 1731 2 0x00000000
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928218]
> ffff880213693180 0000000000000046 ffff880200000000 ffff880216d04180
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928224]
> 0000000000012780 ffff880213df3fd8 ffff880213df3fd8 ffff880213693180
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928230]
> 0000000000000000 00000001135a0d70 ffff8802135a0d60 ffff8802135a0d70
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928236] Call Trace:
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928243]
> [<ffffffffa0123804>] ? get_active_stripe+0x24c/0x505 [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928248]
> [<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928255]
> [<ffffffffa01258c8>] ? make_request+0x1b4/0x37a [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928260]
> [<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928278]
> [<ffffffffa00e8d47>] ? md_make_request+0xee/0x1db [md_mod]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928283]
> [<ffffffff8119a3ec>] ? generic_make_request+0x90/0xcf
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928287]
> [<ffffffff8119a4fe>] ? submit_bio+0xd3/0xf1
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928293]
> [<ffffffff81121b78>] ? bio_alloc_bioset+0x43/0xb6
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928297]
> [<ffffffff8111da68>] ? submit_bh+0xe2/0xff
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928304]
> [<ffffffffa0167674>] ? jbd2_journal_commit_transaction+0x803/0x10bf [jbd2]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928309]
> [<ffffffff8100d02f>] ? load_TLS+0x7/0xa
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928313]
> [<ffffffff8100d69e>] ? __switch_to+0x133/0x258
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928318]
> [<ffffffff81350dd1>] ? _raw_spin_lock_irqsave+0x9/0x25
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928323]
> [<ffffffff8105267a>] ? lock_timer_base.isra.29+0x23/0x47
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928330]
> [<ffffffffa016b166>] ? kjournald2+0xc0/0x20a [jbd2]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928334]
> [<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928341]
> [<ffffffffa016b0a6>] ? commit_timeout+0x5/0x5 [jbd2]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928345]
> [<ffffffff8105f7a1>] ? kthread+0x76/0x7e
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928349]
> [<ffffffff81357ff4>] ? kernel_thread_helper+0x4/0x10
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928354]
> [<ffffffff8105f72b>] ? kthread_worker_fn+0x139/0x139
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928358]
> [<ffffffff81357ff0>] ? gs_change+0x13/0x13
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928408] smbd D
> ffff88021fc12780 0 3063 25481 0x00000000
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928413]
> ffff880213e07780 0000000000000082 0000000000000000 ffffffff8160d020
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928418]
> 0000000000012780 ffff880003cabfd8 ffff880003cabfd8 ffff880213e07780
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928424]
> 0000000000000000 00000001135a0d70 ffff8802135a0d60 ffff8802135a0d70
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928429] Call Trace:
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928435]
> [<ffffffffa0123804>] ? get_active_stripe+0x24c/0x505 [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928439]
> [<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928445]
> [<ffffffffa01258c8>] ? make_request+0x1b4/0x37a [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928450]
> [<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928457]
> [<ffffffffa00e8d47>] ? md_make_request+0xee/0x1db [md_mod]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928468]
> [<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928473]
> [<ffffffff8119a3ec>] ? generic_make_request+0x90/0xcf
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928477]
> [<ffffffff8119a4fe>] ? submit_bio+0xd3/0xf1
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928482]
> [<ffffffff810bedab>] ? __lru_cache_add+0x2b/0x51
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928486]
> [<ffffffff811259dd>] ? mpage_readpages+0x113/0x134
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928496]
> [<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928500]
> [<ffffffff81109033>] ? poll_freewait+0x97/0x97
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928505]
> [<ffffffff81036628>] ? should_resched+0x5/0x23
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928508]
> [<ffffffff8134fa44>] ? _cond_resched+0x7/0x1c
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928513]
> [<ffffffff810bdd31>] ? __do_page_cache_readahead+0x11e/0x1c3
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928517]
> [<ffffffff810be02e>] ? ra_submit+0x19/0x1d
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928522]
> [<ffffffff810b689b>] ? generic_file_aio_read+0x282/0x5cf
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928528]
> [<ffffffff810fadc4>] ? do_sync_read+0xb4/0xec
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928532]
> [<ffffffff810fb4af>] ? vfs_read+0x9f/0xe6
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928536]
> [<ffffffff810fb61f>] ? sys_pread64+0x53/0x6e
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928540]
> [<ffffffff81355e92>] ? system_call_fastpath+0x16/0x1b
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928549] imap D
> ffff88021fc12780 0 3121 4613 0x00000000
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928554]
> ffff880216db1100 0000000000000082 ffffea0000000000 ffffffff8160d020
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928559]
> 0000000000012780 ffff8800cf5b1fd8 ffff8800cf5b1fd8 ffff880216db1100
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928564]
> 0000000000000000 00000001135a0d70 ffff8802135a0d60 ffff8802135a0d70
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928569] Call Trace:
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928576]
> [<ffffffffa0123804>] ? get_active_stripe+0x24c/0x505 [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928580]
> [<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928585]
> [<ffffffffa01258c8>] ? make_request+0x1b4/0x37a [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928590]
> [<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928597]
> [<ffffffffa00e8d47>] ? md_make_request+0xee/0x1db [md_mod]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928607]
> [<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928611]
> [<ffffffff8119a3ec>] ? generic_make_request+0x90/0xcf
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928615]
> [<ffffffff8119a4fe>] ? submit_bio+0xd3/0xf1
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928619]
> [<ffffffff810bedab>] ? __lru_cache_add+0x2b/0x51
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928623]
> [<ffffffff811259dd>] ? mpage_readpages+0x113/0x134
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928633]
> [<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928637]
> [<ffffffff8110b27f>] ? dput+0x27/0xee
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928641]
> [<ffffffff811110df>] ? mntput_no_expire+0x1e/0xc9
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928646]
> [<ffffffff810bdd31>] ? __do_page_cache_readahead+0x11e/0x1c3
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928650]
> [<ffffffff810bdff1>] ? force_page_cache_readahead+0x5f/0x83
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928654]
> [<ffffffff810b85e5>] ? sys_fadvise64_64+0x141/0x1e2
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928658]
> [<ffffffff81355e92>] ? system_call_fastpath+0x16/0x1b
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928667] smbd D
> ffff88021fc12780 0 3155 25481 0x00000000
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928672]
> ffff8802135d8780 0000000000000086 0000000000000000 ffffffff8160d020
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928677]
> 0000000000012780 ffff880005267fd8 ffff880005267fd8 ffff8802135d8780
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928683]
> 0000000000000000 00000001135a0d70 ffff8802135a0d60 ffff8802135a0d70
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928688] Call Trace:
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928694]
> [<ffffffffa0123804>] ? get_active_stripe+0x24c/0x505 [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928698]
> [<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928704]
> [<ffffffffa01258c8>] ? make_request+0x1b4/0x37a [raid456]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928708]
> [<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928715]
> [<ffffffffa00e8d47>] ? md_make_request+0xee/0x1db [md_mod]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928725]
> [<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928729]
> [<ffffffff8119a3ec>] ? generic_make_request+0x90/0xcf
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928733]
> [<ffffffff8119a4fe>] ? submit_bio+0xd3/0xf1
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928737]
> [<ffffffff810bedab>] ? __lru_cache_add+0x2b/0x51
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928741]
> [<ffffffff811259dd>] ? mpage_readpages+0x113/0x134
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928751]
> [<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928755]
> [<ffffffff81109033>] ? poll_freewait+0x97/0x97
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928759]
> [<ffffffff81036628>] ? should_resched+0x5/0x23
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928762]
> [<ffffffff8134fa44>] ? _cond_resched+0x7/0x1c
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928767]
> [<ffffffff810bdd31>] ? __do_page_cache_readahead+0x11e/0x1c3
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928771]
> [<ffffffff810be02e>] ? ra_submit+0x19/0x1d
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928775]
> [<ffffffff810b689b>] ? generic_file_aio_read+0x282/0x5cf
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928780]
> [<ffffffff810fadc4>] ? do_sync_read+0xb4/0xec
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928784]
> [<ffffffff810fb4af>] ? vfs_read+0x9f/0xe6
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928788]
> [<ffffffff810fb61f>] ? sys_pread64+0x53/0x6e
> Oct 17 19:04:46 bazsalikom kernel: [7875480.928792]
> [<ffffffff81355e92>] ? system_call_fastpath+0x16/0x1b
>
> From here on, things went downhill pretty damn fast. I was not able to
> unmount the file-system, stop or re-start the array (/proc/mdstat went
> away), any process trying to touch /dev/md1 hung, so eventually, I run
> out of options and hit the reset button on the machine.
>
> Upon reboot, the array wouldn't assemble, it was complaining that SDA
> and SDA1 had the same superblock info on it.
>
> mdadm: WARNING /dev/sda and /dev/sda1 appear to have very similar
> superblocks.
> If they are really different, please --zero the superblock on one
> If they are the same or overlap, please remove one from the
> DEVICE list in mdadm.conf.
>
> At this point, I looked at the drives and it appeared that the drive
> letters got re-arranged by the kernel. My three new HDD-s (which used to
> be SDH, SDI, SDJ) now appear as SDA, SDB and SDD.
>
> I've read up on this a little and everyone seemed to suggest that you
> repair this super-block corruption by zeroing out the suport-block, so I
> did:
>
> mdadm --zero-superblock /dev/sda1
>
> At this point mdadm started complaining about the super-block on SDB
> (and later SDD) so I ended up zeroing out the superblock on all three of
> the new hard-drives:
>
> mdadm --zero-superblock /dev/sdb1
> mdadm --zero-superblock /dev/sdd1
>
> After this, the array would assemble, but wouldn't start, stating that
> it doesn't have enough disks in it - which is correct for the new array:
> I just removed 3 drives from a RAID6.
>
> Right now, /proc/mdstat says:
>
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md1 : inactive sdh2[0](S) sdc2[6](S) sdj1[5](S) sde1[4](S)
> sdg1[3](S) sdi1[2](S) sdf2[1](S)
> 10744335040 blocks super 0.91
>
> mdadm -E /dev/sdc2 says:
> /dev/sdc2:
> Magic : a92b4efc
> Version : 0.91.00
> UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
> Creation Time : Sat Oct 2 07:21:53 2010
> Raid Level : raid6
> Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
> Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 1
>
>
> Reshape pos'n : 4096
> Delta Devices : 3 (7->10)
>
>
> Update Time : Sat Oct 17 18:59:50 2015
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : fad60788 - correct
> Events : 2579239
>
>
> Layout : left-symmetric
> Chunk Size : 64K
>
>
> Number Major Minor RaidDevice State
> this 6 8 98 6 active sync
>
>
> 0 0 8 50 0 active sync
> 1 1 8 18 1 active sync
> 2 2 8 65 2 active sync /dev/sde1
> 3 3 8 33 3 active sync /dev/sdc1
> 4 4 8 1 4 active sync /dev/sda1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 98 6 active sync
> 7 7 8 145 7 active sync /dev/sdj1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 113 9 active sync /dev/sdh1
>
> So, if I read this right, the superblock here states that the array is
> in the middle of a reshape from 7 to 10 devices, but it just started
> (4096 is the position).
> What's interesting is the device names listed here don't match the ones
> reported by /proc/mdstat, and are actually incorrect. The right
> partition numbers are in /proc/mdstat.
>
> The superblocks on the 6 other original disks match, except for of
> course which one they mark as 'this' and the checksum.
>
> I've read in here (http://ubuntuforums.org/showthread.php?t=2133576)
> among many other places that it might be possible to recover the data on
> the array by trying to re-create it to the state before the re-shape.
>
> I've also read that if I want to re-create an array in read-only mode, I
> should re-create it degraded.
>
> So, what I thought I would do is this:
>
> mdadm --create /dev/md1 --level=6 --raid-devices=7 /dev/sdh2
> /dev/sdf2 /dev/sdi1 /dev/sdg1 /dev/sde1 missing missing
>
> Obviously, at this point, I'm trying to be as cautious as possible in
> not causing any further damage, if that's at all possible.
>
> It seems that this issue has some similarities to this bug:
> https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1001019
>
> So, please all mdadm gurus, help me out! How can I recover as much of
> the data on this volume as possible?
>
> Thanks again,
> Andras Tantos
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* How to recover after md crash during reshape?
From: andras @ 2015-10-20 2:35 UTC (permalink / raw)
To: linux-raid
Dear all,
I have a serious (to me) problem, and I'm seeking some pro advice in
recovering a RAID6 volume after a crash at the beginning of a reshape.
Thank you all in advance for any help!
The details:
I'm running Debian.
uname -r says:
kernel 3.2.0-4-amd64
dmsg says:
Linux version 3.2.0-4-amd64 (debian-kernel@lists.debian.org)
(gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.68-1+deb7u3
mdadm -v says:
mdadm - v3.2.5 - 18th May 2012
I used to have a RAID6 volume with 7 disks on it. I've recently bought
another 3 new HDD-s and was trying to add them to the array.
I've put them in the machine (hot-plug), partitioned them then did:
mdadm --add /dev/md1 /dev/sdh1 /dev/sdi1 /dev/sdj1
This worked fine, /proc/mdstat showed them as three spares. Then I did:
mdadm --grow --raid-devices=10 /dev/md1
Yes, I was dumb enough to start the process without a backup option -
(copy-paste error from https://raid.wiki.kernel.org/index.php/Growing).
This immediately (well, after 2 seconds) crashed the MD driver:
Oct 17 17:30:27 bazsalikom kernel: [7869821.514718] sd 0:0:0:0:
[sdj] Attached SCSI disk
Oct 17 18:39:21 bazsalikom kernel: [7873955.418679] sdh: sdh1
Oct 17 18:39:37 bazsalikom kernel: [7873972.155084] sdi: sdi1
Oct 17 18:39:49 bazsalikom kernel: [7873983.916038] sdj: sdj1
Oct 17 18:40:33 bazsalikom kernel: [7874027.963430] md: bind<sdh1>
Oct 17 18:40:34 bazsalikom kernel: [7874028.263656] md: bind<sdi1>
Oct 17 18:40:34 bazsalikom kernel: [7874028.361112] md: bind<sdj1>
Oct 17 18:59:48 bazsalikom kernel: [7875182.667815] md: reshape of
RAID array md1
Oct 17 18:59:48 bazsalikom kernel: [7875182.667818] md: minimum
_guaranteed_ speed: 1000 KB/sec/disk.
Oct 17 18:59:48 bazsalikom kernel: [7875182.667821] md: using
maximum available idle IO bandwidth (but not more than 200000 KB/sec)
for reshape.
Oct 17 18:59:48 bazsalikom kernel: [7875182.667831] md: using 128k
window, over a total of 1465135936k.
--> Oct 17 18:59:50 bazsalikom kernel: [7875184.326245] md: md_do_sync()
got signal ... exiting
Oct 17 19:02:46 bazsalikom kernel: [7875360.928059] md1_raid6
D ffff88021fc12780 0 282 2 0x00000000
Oct 17 19:02:46 bazsalikom kernel: [7875360.928066]
ffff880213fd9140 0000000000000046 ffff8800aa80c140 ffff880201fe08c0
Oct 17 19:02:46 bazsalikom kernel: [7875360.928073]
0000000000012780 ffff880211845fd8 ffff880211845fd8 ffff880213fd9140
Oct 17 19:02:46 bazsalikom kernel: [7875360.928079]
ffff8800a77d8a40 ffffffff81071331 0000000000000046 ffff8802135a0c00
Oct 17 19:02:46 bazsalikom kernel: [7875360.928085] Call Trace:
Oct 17 19:02:46 bazsalikom kernel: [7875360.928095]
[<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
Oct 17 19:02:46 bazsalikom kernel: [7875360.928111]
[<ffffffffa0124c6c>] ? check_reshape+0x27b/0x51a [raid456]
Oct 17 19:02:46 bazsalikom kernel: [7875360.928128]
[<ffffffffa013ade4>] ? scsi_request_fn+0x443/0x51e [scsi_mod]
Oct 17 19:02:46 bazsalikom kernel: [7875360.928134]
[<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
Oct 17 19:02:46 bazsalikom kernel: [7875360.928144]
[<ffffffffa00ef3b8>] ? md_check_recovery+0x2a5/0x514 [md_mod]
Oct 17 19:02:46 bazsalikom kernel: [7875360.928151]
[<ffffffffa01286c7>] ? raid5d+0x1c/0x483 [raid456]
Oct 17 19:02:46 bazsalikom kernel: [7875360.928156]
[<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
Oct 17 19:02:46 bazsalikom kernel: [7875360.928160]
[<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
Oct 17 19:02:46 bazsalikom kernel: [7875360.928169]
[<ffffffffa00e9256>] ? md_thread+0x114/0x132 [md_mod]
Oct 17 19:02:46 bazsalikom kernel: [7875360.928174]
[<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
Oct 17 19:02:46 bazsalikom kernel: [7875360.928183]
[<ffffffffa00e9142>] ? md_rdev_init+0xea/0xea [md_mod]
Oct 17 19:02:46 bazsalikom kernel: [7875360.928188]
[<ffffffff8105f7a1>] ? kthread+0x76/0x7e
Oct 17 19:02:46 bazsalikom kernel: [7875360.928194]
[<ffffffff81357ff4>] ? kernel_thread_helper+0x4/0x10
Oct 17 19:02:46 bazsalikom kernel: [7875360.928199]
[<ffffffff8105f72b>] ? kthread_worker_fn+0x139/0x139
Oct 17 19:02:46 bazsalikom kernel: [7875360.928204]
[<ffffffff81357ff0>] ? gs_change+0x13/0x13
Oct 17 19:04:46 bazsalikom kernel: [7875480.928055] md1_raid6
D ffff88021fc12780 0 282 2 0x00000000
Oct 17 19:04:46 bazsalikom kernel: [7875480.928062]
ffff880213fd9140 0000000000000046 ffff8800aa80c140 ffff880201fe08c0
Oct 17 19:04:46 bazsalikom kernel: [7875480.928069]
0000000000012780 ffff880211845fd8 ffff880211845fd8 ffff880213fd9140
Oct 17 19:04:46 bazsalikom kernel: [7875480.928075]
ffff8800a77d8a40 ffffffff81071331 0000000000000046 ffff8802135a0c00
Oct 17 19:04:46 bazsalikom kernel: [7875480.928082] Call Trace:
Oct 17 19:04:46 bazsalikom kernel: [7875480.928091]
[<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
Oct 17 19:04:46 bazsalikom kernel: [7875480.928108]
[<ffffffffa0124c6c>] ? check_reshape+0x27b/0x51a [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928124]
[<ffffffffa013ade4>] ? scsi_request_fn+0x443/0x51e [scsi_mod]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928130]
[<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
Oct 17 19:04:46 bazsalikom kernel: [7875480.928141]
[<ffffffffa00ef3b8>] ? md_check_recovery+0x2a5/0x514 [md_mod]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928148]
[<ffffffffa01286c7>] ? raid5d+0x1c/0x483 [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928153]
[<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
Oct 17 19:04:46 bazsalikom kernel: [7875480.928157]
[<ffffffff81071331>] ? arch_local_irq_save+0x11/0x17
Oct 17 19:04:46 bazsalikom kernel: [7875480.928166]
[<ffffffffa00e9256>] ? md_thread+0x114/0x132 [md_mod]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928171]
[<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
Oct 17 19:04:46 bazsalikom kernel: [7875480.928180]
[<ffffffffa00e9142>] ? md_rdev_init+0xea/0xea [md_mod]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928185]
[<ffffffff8105f7a1>] ? kthread+0x76/0x7e
Oct 17 19:04:46 bazsalikom kernel: [7875480.928191]
[<ffffffff81357ff4>] ? kernel_thread_helper+0x4/0x10
Oct 17 19:04:46 bazsalikom kernel: [7875480.928196]
[<ffffffff8105f72b>] ? kthread_worker_fn+0x139/0x139
Oct 17 19:04:46 bazsalikom kernel: [7875480.928200]
[<ffffffff81357ff0>] ? gs_change+0x13/0x13
Oct 17 19:04:46 bazsalikom kernel: [7875480.928212] jbd2/md1-8
D ffff88021fc92780 0 1731 2 0x00000000
Oct 17 19:04:46 bazsalikom kernel: [7875480.928218]
ffff880213693180 0000000000000046 ffff880200000000 ffff880216d04180
Oct 17 19:04:46 bazsalikom kernel: [7875480.928224]
0000000000012780 ffff880213df3fd8 ffff880213df3fd8 ffff880213693180
Oct 17 19:04:46 bazsalikom kernel: [7875480.928230]
0000000000000000 00000001135a0d70 ffff8802135a0d60 ffff8802135a0d70
Oct 17 19:04:46 bazsalikom kernel: [7875480.928236] Call Trace:
Oct 17 19:04:46 bazsalikom kernel: [7875480.928243]
[<ffffffffa0123804>] ? get_active_stripe+0x24c/0x505 [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928248]
[<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
Oct 17 19:04:46 bazsalikom kernel: [7875480.928255]
[<ffffffffa01258c8>] ? make_request+0x1b4/0x37a [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928260]
[<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
Oct 17 19:04:46 bazsalikom kernel: [7875480.928278]
[<ffffffffa00e8d47>] ? md_make_request+0xee/0x1db [md_mod]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928283]
[<ffffffff8119a3ec>] ? generic_make_request+0x90/0xcf
Oct 17 19:04:46 bazsalikom kernel: [7875480.928287]
[<ffffffff8119a4fe>] ? submit_bio+0xd3/0xf1
Oct 17 19:04:46 bazsalikom kernel: [7875480.928293]
[<ffffffff81121b78>] ? bio_alloc_bioset+0x43/0xb6
Oct 17 19:04:46 bazsalikom kernel: [7875480.928297]
[<ffffffff8111da68>] ? submit_bh+0xe2/0xff
Oct 17 19:04:46 bazsalikom kernel: [7875480.928304]
[<ffffffffa0167674>] ? jbd2_journal_commit_transaction+0x803/0x10bf
[jbd2]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928309]
[<ffffffff8100d02f>] ? load_TLS+0x7/0xa
Oct 17 19:04:46 bazsalikom kernel: [7875480.928313]
[<ffffffff8100d69e>] ? __switch_to+0x133/0x258
Oct 17 19:04:46 bazsalikom kernel: [7875480.928318]
[<ffffffff81350dd1>] ? _raw_spin_lock_irqsave+0x9/0x25
Oct 17 19:04:46 bazsalikom kernel: [7875480.928323]
[<ffffffff8105267a>] ? lock_timer_base.isra.29+0x23/0x47
Oct 17 19:04:46 bazsalikom kernel: [7875480.928330]
[<ffffffffa016b166>] ? kjournald2+0xc0/0x20a [jbd2]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928334]
[<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
Oct 17 19:04:46 bazsalikom kernel: [7875480.928341]
[<ffffffffa016b0a6>] ? commit_timeout+0x5/0x5 [jbd2]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928345]
[<ffffffff8105f7a1>] ? kthread+0x76/0x7e
Oct 17 19:04:46 bazsalikom kernel: [7875480.928349]
[<ffffffff81357ff4>] ? kernel_thread_helper+0x4/0x10
Oct 17 19:04:46 bazsalikom kernel: [7875480.928354]
[<ffffffff8105f72b>] ? kthread_worker_fn+0x139/0x139
Oct 17 19:04:46 bazsalikom kernel: [7875480.928358]
[<ffffffff81357ff0>] ? gs_change+0x13/0x13
Oct 17 19:04:46 bazsalikom kernel: [7875480.928408] smbd
D ffff88021fc12780 0 3063 25481 0x00000000
Oct 17 19:04:46 bazsalikom kernel: [7875480.928413]
ffff880213e07780 0000000000000082 0000000000000000 ffffffff8160d020
Oct 17 19:04:46 bazsalikom kernel: [7875480.928418]
0000000000012780 ffff880003cabfd8 ffff880003cabfd8 ffff880213e07780
Oct 17 19:04:46 bazsalikom kernel: [7875480.928424]
0000000000000000 00000001135a0d70 ffff8802135a0d60 ffff8802135a0d70
Oct 17 19:04:46 bazsalikom kernel: [7875480.928429] Call Trace:
Oct 17 19:04:46 bazsalikom kernel: [7875480.928435]
[<ffffffffa0123804>] ? get_active_stripe+0x24c/0x505 [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928439]
[<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
Oct 17 19:04:46 bazsalikom kernel: [7875480.928445]
[<ffffffffa01258c8>] ? make_request+0x1b4/0x37a [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928450]
[<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
Oct 17 19:04:46 bazsalikom kernel: [7875480.928457]
[<ffffffffa00e8d47>] ? md_make_request+0xee/0x1db [md_mod]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928468]
[<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928473]
[<ffffffff8119a3ec>] ? generic_make_request+0x90/0xcf
Oct 17 19:04:46 bazsalikom kernel: [7875480.928477]
[<ffffffff8119a4fe>] ? submit_bio+0xd3/0xf1
Oct 17 19:04:46 bazsalikom kernel: [7875480.928482]
[<ffffffff810bedab>] ? __lru_cache_add+0x2b/0x51
Oct 17 19:04:46 bazsalikom kernel: [7875480.928486]
[<ffffffff811259dd>] ? mpage_readpages+0x113/0x134
Oct 17 19:04:46 bazsalikom kernel: [7875480.928496]
[<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928500]
[<ffffffff81109033>] ? poll_freewait+0x97/0x97
Oct 17 19:04:46 bazsalikom kernel: [7875480.928505]
[<ffffffff81036628>] ? should_resched+0x5/0x23
Oct 17 19:04:46 bazsalikom kernel: [7875480.928508]
[<ffffffff8134fa44>] ? _cond_resched+0x7/0x1c
Oct 17 19:04:46 bazsalikom kernel: [7875480.928513]
[<ffffffff810bdd31>] ? __do_page_cache_readahead+0x11e/0x1c3
Oct 17 19:04:46 bazsalikom kernel: [7875480.928517]
[<ffffffff810be02e>] ? ra_submit+0x19/0x1d
Oct 17 19:04:46 bazsalikom kernel: [7875480.928522]
[<ffffffff810b689b>] ? generic_file_aio_read+0x282/0x5cf
Oct 17 19:04:46 bazsalikom kernel: [7875480.928528]
[<ffffffff810fadc4>] ? do_sync_read+0xb4/0xec
Oct 17 19:04:46 bazsalikom kernel: [7875480.928532]
[<ffffffff810fb4af>] ? vfs_read+0x9f/0xe6
Oct 17 19:04:46 bazsalikom kernel: [7875480.928536]
[<ffffffff810fb61f>] ? sys_pread64+0x53/0x6e
Oct 17 19:04:46 bazsalikom kernel: [7875480.928540]
[<ffffffff81355e92>] ? system_call_fastpath+0x16/0x1b
Oct 17 19:04:46 bazsalikom kernel: [7875480.928549] imap
D ffff88021fc12780 0 3121 4613 0x00000000
Oct 17 19:04:46 bazsalikom kernel: [7875480.928554]
ffff880216db1100 0000000000000082 ffffea0000000000 ffffffff8160d020
Oct 17 19:04:46 bazsalikom kernel: [7875480.928559]
0000000000012780 ffff8800cf5b1fd8 ffff8800cf5b1fd8 ffff880216db1100
Oct 17 19:04:46 bazsalikom kernel: [7875480.928564]
0000000000000000 00000001135a0d70 ffff8802135a0d60 ffff8802135a0d70
Oct 17 19:04:46 bazsalikom kernel: [7875480.928569] Call Trace:
Oct 17 19:04:46 bazsalikom kernel: [7875480.928576]
[<ffffffffa0123804>] ? get_active_stripe+0x24c/0x505 [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928580]
[<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
Oct 17 19:04:46 bazsalikom kernel: [7875480.928585]
[<ffffffffa01258c8>] ? make_request+0x1b4/0x37a [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928590]
[<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
Oct 17 19:04:46 bazsalikom kernel: [7875480.928597]
[<ffffffffa00e8d47>] ? md_make_request+0xee/0x1db [md_mod]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928607]
[<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928611]
[<ffffffff8119a3ec>] ? generic_make_request+0x90/0xcf
Oct 17 19:04:46 bazsalikom kernel: [7875480.928615]
[<ffffffff8119a4fe>] ? submit_bio+0xd3/0xf1
Oct 17 19:04:46 bazsalikom kernel: [7875480.928619]
[<ffffffff810bedab>] ? __lru_cache_add+0x2b/0x51
Oct 17 19:04:46 bazsalikom kernel: [7875480.928623]
[<ffffffff811259dd>] ? mpage_readpages+0x113/0x134
Oct 17 19:04:46 bazsalikom kernel: [7875480.928633]
[<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928637]
[<ffffffff8110b27f>] ? dput+0x27/0xee
Oct 17 19:04:46 bazsalikom kernel: [7875480.928641]
[<ffffffff811110df>] ? mntput_no_expire+0x1e/0xc9
Oct 17 19:04:46 bazsalikom kernel: [7875480.928646]
[<ffffffff810bdd31>] ? __do_page_cache_readahead+0x11e/0x1c3
Oct 17 19:04:46 bazsalikom kernel: [7875480.928650]
[<ffffffff810bdff1>] ? force_page_cache_readahead+0x5f/0x83
Oct 17 19:04:46 bazsalikom kernel: [7875480.928654]
[<ffffffff810b85e5>] ? sys_fadvise64_64+0x141/0x1e2
Oct 17 19:04:46 bazsalikom kernel: [7875480.928658]
[<ffffffff81355e92>] ? system_call_fastpath+0x16/0x1b
Oct 17 19:04:46 bazsalikom kernel: [7875480.928667] smbd
D ffff88021fc12780 0 3155 25481 0x00000000
Oct 17 19:04:46 bazsalikom kernel: [7875480.928672]
ffff8802135d8780 0000000000000086 0000000000000000 ffffffff8160d020
Oct 17 19:04:46 bazsalikom kernel: [7875480.928677]
0000000000012780 ffff880005267fd8 ffff880005267fd8 ffff8802135d8780
Oct 17 19:04:46 bazsalikom kernel: [7875480.928683]
0000000000000000 00000001135a0d70 ffff8802135a0d60 ffff8802135a0d70
Oct 17 19:04:46 bazsalikom kernel: [7875480.928688] Call Trace:
Oct 17 19:04:46 bazsalikom kernel: [7875480.928694]
[<ffffffffa0123804>] ? get_active_stripe+0x24c/0x505 [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928698]
[<ffffffff8103f6e2>] ? try_to_wake_up+0x197/0x197
Oct 17 19:04:46 bazsalikom kernel: [7875480.928704]
[<ffffffffa01258c8>] ? make_request+0x1b4/0x37a [raid456]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928708]
[<ffffffff8105fdf3>] ? add_wait_queue+0x3c/0x3c
Oct 17 19:04:46 bazsalikom kernel: [7875480.928715]
[<ffffffffa00e8d47>] ? md_make_request+0xee/0x1db [md_mod]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928725]
[<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928729]
[<ffffffff8119a3ec>] ? generic_make_request+0x90/0xcf
Oct 17 19:04:46 bazsalikom kernel: [7875480.928733]
[<ffffffff8119a4fe>] ? submit_bio+0xd3/0xf1
Oct 17 19:04:46 bazsalikom kernel: [7875480.928737]
[<ffffffff810bedab>] ? __lru_cache_add+0x2b/0x51
Oct 17 19:04:46 bazsalikom kernel: [7875480.928741]
[<ffffffff811259dd>] ? mpage_readpages+0x113/0x134
Oct 17 19:04:46 bazsalikom kernel: [7875480.928751]
[<ffffffffa017d19a>] ? noalloc_get_block_write+0x17/0x17 [ext4]
Oct 17 19:04:46 bazsalikom kernel: [7875480.928755]
[<ffffffff81109033>] ? poll_freewait+0x97/0x97
Oct 17 19:04:46 bazsalikom kernel: [7875480.928759]
[<ffffffff81036628>] ? should_resched+0x5/0x23
Oct 17 19:04:46 bazsalikom kernel: [7875480.928762]
[<ffffffff8134fa44>] ? _cond_resched+0x7/0x1c
Oct 17 19:04:46 bazsalikom kernel: [7875480.928767]
[<ffffffff810bdd31>] ? __do_page_cache_readahead+0x11e/0x1c3
Oct 17 19:04:46 bazsalikom kernel: [7875480.928771]
[<ffffffff810be02e>] ? ra_submit+0x19/0x1d
Oct 17 19:04:46 bazsalikom kernel: [7875480.928775]
[<ffffffff810b689b>] ? generic_file_aio_read+0x282/0x5cf
Oct 17 19:04:46 bazsalikom kernel: [7875480.928780]
[<ffffffff810fadc4>] ? do_sync_read+0xb4/0xec
Oct 17 19:04:46 bazsalikom kernel: [7875480.928784]
[<ffffffff810fb4af>] ? vfs_read+0x9f/0xe6
Oct 17 19:04:46 bazsalikom kernel: [7875480.928788]
[<ffffffff810fb61f>] ? sys_pread64+0x53/0x6e
Oct 17 19:04:46 bazsalikom kernel: [7875480.928792]
[<ffffffff81355e92>] ? system_call_fastpath+0x16/0x1b
From here on, things went downhill pretty damn fast. I was not able to
unmount the file-system, stop or re-start the array (/proc/mdstat went
away), any process trying to touch /dev/md1 hung, so eventually, I run
out of options and hit the reset button on the machine.
Upon reboot, the array wouldn't assemble, it was complaining that SDA
and SDA1 had the same superblock info on it.
mdadm: WARNING /dev/sda and /dev/sda1 appear to have very similar
superblocks.
If they are really different, please --zero the superblock on one
If they are the same or overlap, please remove one from the
DEVICE list in mdadm.conf.
At this point, I looked at the drives and it appeared that the drive
letters got re-arranged by the kernel. My three new HDD-s (which used to
be SDH, SDI, SDJ) now appear as SDA, SDB and SDD.
I've read up on this a little and everyone seemed to suggest that you
repair this super-block corruption by zeroing out the suport-block, so I
did:
mdadm --zero-superblock /dev/sda1
At this point mdadm started complaining about the super-block on SDB
(and later SDD) so I ended up zeroing out the superblock on all three of
the new hard-drives:
mdadm --zero-superblock /dev/sdb1
mdadm --zero-superblock /dev/sdd1
After this, the array would assemble, but wouldn't start, stating that
it doesn't have enough disks in it - which is correct for the new array:
I just removed 3 drives from a RAID6.
Right now, /proc/mdstat says:
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : inactive sdh2[0](S) sdc2[6](S) sdj1[5](S) sde1[4](S)
sdg1[3](S) sdi1[2](S) sdf2[1](S)
10744335040 blocks super 0.91
mdadm -E /dev/sdc2 says:
/dev/sdc2:
Magic : a92b4efc
Version : 0.91.00
UUID : 5e57a17d:43eb0786:42ea8b6c:723593c7
Creation Time : Sat Oct 2 07:21:53 2010
Raid Level : raid6
Used Dev Size : 1465135936 (1397.26 GiB 1500.30 GB)
Array Size : 11721087488 (11178.10 GiB 12002.39 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 1
Reshape pos'n : 4096
Delta Devices : 3 (7->10)
Update Time : Sat Oct 17 18:59:50 2015
State : active
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Checksum : fad60788 - correct
Events : 2579239
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 98 6 active sync
0 0 8 50 0 active sync
1 1 8 18 1 active sync
2 2 8 65 2 active sync /dev/sde1
3 3 8 33 3 active sync /dev/sdc1
4 4 8 1 4 active sync /dev/sda1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 98 6 active sync
7 7 8 145 7 active sync /dev/sdj1
8 8 8 129 8 active sync /dev/sdi1
9 9 8 113 9 active sync /dev/sdh1
So, if I read this right, the superblock here states that the array is
in the middle of a reshape from 7 to 10 devices, but it just started
(4096 is the position).
What's interesting is the device names listed here don't match the ones
reported by /proc/mdstat, and are actually incorrect. The right
partition numbers are in /proc/mdstat.
The superblocks on the 6 other original disks match, except for of
course which one they mark as 'this' and the checksum.
I've read in here (http://ubuntuforums.org/showthread.php?t=2133576)
among many other places that it might be possible to recover the data on
the array by trying to re-create it to the state before the re-shape.
I've also read that if I want to re-create an array in read-only mode, I
should re-create it degraded.
So, what I thought I would do is this:
mdadm --create /dev/md1 --level=6 --raid-devices=7 /dev/sdh2
/dev/sdf2 /dev/sdi1 /dev/sdg1 /dev/sde1 missing missing
Obviously, at this point, I'm trying to be as cautious as possible in
not causing any further damage, if that's at all possible.
It seems that this issue has some similarities to this bug:
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1001019
So, please all mdadm gurus, help me out! How can I recover as much of
the data on this volume as possible?
Thanks again,
Andras Tantos
^ permalink raw reply
* Re: [PATCH v2 10/12] block: move blk_integrity to request_queue
From: Martin K. Petersen @ 2015-10-20 2:24 UTC (permalink / raw)
To: Williams, Dan J
Cc: martin.petersen@oracle.com, linux-raid@vger.kernel.org,
dm-devel@redhat.com, linux-nvdimm@lists.01.org,
linux-nvme@lists.infradead.org, axboe@fb.com, hch@lst.de
In-Reply-To: <1444956139.2737.7.camel@intel.com>
>>>>> "Dan" == Williams, Dan J <dan.j.williams@intel.com> writes:
Dan> Martin pointed out that I broke compatibility by changing the sysfs
Dan> layout of the integrity attributes. Rather than add a sysfs-link
Dan> we can make this patch simpler by only moving struct blk_integrity
Dan> to request_queue and leave integrity_kobj where it is in gendisk.
Dan> Also undo the conversion of blk_integrity apis take a
Dan> request_queue. Through this simplification I also noticed that I
Dan> had broken blk_integrity_compare()
Dan> Here's the much smaller v3, and I also refreshed
Dan> for-4.4/blk-integrity
Much better.
I ran some test on our combined trees over the weekend and tripped a bug
on some SCSI configs. Turned out to be my fault so I'll repost with a
fixed patch 2.
You can either rebase on top of the new series or maybe Jens will merge
my patches first. Either way works for me.
In any case I'm OK with your latest changes. Feel free to add my
Acked-by.
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox