* Re: raid1 recoverable after system crash?
From: Roman Mamedov @ 2016-04-07 13:00 UTC (permalink / raw)
To: Brian J. Murrell; +Cc: linux-raid
In-Reply-To: <1460033086.27740.145.camel@interlinx.bc.ca>
[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]
On Thu, 07 Apr 2016 08:44:46 -0400
"Brian J. Murrell" <brian@interlinx.bc.ca> wrote:
> # cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md0 : active raid1 sdd[0]
> 1953514496 blocks [2/1] [U_]
>
> md1 : active raid0 sdc[1] sdb[0]
> 1953524736 blocks super 1.2 512k chunks
...
> Is my only option here to fail/remove /dev/md1 from the array and re-
> add it that way or is there a more graceful recovery possible here?
You do not have a write intent bitmap at md0, so re-add will not work. Seems
like you should --add it now, then after it rebuilds use --grow to add a
bitmap, so that in the future you could use -re-add.
As to why the situation occured in the first place, you should ensure that md1
assembles before md0. Perhaps by listing both arrays in /etc/mdadm/mdadm.conf
in the order you need (and don't forget to re-generate initrd.img).
--
With respect,
Roman
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* raid1 recoverable after system crash?
From: Brian J. Murrell @ 2016-04-07 12:44 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 4193 bytes --]
Hi
I had a system crash. When it came back up, one of my arrays was
degraded:
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdd[0]
1953514496 blocks [2/1] [U_]
md1 : active raid0 sdc[1] sdb[0]
1953524736 blocks super 1.2 512k chunks
unused devices: <none>
md1 is supposed to be a member of md0 but it's not currently:
/dev/md0:
Version : 0.90
Creation Time : Mon Jan 26 19:51:38 2009
Raid Level : raid1
Array Size : 1953514496 (1863.02 GiB 2000.40 GB)
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Thu Apr 7 08:19:25 2016
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
UUID : 2f8fc5e0:a0eb646a:2303d005:33f25f21
Events : 0.5030755
Number Major Minor RaidDevice State
0 8 48 0 active sync /dev/sdd
1 0 0 1 removed
It doesn't seem to be re-addable:
# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --manage /dev/md0 --re-add /dev/md1
mdadm: --re-add for /dev/md1 to /dev/md0 is not possible
It doesn't seem to be assemble-able:
# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble /dev/md0 /dev/sdd /dev/md1
mdadm: /dev/md0 has been started with 1 drive (out of 2).
even when forced:
# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble --force /dev/md0 /dev/sdd /dev/md1
mdadm: /dev/md0 has been started with 1 drive (out of 2).
Is my only option here to fail/remove /dev/md1 from the array and re-
add it that way or is there a more graceful recovery possible here?
Some additional info:
mdadm --examine /dev/md1
/dev/md1:
Magic : a92b4efc
Version : 0.90.00
UUID : 2f8fc5e0:a0eb646a:2303d005:33f25f21
Creation Time : Mon Jan 26 19:51:38 2009
Raid Level : raid1
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Array Size : 1953514496 (1863.02 GiB 2000.40 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Update Time : Thu Apr 7 00:18:47 2016
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Checksum : e5ddeac5 - correct
Events : 5030737
Number Major Minor RaidDevice State
this 1 9 1 1 active sync /dev/md1
0 0 8 48 0 active sync /dev/sdd
1 1 9 1 1 active sync /dev/md1
# mdadm --examine /dev/sdd
/dev/sdd:
Magic : a92b4efc
Version : 0.90.00
UUID : 2f8fc5e0:a0eb646a:2303d005:33f25f21
Creation Time : Mon Jan 26 19:51:38 2009
Raid Level : raid1
Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
Array Size : 1953514496 (1863.02 GiB 2000.40 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0
Update Time : Thu Apr 7 08:37:04 2016
State : clean
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
Checksum : e62b232b - correct
Events : 5030757
Number Major Minor RaidDevice State
this 0 8 48 0 active sync /dev/sdd
0 0 8 48 0 active sync /dev/sdd
1 1 0 0 1 faulty removed
Most happy to provide any additional information needed.
Cheers,
b.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply
* Re: recovery from selinux blocking --backup-file during RAID5->6
From: George Rapp @ 2016-04-06 16:32 UTC (permalink / raw)
To: Noah Beck; +Cc: Linux-RAID
In-Reply-To: <CAFAW0zzauYrAd-v3Yq03-XSpO9KCSXcUtXnGWKMvd-VYQj5O5g@mail.gmail.com>
On Wed, Apr 6, 2016 at 10:16 AM, Noah Beck <noah.b.beck@gmail.com> wrote:
> Update:
>
> I backed up locally all data I cared about from the "raid5" array while it was
> stuck in the state:
>
> md127 : active raid6 sde1[3] sda1[2] sdd1[0] sdf1[1]
> 5860535808 blocks super 0.91 level 6, 64k chunk, algorithm 18
> [5/4] [UUUU_]
> [>....................] reshape = 0.0% (1/1953511936)
> finish=1895.2min speed=16642K/sec
>
> <....snip....>
> I stopped the array:
> # mdadm --stop /dev/md127
>
> Then tried re-assembling it (using the locally-built mdadm):
> # mdadm --assemble --verbose --update=revert-reshape /dev/md127 $devices
> mdadm: looking for devices for /dev/md127
> mdadm: /dev/sdd1: Can only revert reshape which changes number of devices
>
> Is the mdadm code only looking for the case where a new device was added but
> the raid level was not modified? Recall, this was a 4-device raid5 that was
> attempted to be converted to a 5-device raid6.
Noah -
That's what I was afraid of. NeilBrown's patch was specific to the
corner case I encountered (SELinux' interruption of a RAID 6 change in
number of devices).
However, I was worse off than you are - I couldn't even find a way to
mount the filesystem to recover the data.
> Out of curiosity, from looking at the patch Neil committed to the tree, I also
> tried adding the --invalid-backup option:
>
> # ./md127/mdadm --assemble --verbose --update=revert-reshape
> --invalid-backup /dev/md127 $devices
> mdadm: looking for devices for /dev/md127
> mdadm: --update=revert-reshape not understood for 0.90 metadata
>
> I see the current metadata version is something like 1.2 now? This array (now
> running on a Fedora 22 system) was originally created on a much older Fedora,
> at least as old as Fedora 9.
This is another delta from my situation. My RAID metadata was (and is)
version 1.2.
> I can create a new array out of the disks and dump my data back onto it if the
> array is really stuck in a state it can't get out of. Is there anything else I
> should try first, or any other experiment to run?
I'll let others weigh in (I wouldn't say "never" until Neil says it
first 8^) -- but I can't see any easy outs.
George
--
George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)
^ permalink raw reply
* Re: recovery from selinux blocking --backup-file during RAID5->6
From: Noah Beck @ 2016-04-06 14:16 UTC (permalink / raw)
To: George Rapp; +Cc: Linux-RAID
In-Reply-To: <CAFAW0zyNS3m4ZZ59xCkznxopo0d4soWQZrd2eBVfovfNCoX6Sg@mail.gmail.com>
Update:
I backed up locally all data I cared about from the "raid5" array while it was
stuck in the state:
md127 : active raid6 sde1[3] sda1[2] sdd1[0] sdf1[1]
5860535808 blocks super 0.91 level 6, 64k chunk, algorithm 18
[5/4] [UUUU_]
[>....................] reshape = 0.0% (1/1953511936)
finish=1895.2min speed=16642K/sec
I found that the previous patch (in
https://marc.info/?l=linux-raid&m=145187378405337&w=2) of course does not apply
cleanly to the top of the current git tree. Looking through the change logs, I
found that a slightly modified version of said patch was included just before
the mdadm-3.4 release. So instead I grabbed the git repo tagged mdadm-3.4
(http://git.neil.brown.name/?p=mdadm.git;a=snapshot;h=c61b1c0bb5ee7a09bb25250e6c12bcd4d4cafb0c;sf=tgz)
and built mdadm from there.
Starting point after unmounting filesystems and a vgchange -an (md127
is a physical volume in lvm):
# mdadm --detail /dev/md127
/dev/md127:
Version : 0.91
Creation Time : Sat Dec 17 23:41:15 2011
Raid Level : raid6
Array Size : 5860535808 (5589.04 GiB 6001.19 GB)
Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
Raid Devices : 5
Total Devices : 4
Preferred Minor : 127
Persistence : Superblock is persistent
Update Time : Wed Apr 6 08:33:26 2016
State : clean, degraded, reshaping
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric-6
Chunk Size : 64K
Reshape Status : 0% complete
New Layout : left-symmetric
UUID : 31838cca:af76c356:b4981550:b0a7388d
Events : 0.184874
Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 81 1 active sync /dev/sdf1
2 8 1 2 active sync /dev/sda1
3 8 65 3 active sync /dev/sde1
8 0 0 8 removed
I stopped the array:
# mdadm --stop /dev/md127
Then tried re-assembling it (using the locally-built mdadm):
# mdadm --assemble --verbose --update=revert-reshape /dev/md127 $devices
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdd1: Can only revert reshape which changes number of devices
Is the mdadm code only looking for the case where a new device was added but
the raid level was not modified? Recall, this was a 4-device raid5 that was
attempted to be converted to a 5-device raid6.
Out of curiosity, from looking at the patch Neil committed to the tree, I also
tried adding the --invalid-backup option:
# ./md127/mdadm --assemble --verbose --update=revert-reshape
--invalid-backup /dev/md127 $devices
mdadm: looking for devices for /dev/md127
mdadm: --update=revert-reshape not understood for 0.90 metadata
I see the current metadata version is something like 1.2 now? This array (now
running on a Fedora 22 system) was originally created on a much older Fedora,
at least as old as Fedora 9.
I can create a new array out of the disks and dump my data back onto it if the
array is really stuck in a state it can't get out of. Is there anything else I
should try first, or any other experiment to run?
Noah
^ permalink raw reply
* Re: [PATCH] [RFC] fix potential access after free: return value of blk_check_plugged() must be used schedule() safe
From: Lars Ellenberg @ 2016-04-06 12:01 UTC (permalink / raw)
To: NeilBrown
Cc: Shaohua Li, Jens Axboe, Chris Mason, Josef Bacik, David Sterba,
linux-raid, linux-kernel, linux-btrfs
In-Reply-To: <87k2kbo2im.fsf@notabene.neil.brown.name>
On Wed, Apr 06, 2016 at 01:10:57PM +1000, NeilBrown wrote:
> On Wed, Apr 06 2016, Shaohua Li wrote:
>
> > On Tue, Apr 05, 2016 at 03:36:57PM +0200, Lars Ellenberg wrote:
> >> blk_check_plugged() will return a pointer
> >> to an object linked on current->plug->cb_list.
> >>
> >> That list may "at any time" be implicitly cleared by
> >> blk_flush_plug_list()
> >> flush_plug_callbacks()
> >> either as a result of blk_finish_plug(),
> >> or implicitly by schedule() [and maybe other implicit mechanisms?]
> >>
> >> If there is no protection against an implicit unplug
> >> between the call to blk_check_plug() and using its return value,
> >> that implicit unplug may have already happened,
> >> even before the plug is actually initialized or populated,
> >> and we may be using a pointer to already free()d data.
> >
> > This isn't correct. flush plug is never called in preemption, which is designed
> > only called when the task is going to sleep. See sched_submit_work. Am I
> > missing anything?
>
> Ahh yes, thanks.
>
> Only two places call blk_schedule_flush_plug().
> One is io_schedule_timeout() which must be called explicitly.
> There other is, as you say, sched_submit_work(). It starts:
>
> static inline void sched_submit_work(struct task_struct *tsk)
> {
> if (!tsk->state || tsk_is_pi_blocked(tsk))
> return;
>
> so if the task is runnable, then as
> include/linux/sched.h:#define TASK_RUNNING 0
>
> it will never call blk_schedule_flush_plug().
>
> So I don't think you are missing anything, we were.
>
> Lars: have your concerns been relieved or do you still have reason to
> think there is a problem?
So just don't call anything that might_sleep() between
blk_check_plug() and using its return value.
All good.
I thought I must have overlooked something.
Thanks,
Lars
^ permalink raw reply
* Re: Repairing R1: Part tabl, & precise command
From: Étienne Buira @ 2016-04-06 10:31 UTC (permalink / raw)
To: Ron Leach; +Cc: Linux RAID Mailing List
In-Reply-To: <5704E16A.3060806@tesco.net>
On Wed, Apr 06, 2016 at 11:14:02AM +0100, Ron Leach wrote:
> On 06/04/2016 00:47, Adam Goryachev wrote:
../..
> I thought this was good advice but apt-get isn't finding smartctl. I
> think your surmise is likely, though, that this disk has 'reduced' in
> size because of faults, and I've a new 3TB drive on its way. I'll
> replace this suspect disk as soon as it arrives.
Hi,
Package is named smartmontools on debian, and IMHO you want to regularly
check HDD health status (smartctl -a /dev/sdX).
Regards
^ permalink raw reply
* Re: Repairing R1: Part tabl, & precise command
From: Ron Leach @ 2016-04-06 10:14 UTC (permalink / raw)
To: Linux RAID Mailing List
In-Reply-To: <57044EA8.2010506@websitemanagers.com.au>
On 06/04/2016 00:47, Adam Goryachev wrote:
> That is one option (reduce the swap partition size). You might also
> look at the mdadm information of the array, generally it is possible
> to create a raid1 array across two devices that are different size,
> and mdadm will automatically ignore the "excess" space of the larger
> drive.
>
> eg:
> sda1 1000M
> sdb1 1050M
>
> The disks and partition tables will show both disks 100% full, because
> the partition fills the disk
> mdadm will ignore the extra 50M on sdb1 and create a raid1 array of 1000M
> LVM (or whatever you put onto the raid1) will show 1000M as the total
> size, and will know nothing about the extra 50M
>
> I think mdadm is silent about size differences if the difference is
> less than 10% (or some other percentage value).
Adam, thank you for confirming that reducing swap might work. I'd
prefer that because I worry that if the md beneath the LVM reduces in
size, then I could lose data that is in that LVM at present.
I checked fstab and realised that the swap partitions on /dev/sdb are
not used, so I have gone ahead and reduced the swap partition, and
rebuilt a good gpt table. That went ok. mdadm is now 'add'ing the
partner partitions to the 3 md devices in the system.
>
> [snip]
>
> Can you provide full output of smartctl, it should show more details
> on the status of the drive, what damage it might have/etc...
>
I thought this was good advice but apt-get isn't finding smartctl. I
think your surmise is likely, though, that this disk has 'reduced' in
size because of faults, and I've a new 3TB drive on its way. I'll
replace this suspect disk as soon as it arrives.
Grateful for the help
regards, Ron
^ permalink raw reply
* Re: [PATCH] badblocks: fix wrong return value when badblocks are disabled
From: Johannes Thumshirn @ 2016-04-06 9:26 UTC (permalink / raw)
To: Artur Paszkiewicz
Cc: axboe, linux-block, vishal.l.verma, linux-raid, linux-block-owner
In-Reply-To: <1459861174-9238-1-git-send-email-artur.paszkiewicz@intel.com>
On 2016-04-05 14:59, Artur Paszkiewicz wrote:
> The return value of md_set_badblocks() was inverted when the code was
> taken out of md, but the case when badblocks are disabled was left
> unchanged. This causes silent ignoring of I/O errors and other
> unpredictable behavior on md arrays that do not support badblocks (any
> array not using v1.x metadata).
>
> Fixes: 9e0e252a048b ("badblocks: Add core badblock management code")
> Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
^ permalink raw reply
* Re: [PATCH] badblocks: fix wrong return value when badblocks are disabled
From: Artur Paszkiewicz @ 2016-04-06 9:13 UTC (permalink / raw)
To: Verma, Vishal L, axboe@kernel.dk
Cc: linux-raid@vger.kernel.org, linux-block@vger.kernel.org
In-Reply-To: <1459877821.4741.4.camel@intel.com>
On 04/05/2016 07:37 PM, Verma, Vishal L wrote:
> On Tue, 2016-04-05 at 14:59 +0200, Artur Paszkiewicz wrote:
>> The return value of md_set_badblocks() was inverted when the code was
>> taken out of md, but the case when badblocks are disabled was left
>> unchanged. This causes silent ignoring of I/O errors and other
>> unpredictable behavior on md arrays that do not support badblocks (any
>> array not using v1.x metadata).
>>
>> Fixes: 9e0e252a048b ("badblocks: Add core badblock management code")
>> Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
>> ---
>> block/badblocks.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> Good find, thanks!
>
> Acked-by: Vishal Verma <vishal.l.verma@intel.com>
>
> Should this also be tagged for stable?
Yes, I think it should be, since it affects 4.5.
Artur
^ permalink raw reply
* Re: [PATCH] [RFC] fix potential access after free: return value of blk_check_plugged() must be used schedule() safe
From: NeilBrown @ 2016-04-06 3:10 UTC (permalink / raw)
To: Shaohua Li, Lars Ellenberg
Cc: Jens Axboe, Chris Mason, Josef Bacik, David Sterba, linux-raid,
linux-kernel, linux-btrfs
In-Reply-To: <20160406004956.GA102852@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 1592 bytes --]
On Wed, Apr 06 2016, Shaohua Li wrote:
> On Tue, Apr 05, 2016 at 03:36:57PM +0200, Lars Ellenberg wrote:
>> blk_check_plugged() will return a pointer
>> to an object linked on current->plug->cb_list.
>>
>> That list may "at any time" be implicitly cleared by
>> blk_flush_plug_list()
>> flush_plug_callbacks()
>> either as a result of blk_finish_plug(),
>> or implicitly by schedule() [and maybe other implicit mechanisms?]
>>
>> If there is no protection against an implicit unplug
>> between the call to blk_check_plug() and using its return value,
>> that implicit unplug may have already happened,
>> even before the plug is actually initialized or populated,
>> and we may be using a pointer to already free()d data.
>
> This isn't correct. flush plug is never called in preemption, which is designed
> only called when the task is going to sleep. See sched_submit_work. Am I
> missing anything?
Ahh yes, thanks.
Only two places call blk_schedule_flush_plug().
One is io_schedule_timeout() which must be called explicitly.
There other is, as you say, sched_submit_work(). It starts:
static inline void sched_submit_work(struct task_struct *tsk)
{
if (!tsk->state || tsk_is_pi_blocked(tsk))
return;
so if the task is runnable, then as
include/linux/sched.h:#define TASK_RUNNING 0
it will never call blk_schedule_flush_plug().
So I don't think you are missing anything, we were.
Lars: have your concerns been relieved or do you still have reason to
think there is a problem?
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH] [RFC] fix potential access after free: return value of blk_check_plugged() must be used schedule() safe
From: Shaohua Li @ 2016-04-06 0:49 UTC (permalink / raw)
To: Lars Ellenberg
Cc: Neil Brown, Jens Axboe, Chris Mason, Josef Bacik, David Sterba,
linux-raid, linux-kernel, linux-btrfs
In-Reply-To: <20160405133657.GA3078@soda.linbit>
On Tue, Apr 05, 2016 at 03:36:57PM +0200, Lars Ellenberg wrote:
> blk_check_plugged() will return a pointer
> to an object linked on current->plug->cb_list.
>
> That list may "at any time" be implicitly cleared by
> blk_flush_plug_list()
> flush_plug_callbacks()
> either as a result of blk_finish_plug(),
> or implicitly by schedule() [and maybe other implicit mechanisms?]
>
> If there is no protection against an implicit unplug
> between the call to blk_check_plug() and using its return value,
> that implicit unplug may have already happened,
> even before the plug is actually initialized or populated,
> and we may be using a pointer to already free()d data.
This isn't correct. flush plug is never called in preemption, which is designed
only called when the task is going to sleep. See sched_submit_work. Am I
missing anything?
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH 11/27] bcache: io.c: use bio_set_vec_table
From: Kent Overstreet @ 2016-04-06 0:35 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Ming Lei, Jens Axboe, linux-kernel, linux-block, Boaz Harrosh,
Shaohua Li, open list:BCACHE (BLOCK LAYER CACHE),
open list:SOFTWARE RAID (Multiple Disks) SUPPORT
In-Reply-To: <20160405124902.GA11986@infradead.org>
On Tue, Apr 05, 2016 at 05:49:02AM -0700, Christoph Hellwig wrote:
> On Tue, Apr 05, 2016 at 07:56:56PM +0800, Ming Lei wrote:
> > diff --git a/drivers/md/bcache/io.c b/drivers/md/bcache/io.c
> > index 86a0bb8..1c48462 100644
> > --- a/drivers/md/bcache/io.c
> > +++ b/drivers/md/bcache/io.c
> > @@ -26,8 +26,7 @@ struct bio *bch_bbio_alloc(struct cache_set *c)
> >
> > bio_init(bio);
> > bio->bi_flags |= BIO_POOL_NONE << BIO_POOL_OFFSET;
> > - bio->bi_max_vecs = bucket_pages(c);
> > - bio->bi_io_vec = bio->bi_inline_vecs;
> > + bio_set_vec_table(bio, bio->bi_inline_vecs, bucket_pages(c));
>
> All this bcache code needs to move away from bio_init on a bio
> embedded in a driver private structure toward properly using
> bio_alloc / bio_alloc_bioset. That will also fix the crash
> with bcache over md that Shaohua reported, so I'd suggest to fast
> track this part of the series.
Why?
bio_init() is a publicly exported function, it's always been one and bcache is
ot the only driver to use it directly.
bios with > BIO_MAX_PAGES bvecs is a separate issue; I would argue that the bug
is in md's queue_limits; it uses blk_set_stacking_limits() which sets
max_segments = USHRT_MAX, which is wrong if it's going to clone the biovec.
^ permalink raw reply
* Re: [PATCH] [RFC] fix potential access after free: return value of blk_check_plugged() must be used schedule() safe
From: Chris Mason @ 2016-04-06 0:30 UTC (permalink / raw)
To: Lars Ellenberg
Cc: Neil Brown, Jens Axboe, Josef Bacik, David Sterba, linux-raid,
linux-kernel, linux-btrfs
In-Reply-To: <20160405133657.GA3078@soda.linbit>
On Tue, Apr 05, 2016 at 03:36:57PM +0200, Lars Ellenberg wrote:
> blk_check_plugged() will return a pointer
> to an object linked on current->plug->cb_list.
>
> That list may "at any time" be implicitly cleared by
> blk_flush_plug_list()
> flush_plug_callbacks()
> either as a result of blk_finish_plug(),
> or implicitly by schedule() [and maybe other implicit mechanisms?]
>
> If there is no protection against an implicit unplug
> between the call to blk_check_plug() and using its return value,
> that implicit unplug may have already happened,
> even before the plug is actually initialized or populated,
> and we may be using a pointer to already free()d data.
>
> I suggest that both raid1 and raid10 can easily be fixed
> by moving the call to blk_check_plugged() inside the spinlock.
>
> For md/raid5 and btrfs/raid56,
> I'm unsure how (if) this needs to be fixed.
I think you're right, digging in to see if there's something I missed.
But as Neil said, it looks like we just got saved by preemption being
off by default.
-chris
^ permalink raw reply
* Re: Repairing R1: Part tabl, & precise command
From: Adam Goryachev @ 2016-04-05 23:47 UTC (permalink / raw)
To: Ron Leach, Linux RAID Mailing List
In-Reply-To: <5703E909.8090903@tesco.net>
On 06/04/16 02:34, Ron Leach wrote:
> On 05/04/2016 16:28, Phil Turmel wrote:
>
>>
>> If your array has write-intent bitmaps, use --re-add instead of --add.
>> It'll be quick. Otherwise just --add and let it rebuild.
>>
>
> Phil, thanks for the advice.
>
> I hit an unexpected problem fixing the partition table on /dev/sdb,
> the disk that dropped from the Raid1 array. The problem is caused by
> /dev/sdb being *smaller* than /dev/sdc (the working array member) -
> despite the disks being identical products from WD. gdisk complains
> that partition 5 (/dev/sdb5), which is to be the Raid1 partner for the
> LVM containing all our backed up files, is too big (together with the
> other partitions) for the /dev/sdb disk.
>
> Presumably, raid1 doesn't work if an 'add'ed disk partition is smaller
> than the existing, running, degraded array? Am I right in thinking
> that the LVM won't be able to be carried securely on the underlying md
> system? lsdrv is reporting that /dev/md127 has 0 free, so it seems
> that the LVM is occupying the complete space of /dev/md127, and it
> must be using the complete space of the underlying /dev/sdc5 because
> only sdc is active, at the moment (the Raid1 being still degraded).
>
> To protect the LVM, what would be a good thing to do? Should I define
> a slightly shorter 'partner' partition on the failed disk (/dev/sdb) -
> I would think not, but I would welcome advice.
>
> I did think about reducing the size of one of the other partitions on
> /dev/sdb - there's a swap partition of 2G which could become 1.5G,
> because there's another 2G on the working disk anyway. Doing that, the
> partner partitions for the real data could be the same size, though
> not in exactly the same place on both disks. I think this might work?
>
> regards, Ron
Hi Ron,
That is one option (reduce the swap partition size). You might also look
at the mdadm information of the array, generally it is possible to
create a raid1 array across two devices that are different size, and
mdadm will automatically ignore the "excess" space of the larger drive.
eg:
sda1 1000M
sdb1 1050M
The disks and partition tables will show both disks 100% full, because
the partition fills the disk
mdadm will ignore the extra 50M on sdb1 and create a raid1 array of 1000M
LVM (or whatever you put onto the raid1) will show 1000M as the total
size, and will know nothing about the extra 50M
I think mdadm is silent about size differences if the difference is less
than 10% (or some other percentage value).
Another concern I have is that the drive has a number of damaged
sectors, has used up all the "spare" sectors that it has for
re-allocation, and is now reporting a smaller size because it knows that
a number of sectors are bad. I don't think drives do this, but it is a
failed drive, and manufacturers might do some strange things.
Can you provide full output of smartctl, it should show more details on
the status of the drive, what damage it might have/etc...
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: [PATCH] [RFC] fix potential access after free: return value of blk_check_plugged() must be used schedule() safe
From: NeilBrown @ 2016-04-05 22:49 UTC (permalink / raw)
To: Lars Ellenberg, Jens Axboe
Cc: Chris Mason, Josef Bacik, David Sterba, linux-raid, linux-kernel,
linux-btrfs, Shaohua Li
In-Reply-To: <20160405133657.GA3078@soda.linbit>
[-- Attachment #1: Type: text/plain, Size: 7176 bytes --]
On Tue, Apr 05 2016, Lars Ellenberg wrote:
> blk_check_plugged() will return a pointer
> to an object linked on current->plug->cb_list.
>
> That list may "at any time" be implicitly cleared by
> blk_flush_plug_list()
> flush_plug_callbacks()
> either as a result of blk_finish_plug(),
> or implicitly by schedule() [and maybe other implicit mechanisms?]
I think the only risk here is preemption, so
preempt_disable() / preempt_enable()
or as you say a spinlock, is sufficient protection.
I would suggest preempt_{dis,en}able for the raid5 code.
Maybe for raid1/raid10 too just for consistency.
>
> If there is no protection against an implicit unplug
> between the call to blk_check_plug() and using its return value,
> that implicit unplug may have already happened,
> even before the plug is actually initialized or populated,
> and we may be using a pointer to already free()d data.
>
> I suggest that both raid1 and raid10 can easily be fixed
> by moving the call to blk_check_plugged() inside the spinlock.
>
> For md/raid5 and btrfs/raid56,
> I'm unsure how (if) this needs to be fixed.
>
> The other current in-tree users of blk_check_plugged()
> are mm_check_plugged(), and mddev_check_plugged().
>
> mm_check_plugged() is already used safely inside a spinlock.
>
> with mddev_check_plugged() I'm unsure, at least on a preempt kernel.
I think this is only an issue on a preempt kernel, and in that case: yes
- mddev_check_plugged() needs protection. Maybe preempt enable/disable
could be done in blk_check_plugged() so those calls which don't
dereference the pointer don't need further protection.
Or maybe blk_check_plugged should have WARN_ON_ONCE(!in_atomic());
>
> Did I overlook any magic that protects against such implicit unplug?
Just the fortunate lack of preemption probably.
>
> Also, why pretend that a custom plug struct (such as raid1_plug_cb)
> may have its member "struct blk_plug_cb cb" at an arbitrary offset?
> As it is, raid1_check_plugged() below is actually just a cast.
Fair point. I generally prefer container_of to casts because it is more
obviously correct, and type checked.
However as blk_check_plugged performs the allocation, the blk_plug_cb
must be at the start of the containing structure, so the complex tests
for handling NULL are just noise.
I'd be happy for that to be changed.
Thanks,
NeilBrown
>
> Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
> ---
> drivers/md/raid1.c | 19 +++++++++++++------
> drivers/md/raid10.c | 21 +++++++++++++--------
> drivers/md/raid5.c | 5 +++++
> fs/btrfs/raid56.c | 5 +++++
> 4 files changed, 36 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index 39fb21e..55dc960 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -1044,6 +1044,18 @@ static void raid1_unplug(struct blk_plug_cb *cb, bool from_schedule)
> kfree(plug);
> }
>
> +static struct raid1_plug_cb *raid1_check_plugged(struct mddev *mddev)
> +{
> + /* return (struct raid1_plug_cb*)blk_check_plugged(...); */
> + struct blk_plug_cb *cb;
> + struct raid1_plug_cb *plug = NULL;
> +
> + cb = blk_check_plugged(raid1_unplug, mddev, sizeof(*plug));
> + if (cb)
> + plug = container_of(cb, struct raid1_plug_cb, cb);
> + return plug;
> +}
> +
> static void raid1_make_request(struct mddev *mddev, struct bio * bio)
> {
> struct r1conf *conf = mddev->private;
> @@ -1060,7 +1072,6 @@ static void raid1_make_request(struct mddev *mddev, struct bio * bio)
> & (REQ_DISCARD | REQ_SECURE));
> const unsigned long do_same = (bio->bi_rw & REQ_WRITE_SAME);
> struct md_rdev *blocked_rdev;
> - struct blk_plug_cb *cb;
> struct raid1_plug_cb *plug = NULL;
> int first_clone;
> int sectors_handled;
> @@ -1382,12 +1393,8 @@ read_again:
>
> atomic_inc(&r1_bio->remaining);
>
> - cb = blk_check_plugged(raid1_unplug, mddev, sizeof(*plug));
> - if (cb)
> - plug = container_of(cb, struct raid1_plug_cb, cb);
> - else
> - plug = NULL;
> spin_lock_irqsave(&conf->device_lock, flags);
> + plug = raid1_check_plugged(mddev);
> if (plug) {
> bio_list_add(&plug->pending, mbio);
> plug->pending_cnt++;
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index e3fd725..d7d4397 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -1052,6 +1052,18 @@ static void raid10_unplug(struct blk_plug_cb *cb, bool from_schedule)
> kfree(plug);
> }
>
> +static struct raid10_plug_cb *raid10_check_plugged(struct mddev *mddev)
> +{
> + /* return (struct raid1_plug_cb*)blk_check_plugged(...); */
> + struct blk_plug_cb *cb;
> + struct raid10_plug_cb *plug = NULL;
> +
> + cb = blk_check_plugged(raid10_unplug, mddev, sizeof(*plug));
> + if (cb)
> + plug = container_of(cb, struct raid10_plug_cb, cb);
> + return plug;
> +}
> +
> static void __make_request(struct mddev *mddev, struct bio *bio)
> {
> struct r10conf *conf = mddev->private;
> @@ -1066,7 +1078,6 @@ static void __make_request(struct mddev *mddev, struct bio *bio)
> const unsigned long do_same = (bio->bi_rw & REQ_WRITE_SAME);
> unsigned long flags;
> struct md_rdev *blocked_rdev;
> - struct blk_plug_cb *cb;
> struct raid10_plug_cb *plug = NULL;
> int sectors_handled;
> int max_sectors;
> @@ -1369,14 +1380,8 @@ retry_write:
>
> atomic_inc(&r10_bio->remaining);
>
> - cb = blk_check_plugged(raid10_unplug, mddev,
> - sizeof(*plug));
> - if (cb)
> - plug = container_of(cb, struct raid10_plug_cb,
> - cb);
> - else
> - plug = NULL;
> spin_lock_irqsave(&conf->device_lock, flags);
> + plug = raid10_check_plugged(mddev);
> if (plug) {
> bio_list_add(&plug->pending, mbio);
> plug->pending_cnt++;
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 8ab8b65..4e3b02b 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -5034,6 +5034,11 @@ static void release_stripe_plug(struct mddev *mddev,
> }
>
> cb = container_of(blk_cb, struct raid5_plug_cb, cb);
> +/* FIXME
> + * Nothing protects current from being scheduled, which means cb, aka plug,
> + * may implicitly be "unplugged" any time now, before it even is initialized,
> + * and will then be a pointer to free()d space.
> + */
>
> if (cb->list.next == NULL) {
> int i;
> diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
> index 0b7792e..17757d4 100644
> --- a/fs/btrfs/raid56.c
> +++ b/fs/btrfs/raid56.c
> @@ -1774,6 +1774,11 @@ int raid56_parity_write(struct btrfs_root *root, struct bio *bio,
> cb = blk_check_plugged(btrfs_raid_unplug, root->fs_info,
> sizeof(*plug));
> if (cb) {
> +/* FIXME
> + * Nothing protects current from being scheduled, which means cb, aka plug,
> + * may implicitly be "unplugged" any time now, before it even is initialized,
> + * and will then be a pointer to free()d space.
> + */
> plug = container_of(cb, struct btrfs_plug_cb, cb);
> if (!plug->info) {
> plug->info = root->fs_info;
> --
> 1.9.1
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 20/27] dm: dm-bufio.c: use bio_set_vec_table()
From: Mike Snitzer @ 2016-04-05 18:11 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Ming Lei, Jens Axboe, linux-kernel, linux-block, Boaz Harrosh,
Alasdair Kergon, maintainer:DEVICE-MAPPER (LVM), Shaohua Li,
open list:SOFTWARE RAID (Multiple Disks) SUPPORT
In-Reply-To: <20160405130402.GD32576@infradead.org>
On Tue, Apr 05 2016 at 9:04am -0400,
Christoph Hellwig <hch@infradead.org> wrote:
> On Tue, Apr 05, 2016 at 08:07:35PM +0800, Ming Lei wrote:
> > Signed-off-by: Ming Lei <tom.leiming@gmail.com>
> > ---
> > drivers/md/dm-bufio.c | 3 +--
> > 1 file changed, 1 insertion(+), 2 deletions(-)
> >
> > diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c
> > index cd77216..0e48ad7 100644
> > --- a/drivers/md/dm-bufio.c
> > +++ b/drivers/md/dm-bufio.c
> > @@ -624,8 +624,7 @@ static void use_inline_bio(struct dm_buffer *b, int rw, sector_t block,
> > int len;
> >
> > bio_init(&b->bio);
> > - b->bio.bi_io_vec = b->bio_vec;
> > - b->bio.bi_max_vecs = DM_BUFIO_INLINE_VECS;
> > + bio_set_vec_table(&b->bio, b->bio_vec, DM_BUFIO_INLINE_VECS);
> > b->bio.bi_iter.bi_sector = block << b->c->sectors_per_block_bits;
> > b->bio.bi_bdev = b->c->bdev;
> > b->bio.bi_end_io = inline_endio;
>
> Should be switched to use bio_alloc instead.
Why does the use of bio_init() vs bio_alloc() bother you?
'struct dm_buffer' has a 'struct bio'. That bio is allocated as part of
the dm_buffer in drivers/md/dmbufio.c:alloc_buffer() -- which is called
by various other bufio interfaces (e.g. __alloc_buffer_wait). Bufio is
in control of ensuring forward progress by carefully managing the memory
associated with these buffers. I don't see the benefit of bio_alloc()
here. What am I missing?
^ permalink raw reply
* Re: [PATCH] badblocks: fix wrong return value when badblocks are disabled
From: Verma, Vishal L @ 2016-04-05 17:37 UTC (permalink / raw)
To: Paszkiewicz, Artur, axboe@kernel.dk
Cc: linux-raid@vger.kernel.org, linux-block@vger.kernel.org
In-Reply-To: <1459861174-9238-1-git-send-email-artur.paszkiewicz@intel.com>
On Tue, 2016-04-05 at 14:59 +0200, Artur Paszkiewicz wrote:
> The return value of md_set_badblocks() was inverted when the code was
> taken out of md, but the case when badblocks are disabled was left
> unchanged. This causes silent ignoring of I/O errors and other
> unpredictable behavior on md arrays that do not support badblocks (any
> array not using v1.x metadata).
>
> Fixes: 9e0e252a048b ("badblocks: Add core badblock management code")
> Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
> ---
> block/badblocks.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
Good find, thanks!
Acked-by: Vishal Verma <vishal.l.verma@intel.com>
Should this also be tagged for stable?
-Vishal
^ permalink raw reply
* Re: [PATCH 11/27] bcache: io.c: use bio_set_vec_table
From: Christoph Hellwig @ 2016-04-05 17:31 UTC (permalink / raw)
To: Ming Lei
Cc: Christoph Hellwig, Jens Axboe, Linux Kernel Mailing List,
linux-block, Boaz Harrosh, Kent Overstreet, Shaohua Li,
open list:BCACHE (BLOCK LAYER CACHE),
open list:SOFTWARE RAID (Multiple Disks) SUPPORT
In-Reply-To: <CACVXFVN+SfQ=mkdKeAMmPrkzQtrYcNxDPrT6bxNjC==xnuaxhg@mail.gmail.com>
On Tue, Apr 05, 2016 at 11:24:30PM +0800, Ming Lei wrote:
> - bio can be embedded into one biger instance, which is often allocated
> dynamically, so one extra allocation for bio can be avoided.
We can also do this the other way around with the bios front_pad,
which avoid the caller poking into bio details.
> - we should support arbitrary bio size by this way, at least bio_add_page()
> supports this usage. Also code gets lots of simplication with arbitrary bio
> size support, such as prio_io(): bcache
There is no reason for not supporting huge bios in the core bio code,
in fact using bio_kmalloc you can already allocate huges bios
dynamically right now. Except that you can't really use it, because the
layers below don't expect that. Bios based drivers expect to be able to
call bio_clone and friends called on bios passed to them, and might
also make assumptions about the max number of bios segments for now.
> BTW, the root cause for bcache crash still isn't clear now because
> blk_bio_segment_split() should split big bio into proper size with
> all queue's limits. Maybe the max segment limit isn't figured out correctly.
The root cause is pretty simple: The queue limits matter for request
based drivers, which are the only ones getting bios > BIO_MAX_PAGES
except for the buggy bcache use case. You'll need to either adjust the
limit for all bio based drivers to or get rid of that one magic caller
not playing by the rules.
^ permalink raw reply
* Re: Repairing R1: Part tabl, & precise command
From: Ron Leach @ 2016-04-05 16:34 UTC (permalink / raw)
To: Linux RAID Mailing List
In-Reply-To: <5703D98F.4020503@turmel.org>
On 05/04/2016 16:28, Phil Turmel wrote:
>
> If your array has write-intent bitmaps, use --re-add instead of --add.
> It'll be quick. Otherwise just --add and let it rebuild.
>
Phil, thanks for the advice.
I hit an unexpected problem fixing the partition table on /dev/sdb,
the disk that dropped from the Raid1 array. The problem is caused by
/dev/sdb being *smaller* than /dev/sdc (the working array member) -
despite the disks being identical products from WD. gdisk complains
that partition 5 (/dev/sdb5), which is to be the Raid1 partner for the
LVM containing all our backed up files, is too big (together with the
other partitions) for the /dev/sdb disk.
Presumably, raid1 doesn't work if an 'add'ed disk partition is smaller
than the existing, running, degraded array? Am I right in thinking
that the LVM won't be able to be carried securely on the underlying md
system? lsdrv is reporting that /dev/md127 has 0 free, so it seems
that the LVM is occupying the complete space of /dev/md127, and it
must be using the complete space of the underlying /dev/sdc5 because
only sdc is active, at the moment (the Raid1 being still degraded).
To protect the LVM, what would be a good thing to do? Should I define
a slightly shorter 'partner' partition on the failed disk (/dev/sdb) -
I would think not, but I would welcome advice.
I did think about reducing the size of one of the other partitions on
/dev/sdb - there's a swap partition of 2G which could become 1.5G,
because there's another 2G on the working disk anyway. Doing that,
the partner partitions for the real data could be the same size,
though not in exactly the same place on both disks. I think this
might work?
regards, Ron
^ permalink raw reply
* Re: Setting up fakeraid with mdadm / dmraid
From: Christoph Pleger @ 2016-04-05 15:43 UTC (permalink / raw)
To: linux-raid
Hello,
> Christoph> I have a machine with an LSI Megaraid Sofware RAID
> Christoph> fakeraid, which uses ddf format. About two weeks ago, when
Christoph> I wanted to install the machine, I configured a RAID 1
Christoph> array in the BIOS config utility and then booted the
> Christoph> machine from PXE with an NFSROOT. mdadm and dmraid are
Christoph> installed in the NFSROOT, dracut is used for initrd
> Christoph> generation. As dracut prefers mdadm over dmraid, mdadm was
Christoph> chosen for RAID management.
> It's also almost entirely black magic to support these things, since the
vendors don't generally share the details on the on-disk format that I'm
aware of. I'm probably wrong in the details.
The same fakeraid works quite well in Linux when using dmraid instead of
mdadm, so the ddf format used is obviously not secret. I just would like
to switch to mdadm because its documentation states that it supports the
ddf format and because many sites on the internet recommend to move from
dmraid to mdadm for fakeraid management.
> But again, please just use the controller in JBOD (Just a Bunch of
Disks) mode, and let mdadm mirror the disks for you.
The purpose of my NFSROOT is to partition disks, create filesystems and
unpack a base system to the disks. This should support a great variety of
machines, so the installation system has to be quite general. Though it is
not the case here, what if Linux had to be installed on the RAID together
with another OS? I think, that then all installed OSes should have the
same view on the RAID - the view which if offered by the fakeraid.
And if I had not intended to reinstall the machine anyway, but wanted to
use the NFSROOT as a rescue system, my data had been lost without
intention. So, even if mdadm does not support ddf fakeraids (correctly),
it should at least not destroy data.
> Crappy implementation of semi-secret on-disk format? It could be that
mdadm over-wrote something on the disk, or you partitioned it and then
overwrote something the FakeRAID wanted kept around.
Like I wrote before, I did not perform any action on the disks. No
mounting, no partitioning, no mkfs. Just booting with an NFSROOT which has
mdadm and dmraid installed.
Regards
Christoph
^ permalink raw reply
* Re: Repairing R1: Part tabl, & precise command
From: Phil Turmel @ 2016-04-05 15:28 UTC (permalink / raw)
To: Ron Leach; +Cc: Linux RAID Mailing List
In-Reply-To: <5703AE18.700@tesco.net>
On 04/05/2016 08:22 AM, Ron Leach wrote:
>
> mdadm -- manage .. --add ..
>
> commands and whether mdadm needs to be inhibited from taking any
> automatic remedial action.
If your array has write-intent bitmaps, use --re-add instead of --add.
It'll be quick. Otherwise just --add and let it rebuild.
Phil
^ permalink raw reply
* Re: [PATCH 11/27] bcache: io.c: use bio_set_vec_table
From: Ming Lei @ 2016-04-05 15:24 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Jens Axboe, Linux Kernel Mailing List, linux-block, Boaz Harrosh,
Kent Overstreet, Shaohua Li, open list:BCACHE (BLOCK LAYER CACHE),
open list:SOFTWARE RAID (Multiple Disks) SUPPORT
In-Reply-To: <20160405124902.GA11986@infradead.org>
On Tue, Apr 5, 2016 at 8:49 PM, Christoph Hellwig <hch@infradead.org> wrote:
> On Tue, Apr 05, 2016 at 07:56:56PM +0800, Ming Lei wrote:
>> diff --git a/drivers/md/bcache/io.c b/drivers/md/bcache/io.c
>> index 86a0bb8..1c48462 100644
>> --- a/drivers/md/bcache/io.c
>> +++ b/drivers/md/bcache/io.c
>> @@ -26,8 +26,7 @@ struct bio *bch_bbio_alloc(struct cache_set *c)
>>
>> bio_init(bio);
>> bio->bi_flags |= BIO_POOL_NONE << BIO_POOL_OFFSET;
>> - bio->bi_max_vecs = bucket_pages(c);
>> - bio->bi_io_vec = bio->bi_inline_vecs;
>> + bio_set_vec_table(bio, bio->bi_inline_vecs, bucket_pages(c));
>
> All this bcache code needs to move away from bio_init on a bio
> embedded in a driver private structure toward properly using
> bio_alloc / bio_alloc_bioset. That will also fix the crash
> with bcache over md that Shaohua reported, so I'd suggest to fast
> track this part of the series.
I suggest to keep this usage for the following reasons:
- bio can be embedded into one biger instance, which is often allocated
dynamically, so one extra allocation for bio can be avoided.
- we should support arbitrary bio size by this way, at least bio_add_page()
supports this usage. Also code gets lots of simplication with arbitrary bio
size support, such as prio_io(): bcache
BTW, the root cause for bcache crash still isn't clear now because
blk_bio_segment_split() should split big bio into proper size with
all queue's limits. Maybe the max segment limit isn't figured out correctly.
Thanks,
Ming Lei
^ permalink raw reply
* Re: recovery from selinux blocking --backup-file during RAID5->6
From: Noah Beck @ 2016-04-05 15:01 UTC (permalink / raw)
To: Wols Lists; +Cc: Linux-RAID
In-Reply-To: <57039E1F.3060307@youngman.org.uk>
On Tue, Apr 5, 2016 at 7:14 AM, Wols Lists <antlists@youngman.org.uk> wrote:
> DO YOU HAVE A BACKUP :-)
There are three categories of stuff on this array:
1) my own pictures and home videos, which are backed up to crashplan's
servers. So they're available and current but incredibly inconvenient
to get to through my 1.7Mbps DSL (yes, DSL exists in 2016).
2) local crashplan backups from other computers in the house. If I
lose this then crashplan just re-dumps the data from its source again.
3) mythtv recordings of shows going back many years. This is ~2TB by
itself and since the aforementioned DSL is my automatic backup path,
and I don't feel I lose much if it dies, it's not backed up.
Primarily my wife wants these kept, so for domestic harmony I'll need
to find 2TB elsewhere to dump this to before I do anything
experimental.
> Thing is, when one drive fails, it should be ringing alarm bells that
> another one is on its last legs - these things have an annoying habit of
> failing in bunches. Which says that you really need a *second* spare
> drive handy - is the rebuild going to tip one of your live drives over
> the edge? I'd say the chances of you ending up with a 5-device raid-6
> with one device failed is a lot higher than you'd like :-(
Quite true. That's why I wanted to switch to RAID6 in the first
place. This is an incredibly old array, dating back a little over 10
years. All of the original drives have been replaced (with larger
ones) along the way and there have been a number of plain old failed
drives that had to be replaced as well. Having no redundancy
available during the rebuild is a nail-biting experience I want to
stop having. :)
Noah
^ permalink raw reply
* Re: recovery from selinux blocking --backup-file during RAID5->6
From: Noah Beck @ 2016-04-05 14:44 UTC (permalink / raw)
To: George Rapp; +Cc: Linux-RAID
In-Reply-To: <CAF-KpgYqa9hB7m=pNAc8GFrTuKF8YwToHXvU4U2uCncs77Jx5g@mail.gmail.com>
[re-send including linux-raid this time]
On Mon, Apr 4, 2016 at 11:58 PM, George Rapp <george.rapp@gmail.com> wrote:
>> The previous thread resulted in a patch (in
>> https://marc.info/?l=linux-raid&m=145187378405337&w=2 ). If I want to
>> go back to having a 4-device RAID5 array before I shut this system
>> down to replace the bad disk, is the right thing to do still to apply
>> that patch to mdadm, stop /dev/md127, and assemble again with
>> --update=revert-reshape? Or does the info above indicate I should use
>> any different solution?
>
> Noah -
>
> I was the one bitten by SELinux in the thread you linked above. However, my
> starting point was different, as I was growing a 5-disk RAID 6 array to six
> disks. Otherwise, what you described was exactly what I experienced.
Heh. I read 5-disk RAID 6 as RAID5 in your other thread. Apparently
wishful thinking on my part, but still pretty similar like you say.
> If you want to try NeilBrown's patch
> (https://marc.info/?l=linux-raid&m=145187378405337&w=2), I'd strongly
> suggest testing it nondestructively first, using the overlay strategy
> detailed at
> https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID
The overlay seems like a safe way to experiment. I was wondering though,
since the array is still running, is it supposed to work to apply the
--update=revert-reshape to the system without shutting it down?
Or is it required to apply revert-reshape on an assemble operation?
> The steps you proposed above are correct. More detail on the exact commands
> I used: https://marc.info/?l=linux-raid&m=145349072305613&w=2
>
> Good luck. Please report success or failure to the list.
Yes I saw that. I'm headed out to get a replacement disk then I'll
start messing
with the system. Hopefully I'll report back with something later today.
Thanks for the help,
Noah
^ permalink raw reply
* Re: Setting up fakeraid with mdadm / dmraid
From: John Stoffel @ 2016-04-05 13:58 UTC (permalink / raw)
To: Christoph Pleger; +Cc: linux-raid
In-Reply-To: <b6ced30ee931f9a6faf2541db2462401.squirrel@postweb.cs.tu-dortmund.de>
Christoph> I have a machine with an LSI Megaraid Sofware RAID
Christoph> fakeraid, which uses ddf format. About two weeks ago, when
Christoph> I wanted to install the machine, I configured a RAID 1
Christoph> array in the BIOS config utility and then booted the
Christoph> machine from PXE with an NFSROOT. mdadm and dmraid are
Christoph> installed in the NFSROOT, dracut is used for initrd
Christoph> generation. As dracut prefers mdadm over dmraid, mdadm was
Christoph> chosen for RAID management.
First off, I would strongly suggest that you turn off the FakeRAID
entirely, just expose both disks to the OS and have the OS do the
mirroring. FakeRAID was designed when CPUs were much slower, but RAID
HW ASICs were also expensive. So you got the worst of both worlds!
*grin*
It's also almost entirely black magic to support these things, since
the vendors don't generally share the details on the on-disk format
that I'm aware of. I'm probably wrong in the details.
But again, please just use the controller in JBOD (Just a Bunch of
Disks) mode, and let mdadm mirror the disks for you.
Christoph> But though mdadm detected the RAID (it created container
Christoph> device /dev/md127 and raid device /dev/md126), it destroyed
Christoph> it - that is, though I did not perform any action on the
Christoph> raid disks, at the next boot, the BIOS RAID utility had
Christoph> "forgotten" about the RAID configuration it had created
Christoph> before.
Christoph> Why are the possible reasons for that?
Crappy implementation of semi-secret on-disk format? It could be
that mdadm over-wrote something on the disk, or you partitioned it and
then overwrote something the FakeRAID wanted kept around.
In general, just don't go the fake RAID route, it's not worth the
hassle.
John
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox