* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Piergiorgio Sartor @ 2014-10-30 17:25 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: Ethan Wilson, linux-raid
In-Reply-To: <CAK-d5dZgsoy+PF2urM=N1F67w7nLGyc83muyNoT2h+gQt5ODnQ@mail.gmail.com>
On Thu, Oct 30, 2014 at 08:27:27PM +0530, Anshuman Aggarwal wrote:
> What you are suggesting will work for delaying writing the checksum
> (but still making 2 disks work non stop and lead to failure, cost
> etc).
Hi Anshuman,
I'm a bit missing the point here.
In my experience, with my storage systems, I change
disks because they're too small, way long before they
are too old (way long before they fail).
That's why I end up with a collection of small HDDs.
which, in turn, I recycled in some custom storage
system (using disks of different size, like explained
in one of the links posted before).
Honestly, the only reason to spin down the disks, still
in my experience, is for reducing power consumption.
And this can be done with a RAID-6 without problems
and in a extremely flexible way.
So, the bottom line, still in my experience, is that
this you're describing seems quite a nice situation.
Or, I did not understood what you're proposing.
Thanks,
bye,
pg
> I am proposing N independent disks which are rarely accessed. When
> parity has to be written to the remaining 1,2 ...X disks ...it is
> batched up (bcache is feasible) and written out once in a while
> depending on how much write is happening. N-1 disks stay spun down and
> only X disks wake up periodically to get checksum written to (this
> would be tweaked by the user based on how up to date he needs the
> parity to be (tolerance of rebuilding parity in case of crash) and vs
> disk access for each parity write)
>
> It can't be done using any RAID6 because RAID5/6 will stripe all the
> data across the devices making any read access wake up all the
> devices. Ditto for writing to parity on every write to a single disk.
>
> The architecture being proposed is a lazy write to manage parity for
> individual disks which won't suffer from RAID catastrophic data loss
> and concurrent disk.
>
>
>
>
> On 30 October 2014 00:57, Ethan Wilson <ethan.wilson@shiftmail.org> wrote:
> > On 29/10/2014 10:25, Anshuman Aggarwal wrote:
> >>
> >> Right on most counts but please see comments below.
> >>
> >> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
> >>>
> >>> Just to be sure I understand, you would have N + X devices. Each of the
> >>> N
> >>> devices contains an independent filesystem and could be accessed directly
> >>> if
> >>> needed. Each of the X devices contains some codes so that if at most X
> >>> devices in total died, you would still be able to recover all of the
> >>> data.
> >>> If more than X devices failed, you would still get complete data from the
> >>> working devices.
> >>>
> >>> Every update would only write to the particular N device on which it is
> >>> relevant, and all of the X devices. So N needs to be quite a bit bigger
> >>> than X for the spin-down to be really worth it.
> >>>
> >>> Am I right so far?
> >>
> >> Perfectly right so far. I typically have a N to X ratio of 4 (4
> >> devices to 1 data) so spin down is totally worth it for data
> >> protection but more on that below.
> >>
> >>> For some reason the writes to X are delayed... I don't really understand
> >>> that part.
> >>
> >> This delay is basically designed around archival devices which are
> >> rarely read from and even more rarely written to. By delaying writes
> >> on 2 criteria ( designated cache buffer filling up or preset time
> >> duration from last write expiring) we can significantly reduce the
> >> writes on the parity device. This assumes that we are ok to lose a
> >> movie or two in case the parity disk is not totally up to date but are
> >> more interested in device longevity.
> >>
> >>> Sounds like multi-parity RAID6 with no parity rotation and
> >>> chunksize == devicesize
> >>
> >> RAID6 would present us with a joint device and currently only allows
> >> writes to that directly, yes? Any writes will be striped.
> >
> >
> > I am not totally sure I understand your design, but it seems to me that the
> > following solution could work for you:
> >
> > MD raid-6, maybe multi-parity (multi-parity not implemented yet in MD yet,
> > but just do a periodic scrub and 2 parities can be fine. Wake-up is not so
> > expensive that you can't scrub)
> >
> > Over that you put a raid1 of 2 x 4TB disks as a bcache cache device (those
> > two will never spin-down) in writeback mode with writeback_running=off .
> > This will prevent writes to backend and leave the backend array spun down.
> > When bcache is almost full (poll dirty_data), switch to writeback_running=on
> > and writethrough: it will wake up the backend raid6 array and flush all
> > dirty data. You can then then revert to writeback and writeback_running=off.
> > After this you can spin-down the backend array again.
> >
> > You also get read caching for free, which helps the backend array to stay
> > spun down as much as possible.
> >
> > Maybe you can modify bcache slightly so to implement an automatic switching
> > between the modes as described above, instead of polling the state from
> > outside.
> >
> > Would that work, or you are asking something different?
> >
> > EW
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-10-30 15:00 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <CAK-d5dah-NyQzVNBScYoVSo2cpGA8F3vuK_Zh1YzQn5Mr+_-oQ@mail.gmail.com>
Would chunksize==disksize work? Wouldn't that lead to the entire
parity be invalidated for any write to any of the disks (assuming md
operates at a chunk level)...also please see my reply below
On 29 October 2014 14:55, Anshuman Aggarwal <anshuman.aggarwal@gmail.com> wrote:
> Right on most counts but please see comments below.
>
> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
>> Just to be sure I understand, you would have N + X devices. Each of the N
>> devices contains an independent filesystem and could be accessed directly if
>> needed. Each of the X devices contains some codes so that if at most X
>> devices in total died, you would still be able to recover all of the data.
>> If more than X devices failed, you would still get complete data from the
>> working devices.
>>
>> Every update would only write to the particular N device on which it is
>> relevant, and all of the X devices. So N needs to be quite a bit bigger
>> than X for the spin-down to be really worth it.
>>
>> Am I right so far?
>
> Perfectly right so far. I typically have a N to X ratio of 4 (4
> devices to 1 data) so spin down is totally worth it for data
> protection but more on that below.
>
>>
>> For some reason the writes to X are delayed... I don't really understand
>> that part.
>
> This delay is basically designed around archival devices which are
> rarely read from and even more rarely written to. By delaying writes
> on 2 criteria ( designated cache buffer filling up or preset time
> duration from last write expiring) we can significantly reduce the
> writes on the parity device. This assumes that we are ok to lose a
> movie or two in case the parity disk is not totally up to date but are
> more interested in device longevity.
>
>>
>> Sounds like multi-parity RAID6 with no parity rotation and
>> chunksize == devicesize
> RAID6 would present us with a joint device and currently only allows
> writes to that directly, yes? Any writes will be striped.
> In any case would md raid allow the underlying device to be written to
> directly? Also how would it know that the device has been written to
> and hence parity has to be updated? What about the superblock which
> the FS would not know about?
>
> Also except for the delayed checksum writing part which would be
> significant if one of the objectives is to reduce the amount of
> writes. Can we delay that in the code currently for RAID6? I
> understand the objective of RAID6 is to ensure data recovery and we
> are looking at a compromise in this case.
>
> If feasible, this can be an enhancement to MD RAID as well where N
> devices are presented instead of a single joint device in case of
> raid6 (maybe the multi part device can be individual disks?)
>
> It will certainly solve my problem of where to store the metadata. I
> was currently hoping to just store it as a configuration file to be
> read by the initramfs since in this case worst case scenario the
> checksum goes out of sync and is rebuilt from scratch.
>
>>
>> I wouldn't use device-mapper myself, but you are unlikely to get an entirely
>> impartial opinion from me on that topic.
>
> I haven't hacked around the kernel internals much so far so will have
> to dig out that history. I will welcome any particular links/mail
> threads I should look at for guidance (with both yours and opposing
> points of view)
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-10-30 14:57 UTC (permalink / raw)
To: Ethan Wilson; +Cc: linux-raid
In-Reply-To: <54513FB8.2050407@shiftmail.org>
What you are suggesting will work for delaying writing the checksum
(but still making 2 disks work non stop and lead to failure, cost
etc).
I am proposing N independent disks which are rarely accessed. When
parity has to be written to the remaining 1,2 ...X disks ...it is
batched up (bcache is feasible) and written out once in a while
depending on how much write is happening. N-1 disks stay spun down and
only X disks wake up periodically to get checksum written to (this
would be tweaked by the user based on how up to date he needs the
parity to be (tolerance of rebuilding parity in case of crash) and vs
disk access for each parity write)
It can't be done using any RAID6 because RAID5/6 will stripe all the
data across the devices making any read access wake up all the
devices. Ditto for writing to parity on every write to a single disk.
The architecture being proposed is a lazy write to manage parity for
individual disks which won't suffer from RAID catastrophic data loss
and concurrent disk.
On 30 October 2014 00:57, Ethan Wilson <ethan.wilson@shiftmail.org> wrote:
> On 29/10/2014 10:25, Anshuman Aggarwal wrote:
>>
>> Right on most counts but please see comments below.
>>
>> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
>>>
>>> Just to be sure I understand, you would have N + X devices. Each of the
>>> N
>>> devices contains an independent filesystem and could be accessed directly
>>> if
>>> needed. Each of the X devices contains some codes so that if at most X
>>> devices in total died, you would still be able to recover all of the
>>> data.
>>> If more than X devices failed, you would still get complete data from the
>>> working devices.
>>>
>>> Every update would only write to the particular N device on which it is
>>> relevant, and all of the X devices. So N needs to be quite a bit bigger
>>> than X for the spin-down to be really worth it.
>>>
>>> Am I right so far?
>>
>> Perfectly right so far. I typically have a N to X ratio of 4 (4
>> devices to 1 data) so spin down is totally worth it for data
>> protection but more on that below.
>>
>>> For some reason the writes to X are delayed... I don't really understand
>>> that part.
>>
>> This delay is basically designed around archival devices which are
>> rarely read from and even more rarely written to. By delaying writes
>> on 2 criteria ( designated cache buffer filling up or preset time
>> duration from last write expiring) we can significantly reduce the
>> writes on the parity device. This assumes that we are ok to lose a
>> movie or two in case the parity disk is not totally up to date but are
>> more interested in device longevity.
>>
>>> Sounds like multi-parity RAID6 with no parity rotation and
>>> chunksize == devicesize
>>
>> RAID6 would present us with a joint device and currently only allows
>> writes to that directly, yes? Any writes will be striped.
>
>
> I am not totally sure I understand your design, but it seems to me that the
> following solution could work for you:
>
> MD raid-6, maybe multi-parity (multi-parity not implemented yet in MD yet,
> but just do a periodic scrub and 2 parities can be fine. Wake-up is not so
> expensive that you can't scrub)
>
> Over that you put a raid1 of 2 x 4TB disks as a bcache cache device (those
> two will never spin-down) in writeback mode with writeback_running=off .
> This will prevent writes to backend and leave the backend array spun down.
> When bcache is almost full (poll dirty_data), switch to writeback_running=on
> and writethrough: it will wake up the backend raid6 array and flush all
> dirty data. You can then then revert to writeback and writeback_running=off.
> After this you can spin-down the backend array again.
>
> You also get read caching for free, which helps the backend array to stay
> spun down as much as possible.
>
> Maybe you can modify bcache slightly so to implement an automatic switching
> between the modes as described above, instead of polling the state from
> outside.
>
> Would that work, or you are asking something different?
>
> EW
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Grub2/raid quickie ...
From: Chris Murphy @ 2014-10-30 1:21 UTC (permalink / raw)
To: Wols Lists; +Cc: linux-raid
In-Reply-To: <5451818B.6010809@youngman.org.uk>
On Oct 29, 2014, at 6:08 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> But the other one, the "root=UUID=...", I'm damned if
> I can find that id anywhere! Does anyone know a command, that will tell
> me what that id is, so I can fix grub.cfg on my development system?
blkid
The UUID being referenced by grub-mkconfig is the first UUID reported by blkid, which is the filesystem volume UUID. It's not the same thing as the partition UUID found on GPT disks (a.k.a. unique partition GUID).
>
> (As an aside, how do I retrospectively force a name onto an array so
> that I can guarantee it will be called /dev/md100, or /dev/mdhome, or
> whatever? I daren't use the /dev/mdxxx name at the moment, because they
> change and the result is a system that fails to boot, or is unusable, or
> some other malarkey. Dead easy for me to fix, but my wife will go spare!)
Pretty sure that goes in mdadm.conf and then you need to rebuild the initramfs since that mdadm.conf gets pulled in and referenced during early boot. Or I'm 100% wrong, and hopefully someone else says what's correct.
Chris Murphy
^ permalink raw reply
* Re: Grub2/raid quickie ...
From: NeilBrown @ 2014-10-30 1:03 UTC (permalink / raw)
To: Wols Lists; +Cc: linux-raid
In-Reply-To: <5451818B.6010809@youngman.org.uk>
[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]
On Thu, 30 Oct 2014 00:08:43 +0000 Wols Lists <antlists@youngman.org.uk>
wrote:
> After my last shenanigans, I've now got my main system booting
> successfully from a raid / partition. HOWEVER.
>
> I'm somewhat puzzled by grub.cfg, and this is in part behind why my dev
> system won't boot raid properly, I think. Grub uses two different UUIDs!?!?
>
> One of them, (mduuid/...) is dead easy to find, "mdadm -D" gives the md
> uuid of the array. But the other one, the "root=UUID=...", I'm damned if
> I can find that id anywhere! Does anyone know a command, that will tell
> me what that id is, so I can fix grub.cfg on my development system?
blkid /dev/whatever
>
> (As an aside, how do I retrospectively force a name onto an array so
> that I can guarantee it will be called /dev/md100, or /dev/mdhome, or
> whatever? I daren't use the /dev/mdxxx name at the moment, because they
> change and the result is a system that fails to boot, or is unusable, or
> some other malarkey. Dead easy for me to fix, but my wife will go spare!)
mdadm --assemble /dev/mdhome --update=name list-of-devices
You cannot do this without stopping the array, sorry. Boot off a usb or CD
is necessary.
NeilBrown
>
> Cheers,
> Wol
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Grub2/raid quickie ...
From: Wols Lists @ 2014-10-30 0:08 UTC (permalink / raw)
To: linux-raid
After my last shenanigans, I've now got my main system booting
successfully from a raid / partition. HOWEVER.
I'm somewhat puzzled by grub.cfg, and this is in part behind why my dev
system won't boot raid properly, I think. Grub uses two different UUIDs!?!?
One of them, (mduuid/...) is dead easy to find, "mdadm -D" gives the md
uuid of the array. But the other one, the "root=UUID=...", I'm damned if
I can find that id anywhere! Does anyone know a command, that will tell
me what that id is, so I can fix grub.cfg on my development system?
(As an aside, how do I retrospectively force a name onto an array so
that I can guarantee it will be called /dev/md100, or /dev/mdhome, or
whatever? I daren't use the /dev/mdxxx name at the moment, because they
change and the result is a system that fails to boot, or is unusable, or
some other malarkey. Dead easy for me to fix, but my wife will go spare!)
Cheers,
Wol
^ permalink raw reply
* Re: Raid1 element stuck in (S) state
From: NeilBrown @ 2014-10-29 22:47 UTC (permalink / raw)
To: micah; +Cc: micah anderson, linux-raid
In-Reply-To: <87zjcepnno.fsf@muck.riseup.net>
[-- Attachment #1: Type: text/plain, Size: 3757 bytes --]
On Wed, 29 Oct 2014 17:32:43 -0400 micah <micah@riseup.net> wrote:
> NeilBrown <neilb@suse.de> writes:
>
> > On Wed, 29 Oct 2014 10:03:16 -0400 micah <micah@riseup.net> wrote:
> >
> >> NeilBrown <neilb@suse.de> writes:
> >>
> >> > On Mon, 27 Oct 2014 10:18:47 -0400 micah anderson <micah@debian.org> wrote:
> >> >
> >> >>
> >> >> Hi,
> >> >>
> >> >> i've got a raid1 setup, where one drive died, it was replaced with a new
> >> >> one, but its stuck in a (S) state and I can't seem to get it added into
> >> >> the array, /proc/mdstat looks like this:
> >> >>
> >> >> md3 : active raid1 sdc1[2](S) sdd1[1]
> >> >> 976759672 blocks super 1.2 [2/1] [_U]
> >> >>
> >> >> where sdc1 is the replaced drive.
> >> >>
> >> >> What is the right way to get this added back?
> >> >>
> >> >
> >> > I've a feeling this bug might have been fixed.
> >> > What versions of mdadm and Linux are you using?
> >>
> >> I'm using squeeze here, and had 3.1.4-1+8efb9d1+squeeze1 installed, I
> >> just installed the backport, which is 3.2.5-3~bpo60+1.
> >
> > Is assume that is the version of mdadm. You didn't say what version of Linux.
>
> Yes, that is the version of mdadm. I am running squeeze, which is a
> 2.6.32-5 version of the kernel, and it is an amd64 machine.
Wow.... a 5 year old kernel.
I suspect this is a kernel bug you are hitting. I vaguely remember something
like that - spares not becoming properly activated after recovery.
I don't remember the details and a quick look at commit logs doesn't show
anything obvious.
And maybe Debian has backported something which broke something.
Can you try a newer kernel at all?
NeilBrown
>
> >> > Are there any errors in the kernel logs when you --add the device?
> >
> > You didn't answer this question either. Are there any messages in the
> > kernel log: /var/log/kern.log on debian.
> > Or in the output of "dmesg".
>
> The only thing I see in the log is:
>
> [307932.328420] mdadm: sending ioctl 1261 to a partition!
> [307932.328425] mdadm: sending ioctl 1261 to a partition!
> [307932.346642] mdadm: sending ioctl 1261 to a partition!
> [307932.346648] mdadm: sending ioctl 1261 to a partition!
> [307932.352466] mdadm: sending ioctl 1261 to a partition!
> [307932.352468] mdadm: sending ioctl 1261 to a partition!
> [307932.376821] mdadm: sending ioctl 1261 to a partition!
> [307932.376824] mdadm: sending ioctl 1261 to a partition!
> [307932.377623] mdadm: sending ioctl 1261 to a partition!
> [307932.377630] mdadm: sending ioctl 1261 to a partition!
> [307932.467292] md: bind<sdc1>
> [307932.588154] RAID1 conf printout:
> [307932.588159] --- wd:1 rd:2
> [307932.588164] disk 0, wo:1, o:1, dev:sdc1
> [307932.588167] disk 1, wo:0, o:1, dev:sdd1
> [307932.588248] md: recovery of RAID array md3
> [307932.588251] md: minimum _guaranteed_ speed: 50000 KB/sec/disk.
> [307932.588254] md: using maximum available idle IO bandwidth (but not more than 2000000 KB/sec) for recovery.
> [307932.588260] md: using 128k window, over a total of 976759672 blocks.
>
> but this is just when the device is added, after that it appears that
> logrotation failed and I have a zero byte kern.log, and firewall spew
> has filled up my dmesg ring.
>
> >> Can I just zero the superblock of that device and re-add it in order to
> >> resolve this?
> >
> >
> > If it resyncs and the is still spare, there was almost certainly some sort of
> > failure. There really must be something in the kernel logs at that time.
>
> It did resync, and is still a spare.... Now that I've fixed the logs,
> I'm going to try it again to see if there is any error that happens
> after the sync finishes.
>
> micah
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Raid1 element stuck in (S) state
From: micah @ 2014-10-29 21:32 UTC (permalink / raw)
To: NeilBrown; +Cc: micah anderson, linux-raid
In-Reply-To: <20141030071019.1a5c57e7@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Wed, 29 Oct 2014 10:03:16 -0400 micah <micah@riseup.net> wrote:
>
>> NeilBrown <neilb@suse.de> writes:
>>
>> > On Mon, 27 Oct 2014 10:18:47 -0400 micah anderson <micah@debian.org> wrote:
>> >
>> >>
>> >> Hi,
>> >>
>> >> i've got a raid1 setup, where one drive died, it was replaced with a new
>> >> one, but its stuck in a (S) state and I can't seem to get it added into
>> >> the array, /proc/mdstat looks like this:
>> >>
>> >> md3 : active raid1 sdc1[2](S) sdd1[1]
>> >> 976759672 blocks super 1.2 [2/1] [_U]
>> >>
>> >> where sdc1 is the replaced drive.
>> >>
>> >> What is the right way to get this added back?
>> >>
>> >
>> > I've a feeling this bug might have been fixed.
>> > What versions of mdadm and Linux are you using?
>>
>> I'm using squeeze here, and had 3.1.4-1+8efb9d1+squeeze1 installed, I
>> just installed the backport, which is 3.2.5-3~bpo60+1.
>
> Is assume that is the version of mdadm. You didn't say what version of Linux.
Yes, that is the version of mdadm. I am running squeeze, which is a
2.6.32-5 version of the kernel, and it is an amd64 machine.
>> > Are there any errors in the kernel logs when you --add the device?
>
> You didn't answer this question either. Are there any messages in the
> kernel log: /var/log/kern.log on debian.
> Or in the output of "dmesg".
The only thing I see in the log is:
[307932.328420] mdadm: sending ioctl 1261 to a partition!
[307932.328425] mdadm: sending ioctl 1261 to a partition!
[307932.346642] mdadm: sending ioctl 1261 to a partition!
[307932.346648] mdadm: sending ioctl 1261 to a partition!
[307932.352466] mdadm: sending ioctl 1261 to a partition!
[307932.352468] mdadm: sending ioctl 1261 to a partition!
[307932.376821] mdadm: sending ioctl 1261 to a partition!
[307932.376824] mdadm: sending ioctl 1261 to a partition!
[307932.377623] mdadm: sending ioctl 1261 to a partition!
[307932.377630] mdadm: sending ioctl 1261 to a partition!
[307932.467292] md: bind<sdc1>
[307932.588154] RAID1 conf printout:
[307932.588159] --- wd:1 rd:2
[307932.588164] disk 0, wo:1, o:1, dev:sdc1
[307932.588167] disk 1, wo:0, o:1, dev:sdd1
[307932.588248] md: recovery of RAID array md3
[307932.588251] md: minimum _guaranteed_ speed: 50000 KB/sec/disk.
[307932.588254] md: using maximum available idle IO bandwidth (but not more than 2000000 KB/sec) for recovery.
[307932.588260] md: using 128k window, over a total of 976759672 blocks.
but this is just when the device is added, after that it appears that
logrotation failed and I have a zero byte kern.log, and firewall spew
has filled up my dmesg ring.
>> Can I just zero the superblock of that device and re-add it in order to
>> resolve this?
>
>
> If it resyncs and the is still spare, there was almost certainly some sort of
> failure. There really must be something in the kernel logs at that time.
It did resync, and is still a spare.... Now that I've fixed the logs,
I'm going to try it again to see if there is any error that happens
after the sync finishes.
micah
^ permalink raw reply
* Re: mdadm creates corrupt superblock
From: Hans Kraus @ 2014-10-29 21:11 UTC (permalink / raw)
To: NeilBrown; +Cc: Linux-RAID
In-Reply-To: <20141029080019.272bc844@notabene.brown>
Hi Neil,
many thanks for your response. I rebooted the machine and got new drive
assignments. The drive in question is now sdc1. mdadm gave an error with
the dump, results below (nothing to report by dmesg) [by the way, I
wasn'nt even able to create an array on that drive with
'mdadm --create --level=1 -n 2 /dev/sdc1 missing', same error(s)]:
-------------------------------------------------------------------------
root@nashorn:~# mdadm -E /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
Name : nashorn:126 (local to host nashorn)
Creation Time : Mon Oct 27 15:58:38 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
Array Size : 976630488 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=0 sectors
State : clean
Device UUID : f7a91c46:e4cda0c0:1b770fd7:876634f8
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Oct 29 18:42:16 2014
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 939b22d1 - correct
Events : 3325
Device Role : Active device 0
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
root@nashorn:~#
-------------------------------------------------------------------------
root@nashorn:~# mdadm -E /dev/sdh1
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
Name : nashorn:126 (local to host nashorn)
Creation Time : Mon Oct 27 15:58:38 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
Array Size : 976630488 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262046 sectors, after=0 sectors
State : clean
Device UUID : 089d1d0e:4d5ca222:dd716acb:0813815c
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Oct 29 18:42:16 2014
Bad Block Log : 512 entries available at offset 82 sectors
Checksum : 16454b96 - correct
Events : 3325
Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
root@nashorn:~#
-------------------------------------------------------------------------
root@nashorn:~# mdadm -E /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
Name : nashorn:126 (local to host nashorn)
Creation Time : Mon Oct 27 15:58:38 2014
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
Array Size : 0
Used Dev Size : 0
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=1953260976 sectors
State : clean
Device UUID : 4d1420c6:79978477:e86412b1:24beebfc
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Oct 28 11:53:49 2014
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6f02d040 - expected 891eedff
Events : 0
Device Role : spare
Array State : RR ('A' == active, '.' == missing, 'R' == replacing)
-------------------------------------------------------------------------
root@nashorn:~# mkdir -p /tmp/dump
root@nashorn:~# mdadm --dump /tmp/dump /dev/sdc1
mdadm: Failed to copy metadata from /dev/sdc1 to /tmp/dump/sdc1
-------------------------------------------------------------------------
Kind regards, Hans
Am 28.10.2014 22:00, schrieb NeilBrown:
> On Tue, 28 Oct 2014 19:06:20 +0100 Hans Kraus <hans@hanswkraus.com> wrote:
>
>> Hi,
>>
>> I debugged my problem a bit more: it seems that mdadm creates a defect
>> superblock. That's repeatedly happening, but only for one drive. I
>> copied zeros to that drive (via ddrescue /dev/zero ...) and the drive
>> looks OK. The info I got:
>>
>
> and you send me the metadata of all your devices please?
>
> mkdir /tmp/dump
> mdadm --dump /tmp/dump /dev/sde1
> tar czvf /tmp/dump.tgz /tmp/dump
>
> and then send /tmp/dump.tgz
>
> Thanks.
>
> NeilBrown
>
> =================================================================================================================================================================================================
>> root@nashorn:/home/kraush/work/smartctrl# mdadm --zero-superblock /dev/sde1
>> mdadm: Unrecognised md component device - /dev/sde1
>> root@nashorn:/home/kraush/work/smartctrl# mdadm --add /dev/md126 /dev/sde1
>> mdadm: add new device failed for /dev/sde1 as 4: Invalid argument
>> root@nashorn:/home/kraush/work/smartctrl# mdadm -E /dev/sde1
>> /dev/sde1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x1
>> Array UUID : e7caa5d1:b33fd2a5:7782fb0c:9d8d9d5b
>> Name : nashorn:126 (local to host nashorn)
>> Creation Time : Mon Oct 27 15:58:38 2014
>> Raid Level : raid1
>> Raid Devices : 2
>>
>> Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
>> Array Size : 0
>> Used Dev Size : 0
>> Data Offset : 262144 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=262056 sectors, after=1953260976 sectors
>> State : clean
>> Device UUID : 4d1420c6:79978477:e86412b1:24beebfc
>>
>> Internal Bitmap : 8 sectors from superblock
>> Update Time : Tue Oct 28 11:53:49 2014
>> Bad Block Log : 512 entries available at offset 72 sectors
>> Checksum : 6f02d040 - expected 891eedff
>> Events : 0
>>
>>
>> Device Role : spare
>> Array State : RR ('A' == active, '.' == missing, 'R' == replacing)
>> root@nashorn:/home/kraush/work/smartctrl#
>> -----------------------------------------------------------------------------------------------------------
>> root@nashorn:/home/kraush/work/smartctrl# cat /proc/mdstat
>> Personalities : [raid1] [raid6] [raid5] [raid4]
>> md127 : active (auto-read-only) raid6 sda1[0] sdk1[6](S) sdl2[7](S)
>> sdh1[5] sdg1[4] sdf1[3] sdc1[2] sdb1[1]
>> 1953017856 blocks super 1.2 level 6, 512k chunk, algorithm 2
>> [6/6] [UUUUUU]
>> bitmap: 0/4 pages [0KB], 65536KB chunk
>>
>> md10 : active (auto-read-only) raid1 sdi2[0] sdl3[1]
>> 87833408 blocks super 1.2 [2/2] [UU]
>> bitmap: 0/1 pages [0KB], 65536KB chunk
>>
>> md126 : active raid1 sdd1[3] sdj1[2]
>> 488254464 blocks super 1.2 [2/2] [UU]
>> bitmap: 0/4 pages [0KB], 65536KB chunk
>>
>> md0 : active raid1 sdl1[4] sdd2[3] sdi1[5]
>> 156157824 blocks super 1.2 [3/1] [U__]
>> [===================>.] recovery = 95.6% (149290048/156157824)
>> finish=8.4min speed=13496K/sec
>> bitmap: 1/2 pages [4KB], 65536KB chunk
>>
>> unused devices: <none>
>> root@nashorn:/home/kraush/work/smartctrl#
>> -----------------------------------------------------------------------------------------------------------
>> dmesg:
>> [33299.387382] md: invalid superblock checksum on sde1
>> [33299.387385] md: sde1 does not have a valid v1.2 superblock, not
>> importing!
>> [33299.387408] md: md_import_device returned -22
>> root@nashorn:/home/kraush/work/smartctrl#
>> -----------------------------------------------------------------------------------------------------------
>> root@nashorn:/home/kraush/work/mdadm# mdadm --version
>> mdadm - v3.3-161-gfed12d4 - 21st August 2014
>> =================================================================================================================================================================================================
>>
>> Kind regards, Hans
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.
From: Jason Keltz @ 2014-10-29 20:56 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141029135749.241f9e50@notabene.brown>
On 10/28/2014 10:57 PM, NeilBrown wrote:
> On Tue, 28 Oct 2014 22:34:07 -0400 Jason Keltz <jas@cse.yorku.ca> wrote:
>
>> On 28/10/2014 6:38 PM, NeilBrown wrote:
>>> On Mon, 20 Oct 2014 17:07:38 -0400 Jason Keltz<jas@cse.yorku.ca> wrote:
>>>
>>>> On 10/20/2014 12:19 PM, Jason Keltz wrote:
>>>>> Hi.
>>>>>
>>>>> I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system.
>>>>> I've experimented with setting "speed_limit_min" and "speed_limit_max"
>>>>> kernel variables so that I get the best balance of performance during
>>>>> a RAID rebuild of one of the RAID1 pairs. If, for example, I set
>>>>> speed_limit_min AND speed_limit_max to 80000 then fail a disk when
>>>>> there is no other disk activity, then I do get a rebuild rate of
>>>>> around 80 MB/s. However, if I then start up a write intensive
>>>>> operation on the MD array (eg. a dd, or a mkfs on an LVM logical
>>>>> volume that is created on that MD), then, my write operation seems to
>>>>> get "full power", and my rebuild drops to around 25 MB/s. This means
>>>>> that the rebuild of my RAID10 disk is going to take a huge amount of
>>>>> time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to
>>>>> the same value, am I not guaranteeing the rebuild speed? Is this a bug
>>>>> that I should be reporting to Red Hat, or a "feature"?
>>>>>
>>>>> Thanks in advance for any help that you can provide...
>>>>>
>>>>> Jason.
>>>> I would like to add that I downloaded the latest version of Ubuntu, and
>>>> am running it on the same server with the same MD.
>>>> When I set speed_limit_min and speed_limit_max to 80000, I was able to
>>>> start two large dds on the md array, and the rebuild stuck at around 71
>>>> MB/s, which is close enough. This leads me to believe that the problem
>>>> above is probably a RHEL6 issue. However, after I stopped the two dd
>>>> operations, and raised both speed_limit_min and speed_limit_max to
>>>> 120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes
>>>> .. now it seems to be at 100 MB/s... but doesn't seem to get any higher
>>>> (even though I had 120 MB/s and above on the RHEL system without any
>>>> load)... Hmm.
>>>>
>>> md certainly cannot "guarantee" any speed - it can only deliver what the
>>> underlying devices deliver.
>>> I know the kernels logs say something about a "guarantee". That was added
>>> before my time and I haven't had occasion to remove it.
>>>
>>> md will normally just try to recover as fast as it can unless that exceeds
>>> one of the limits - then it will back-off.
>>> What speed it actually achieved depends on other load and the behaviour of
>>> the IO scheduler.
>>>
>>> "RHEL6" and "Ubuntu" don't mean a lot to me. Specific kernel version might,
>>> though in the case of Redhat I know that backport lots of stuff so even the
>>> kernel version isn't very helpful. I'm must prefer having report against
>>> mainline kernels.
>>>
>>> Rotating drives do get lower transfer speeds at higher addresses. That might
>>> explain the 120 / 100 difference.
>> Hi Neil,
>> Thanks very much for your response.
>> I must say that I'm a little puzzled though. I'm coming from using a
>> 3Ware hardware RAID controller where I could configure how much of the
>> disk bandwidth is to be used for a rebuild versus I/O. From what I
>> understand, you're saying that MD can only use the disk bandwidth
>> available to it. It seems that it doesn't take any priority in the I/O
>> chain. It will only attempt to use no less than min bandwidth, and no
>> more than max bandwidth for the rebuild, but if you're on a busy system,
>> and other system I/O needs that disk bandwidth, then there's nothing it
>> can do about it. I guess I just don't understand why. Why can't md be
>> given a priority in the kernel to allow the admin to decide how much
>> bandwidth goes to system I/O versus rebuild I/O. Even in a busy system,
>> I still want to allocate at least some minimum bandwidth to MD. In
>> fact, in the event of a disk failure, I want to have a whole lot of the
>> disk bandwidth dedicated to MD. It's something about short term pain
>> for long term gain? I'd rather not have the users suffer at all, but if
>> they do have to suffer, I'd rather them suffer for a few hours, knowing
>> that after that, the RAID system is in a perfectly good state with no
>> bad disks as opposed to letting a bad disk resync take days because the
>> system is really busy... days during which another failure might occur!
>>
>> Jason.
> It isn't so much "that MD can only use..." but rather "that MD does only
> use ...".
Got it..
> This is how the code has "always" worked and no-one has ever bothered to
> change it, or to ask for it to be changed (that I recall).
I'm actually not surprised to hear that since I spent a considerable
time trying to find articles talking about this topic, and couldn't find
a single thing! :) I, on the other hand, am replacing a hardware RAID
system with MD, so I've been "spoiled" already! (not with the
performance of hardware RAID, but the functionality) :)
> There are difficulties in guaranteeing a minimum when the array uses
> partitions from devices on which other partitions are used for other things.
> In that case I don't think it is practical to make guarantees, but that
> needn't stop us making guarantees when we can I guess.
>
> If the configured bandwidth exceeded the physically available bandwidth I
> don't think we would want to exclude non-resync IO completely, so the
> guaranty would have to be:
> N MB/sec or M% of available, whichever is less
>
> We could even implement the different approach in a back-compatible way.
> Introduce a new setting "max_sync_percent". By default that is unset and the
> current algorithm applies.
> If it is set to something below 100, non-resync IO is throttled to
> an appropriate fraction of the actual resync throughput whenever that is
> below sync_speed_min.
>
> Or something like that.
That actually sounds great! I can certainly understand and appreciate
how it would be difficult to handle arrays using partitions from
multiple devices. Maybe the functionality only works if you're using
full disk devices. :) (Okay, on second thought, that's about 99% of the
people using MD -- apparently, word has it that only you and I are using
MD full disk devices).
> Some care would be needed in comparing throughput and sync throughput is
> measured per-device, while non-resync throughput might be measured per-array.
> Maybe the throttling would happen per-device??
>
> All we need now is for someone to firm up the design and then write the code.
Maybe someone on the list will step forward :D
The truth is, as people start to combine larger and larger disks, and
rebuild times go up and up and up, this type of request will become more
common....
Jas.
^ permalink raw reply
* Re: Raid1 element stuck in (S) state
From: NeilBrown @ 2014-10-29 20:10 UTC (permalink / raw)
To: micah; +Cc: micah anderson, linux-raid
In-Reply-To: <8761f3q8gr.fsf@muck.riseup.net>
[-- Attachment #1: Type: text/plain, Size: 1796 bytes --]
On Wed, 29 Oct 2014 10:03:16 -0400 micah <micah@riseup.net> wrote:
> NeilBrown <neilb@suse.de> writes:
>
> > On Mon, 27 Oct 2014 10:18:47 -0400 micah anderson <micah@debian.org> wrote:
> >
> >>
> >> Hi,
> >>
> >> i've got a raid1 setup, where one drive died, it was replaced with a new
> >> one, but its stuck in a (S) state and I can't seem to get it added into
> >> the array, /proc/mdstat looks like this:
> >>
> >> md3 : active raid1 sdc1[2](S) sdd1[1]
> >> 976759672 blocks super 1.2 [2/1] [_U]
> >>
> >> where sdc1 is the replaced drive.
> >>
> >> What is the right way to get this added back?
> >>
> >
> > I've a feeling this bug might have been fixed.
> > What versions of mdadm and Linux are you using?
>
> I'm using squeeze here, and had 3.1.4-1+8efb9d1+squeeze1 installed, I
> just installed the backport, which is 3.2.5-3~bpo60+1.
Is assume that is the version of mdadm. You didn't say what version of Linux.
>
> > Are there any errors in the kernel logs when you --add the device?
You didn't answer this question either. Are there any messages in the
kernel log: /var/log/kern.log on debian.
Or in the output of "dmesg".
>
> After installing the backported 3.2.5, I tried to add it, and it said:
>
> # mdadm --add /dev/md3 /dev/sdc1
> mdadm: Cannot open /dev/sdc1: Device or resource busy
>
> so I did a --remove of the drive, and then added it, it then proceeded
> to sync the array, and after that finished, it is now back in the (S)
> state.
>
> Can I just zero the superblock of that device and re-add it in order to
> resolve this?
If it resyncs and the is still spare, there was almost certainly some sort of
failure. There really must be something in the kernel logs at that time.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Ethan Wilson @ 2014-10-29 19:27 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAK-d5dah-NyQzVNBScYoVSo2cpGA8F3vuK_Zh1YzQn5Mr+_-oQ@mail.gmail.com>
On 29/10/2014 10:25, Anshuman Aggarwal wrote:
> Right on most counts but please see comments below.
>
> On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
>> Just to be sure I understand, you would have N + X devices. Each of the N
>> devices contains an independent filesystem and could be accessed directly if
>> needed. Each of the X devices contains some codes so that if at most X
>> devices in total died, you would still be able to recover all of the data.
>> If more than X devices failed, you would still get complete data from the
>> working devices.
>>
>> Every update would only write to the particular N device on which it is
>> relevant, and all of the X devices. So N needs to be quite a bit bigger
>> than X for the spin-down to be really worth it.
>>
>> Am I right so far?
> Perfectly right so far. I typically have a N to X ratio of 4 (4
> devices to 1 data) so spin down is totally worth it for data
> protection but more on that below.
>
>> For some reason the writes to X are delayed... I don't really understand
>> that part.
> This delay is basically designed around archival devices which are
> rarely read from and even more rarely written to. By delaying writes
> on 2 criteria ( designated cache buffer filling up or preset time
> duration from last write expiring) we can significantly reduce the
> writes on the parity device. This assumes that we are ok to lose a
> movie or two in case the parity disk is not totally up to date but are
> more interested in device longevity.
>
>> Sounds like multi-parity RAID6 with no parity rotation and
>> chunksize == devicesize
> RAID6 would present us with a joint device and currently only allows
> writes to that directly, yes? Any writes will be striped.
I am not totally sure I understand your design, but it seems to me that
the following solution could work for you:
MD raid-6, maybe multi-parity (multi-parity not implemented yet in MD
yet, but just do a periodic scrub and 2 parities can be fine. Wake-up is
not so expensive that you can't scrub)
Over that you put a raid1 of 2 x 4TB disks as a bcache cache device
(those two will never spin-down) in writeback mode with
writeback_running=off . This will prevent writes to backend and leave
the backend array spun down.
When bcache is almost full (poll dirty_data), switch to
writeback_running=on and writethrough: it will wake up the backend raid6
array and flush all dirty data. You can then then revert to writeback
and writeback_running=off. After this you can spin-down the backend
array again.
You also get read caching for free, which helps the backend array to
stay spun down as much as possible.
Maybe you can modify bcache slightly so to implement an automatic
switching between the modes as described above, instead of polling the
state from outside.
Would that work, or you are asking something different?
EW
^ permalink raw reply
* Re: RAID1 removing failed disk returns EBUSY
From: Joe Lawrence @ 2014-10-29 17:36 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Bill Kuzeja
In-Reply-To: <20141029084113.57f6ae6a@notabene.brown>
On Wed, 29 Oct 2014 08:41:13 +1100
NeilBrown <neilb@suse.de> wrote:
> On Mon, 27 Oct 2014 16:27:48 -0400 Joe Lawrence <joe.lawrence@stratus.com>
> wrote:
>
> > Hi Neil,
> >
> > We've encountered changes in MD and mdadm that have broken our automated
> > disk removal script. In the past, we've been able to run the following
> > after a RAID1 disk component removal:
> >
> > % echo fail > /sys/block/md3/md/dev-sdr5/state
> > % echo remove > /sys/block/md3/md/dev-sdr5/state
> >
> > However, the latest RHEL6.6 code drop has rebased to sufficiently recent
> > MD kernel and mdadm changes, in which the previous commands occasionally
> > fail like so:
> >
> > * MD array is usually resyncing or checking
> > * Component disk /dev/sdr removed via HBA sysfs PCI removal
> > * Following UDEV rule fires:
> >
> > SUBSYSTEM=="block", ACTION=="remove", ENV{ID_PATH}=="?*", \
> > RUN+="/sbin/mdadm -If $name --path $env{ID_PATH}"
> >
> > % mdadm --detail /dev/md3
> > /dev/md3:
> > Version : 1.1
> > Creation Time : Tue Oct 14 17:31:59 2014
> > Raid Level : raid1
> > Array Size : 25149440 (23.98 GiB 25.75 GB)
> > Used Dev Size : 25149440 (23.98 GiB 25.75 GB)
> > Raid Devices : 2
> > Total Devices : 2
> > Persistence : Superblock is persistent
> >
> > Intent Bitmap : Internal
> >
> > Update Time : Wed Oct 15 14:22:34 2014
> > State : active, degraded
> > Active Devices : 1
> > Working Devices : 1
> > Failed Devices : 1
> > Spare Devices : 0
> >
> > Name : localhost.localdomain:3
> > UUID : 40ed68ee:ba41d4cd:28c361ed:be7470b8
> > Events : 142
> >
> > Number Major Minor RaidDevice State
> > 0 65 21 0 faulty
> > 1 65 5 1 active sync /dev/sdj5
> >
> > All attempts to remove this device fail:
> >
> > % echo remove > /sys/block/md3/md/dev-sdr5/state
> > -bash: echo: write error: Device or resource busy
> >
> > This can be traced to state_store():
> >
> > } else if (cmd_match(buf, "remove")) {
> > if (rdev->raid_disk >= 0)
> > err = -EBUSY;
> >
> > After much debugging and systemtapping, I think I've figured out that the
> > sysfs scripting may fail after the following combination of changes:
> >
> > mdadm 8af530b07fce "Enhance incremental removal."
> > kernel 30b8feb730f9 "md/raid5: avoid deadlock when raid5 array has unack
> > badblocks during md_stop_writes"
> >
> > With these two changes:
> >
> > 1 - On the user side, mdadm is trying to set the array_state to read-auto
> > on incremental removal (as invoked by UDEV rule).
> >
> > 2 - Kernel side, md_set_readonly() will set the MD_RECOVERY_FROZEN flag,
> > wake up the mddev->thread and if there is a sync_thread, it will set
> > MD_RECOVERY_INTR and then wait until the sync_thread is set to NULL.
> >
> > When md_check_recovery() gets a chance to run as part of the
> > raid1d() mddev->thread, it may or may not ever get to
> > an invocation of remove_and_add_spares(), for there are but *many*
> > conditional early exits along the way -- for example, if
> > MD_RECOVERY_FROZEN is set, the following condition will bounce out of
> > the routine:
> >
> > if (!test_and_clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
> > test_bit(MD_RECOVERY_FROZEN, &mddev->recovery))
> > goto unlock;
> >
> > the next time around, MD_RECOVERY_NEEDED will have been cleared, so
> > all future tests will return 0 and the negation will always take the
> > early exit path.
> >
> > Back in md_set_readonly(), it may notice that the MD is still in use,
> > so it clears the MD_RECOVERY_FROZEN and then returns -EBUSY, without
> > setting mddev->ro. But the damage has been done as conditions have
> > been set such that md_check_recovery() will never call
> > remove_and_add_spares().
> >
> > This would also explain why an "idle" sync_action clears the wedge: it
> > sets MD_RECOVERY_NEEDED allowing md_check_recovery() to continue executing
> > to remove_and_add_spares().
> >
> > As far as I can tell, this is what is happening to prevent the "remove"
> > write to /sys/block/md3/md/dev-sdr5/state from succeeding. There are
> > certainly a lot of little bit-states between disk removal, UDEV mdadm, and
> > various MD kernel threads, so apologies if I missed an important
> > transition.
> >
> > Would you consider writing "idle" to the MD array sync_action file as a
> > safe and reasonable intermediate workaround step for our script?
> >
> > And of course, any suggestions to whether this is intended behavior (ie,
> > the removed component disk is failed, but stuck in the array)?
> >
> > This is fairly easy for us to reproduce with multiple MD arrays per disk
> > (one per partition) and interrupting a raid check on all of them
> > (especially when they are delayed waiting for the first to finish) by
> > removing the component disk via sysfs PCI removal. We can provide
> > additional debug or testing if required.
> >
>
> Hi Joe,
> thanks for the details analysis!!
>
> I think the correct fix would be that MD_RECOVERY_NEEDED should be set after
> clearing MD_RECOVERY_FROZEN, like the patch below.
> Can you confirm that it works for you?
>
> Writing 'idle' should in general be safe, so that could be used as an interim.
>
> Thanks,
> NeilBrown
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index c03d87b6890a..2c73fcb82593 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -5261,6 +5261,7 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev)
> printk("md: %s still in use.\n",mdname(mddev));
> if (did_freeze) {
> clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> md_wakeup_thread(mddev->thread);
> }
> err = -EBUSY;
> @@ -5275,6 +5276,8 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev)
> mddev->ro = 1;
> set_disk_ro(mddev->gendisk, 1);
> clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> + md_wakeup_thread(mddev->thread);
> sysfs_notify_dirent_safe(mddev->sysfs_state);
> err = 0;
> }
> @@ -5318,6 +5321,7 @@ static int do_md_stop(struct mddev *mddev, int mode,
> mutex_unlock(&mddev->open_mutex);
> if (did_freeze) {
> clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> md_wakeup_thread(mddev->thread);
> }
> return -EBUSY;
Hi Neil,
In my tests, the UDEV "mdadm -If" invocation fails *and* removes the
pulled disk from the MD array. This is okay for our intentions, but I
wanted to make sure that it's okay to skip any failed-but-not-removed
state.
Tested-by: Joe Lawrence <joe.lawrence@stratus.com>
and should this have a
Fixes: 30b8feb730f9 ("md/raid5: avoid deadlock when raid5 array has unack badblocks during md_stop_writes")
tag to mark for stable?
Thanks,
-- Joe
^ permalink raw reply
* Re: Raid1 element stuck in (S) state
From: micah @ 2014-10-29 14:03 UTC (permalink / raw)
To: NeilBrown, micah anderson; +Cc: linux-raid
In-Reply-To: <20141029084224.6d92d8be@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Mon, 27 Oct 2014 10:18:47 -0400 micah anderson <micah@debian.org> wrote:
>
>>
>> Hi,
>>
>> i've got a raid1 setup, where one drive died, it was replaced with a new
>> one, but its stuck in a (S) state and I can't seem to get it added into
>> the array, /proc/mdstat looks like this:
>>
>> md3 : active raid1 sdc1[2](S) sdd1[1]
>> 976759672 blocks super 1.2 [2/1] [_U]
>>
>> where sdc1 is the replaced drive.
>>
>> What is the right way to get this added back?
>>
>
> I've a feeling this bug might have been fixed.
> What versions of mdadm and Linux are you using?
I'm using squeeze here, and had 3.1.4-1+8efb9d1+squeeze1 installed, I
just installed the backport, which is 3.2.5-3~bpo60+1.
> Are there any errors in the kernel logs when you --add the device?
After installing the backported 3.2.5, I tried to add it, and it said:
# mdadm --add /dev/md3 /dev/sdc1
mdadm: Cannot open /dev/sdc1: Device or resource busy
so I did a --remove of the drive, and then added it, it then proceeded
to sync the array, and after that finished, it is now back in the (S)
state.
Can I just zero the superblock of that device and re-add it in order to
resolve this?
thanks!
micah
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-10-29 9:25 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141029200501.1f01269d@notabene.brown>
Right on most counts but please see comments below.
On 29 October 2014 14:35, NeilBrown <neilb@suse.de> wrote:
> Just to be sure I understand, you would have N + X devices. Each of the N
> devices contains an independent filesystem and could be accessed directly if
> needed. Each of the X devices contains some codes so that if at most X
> devices in total died, you would still be able to recover all of the data.
> If more than X devices failed, you would still get complete data from the
> working devices.
>
> Every update would only write to the particular N device on which it is
> relevant, and all of the X devices. So N needs to be quite a bit bigger
> than X for the spin-down to be really worth it.
>
> Am I right so far?
Perfectly right so far. I typically have a N to X ratio of 4 (4
devices to 1 data) so spin down is totally worth it for data
protection but more on that below.
>
> For some reason the writes to X are delayed... I don't really understand
> that part.
This delay is basically designed around archival devices which are
rarely read from and even more rarely written to. By delaying writes
on 2 criteria ( designated cache buffer filling up or preset time
duration from last write expiring) we can significantly reduce the
writes on the parity device. This assumes that we are ok to lose a
movie or two in case the parity disk is not totally up to date but are
more interested in device longevity.
>
> Sounds like multi-parity RAID6 with no parity rotation and
> chunksize == devicesize
RAID6 would present us with a joint device and currently only allows
writes to that directly, yes? Any writes will be striped.
In any case would md raid allow the underlying device to be written to
directly? Also how would it know that the device has been written to
and hence parity has to be updated? What about the superblock which
the FS would not know about?
Also except for the delayed checksum writing part which would be
significant if one of the objectives is to reduce the amount of
writes. Can we delay that in the code currently for RAID6? I
understand the objective of RAID6 is to ensure data recovery and we
are looking at a compromise in this case.
If feasible, this can be an enhancement to MD RAID as well where N
devices are presented instead of a single joint device in case of
raid6 (maybe the multi part device can be individual disks?)
It will certainly solve my problem of where to store the metadata. I
was currently hoping to just store it as a configuration file to be
read by the initramfs since in this case worst case scenario the
checksum goes out of sync and is rebuilt from scratch.
>
> I wouldn't use device-mapper myself, but you are unlikely to get an entirely
> impartial opinion from me on that topic.
I haven't hacked around the kernel internals much so far so will have
to dig out that history. I will welcome any particular links/mail
threads I should look at for guidance (with both yours and opposing
points of view)
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: NeilBrown @ 2014-10-29 9:05 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: linux-raid
In-Reply-To: <CAK-d5dbdF160hoa1==jWxEQZRpwQ7Sa76=9MREmp2V6Y24U8Kw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3348 bytes --]
On Wed, 29 Oct 2014 12:45:34 +0530 Anshuman Aggarwal
<anshuman.aggarwal@gmail.com> wrote:
> I'm outlining below a proposal for a RAID device mapper virtual block
> device for the kernel which adds "split raid" functionality on an
> incremental batch basis for a home media server/archived content which
> is rarely accessed.
>
> Given a set of N+X block devices (of the same size but smallest common
> size wins)
>
> the SplitRAID device mapper device generates virtual devices which are
> passthrough for N devices and write a Batched/Delayed checksum into
> the X devices so as to allow offline recovery of block on the N
> devices in case of a single disk failure.
>
> Advantages over conventional RAID:
>
> - Disks can be spun down reducing wear and tear over MD RAID Levels
> (such as 1, 10, 5,6) in the case of rarely accessed archival content
>
> - Prevent catastrophic data loss for multiple device failure since
> each block device is independent and hence unlike MD RAID will only
> lose data incrementally.
>
> - Performance degradation for writes can be achieved by keeping the
> checksum update asynchronous and delaying the fsync to the checksum
> block device.
>
> In the event of improper shutdown the checksum may not have all the
> updated data but will be mostly up to date which is often acceptable
> for home media server requirements. A flag can be set in case the
> checksum block device was shutdown properly indicating that a full
> checksum rebuild is not required.
>
> Existing solutions considered:
>
> - SnapRAID (http://snapraid.sourceforge.net/) which is a snapshot
> based scheme (Its advantages are that its in user space and has cross
> platform support but has the huge disadvantage of every checksum being
> done from scratch slowing the system, causing immense wear and tear on
> every snapshot and also losing any information updates upto the
> snapshot point etc)
>
> I'd like to get opinions on the pros and cons of this proposal from
> more experienced people on the list to redirect suitably on the
> following questions:
>
> - Maybe this can already be done using the block devices available in
> the kernel?
>
> - If not, Device mapper the right API to use? (I think so)
>
> - What would be the best block devices code to look at to implement?
>
> Neil, would appreciate your weighing in on this.
Just to be sure I understand, you would have N + X devices. Each of the N
devices contains an independent filesystem and could be accessed directly if
needed. Each of the X devices contains some codes so that if at most X
devices in total died, you would still be able to recover all of the data.
If more than X devices failed, you would still get complete data from the
working devices.
Every update would only write to the particular N device on which it is
relevant, and all of the X devices. So N needs to be quite a bit bigger
than X for the spin-down to be really worth it.
Am I right so far?
For some reason the writes to X are delayed... I don't really understand
that part.
Sounds like multi-parity RAID6 with no parity rotation and
chunksize == devicesize
I wouldn't use device-mapper myself, but you are unlikely to get an entirely
impartial opinion from me on that topic.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-10-29 8:31 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-raid
In-Reply-To: <20141029123235.56910b68@natsu>
Actually I already use a combination of these solutions (MD raid,
multiple devices + LVM2 to join). Unfortunately, none of these
solutions address the following:
- Full data loss in case of disk failure beyond the raid level ( 2
disks in raid5, 3 disks in raid6). This solution allows for single
disk data loss
- Continous read/write to all disks causing wear and tear reducing
life and increasing end user cost
mhddfs (or something like it) will probably be used on top of the N
devices in this proposal to join but that is upto the requirement of
the user.
On 29 October 2014 13:02, Roman Mamedov <rm@romanrm.net> wrote:
> On Wed, 29 Oct 2014 12:45:34 +0530
> Anshuman Aggarwal <anshuman.aggarwal@gmail.com> wrote:
>
>> I'm outlining below a proposal for a RAID device mapper virtual block
>> device for the kernel which adds "split raid" functionality on an
>> incremental batch basis for a home media server/archived content which
>> is rarely accessed.
>
>> Existing solutions considered:
>
> Some of the already-available "home media server" setup schemes you did not
> mention:
>
> http://linuxconfig.org/prouhd-raid-for-the-end-user
> a smart way of managing MD RAID given multiple devices of various sizes;
>
> http://louwrentius.com/building-a-raid-6-array-of-mixed-drives.html
> what to do with a set of mixed-size drives, in simpler terms;
>
> https://romanrm.net/mhddfs
> File-level "concatenation" of disks, with smart distribution of new files;
>
> --
> With respect,
> Roman
^ permalink raw reply
* Re: Split RAID: Proposal for archival RAID using incremental batch checksum
From: Roman Mamedov @ 2014-10-29 7:32 UTC (permalink / raw)
To: Anshuman Aggarwal; +Cc: linux-raid
In-Reply-To: <CAK-d5dbdF160hoa1==jWxEQZRpwQ7Sa76=9MREmp2V6Y24U8Kw@mail.gmail.com>
On Wed, 29 Oct 2014 12:45:34 +0530
Anshuman Aggarwal <anshuman.aggarwal@gmail.com> wrote:
> I'm outlining below a proposal for a RAID device mapper virtual block
> device for the kernel which adds "split raid" functionality on an
> incremental batch basis for a home media server/archived content which
> is rarely accessed.
> Existing solutions considered:
Some of the already-available "home media server" setup schemes you did not
mention:
http://linuxconfig.org/prouhd-raid-for-the-end-user
a smart way of managing MD RAID given multiple devices of various sizes;
http://louwrentius.com/building-a-raid-6-array-of-mixed-drives.html
what to do with a set of mixed-size drives, in simpler terms;
https://romanrm.net/mhddfs
File-level "concatenation" of disks, with smart distribution of new files;
--
With respect,
Roman
^ permalink raw reply
* Split RAID: Proposal for archival RAID using incremental batch checksum
From: Anshuman Aggarwal @ 2014-10-29 7:15 UTC (permalink / raw)
To: linux-raid
I'm outlining below a proposal for a RAID device mapper virtual block
device for the kernel which adds "split raid" functionality on an
incremental batch basis for a home media server/archived content which
is rarely accessed.
Given a set of N+X block devices (of the same size but smallest common
size wins)
the SplitRAID device mapper device generates virtual devices which are
passthrough for N devices and write a Batched/Delayed checksum into
the X devices so as to allow offline recovery of block on the N
devices in case of a single disk failure.
Advantages over conventional RAID:
- Disks can be spun down reducing wear and tear over MD RAID Levels
(such as 1, 10, 5,6) in the case of rarely accessed archival content
- Prevent catastrophic data loss for multiple device failure since
each block device is independent and hence unlike MD RAID will only
lose data incrementally.
- Performance degradation for writes can be achieved by keeping the
checksum update asynchronous and delaying the fsync to the checksum
block device.
In the event of improper shutdown the checksum may not have all the
updated data but will be mostly up to date which is often acceptable
for home media server requirements. A flag can be set in case the
checksum block device was shutdown properly indicating that a full
checksum rebuild is not required.
Existing solutions considered:
- SnapRAID (http://snapraid.sourceforge.net/) which is a snapshot
based scheme (Its advantages are that its in user space and has cross
platform support but has the huge disadvantage of every checksum being
done from scratch slowing the system, causing immense wear and tear on
every snapshot and also losing any information updates upto the
snapshot point etc)
I'd like to get opinions on the pros and cons of this proposal from
more experienced people on the list to redirect suitably on the
following questions:
- Maybe this can already be done using the block devices available in
the kernel?
- If not, Device mapper the right API to use? (I think so)
- What would be the best block devices code to look at to implement?
Neil, would appreciate your weighing in on this.
Regards,
Anshuman Aggarwal
^ permalink raw reply
* Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.
From: NeilBrown @ 2014-10-29 2:57 UTC (permalink / raw)
To: Jason Keltz; +Cc: linux-raid
In-Reply-To: <5450521F.8060309@cse.yorku.ca>
[-- Attachment #1: Type: text/plain, Size: 5940 bytes --]
On Tue, 28 Oct 2014 22:34:07 -0400 Jason Keltz <jas@cse.yorku.ca> wrote:
> On 28/10/2014 6:38 PM, NeilBrown wrote:
> > On Mon, 20 Oct 2014 17:07:38 -0400 Jason Keltz<jas@cse.yorku.ca> wrote:
> >
> >> On 10/20/2014 12:19 PM, Jason Keltz wrote:
> >>> Hi.
> >>>
> >>> I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system.
> >>> I've experimented with setting "speed_limit_min" and "speed_limit_max"
> >>> kernel variables so that I get the best balance of performance during
> >>> a RAID rebuild of one of the RAID1 pairs. If, for example, I set
> >>> speed_limit_min AND speed_limit_max to 80000 then fail a disk when
> >>> there is no other disk activity, then I do get a rebuild rate of
> >>> around 80 MB/s. However, if I then start up a write intensive
> >>> operation on the MD array (eg. a dd, or a mkfs on an LVM logical
> >>> volume that is created on that MD), then, my write operation seems to
> >>> get "full power", and my rebuild drops to around 25 MB/s. This means
> >>> that the rebuild of my RAID10 disk is going to take a huge amount of
> >>> time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to
> >>> the same value, am I not guaranteeing the rebuild speed? Is this a bug
> >>> that I should be reporting to Red Hat, or a "feature"?
> >>>
> >>> Thanks in advance for any help that you can provide...
> >>>
> >>> Jason.
> >> I would like to add that I downloaded the latest version of Ubuntu, and
> >> am running it on the same server with the same MD.
> >> When I set speed_limit_min and speed_limit_max to 80000, I was able to
> >> start two large dds on the md array, and the rebuild stuck at around 71
> >> MB/s, which is close enough. This leads me to believe that the problem
> >> above is probably a RHEL6 issue. However, after I stopped the two dd
> >> operations, and raised both speed_limit_min and speed_limit_max to
> >> 120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes
> >> .. now it seems to be at 100 MB/s... but doesn't seem to get any higher
> >> (even though I had 120 MB/s and above on the RHEL system without any
> >> load)... Hmm.
> >>
> > md certainly cannot "guarantee" any speed - it can only deliver what the
> > underlying devices deliver.
> > I know the kernels logs say something about a "guarantee". That was added
> > before my time and I haven't had occasion to remove it.
> >
> > md will normally just try to recover as fast as it can unless that exceeds
> > one of the limits - then it will back-off.
> > What speed it actually achieved depends on other load and the behaviour of
> > the IO scheduler.
> >
> > "RHEL6" and "Ubuntu" don't mean a lot to me. Specific kernel version might,
> > though in the case of Redhat I know that backport lots of stuff so even the
> > kernel version isn't very helpful. I'm must prefer having report against
> > mainline kernels.
> >
> > Rotating drives do get lower transfer speeds at higher addresses. That might
> > explain the 120 / 100 difference.
> Hi Neil,
> Thanks very much for your response.
> I must say that I'm a little puzzled though. I'm coming from using a
> 3Ware hardware RAID controller where I could configure how much of the
> disk bandwidth is to be used for a rebuild versus I/O. From what I
> understand, you're saying that MD can only use the disk bandwidth
> available to it. It seems that it doesn't take any priority in the I/O
> chain. It will only attempt to use no less than min bandwidth, and no
> more than max bandwidth for the rebuild, but if you're on a busy system,
> and other system I/O needs that disk bandwidth, then there's nothing it
> can do about it. I guess I just don't understand why. Why can't md be
> given a priority in the kernel to allow the admin to decide how much
> bandwidth goes to system I/O versus rebuild I/O. Even in a busy system,
> I still want to allocate at least some minimum bandwidth to MD. In
> fact, in the event of a disk failure, I want to have a whole lot of the
> disk bandwidth dedicated to MD. It's something about short term pain
> for long term gain? I'd rather not have the users suffer at all, but if
> they do have to suffer, I'd rather them suffer for a few hours, knowing
> that after that, the RAID system is in a perfectly good state with no
> bad disks as opposed to letting a bad disk resync take days because the
> system is really busy... days during which another failure might occur!
>
> Jason.
It isn't so much "that MD can only use..." but rather "that MD does only
use ...".
This is how the code has "always" worked and no-one has ever bothered to
change it, or to ask for it to be changed (that I recall).
There are difficulties in guaranteeing a minimum when the array uses
partitions from devices on which other partitions are used for other things.
In that case I don't think it is practical to make guarantees, but that
needn't stop us making guarantees when we can I guess.
If the configured bandwidth exceeded the physically available bandwidth I
don't think we would want to exclude non-resync IO completely, so the
guaranty would have to be:
N MB/sec or M% of available, whichever is less
We could even implement the different approach in a back-compatible way.
Introduce a new setting "max_sync_percent". By default that is unset and the
current algorithm applies.
If it is set to something below 100, non-resync IO is throttled to
an appropriate fraction of the actual resync throughput whenever that is
below sync_speed_min.
Or something like that.
Some care would be needed in comparing throughput and sync throughput is
measured per-device, while non-resync throughput might be measured per-array.
Maybe the throttling would happen per-device??
All we need now is for someone to firm up the design and then write the code.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: question about MD raid rebuild performance degradation even with speed_limit_min/speed_limit_max set.
From: Jason Keltz @ 2014-10-29 2:34 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141029093822.79242658@notabene.brown>
On 28/10/2014 6:38 PM, NeilBrown wrote:
> On Mon, 20 Oct 2014 17:07:38 -0400 Jason Keltz<jas@cse.yorku.ca> wrote:
>
>> On 10/20/2014 12:19 PM, Jason Keltz wrote:
>>> Hi.
>>>
>>> I'm creating a 22 x 2 TB SATA disk MD RAID10 on a new RHEL6 system.
>>> I've experimented with setting "speed_limit_min" and "speed_limit_max"
>>> kernel variables so that I get the best balance of performance during
>>> a RAID rebuild of one of the RAID1 pairs. If, for example, I set
>>> speed_limit_min AND speed_limit_max to 80000 then fail a disk when
>>> there is no other disk activity, then I do get a rebuild rate of
>>> around 80 MB/s. However, if I then start up a write intensive
>>> operation on the MD array (eg. a dd, or a mkfs on an LVM logical
>>> volume that is created on that MD), then, my write operation seems to
>>> get "full power", and my rebuild drops to around 25 MB/s. This means
>>> that the rebuild of my RAID10 disk is going to take a huge amount of
>>> time (>12 hours!!!). When I set speed_limit_min and speed_limit_max to
>>> the same value, am I not guaranteeing the rebuild speed? Is this a bug
>>> that I should be reporting to Red Hat, or a "feature"?
>>>
>>> Thanks in advance for any help that you can provide...
>>>
>>> Jason.
>> I would like to add that I downloaded the latest version of Ubuntu, and
>> am running it on the same server with the same MD.
>> When I set speed_limit_min and speed_limit_max to 80000, I was able to
>> start two large dds on the md array, and the rebuild stuck at around 71
>> MB/s, which is close enough. This leads me to believe that the problem
>> above is probably a RHEL6 issue. However, after I stopped the two dd
>> operations, and raised both speed_limit_min and speed_limit_max to
>> 120000, the rebuild stayed between 71-73 Mb/s for more than 10 minutes
>> .. now it seems to be at 100 MB/s... but doesn't seem to get any higher
>> (even though I had 120 MB/s and above on the RHEL system without any
>> load)... Hmm.
>>
> md certainly cannot "guarantee" any speed - it can only deliver what the
> underlying devices deliver.
> I know the kernels logs say something about a "guarantee". That was added
> before my time and I haven't had occasion to remove it.
>
> md will normally just try to recover as fast as it can unless that exceeds
> one of the limits - then it will back-off.
> What speed it actually achieved depends on other load and the behaviour of
> the IO scheduler.
>
> "RHEL6" and "Ubuntu" don't mean a lot to me. Specific kernel version might,
> though in the case of Redhat I know that backport lots of stuff so even the
> kernel version isn't very helpful. I'm must prefer having report against
> mainline kernels.
>
> Rotating drives do get lower transfer speeds at higher addresses. That might
> explain the 120 / 100 difference.
Hi Neil,
Thanks very much for your response.
I must say that I'm a little puzzled though. I'm coming from using a
3Ware hardware RAID controller where I could configure how much of the
disk bandwidth is to be used for a rebuild versus I/O. From what I
understand, you're saying that MD can only use the disk bandwidth
available to it. It seems that it doesn't take any priority in the I/O
chain. It will only attempt to use no less than min bandwidth, and no
more than max bandwidth for the rebuild, but if you're on a busy system,
and other system I/O needs that disk bandwidth, then there's nothing it
can do about it. I guess I just don't understand why. Why can't md be
given a priority in the kernel to allow the admin to decide how much
bandwidth goes to system I/O versus rebuild I/O. Even in a busy system,
I still want to allocate at least some minimum bandwidth to MD. In
fact, in the event of a disk failure, I want to have a whole lot of the
disk bandwidth dedicated to MD. It's something about short term pain
for long term gain? I'd rather not have the users suffer at all, but if
they do have to suffer, I'd rather them suffer for a few hours, knowing
that after that, the RAID system is in a perfectly good state with no
bad disks as opposed to letting a bad disk resync take days because the
system is really busy... days during which another failure might occur!
Jason.
^ permalink raw reply
* Re: [PATCH] Grow: fix resize of array component size to > 32bits
From: NeilBrown @ 2014-10-29 0:05 UTC (permalink / raw)
To: Justin Maggard; +Cc: linux-raid
In-Reply-To: <CAKgsxVQJp7C0_y9ZENG_ZpaSk9Zi0Aa1cf0-qwRFcHx1HB5v0A@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3219 bytes --]
On Tue, 28 Oct 2014 16:47:06 -0700 Justin Maggard <jmaggard10@gmail.com>
wrote:
> On Tue, Oct 28, 2014 at 3:19 PM, NeilBrown <neilb@suse.de> wrote:
> > On Fri, 24 Oct 2014 17:55:02 -0700 Justin Maggard <jmaggard10@gmail.com>
> > wrote:
> >
> >> If the request --size to --grow an array to is larger
> >> than 32bits, then mdadm may make the wrong choice and
> >> use ioctl instead of setting component_size via sysfs
> >> and the change is ignored.
> >
> > Can you explain exactly why the current code is not sufficient? When does
> > it fail?
> > If you include the explanation in a re-submission of the patch, and I am
> > convinced, then I will gladly apply your patch.
> >
>
> To be honest, I can't figure out how the current comparison would have
> ever worked at all. To illustrate, here's a simple test program:
> -----
> $ ./test 0xfffffffff
> cast comparison did not detect truncation
> bit comparison detected truncation
>
> $ cat test.c
> #include <stdio.h>
> #include <stdlib.h>
> #include <stdint.h>
>
> int main(int argc, char **argv)
> {
> unsigned long long ssize = strtoull(argv[1], NULL, 0);
> int asize;
>
> asize = ssize;
> if (asize != (signed)ssize)
> printf("cast comparison detected truncation\n");
> else
> printf("cast comparison did not detect truncation \n");
> if (ssize & ~INT32_MAX)
> printf("bit comparison detected truncation\n");
> else
> printf("bit comparison did not detect truncation \n");
>
> return 0;
> }
> -----
>
> I plugged lots of numbers in there, and I was never able to get the
> current cast comparison to see a difference.
>
> I ran into the issue by trying to grow the component size of a RAID
> array from 1TB to 3TB, and it wouldn't work if I specified the size;
> only using "max" worked.
>
> I'm happy to re-submit if you'd like; I just thought it was a pretty
> straightforward bug. I guess what I'm saying is, I don't understand
> why there *should* be a difference between assigning a unsigned long
> long to an int variable, and casting that unsigned long long to a
> signed type. But
>
> -Justin
>
> > Thanks,
> > NeilBrown
> >
> >>
> >> Instead of using casts to check for a 32-bit overflow,
> >> just check for set bits outside of INT32_MAX.
> >> ---
> >> Grow.c | 2 +-
> >> 1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/Grow.c b/Grow.c
> >> index a9c8589..a614102 100644
> >> --- a/Grow.c
> >> +++ b/Grow.c
> >> @@ -1818,7 +1818,7 @@ int Grow_reshape(char *devname, int fd,
> >> if (s->size == MAX_SIZE)
> >> s->size = 0;
> >> array.size = s->size;
> >> - if (array.size != (signed)s->size) {
> >> + if (s->size & ~INT32_MAX) {
> >> /* got truncated to 32bit, write to
> >> * component_size instead
> >> */
> >
Thanks.
Looks like I broke it in July.
http://git.neil.brown.name/?p=mdadm.git;a=commitdiff;h=4e9a3dd16d656b269f5602624ac4f7109a571368
I probably should have made it
if (s->size != (signed long long)s->size)
I've applied your patch.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply
* Re: [PATCH] Grow: fix resize of array component size to > 32bits
From: Justin Maggard @ 2014-10-28 23:47 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20141029091957.1c4cdeae@notabene.brown>
On Tue, Oct 28, 2014 at 3:19 PM, NeilBrown <neilb@suse.de> wrote:
> On Fri, 24 Oct 2014 17:55:02 -0700 Justin Maggard <jmaggard10@gmail.com>
> wrote:
>
>> If the request --size to --grow an array to is larger
>> than 32bits, then mdadm may make the wrong choice and
>> use ioctl instead of setting component_size via sysfs
>> and the change is ignored.
>
> Can you explain exactly why the current code is not sufficient? When does
> it fail?
> If you include the explanation in a re-submission of the patch, and I am
> convinced, then I will gladly apply your patch.
>
To be honest, I can't figure out how the current comparison would have
ever worked at all. To illustrate, here's a simple test program:
-----
$ ./test 0xfffffffff
cast comparison did not detect truncation
bit comparison detected truncation
$ cat test.c
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
int main(int argc, char **argv)
{
unsigned long long ssize = strtoull(argv[1], NULL, 0);
int asize;
asize = ssize;
if (asize != (signed)ssize)
printf("cast comparison detected truncation\n");
else
printf("cast comparison did not detect truncation \n");
if (ssize & ~INT32_MAX)
printf("bit comparison detected truncation\n");
else
printf("bit comparison did not detect truncation \n");
return 0;
}
-----
I plugged lots of numbers in there, and I was never able to get the
current cast comparison to see a difference.
I ran into the issue by trying to grow the component size of a RAID
array from 1TB to 3TB, and it wouldn't work if I specified the size;
only using "max" worked.
I'm happy to re-submit if you'd like; I just thought it was a pretty
straightforward bug. I guess what I'm saying is, I don't understand
why there *should* be a difference between assigning a unsigned long
long to an int variable, and casting that unsigned long long to a
signed type. But
-Justin
> Thanks,
> NeilBrown
>
>>
>> Instead of using casts to check for a 32-bit overflow,
>> just check for set bits outside of INT32_MAX.
>> ---
>> Grow.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/Grow.c b/Grow.c
>> index a9c8589..a614102 100644
>> --- a/Grow.c
>> +++ b/Grow.c
>> @@ -1818,7 +1818,7 @@ int Grow_reshape(char *devname, int fd,
>> if (s->size == MAX_SIZE)
>> s->size = 0;
>> array.size = s->size;
>> - if (array.size != (signed)s->size) {
>> + if (s->size & ~INT32_MAX) {
>> /* got truncated to 32bit, write to
>> * component_size instead
>> */
>
^ permalink raw reply
* Re: "bitmap file is out of date, doing full recovery"
From: NeilBrown @ 2014-10-28 23:19 UTC (permalink / raw)
To: Alexander Lyakas; +Cc: linux-raid
In-Reply-To: <CAGRgLy69gC2n0oAP8CRNBzTb+tBnwmwoJOCdKb+aRMO=Oeqf3Q@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 3067 bytes --]
On Thu, 23 Oct 2014 19:04:48 +0300 Alexander Lyakas <alex.bolshoy@gmail.com>
wrote:
> Hi Neil,
> I found at least one way of this happening. The problem is that in
> md_update_sb() we allow to decrease the event count:
>
> /* If this is just a dirty<->clean transition, and the array is clean
> * and 'events' is odd, we can roll back to the previous clean state */
> if (nospares
> && (mddev->in_sync && mddev->recovery_cp == MaxSector)
> && mddev->can_decrease_events
> && mddev->events != 1) {
> mddev->events--;
> mddev->can_decrease_events = 0;
>
> Then we call bitmap_update_sb(). If we crash after we update (the
> first or all of) bitmap superblocks, then after reboot, we will see
> that bitmap event count is less than MD superblock event count. Then
> we decide to do full resync.
>
> This can be easily reproduced by hacking bitmap_update_sb() to call
> BUG(), after it calls write_page() in case event count was decreased.
>
> Why we are decreasing the event count??? Can we always increase it?
> u64 is a lot to increase...
The reason for decreasing the event count is so that we don't need to update
the event count on spares - they can be left spun down.
We for simple clean/dirty transitions with increment for clean->dirty and
decrement for dirty->clean. But we should only use this optimisation when
everything is simple.
We really shouldn't do this when the array is degraded.
Do this fix your problem?
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 2c73fcb82593..98fd97b10e13 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2244,6 +2244,7 @@ repeat:
* and 'events' is odd, we can roll back to the previous clean state */
if (nospares
&& (mddev->in_sync && mddev->recovery_cp == MaxSector)
+ && mddev->degraded == 0
&& mddev->can_decrease_events
&& mddev->events != 1) {
mddev->events--;
>
> Some other doubt that I have is that bitmap_unplug() and
> bitmap_daemon_work() call write_page() on page index=0. This page
> contains both the superblock and also some dirty bits (could not we
> waste 4KB on bitmap superblock???). I am not sure, but I wonder
> whether this call can race with md_update_sb (which explicitly calls
> bitmap_update_sb), and somehow write the outdated superblock, after
> bitmap_update_sb has completed writing it.
>
storage.sb_page is exactly the same as storage.filemap[0]
So once an update has happened, the "outdated superblock" doesn't exist
anywhere to be written out from.
> Yet another suspect is when loading the bitmap we basically load it
> from the first up-to-date drive. Maybe we should have scanned all the
> bitmap superblocks, and selected one that has the higher event count
> (although as we saw "higher" does not necessarily mean "more
> up-to-date").
>
> Anyways, back to decrementing the event count. Do you see any issue
> with not doing this and always incrementing?
>
> Thanks,
> Alex.
>
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply related
* Re: Very small bug on assemble --force
From: Ethan Wilson @ 2014-10-28 22:40 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20141029075050.605a11a9@notabene.brown>
On 28/10/2014 21:50, NeilBrown wrote:
> On Tue, 28 Oct 2014 15:27:02 +0100 Ethan Wilson <ethan.wilson@shiftmail.org>
> wrote:
>
>> Very small bug report:
> Thanks for the bug report.
> However, please always quote version of mdadm and version of Linux.
>
> I think this might have been fixed already, in mdadm 3.3.1.
>
> NeilBrown
>
Whops
kernel 3.14.22
mdadm - v3.2.5 - 18th May 2012
mdadm is old indeed so you are probably right.
It is difficult for me to test on the new version because I don't have
degraded arrays anymore right now :-/
Sorry
EW
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox