* Re: Installing Linux directly onto RAID6 Array...........
From: Another Sillyname @ 2015-05-12 14:05 UTC (permalink / raw)
To: Wilson, Jonathan; +Cc: linux-raid
In-Reply-To: <BLU436-SMTP20942BFA85744B69593179D98DA0@phx.gbl>
It's a new z97 board so EFI rules apply.
I have some spare 3TB drives sitting around that I may try to practice
on.......see what will and won't work.
Still open to all suggestions and especially from anyone who's
actually done this.
On 12 May 2015 at 14:27, Wilson, Jonathan <piercing_male@hotmail.com> wrote:
> On Tue, 2015-05-12 at 11:08 +0100, Another Sillyname wrote:
>> I've tried to do some research on this but the information out there
>> seems a bit contradictory (mainly because some is so old).
>>
>> I want to install Fedora directly onto a RAID array (no separate boot disk).
>>
>> My plan is to 'pre configure' the 6 drives as a clean RAID6 array,
>> effectively sd[a-f] without partitions and then attempt to install
>> Fedora 21, from sources it looks like Grub2 should recognise the array
>> and then allow the Kernel to boot thereby 'enabling' the array to
>> become visible and active.
>>
>> However I have not been able to find an actual example of someone
>> trying this......thoughts?
>>
>> The reason to do this is I'm intending to use a Mini ITX board with 6
>> sata ports and want to use 8TB drives in Raid6 to give me a very high
>> density data resilient small form factor storage box.
>
> Grub2 can handle booting into raid6, but some while ago the support was
> sketchy if the array was degraded; this may have improved.
>
> Your problem would be that it would require a biosboot (GPT, type:EF02,
> size:1 MiB) partition to hold part of the loader as it will not fit
> entirely into the "mbr."
>
> A second problem might be that while drives larger than 2 GB can be used
> on (most?) older boards they might not be able to be accessed/read
> correctly/bootable by older non-EFI bios's. My old MB was quite happy to
> boot from a 1 TB drive and linux could see and use my 3TB drives but the
> bios only saw the 3TB drives as 700MB (approx, I recall)
>
> If you are using an EFI system in EFI mode, you will need an EFI
> partition(s) somewhere. On my new system I have all 5 of my 3TB drives
> contain an EFI-dos partition of about 200-500MB and the the rest as one
> large partition for the raid6 containing everything else (/, /home,
> etc.). The only pain in the neck is remembering to copy everything from
> the "live EFI" (default loaded as "/boot") to the backup EFI's when ever
> I change/update stuff in it.
>>
>> Ideas/Suggestions?
>>
>> Thanks
>>
>> Tony
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
^ permalink raw reply
* Re: Installing Linux directly onto RAID6 Array...........
From: Roman Mamedov @ 2015-05-12 13:42 UTC (permalink / raw)
To: Another Sillyname; +Cc: linux-raid
In-Reply-To: <CAOS+5GE1DTzuARuPPRMvhzB46Ka5snF+MZqyM_tbLH7uY9_VuQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 530 bytes --]
On Tue, 12 May 2015 14:12:10 +0100
Another Sillyname <anothersname@googlemail.com> wrote:
> there's an interesting idea in there of effectively installing the
> bootloader onto every drive in the array......
Oh, sure you can install GRUB2 to every drive and also create a 6-member RAID1
with metadata version 0.90 from first partition of each drive (256 MB is
enough) for /boot. That's another way of "booting from the same set drives as
the main storage". I do that on another machine :)
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: Installing Linux directly onto RAID6 Array...........
From: Wilson, Jonathan @ 2015-05-12 13:27 UTC (permalink / raw)
To: Another Sillyname; +Cc: linux-raid
In-Reply-To: <CAOS+5GFwGJouUfZ_RWhtRVXw7wnoP1Ry4doHTJmkruFoccmqRg@mail.gmail.com>
On Tue, 2015-05-12 at 11:08 +0100, Another Sillyname wrote:
> I've tried to do some research on this but the information out there
> seems a bit contradictory (mainly because some is so old).
>
> I want to install Fedora directly onto a RAID array (no separate boot disk).
>
> My plan is to 'pre configure' the 6 drives as a clean RAID6 array,
> effectively sd[a-f] without partitions and then attempt to install
> Fedora 21, from sources it looks like Grub2 should recognise the array
> and then allow the Kernel to boot thereby 'enabling' the array to
> become visible and active.
>
> However I have not been able to find an actual example of someone
> trying this......thoughts?
>
> The reason to do this is I'm intending to use a Mini ITX board with 6
> sata ports and want to use 8TB drives in Raid6 to give me a very high
> density data resilient small form factor storage box.
Grub2 can handle booting into raid6, but some while ago the support was
sketchy if the array was degraded; this may have improved.
Your problem would be that it would require a biosboot (GPT, type:EF02,
size:1 MiB) partition to hold part of the loader as it will not fit
entirely into the "mbr."
A second problem might be that while drives larger than 2 GB can be used
on (most?) older boards they might not be able to be accessed/read
correctly/bootable by older non-EFI bios's. My old MB was quite happy to
boot from a 1 TB drive and linux could see and use my 3TB drives but the
bios only saw the 3TB drives as 700MB (approx, I recall)
If you are using an EFI system in EFI mode, you will need an EFI
partition(s) somewhere. On my new system I have all 5 of my 3TB drives
contain an EFI-dos partition of about 200-500MB and the the rest as one
large partition for the raid6 containing everything else (/, /home,
etc.). The only pain in the neck is remembering to copy everything from
the "live EFI" (default loaded as "/boot") to the backup EFI's when ever
I change/update stuff in it.
>
> Ideas/Suggestions?
>
> Thanks
>
> Tony
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Fwd: Installing Linux directly onto RAID6 Array...........
From: Another Sillyname @ 2015-05-12 13:12 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAOS+5GHhUoYxTTYOWU7cdN6GSdffSMGrhWHU5ZtWEjc4jEm3eg@mail.gmail.com>
---------- Forwarded message ----------
From: Another Sillyname <anothersname@googlemail.com>
Date: 12 May 2015 at 14:11
Subject: Re: Installing Linux directly onto RAID6 Array...........
To: Roman Mamedov <rm@romanrm.net>
Guys
Don't want to use USB as the machine may sometimes be in a non secure
environ, want to boot direct from Raid.
Reading more think there may be a way....
https://help.ubuntu.com/community/Grub2/Installing
there's an interesting idea in there of effectively installing the
bootloader onto every drive in the array......
On 12 May 2015 at 13:31, Roman Mamedov <rm@romanrm.net> wrote:
> On Tue, 12 May 2015 08:20:42 -0400
> Phil Turmel <philip@turmel.org> wrote:
>
>> > Doing something like that... and using network boot to solve those issues
>> >
>> > Last i tried grub2 could not handle a raid6 directly (could be changed now)
>>
>> You could also boot from a USB device, possibly one that memory caches
>> its root filesystem (like System Rescue CD).
>
> Just place GRUB and the /boot partition with kernel and initrd on a USB stick,
> the rest (root FS) can be on RAID6. That's what I do on one machine.
>
> --
> With respect,
> Roman
^ permalink raw reply
* Re: Possible RAID6 regression with ASYNC_TX_DMA enabled in 4.1
From: Maxime Ripard @ 2015-05-12 12:55 UTC (permalink / raw)
To: Shaohua Li
Cc: Neil Brown, linux-raid, linux-kernel, Lior Amsalem,
Thomas Petazzoni, Gregory Clement, Boris Brezillon
In-Reply-To: <20150511062638.GA63893@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 1902 bytes --]
Hi Shaohua,
On Sun, May 10, 2015 at 11:26:38PM -0700, Shaohua Li wrote:
> On Thu, May 07, 2015 at 02:57:02PM +0200, Maxime Ripard wrote:
> > Hi,
> >
> > I'm currently trying to add support for the PQ operations on the
> > marvell XOR engine, in dmaengine, obviously to be able to use async_tx
> > to offload these operations.
> >
> > I'm testing these patches with a RAID6 array with 4 disks.
> >
> > However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full
> > stripe write", every write to that array fails with the following
> > stacktrace.
> >
> > http://code.bulix.org/eh8iew-88342?raw
> >
> > It seems to be generated by that warning here:
> >
> > http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173
> >
> > And indeed, if we dump the status of depend_tx here, it's already been
> > acked.
> >
> > That doesn't happen if ASYNC_TX_DMA is disabled, hence using the
> > software version of it, instead of relying on our XOR engine. It
> > doesn't happen on any commit prior to the one mentionned above, with
> > the exact same changes applied. These changes are meant to be
> > contributed, so I can definitely push them somewhere if needed.
> >
> > I don't really know where to look for though, the change that is
> > causing this is probably the change in ops_run_reconstruct6, but I'm
> > not sure that this partial revert alone would work with regard to the
> > rest of the patch.
>
> I don't have a machine with dmaengine, it's likely there is error in this side.
> Could you please make stripe_can_batch() returns false always and check if the
> error disappear? This should narrow down if it's related to batch issue.
The error indeed disappears if stripe_can_batch always returns false.
Maxime
--
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
* Re: Installing Linux directly onto RAID6 Array...........
From: Roman Mamedov @ 2015-05-12 12:31 UTC (permalink / raw)
To: Phil Turmel; +Cc: Rudy Zijlstra, Another Sillyname, linux-raid
In-Reply-To: <5551F01A.4050705@turmel.org>
[-- Attachment #1: Type: text/plain, Size: 551 bytes --]
On Tue, 12 May 2015 08:20:42 -0400
Phil Turmel <philip@turmel.org> wrote:
> > Doing something like that... and using network boot to solve those issues
> >
> > Last i tried grub2 could not handle a raid6 directly (could be changed now)
>
> You could also boot from a USB device, possibly one that memory caches
> its root filesystem (like System Rescue CD).
Just place GRUB and the /boot partition with kernel and initrd on a USB stick,
the rest (root FS) can be on RAID6. That's what I do on one machine.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: Installing Linux directly onto RAID6 Array...........
From: Phil Turmel @ 2015-05-12 12:20 UTC (permalink / raw)
To: Rudy Zijlstra, Another Sillyname, linux-raid
In-Reply-To: <5551D3F2.8020007@grumpydevil.homelinux.org>
On 05/12/2015 06:20 AM, Rudy Zijlstra wrote:
> Hi
>
> Another Sillyname schreef op 12-05-15 om 12:08:
>> I've tried to do some research on this but the information out there
>> seems a bit contradictory (mainly because some is so old).
>>
>> I want to install Fedora directly onto a RAID array (no separate boot
>> disk).
>
>
> Doing something like that... and using network boot to solve those issues
>
> Last i tried grub2 could not handle a raid6 directly (could be changed now)
You could also boot from a USB device, possibly one that memory caches
its root filesystem (like System Rescue CD).
Phil
^ permalink raw reply
* Re: Possible RAID6 regression with ASYNC_TX_DMA enabled in 4.1
From: Shaohua Li @ 2015-05-12 10:59 UTC (permalink / raw)
To: Maxime Ripard
Cc: Neil Brown, linux-raid, linux-kernel, Lior Amsalem,
Thomas Petazzoni, Gregory Clement, Boris Brezillon
In-Reply-To: <20150512125546.GJ10961@lukather>
On Tue, May 12, 2015 at 02:55:46PM +0200, Maxime Ripard wrote:
> Hi Shaohua,
>
> On Sun, May 10, 2015 at 11:26:38PM -0700, Shaohua Li wrote:
> > On Thu, May 07, 2015 at 02:57:02PM +0200, Maxime Ripard wrote:
> > > Hi,
> > >
> > > I'm currently trying to add support for the PQ operations on the
> > > marvell XOR engine, in dmaengine, obviously to be able to use async_tx
> > > to offload these operations.
> > >
> > > I'm testing these patches with a RAID6 array with 4 disks.
> > >
> > > However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full
> > > stripe write", every write to that array fails with the following
> > > stacktrace.
> > >
> > > http://code.bulix.org/eh8iew-88342?raw
> > >
> > > It seems to be generated by that warning here:
> > >
> > > http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173
> > >
> > > And indeed, if we dump the status of depend_tx here, it's already been
> > > acked.
> > >
> > > That doesn't happen if ASYNC_TX_DMA is disabled, hence using the
> > > software version of it, instead of relying on our XOR engine. It
> > > doesn't happen on any commit prior to the one mentionned above, with
> > > the exact same changes applied. These changes are meant to be
> > > contributed, so I can definitely push them somewhere if needed.
> > >
> > > I don't really know where to look for though, the change that is
> > > causing this is probably the change in ops_run_reconstruct6, but I'm
> > > not sure that this partial revert alone would work with regard to the
> > > rest of the patch.
> >
> > I don't have a machine with dmaengine, it's likely there is error in this side.
> > Could you please make stripe_can_batch() returns false always and check if the
> > error disappear? This should narrow down if it's related to batch issue.
>
> The error indeed disappears if stripe_can_batch always returns false.
Does this fix it?
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 77dfd72..5e820fc 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -1825,7 +1825,7 @@ ops_run_reconstruct6(struct stripe_head *sh, struct raid5_percpu *percpu,
} else
init_async_submit(&submit, 0, tx, NULL, NULL,
to_addr_conv(sh, percpu, j));
- async_gen_syndrome(blocks, 0, count+2, STRIPE_SIZE, &submit);
+ tx = async_gen_syndrome(blocks, 0, count+2, STRIPE_SIZE, &submit);
if (!last_stripe) {
j++;
sh = list_first_entry(&sh->batch_list, struct stripe_head,
^ permalink raw reply related
* Re: Installing Linux directly onto RAID6 Array...........
From: Rudy Zijlstra @ 2015-05-12 10:20 UTC (permalink / raw)
To: Another Sillyname, linux-raid
In-Reply-To: <CAOS+5GFwGJouUfZ_RWhtRVXw7wnoP1Ry4doHTJmkruFoccmqRg@mail.gmail.com>
Hi
Another Sillyname schreef op 12-05-15 om 12:08:
> I've tried to do some research on this but the information out there
> seems a bit contradictory (mainly because some is so old).
>
> I want to install Fedora directly onto a RAID array (no separate boot disk).
Doing something like that... and using network boot to solve those issues
Last i tried grub2 could not handle a raid6 directly (could be changed now)
Cheers
Rudy
^ permalink raw reply
* Installing Linux directly onto RAID6 Array...........
From: Another Sillyname @ 2015-05-12 10:08 UTC (permalink / raw)
To: linux-raid
I've tried to do some research on this but the information out there
seems a bit contradictory (mainly because some is so old).
I want to install Fedora directly onto a RAID array (no separate boot disk).
My plan is to 'pre configure' the 6 drives as a clean RAID6 array,
effectively sd[a-f] without partitions and then attempt to install
Fedora 21, from sources it looks like Grub2 should recognise the array
and then allow the Kernel to boot thereby 'enabling' the array to
become visible and active.
However I have not been able to find an actual example of someone
trying this......thoughts?
The reason to do this is I'm intending to use a Mini ITX board with 6
sata ports and want to use 8TB drives in Raid6 to give me a very high
density data resilient small form factor storage box.
Ideas/Suggestions?
Thanks
Tony
^ permalink raw reply
* Re: [PATCH 0/5] a caching layer for raid 5/6
From: Christoph Hellwig @ 2015-05-12 7:18 UTC (permalink / raw)
To: Shaohua Li
Cc: Christoph Hellwig, linux-raid, Kernel-team, songliubraving,
dan.j.williams, neilb
In-Reply-To: <20150511160349.GA545551@devbig257.prn2.facebook.com>
On Mon, May 11, 2015 at 09:03:51AM -0700, Shaohua Li wrote:
> > - What is the reason for retry_bio_list? If a driver returns an
> > I/O error to the higher levels it already has retried and came
> > to the conclusion this is a permanent error.
>
> The retry_bio_list is to handle io to cache disk. If IO to cache disk
> has error, it's not a permanent error here. The cache disk is a cache,
> We can still dispatch the IO to its final destination, the raid disks.
How does this work in practice? We've filled our cache disk with
dirty data, and it now returns non-correctable write errors. At this
point we had claimed to caller that data is on stable disk, but our
cache disk is toast now. Is it really a good idea to now start a large
window where we do not actually have the cache data on stable storage
we can get back at but pretent business as usual?
IMHO the only sane way is to shut down the array when write to the cache
disk fail. Hopefully the disk will still allow reading from it. Note
that to be on the safe side you'll need a mirrored cache disk anyway.
^ permalink raw reply
* Re: Breaks LSI RAID on C600 chipset
From: David F. @ 2015-05-11 16:31 UTC (permalink / raw)
To: NeilBrown; +Cc: John Stoffel, linux-raid@vger.kernel.org
In-Reply-To: <CAGRSmLsxN_oGm9LQmDJSUVcTvVP7DzSAeWG+aaR+Td-OFc=c=A@mail.gmail.com>
any progress or more information needed on this?
On Mon, Apr 20, 2015 at 10:39 PM, David F. <df7729@gmail.com> wrote:
> If you need access to the system, a network kvm is available - or if
> you have a C600 based system, you should see the same problem there.
>
> On Fri, Apr 10, 2015 at 1:07 PM, David F. <df7729@gmail.com> wrote:
>> Okay, and to confirm, this is happening on other C600 based systems.
>> Boot to Linux with MDADM raid support and the raid is gone after
>> reboot.
>>
>> On Thu, Apr 9, 2015 at 5:50 PM, NeilBrown <neilb@suse.de> wrote:
>>> On Thu, 9 Apr 2015 14:13:05 -0700 "David F." <df7729@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I built a new system, installed new drives (no partitioning, just raw
>>>> new drives), configured RAID 1, boot to linux, created reports
>>>> attached. Rebooted and the system doesn't see any configured raid
>>>> drives (just the two raw drives).
>>>
>>> Thanks. I might take a look, but I would greatly prefer it if you kept the
>>> linux-raid list on the Cc.....
>>>
>>> NeilBrown
>>>
^ permalink raw reply
* Re: [PATCH 0/5] a caching layer for raid 5/6
From: Shaohua Li @ 2015-05-11 16:03 UTC (permalink / raw)
To: Christoph Hellwig
Cc: linux-raid, Kernel-team, songliubraving, dan.j.williams, neilb
In-Reply-To: <20150511122347.GA5082@infradead.org>
On Mon, May 11, 2015 at 05:23:47AM -0700, Christoph Hellwig wrote:
> Hi Shaohua,
>
> here are a couple of notes from reading through the code in a bit more
> detail:
>
Thanks for your time.
> Error retries:
> - What is the reason for retry_bio_list? If a driver returns an
> I/O error to the higher levels it already has retried and came
> to the conclusion this is a permanent error.
The retry_bio_list is to handle io to cache disk. If IO to cache disk
has error, it's not a permanent error here. The cache disk is a cache,
We can still dispatch the IO to its final destination, the raid disks.
> Flushes:
> - no need to allocate a task here
> - no real need to clone the bio either
clone the bio makes the IO retry easier.
> Tasks:
> - the completion argument passed to r5l_queue_bio is always the same,
> the code would be a lot simpler by removing this abstraction.
> - that would also allow allocating the task embedded in the range
> and cut down on memory allocations
Yep, my original idea is the stuff handling caching (r5c*) doesn't need
to know the detail of log device. I'll look this again to check if this
is a over-design.
> - we're not really manipulating the bio payload, so shouldn't a
> _fast cone be fine here?
> In fact why do we clone the bio at all?
The cache disk drive might manipulate the bio payload. The problem is
really why we clone the bio. You are right we don't need to clone the
bio at normal case. The exception is IO error. IO to the cache disk can
fail. In that case, we will try to skip cache disk and dispatch IO to
raid disks directly, which is the retry. Since cache disk driver might
already manipulate the bio, cloning a bio makes retry easier.
> - r5l_queue_task should probably be split into two helpers
> for data vs parity
> - r5l_queue_bio and r5c_copy_bio should probably use bvec iterators
> - r5l_run_task does very different things for data vs metadata,
> it's proably better it.
I'll check these.
>
> Allocations:
> - most metadata pages are allocated as highmem leading to
> constant kmap/kunmap. Maybe just allocate them as GFP_KERNEL
> to simplify things?
Sounds good.
> Misc:
> - where does the ioctl in r5l_ioctl anѕ associated functions come
> from? There are not ioctls handler here so the naming seems
> rather confusing.
Ah, it manages an io really, I'll change a name.
Thanks,
Shaohua
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] DDF: _write_super_to_disk: fix anchor header type
From: Martin Wilck @ 2015-05-11 14:09 UTC (permalink / raw)
To: neilb, linux-raid; +Cc: Martin Wilck
Since commit 30bee0201, the anchor is updated from the active
DDF header. This requires fixing the header type before the
anchor is written.
The LSI Software RAID code will reject DDF meta data with wrong
anchor type and will erase all meta data when it encounters
such a broken anchor. Thus starting Linux md once on a system
with LSI RAID BIOS may cause the meta data to get destroyed.
---
super-ddf.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/super-ddf.c b/super-ddf.c
index d0b2ac7..faaf0a7 100644
--- a/super-ddf.c
+++ b/super-ddf.c
@@ -3158,6 +3158,7 @@ static int _write_super_to_disk(struct ddf_super *ddf, struct dl *d)
memcpy(&ddf->primary, &ddf->anchor, 512);
memcpy(&ddf->secondary, &ddf->anchor, 512);
+ ddf->anchor.type = DDF_HEADER_ANCHOR;
ddf->anchor.openflag = 0xFF; /* 'open' means nothing */
ddf->anchor.seq = cpu_to_be32(0xFFFFFFFF); /* no sequencing in anchor */
ddf->anchor.crc = calc_crc(&ddf->anchor, 512);
--
2.1.0
^ permalink raw reply related
* Re: [PATCH 0/5] a caching layer for raid 5/6
From: Christoph Hellwig @ 2015-05-11 12:23 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Kernel-team, songliubraving, dan.j.williams, neilb
In-Reply-To: <cover.1430954848.git.shli@fb.com>
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1582 bytes --]
Hi Shaohua,
here are a couple of notes from reading through the code in a bit more
detail:
Error retries:
- What is the reason for retry_bio_list? If a driver returns an
I/O error to the higher levels it already has retried and came
to the conclusion this is a permanent error.
Flushes:
- no need to allocate a task here
- no real need to clone the bio either
Tasks:
- the completion argument passed to r5l_queue_bio is always the same,
the code would be a lot simpler by removing this abstraction.
- that would also allow allocating the task embedded in the range
and cut down on memory allocations
- we're not really manipulating the bio payload, so shouldn't a
_fast cone be fine here?
In fact why do we clone the bio at all?
- r5l_queue_task should probably be split into two helpers
for data vs parity
- r5l_queue_bio and r5c_copy_bio should probably use bvec iterators
- r5l_run_task does very different things for data vs metadata,
it's proably better it.
Allocations:
- most metadata pages are allocated as highmem leading to
constant kmap/kunmap. Maybe just allocate them as GFP_KERNEL
to simplify things?
Misc:
- where does the ioctl in r5l_ioctl anѕ associated functions come
from? There are not ioctls handler here so the naming seems
rather confusing.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Possible RAID6 regression with ASYNC_TX_DMA enabled in 4.1
From: Maxime Ripard @ 2015-05-11 9:13 UTC (permalink / raw)
To: Markus Stockhausen
Cc: Neil Brown, Shaohua Li, linux-raid@vger.kernel.org,
linux-kernel@vger.kernel.org, Lior Amsalem, Thomas Petazzoni,
Gregory Clement, Boris Brezillon
In-Reply-To: <12EF8D94C6F8734FB2FF37B9FBEDD1735FC9B959@EXCHANGE.collogia.de>
[-- Attachment #1: Type: text/plain, Size: 2666 bytes --]
Hi Markus,
On Thu, May 07, 2015 at 02:39:07PM +0000, Markus Stockhausen wrote:
> Hi Maxime,
>
> > Von: linux-raid-owner@vger.kernel.org [linux-raid-owner@vger.kernel.org]" im Auftrag von "Maxime Ripard [maxime.ripard@free-electrons.com]
> > Gesendet: Donnerstag, 7. Mai 2015 14:57
> > An: Neil Brown; Shaohua Li
> > Cc: linux-raid@vger.kernel.org; linux-kernel@vger.kernel.org; Lior Amsalem; Thomas Petazzoni; Gregory Clement; Boris Brezillon
> > Betreff: Possible RAID6 regression with ASYNC_TX_DMA enabled in 4.1
> >
> > Hi,
> >
> > I'm currently trying to add support for the PQ operations on the
> > marvell XOR engine, in dmaengine, obviously to be able to use async_tx
> > to offload these operations.
> >
> > I'm testing these patches with a RAID6 array with 4 disks.
> >
> > However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full
> > stripe write", every write to that array fails with the following
> > stacktrace.
> >
> > http://code.bulix.org/eh8iew-88342?raw
>
> I don't know if it might be related. I added support for RAID6 Read-Modify-Write
> in software XOR with some patches. The following commit mangles some lines in
> async_pq.c:
>
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
> id=584acdd49cd2472ca0f5a06adbe979db82d0b4af
>
> I introduced a new flag ASYNC_TX_PQ_XOR_DST that notifies the async layer
> that we want to do a XOR syndrome operation instead of a full calculation.
> This will enforce the software path because I guessed that hardware does not
> support that case. Without hardware to check I might have missed some
> checks in the async layer.
>
> In the upper layer ops_run_reconstruct6 will set the flag if we determined
> that rmw is faster than rcw.
>
> Can you check if rmw_level=0 fixes the issue. See:
>
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
> id=d06f191f8ecaef4d524e765fdb455f96392fbd42
I just gave this a try, and it doesn't fix anything.
One thing I forgot to mention is that our hardware doesn't support the
PQ multiplications and product sums, so one of the patches we have is
to add a new ASYNC_TX flag to be able to identify and bail out of such
transfers.
The patch is here:
https://github.com/MISL-EBU-System-SW/mainline-public/commit/9964fe4a79da10162f83bd527b3fe44da60d7e0f
There might be some interaction between your patch and this one, even
though the async_tx code itself looks to be untouched by your patches.
Thanks!
Maxime
--
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply
* Re: Possible RAID6 regression with ASYNC_TX_DMA enabled in 4.1
From: Shaohua Li @ 2015-05-11 6:26 UTC (permalink / raw)
To: Maxime Ripard
Cc: Neil Brown, Shaohua Li, linux-raid, linux-kernel, Lior Amsalem,
Thomas Petazzoni, Gregory Clement, Boris Brezillon
In-Reply-To: <20150507125702.GI11057@lukather>
On Thu, May 07, 2015 at 02:57:02PM +0200, Maxime Ripard wrote:
> Hi,
>
> I'm currently trying to add support for the PQ operations on the
> marvell XOR engine, in dmaengine, obviously to be able to use async_tx
> to offload these operations.
>
> I'm testing these patches with a RAID6 array with 4 disks.
>
> However, since the commit 59fc630b8b5f ("RAID5: batch adjacent full
> stripe write", every write to that array fails with the following
> stacktrace.
>
> http://code.bulix.org/eh8iew-88342?raw
>
> It seems to be generated by that warning here:
>
> http://lxr.free-electrons.com/source/crypto/async_tx/async_tx.c#L173
>
> And indeed, if we dump the status of depend_tx here, it's already been
> acked.
>
> That doesn't happen if ASYNC_TX_DMA is disabled, hence using the
> software version of it, instead of relying on our XOR engine. It
> doesn't happen on any commit prior to the one mentionned above, with
> the exact same changes applied. These changes are meant to be
> contributed, so I can definitely push them somewhere if needed.
>
> I don't really know where to look for though, the change that is
> causing this is probably the change in ops_run_reconstruct6, but I'm
> not sure that this partial revert alone would work with regard to the
> rest of the patch.
I don't have a machine with dmaengine, it's likely there is error in this side.
Could you please make stripe_can_batch() returns false always and check if the
error disappear? This should narrow down if it's related to batch issue.
Thanks,
Shaohua
^ permalink raw reply
* [GIT PULL REQUEST]
From: NeilBrown @ 2015-05-11 3:27 UTC (permalink / raw)
To: Linus Torvalds; +Cc: linux RAID, lkml, Heinz Mauelshagen, Shaohua Li
[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]
hi Linus
Please pull this collection of bugfixes.
Thanks,
NeilBrown
The following changes since commit 5ebe6afaf0057ac3eaeb98defd5456894b446d22:
Linux 4.1-rc2 (2015-05-03 19:22:23 -0700)
are available in the git repository at:
git://neil.brown.name/md/ tags/md/4.1-rc3-fixes
for you to fetch changes up to bb27051f9fd7643f05d8f0babce3337f0b9b3087:
md/raid5: fix handling of degraded stripes in batches. (2015-05-08 18:47:57 +1000)
----------------------------------------------------------------
A few fixes for md.
Most of these are related to the new "batched stripe writeout",
but there are a few others.
----------------------------------------------------------------
Heinz Mauelshagen (1):
md-raid0: conditional mddev->queue access to suit dm-raid
NeilBrown (6):
md/raid5: new alloc_stripe() to allocate an initialize a stripe.
md/raid5: more incorrect BUG_ON in handle_stripe_fill.
md/raid5: avoid reading parity blocks for full-stripe write to degraded array
md/raid5: don't record new size if resize_stripes fails.
md/raid5: fix allocation of 'scribble' array.
md/raid5: fix handling of degraded stripes in batches.
drivers/md/raid0.c | 5 ++-
drivers/md/raid5.c | 123 ++++++++++++++++++++++++++++++-----------------------
2 files changed, 73 insertions(+), 55 deletions(-)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: raid6: general protection fault in async_copy_data
From: Alexander Lyakas @ 2015-05-10 8:05 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150506142115.1b30da19@notabene.brown>
Thanks, Neil.
On Wed, May 6, 2015 at 6:21 AM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 5 May 2015 10:14:18 +0200 Alexander Lyakas <alex.bolshoy@gmail.com>
> wrote:
>
>> Hi Neil,
>> we had the following crash:
>>
>> [86399.862150] general protection fault: 0000 [#1] SMP
>> [86399.881970] CPU 1
>> [86399.882264] Pid: 17989, comm: md4_raid6 Tainted: GF W O
>> 3.8.13-030813-generic #201305111843 Bochs Bochs
>> [86399.883681] RIP: 0010:[<ffffffff8135d446>] [<ffffffff8135d446>]
>> memcpy+0x6/0x110
>> [86399.884886] RSP: 0018:ffff8800a78e5a80 EFLAGS: 00010286
>> [86399.885629] RAX: 4588966d912cea06 RBX: ffff8800a78e4000 RCX: 0000000000001000
>> [86399.886605] RDX: 0000000000001000 RSI: ffff8800a7ed2000 RDI: 4588966d912cea06
>> [86399.887586] RBP: ffff8800a78e5ae8 R08: 0000000000001000 R09: ffff8800a78e5b20
>> [86399.888603] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
>> [86399.889593] R13: ffff8800a78e5b20 R14: 0000000000001000 R15: 0000000000000000
>> [86399.890551] FS: 0000000000000000(0000) GS:ffff88011fd00000(0000)
>> knlGS:0000000000000000
>> [86399.891648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86399.892504] CR2: 00007f10cb8ae966 CR3: 0000000113bfc000 CR4: 00000000001406e0
>> [86399.893493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [86399.894458] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [86399.895426] Process md4_raid6 (pid: 17989, threadinfo
>> ffff8800a78e4000, task ffff8800a7dc0000)
>> [86399.896629] Stack:
>> [86399.896930] ffffffffa05061c5 ffff88000ab6fa06 ffffffff816ed725
>> ffffea00029fb480
>> [86399.898005] 51160e39b619d7e4 0000000000000000 000000000ab6fa06
>> ffff8800c696b938
>> [86399.899082] 000000003eca7624 ffff880084dc1ac0 0000000000001000
>> 0000000000000002
>> [86399.900293] Call Trace:
>> [86399.900660] [<ffffffffa05061c5>] ? async_memcpy+0x1c5/0x1000 [async_memcpy]
>> [86399.901653] [<ffffffff816ed725>] ? _raw_spin_lock_irq+0x15/0x20
>> [86399.902655] [<ffffffffa05a5090>] async_copy_data+0x100/0x140 [raid456]
>> [86399.903557] [<ffffffffa05abe20>] handle_stripe+0x13e0/0x2380 [raid456]
>> [86399.904531] [<ffffffff815739de>] ? dm_dispatch_request+0x3e/0x70
>> [86399.905388] [<ffffffff81097c33>] ? update_curr+0x143/0x1f0
>> [86399.906151] [<ffffffff816eb03d>] ? mutex_lock+0x1d/0x50
>> [86399.906888] [<ffffffffa05adea5>] handle_active_stripes+0x165/0x200 [raid456]
>> [86399.907857] [<ffffffff8156ab8e>] ? md_check_recovery.part.49+0x3e/0x530
>> [86399.908811] [<ffffffffa05ae28a>] raid5d+0x34a/0x570 [raid456]
>> [86399.909614] [<ffffffff8156344d>] md_thread+0x10d/0x140
>> [86399.910356] [<ffffffff8107fc10>] ? add_wait_queue+0x60/0x60
>> [86399.911149] [<ffffffff81563340>] ? md_rdev_init+0x140/0x140
>> [86399.911955] [<ffffffff8107f050>] kthread+0xc0/0xd0
>> [86399.912668] [<ffffffff8107ef90>] ? flush_kthread_worker+0xb0/0xb0
>> [86399.913528] [<ffffffff816f61ec>] ret_from_fork+0x7c/0xb0
>> [86399.914267] [<ffffffff8107ef90>] ? flush_kthread_worker+0xb0/0xb0
>> [86399.915109] Code: 74 13 48 8b 43 58 48 2b 43 50 88 43 4e 48 83 c4
>> 08 5b 5d c3 90 e8 fb fd ff ff eb e6 90 90 90 90 90 90 90 90 90 48 89
>> f8 48 89 d1 <f3> a4 c3 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 20 4c 8b 06
>> 4c 8b
>> [86399.919028] RIP [<ffffffff8135d446>] memcpy+0x6/0x110
>>
>> Can you maybe advise what is happening here? Our kernel is 3.8.13.
>>
>
> Not really.
> It appears that %RDI is the destination for the memcpy, and it contains a
> garbage address.
> I cannot easily tell if this is a read or a write, but I'd guess a read as it
> is hard to get the address of the page in the stripe_cache wrong.
>
> Maybe something has corrupted the bio??
>
> NeilBrown
^ permalink raw reply
* Re: mdadm Raid level change from 1 to 5?
From: Hans Kraus @ 2015-05-09 8:58 UTC (permalink / raw)
To: robin; +Cc: linux-raid
In-Reply-To: <20150508104722.GA21989@cthulhu.home.robinhill.me.uk>
Thanks a lot, that was it. After fail & remove one HD, set
--raid-devices=2 and re-add the removed HD as spare, the --grow worked.
Regards, Hans
Am 08.05.2015 um 12:47 schrieb Robin Hill:
> On Fri May 08, 2015 at 11:18:44AM +0200, Hans Kraus wrote:
>
>> Hi,
>>
>> I have a Raid 1 with 3 drives, the following layout (the internal
>> bitmap is already removed):
>> -----------------------------------------------------------------
>> md20 : active raid1 sdj1[2] sdi1[1] sdn1[0]
>> 732442432 blocks super 1.2 [3/3] [UUU]
>> -----------------------------------------------------------------
>> I tried to change it to Raid 5, but I'm getting aways an error.
>> -----------------------------------------------------------------
>> mdadm --grow /dev/md20 --level=5 --raid-devices=3 -c 32 \
>> --backup-file=/home/kraush/work/raidLevelChange
>>
>> Impossibly level change request for RAID1
>> -----------------------------------------------------------------
>> I tried it with various permutations (+/- raid-devices, c, ...).
>>
>> Is something in my command wrong or is the level change not
>> possible?
>> This is on Debian 7, mdadm - v3.3-65-g2167de7 - 11th March 2014
>>
> I think you'll need one of the disks to be spare first. Try reducing the
> RAID1 to a 2-disk plus spare configuration, then rerunning the grow.
> Alternately you'd need to add a new disk as a spare first, then grow to
> a 4 disk RAID5.
>
> HTH,
> Robin
>
---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
http://www.avast.com
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* unexpected speed difference at RAID initialisation
From: Christoph Anton Mitterer @ 2015-05-08 22:47 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 10451 bytes --]
Hey.
I'm just deploying some new servers at the faculty, where I made the
following strange observation, which I cannot explain.
All nodes have exactly the same hardware (all recent stuff, some 15k€
Dell servers with 16 disks á 6 TB, plenty of CPU), the same
BIOS/firmware config, the same OS (Debian jessie, except the kernel
4.0.0 from experimental and btrfs-tools 4.0 from sid) with identical
config.
The discs are connected via some Dell PERC RAID controller but for
testing they're exported as JBODs.
Nothing except some standard daemons (haveged, irqbalance and that like)
are running on these nodes.
I created an MD RAID6 over all disks via:
mdadm --create /dev/md/data-test-raid --verbose --metadata=1.2
--size=max --chunk=512K --level=raid6 --bitmap=internal
--name=data-test-raid --raid-devices=16
--spare-devices=0 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp
basically at the same time (few seconds difference) on both nodes.
But looking at the initial rebuild on two nodes, one can see substantial
speed differences:
node A:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid6 sdp[15] sdo[14] sdn[13] sdm[12] sdl[11] sdk[10] sdj[9] sdi[8] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1] sda[0]
82045479936 blocks super 1.2 level 6, 512k chunk, algorithm 2 [16/16] [UUUUUUUUUUUUUUUU]
[=====>...............] resync = 28.8% (1691996416/5860391424) finish=841.8min speed=82526K/sec
bitmap: 32/44 pages [128KB], 65536KB chunk
unused devices: <none>
node B:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid6 sdp[15] sdo[14] sdn[13] sdm[12] sdl[11] sdk[10] sdj[9] sdi[8] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1] sda[0]
82045479936 blocks super 1.2 level 6, 512k chunk, algorithm 2 [16/16] [UUUUUUUUUUUUUUUU]
[====>................] resync = 20.1% (1180137984/5860391424) finish=1496.2min speed=52132K/sec
bitmap: 36/44 pages [144KB], 65536KB chunk
unused devices: <none>
(again taken with only few seconds in between).
As you can see it already shows different speed (~80000K/s for node A
and ~50000K/s for node B).
dmesg on A gives:
[ 8333.919976] md: bind<sda>
[ 8333.920285] md: bind<sdb>
[ 8333.920553] md: bind<sdc>
[ 8333.920831] md: bind<sdd>
[ 8333.921092] md: bind<sde>
[ 8333.921344] md: bind<sdf>
[ 8333.921593] md: bind<sdg>
[ 8333.921853] md: bind<sdh>
[ 8333.922093] md: bind<sdi>
[ 8333.922377] md: bind<sdj>
[ 8333.922694] md: bind<sdk>
[ 8333.922973] md: bind<sdl>
[ 8333.923235] md: bind<sdm>
[ 8333.923502] md: bind<sdn>
[ 8333.923744] md: bind<sdo>
[ 8333.923941] md: bind<sdp>
[ 8334.002228] raid6: sse2x1 2651 MB/s
[ 8334.070159] raid6: sse2x2 7676 MB/s
[ 8334.138090] raid6: sse2x4 13690 MB/s
[ 8334.206027] raid6: avx2x1 18736 MB/s
[ 8334.273958] raid6: avx2x2 21984 MB/s
[ 8334.341893] raid6: avx2x4 24764 MB/s
[ 8334.341904] raid6: using algorithm avx2x4 (24764 MB/s)
[ 8334.341923] raid6: using avx2x2 recovery algorithm
[ 8334.342962] async_tx: api initialized (async)
[ 8334.343648] xor: automatically using best checksumming function:
[ 8334.381855] avx : 30433.000 MB/sec
[ 8334.386213] md: raid6 personality registered for level 6
[ 8334.386235] md: raid5 personality registered for level 5
[ 8334.386255] md: raid4 personality registered for level 4
[ 8334.428030] md/raid:md127: not clean -- starting background reconstruction
[ 8334.428133] md/raid:md127: device sdp operational as raid disk 15
[ 8334.428157] md/raid:md127: device sdo operational as raid disk 14
[ 8334.428180] md/raid:md127: device sdn operational as raid disk 13
[ 8334.428203] md/raid:md127: device sdm operational as raid disk 12
[ 8334.428226] md/raid:md127: device sdl operational as raid disk 11
[ 8334.428248] md/raid:md127: device sdk operational as raid disk 10
[ 8334.428272] md/raid:md127: device sdj operational as raid disk 9
[ 8334.428294] md/raid:md127: device sdi operational as raid disk 8
[ 8334.428756] md/raid:md127: device sdh operational as raid disk 7
[ 8334.429108] md/raid:md127: device sdg operational as raid disk 6
[ 8334.429450] md/raid:md127: device sdf operational as raid disk 5
[ 8334.429754] md/raid:md127: device sde operational as raid disk 4
[ 8334.430062] md/raid:md127: device sdd operational as raid disk 3
[ 8334.430329] md/raid:md127: device sdc operational as raid disk 2
[ 8334.430595] md/raid:md127: device sdb operational as raid disk 1
[ 8334.430845] md/raid:md127: device sda operational as raid disk 0
[ 8334.432415] md/raid:md127: allocated 0kB
[ 8334.432749] md/raid:md127: raid level 6 active with 16 out of 16 devices, algorithm 2
[ 8334.433021] RAID conf printout:
[ 8334.433022] --- level:6 rd:16 wd:16
[ 8334.433023] disk 0, o:1, dev:sda
[ 8334.433024] disk 1, o:1, dev:sdb
[ 8334.433025] disk 2, o:1, dev:sdc
[ 8334.433026] disk 3, o:1, dev:sdd
[ 8334.433027] disk 4, o:1, dev:sde
[ 8334.433028] disk 5, o:1, dev:sdf
[ 8334.433029] disk 6, o:1, dev:sdg
[ 8334.433030] disk 7, o:1, dev:sdh
[ 8334.433031] disk 8, o:1, dev:sdi
[ 8334.433032] disk 9, o:1, dev:sdj
[ 8334.433032] disk 10, o:1, dev:sdk
[ 8334.433033] disk 11, o:1, dev:sdl
[ 8334.433034] disk 12, o:1, dev:sdm
[ 8334.433035] disk 13, o:1, dev:sdn
[ 8334.433036] disk 14, o:1, dev:sdo
[ 8334.433037] disk 15, o:1, dev:sdp
[ 8334.433257] created bitmap (44 pages) for device md127
[ 8334.437094] md127: bitmap initialized from disk: read 3 pages, set 89423 of 89423 bits
[ 8334.492393] md127: detected capacity change from 0 to 84014571454464
[ 8334.492438] md: md127 switched to read-write mode.
[ 8334.518862] md: resync of RAID array md127
[ 8334.519677] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 8334.520329] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 8334.521357] md: using 128k window, over a total of 5860391424k.
and on B:
[ 8269.297090] md: bind<sda>
[ 8269.297414] md: bind<sdb>
[ 8269.297689] md: bind<sdc>
[ 8269.297957] md: bind<sdd>
[ 8269.298230] md: bind<sde>
[ 8269.298487] md: bind<sdf>
[ 8269.298753] md: bind<sdg>
[ 8269.299006] md: bind<sdh>
[ 8269.299260] md: bind<sdi>
[ 8269.299522] md: bind<sdj>
[ 8269.299778] md: bind<sdk>
[ 8269.300061] md: bind<sdl>
[ 8269.300326] md: bind<sdm>
[ 8269.300543] md: bind<sdn>
[ 8269.300912] md: bind<sdo>
[ 8269.301123] md: bind<sdp>
[ 8269.372959] raid6: sse2x1 2622 MB/s
[ 8269.440892] raid6: sse2x2 4685 MB/s
[ 8269.508815] raid6: sse2x4 9777 MB/s
[ 8269.576747] raid6: avx2x1 16486 MB/s
[ 8269.644680] raid6: avx2x2 21875 MB/s
[ 8269.712615] raid6: avx2x4 25242 MB/s
[ 8269.712632] raid6: using algorithm avx2x4 (25242 MB/s)
[ 8269.712652] raid6: using avx2x2 recovery algorithm
[ 8269.713686] async_tx: api initialized (async)
[ 8269.714400] xor: automatically using best checksumming function:
[ 8269.752579] avx : 30506.000 MB/sec
[ 8269.757308] md: raid6 personality registered for level 6
[ 8269.757339] md: raid5 personality registered for level 5
[ 8269.757365] md: raid4 personality registered for level 4
[ 8269.782660] md/raid:md127: not clean -- starting background reconstruction
[ 8269.782767] md/raid:md127: device sdp operational as raid disk 15
[ 8269.782797] md/raid:md127: device sdo operational as raid disk 14
[ 8269.782827] md/raid:md127: device sdn operational as raid disk 13
[ 8269.782855] md/raid:md127: device sdm operational as raid disk 12
[ 8269.782884] md/raid:md127: device sdl operational as raid disk 11
[ 8269.782912] md/raid:md127: device sdk operational as raid disk 10
[ 8269.782941] md/raid:md127: device sdj operational as raid disk 9
[ 8269.782969] md/raid:md127: device sdi operational as raid disk 8
[ 8269.783539] md/raid:md127: device sdh operational as raid disk 7
[ 8269.783965] md/raid:md127: device sdg operational as raid disk 6
[ 8269.784395] md/raid:md127: device sdf operational as raid disk 5
[ 8269.784776] md/raid:md127: device sde operational as raid disk 4
[ 8269.785106] md/raid:md127: device sdd operational as raid disk 3
[ 8269.785425] md/raid:md127: device sdc operational as raid disk 2
[ 8269.785718] md/raid:md127: device sdb operational as raid disk 1
[ 8269.786012] md/raid:md127: device sda operational as raid disk 0
[ 8269.787672] md/raid:md127: allocated 0kB
[ 8269.788026] md/raid:md127: raid level 6 active with 16 out of 16 devices, algorithm 2
[ 8269.788349] RAID conf printout:
[ 8269.788351] --- level:6 rd:16 wd:16
[ 8269.788352] disk 0, o:1, dev:sda
[ 8269.788354] disk 1, o:1, dev:sdb
[ 8269.788355] disk 2, o:1, dev:sdc
[ 8269.788356] disk 3, o:1, dev:sdd
[ 8269.788357] disk 4, o:1, dev:sde
[ 8269.788358] disk 5, o:1, dev:sdf
[ 8269.788359] disk 6, o:1, dev:sdg
[ 8269.788360] disk 7, o:1, dev:sdh
[ 8269.788361] disk 8, o:1, dev:sdi
[ 8269.788362] disk 9, o:1, dev:sdj
[ 8269.788364] disk 10, o:1, dev:sdk
[ 8269.788365] disk 11, o:1, dev:sdl
[ 8269.788366] disk 12, o:1, dev:sdm
[ 8269.788367] disk 13, o:1, dev:sdn
[ 8269.788368] disk 14, o:1, dev:sdo
[ 8269.788369] disk 15, o:1, dev:sdp
[ 8269.788589] created bitmap (44 pages) for device md127
[ 8269.791496] md127: bitmap initialized from disk: read 3 pages, set 89423 of 89423 bits
[ 8269.807348] md127: detected capacity change from 0 to 84014571454464
[ 8269.807896] md: md127 switched to read-write mode.
[ 8269.808387] md: resync of RAID array md127
[ 8269.808685] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 8269.808980] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 8269.809504] md: using 128k window, over a total of 5860391424k.
There's a bigger difference at:
[ 8334.341904] raid6: using algorithm avx2x4 (24764 MB/s)
vs.
[ 8269.712632] raid6: using algorithm avx2x4 (25242 MB/s)
How are these numbers determined?
Further:
node A:
# ps ax | grep md127
7550 ? S 273:47 [md127_raid6]
7552 ? D 79:43 [md127_resync]
node B:
# ps ax | grep md127
7494 ? R 251:30 [md127_raid6]
7495 ? D 63:48 [md127_resync]
Any ideas where this performance difference could come from?
Thanks,
Chris.
[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]
^ permalink raw reply
* Re: [PATCH 7/7] md/raid5: fix handling of degraded stripes in batches.
From: Shaohua Li @ 2015-05-08 19:12 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, linux-nfs
In-Reply-To: <20150508085612.19179.92120.stgit@notabene.brown>
On Fri, May 08, 2015 at 06:56:12PM +1000, NeilBrown wrote:
> There is no need for special handling of stripe-batches when the array
> is degraded.
>
> There may be if there is a failure in the batch, but STRIPE_DEGRADED
> does not imply an error.
>
> So don't set STRIPE_BATCH_ERR in ops_run_io just because the array is
> degraded.
> This actually causes a bug: the STRIPE_DEGRADED flag gets cleared in
> check_break_stripe_batch_list() and so the bitmap bit gets cleared
> when it shouldn't.
>
> So in check_break_stripe_batch_list(), split the batch up completely -
> again STRIPE_DEGRADED isn't meaningful.
>
> Also don't set STRIPE_BATCH_ERR when there is a write error to a
> replacement device. This simply removes the replacement device and
> requires no extra handling.
>
> Signed-off-by: NeilBrown <neilb@suse.de>
> ---
> drivers/md/raid5.c | 17 +++--------------
> 1 file changed, 3 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 3873eaa6fa2e..1ba97fdc6df1 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -1078,9 +1078,6 @@ again:
> pr_debug("skip op %ld on disc %d for sector %llu\n",
> bi->bi_rw, i, (unsigned long long)sh->sector);
> clear_bit(R5_LOCKED, &sh->dev[i].flags);
> - if (sh->batch_head)
> - set_bit(STRIPE_BATCH_ERR,
> - &sh->batch_head->state);
> set_bit(STRIPE_HANDLE, &sh->state);
> }
Patches look good to me. I had a question here. Is it possible some stripes in
a batch become degraded here but some not? Seems possible, then the batch
should be splitted too.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH] md-cluster: avoid deadlock on MESSAGE lock resource
From: Abhijit Bhopatkar @ 2015-05-08 13:14 UTC (permalink / raw)
To: linux-raid, Lidong Zhong, Goldwyn Rodrigues; +Cc: Reese Faucette (rfaucett)
In-Reply-To: <554CB5DB.4020305@cisco.com>
On 08/05/15 6:40 pm, Abhijit Bhopatkar wrote:
>
> Every receiver has CR lock on MESSAGE while processing the message. When
> every receiver releases ACK lock and for some reason fails to grab EX on
> MESSAGE resource in time, a waiting sender could queue an EX on MESSAGE
> instead. Now when receiver queues its up convert request on MESSAGE it
> will end up in a deadlock situation.
>
> Setting NOQUEUE flag on MESSAGE lock resource while grabbing the EX on
> MESSAGE on sender will avoid this deadlock. If sender can not grab
> MESSAGE lock immediately it should retry until the lock is granted.
>
> Signed-off-by: Abhijit Bhopatkar <abhopatk@cisco.com>
> ---
> This has been minimally tested on a three node cluster.
>
I have tested standard mdadm operations (create, assemble etc).
What more testing would you want me to do on this before its considered
ready?
Regards,
Abhijit
> drivers/md/md-cluster.c | 14 ++++++++++++--
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index fcfc4b9..04ac309 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -512,7 +512,10 @@ static void unlock_comm(struct md_cluster_info *cinfo)
> * This function performs the actual sending of the message. This function is
> * usually called after performing the encompassing operation
> * The function:
> - * 1. Grabs the message lockresource in EX mode
> + * 1. Grabs the message lockresource in EX. Do not queue the request if not granted
> + immediately. This avoids deadlock with receivers when receivers try to
> + upconvert CR to EX of message lockresource. The thread will retry until the
> + request is granted.
> * 2. Copies the message to the message LVB
> * 3. Downconverts message lockresource to CR
> * 4. Upconverts ack lock resource from CR to EX. This forces the BAST on other nodes
> @@ -526,12 +529,19 @@ static int __sendmsg(struct md_cluster_info *cinfo, struct cluster_msg *cmsg)
> int slot = cinfo->slot_number - 1;
>
> cmsg->slot = cpu_to_le32(slot);
> - /*get EX on Message*/
> +
> + /* get EX on Message with noqueue flag */
> + cinfo->message_lockres->flags |= DLM_LKF_NOQUEUE;
> +
> +retry:
> error = dlm_lock_sync(cinfo->message_lockres, DLM_LOCK_EX);
> if (error) {
> + if (error == -EAGAIN)
> + goto retry;
> pr_err("md-cluster: failed to get EX on MESSAGE (%d)\n", error);
> goto failed_message;
> }
> + cinfo->message_lockres->flags &= ~DLM_LKF_NOQUEUE;
>
> memcpy(cinfo->message_lockres->lksb.sb_lvbptr, (void *)cmsg,
> sizeof(struct cluster_msg));
> -- 2.1.0
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* [PATCH] md-cluster: avoid deadlock on MESSAGE lock resource
From: Abhijit Bhopatkar @ 2015-05-08 13:10 UTC (permalink / raw)
To: linux-raid, Lidong Zhong, Goldwyn Rodrigues; +Cc: Reese Faucette (rfaucett)
Every receiver has CR lock on MESSAGE while processing the message. When
every receiver releases ACK lock and for some reason fails to grab EX on
MESSAGE resource in time, a waiting sender could queue an EX on MESSAGE
instead. Now when receiver queues its up convert request on MESSAGE it
will end up in a deadlock situation.
Setting NOQUEUE flag on MESSAGE lock resource while grabbing the EX on
MESSAGE on sender will avoid this deadlock. If sender can not grab
MESSAGE lock immediately it should retry until the lock is granted.
Signed-off-by: Abhijit Bhopatkar <abhopatk@cisco.com>
---
This has been minimally tested on a three node cluster.
drivers/md/md-cluster.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index fcfc4b9..04ac309 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -512,7 +512,10 @@ static void unlock_comm(struct md_cluster_info *cinfo)
* This function performs the actual sending of the message. This function is
* usually called after performing the encompassing operation
* The function:
- * 1. Grabs the message lockresource in EX mode
+ * 1. Grabs the message lockresource in EX. Do not queue the request if not granted
+ immediately. This avoids deadlock with receivers when receivers try to
+ upconvert CR to EX of message lockresource. The thread will retry until the
+ request is granted.
* 2. Copies the message to the message LVB
* 3. Downconverts message lockresource to CR
* 4. Upconverts ack lock resource from CR to EX. This forces the BAST on other nodes
@@ -526,12 +529,19 @@ static int __sendmsg(struct md_cluster_info *cinfo, struct cluster_msg *cmsg)
int slot = cinfo->slot_number - 1;
cmsg->slot = cpu_to_le32(slot);
- /*get EX on Message*/
+
+ /* get EX on Message with noqueue flag */
+ cinfo->message_lockres->flags |= DLM_LKF_NOQUEUE;
+
+retry:
error = dlm_lock_sync(cinfo->message_lockres, DLM_LOCK_EX);
if (error) {
+ if (error == -EAGAIN)
+ goto retry;
pr_err("md-cluster: failed to get EX on MESSAGE (%d)\n", error);
goto failed_message;
}
+ cinfo->message_lockres->flags &= ~DLM_LKF_NOQUEUE;
memcpy(cinfo->message_lockres->lksb.sb_lvbptr, (void *)cmsg,
sizeof(struct cluster_msg));
-- 2.1.0
^ permalink raw reply related
* Re: mdadm Raid level change from 1 to 5?
From: Robin Hill @ 2015-05-08 10:47 UTC (permalink / raw)
To: Hans Kraus; +Cc: linux-raid
In-Reply-To: <554C7F74.5010404@hanswkraus.com>
[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]
On Fri May 08, 2015 at 11:18:44AM +0200, Hans Kraus wrote:
> Hi,
>
> I have a Raid 1 with 3 drives, the following layout (the internal
> bitmap is already removed):
> -----------------------------------------------------------------
> md20 : active raid1 sdj1[2] sdi1[1] sdn1[0]
> 732442432 blocks super 1.2 [3/3] [UUU]
> -----------------------------------------------------------------
> I tried to change it to Raid 5, but I'm getting aways an error.
> -----------------------------------------------------------------
> mdadm --grow /dev/md20 --level=5 --raid-devices=3 -c 32 \
> --backup-file=/home/kraush/work/raidLevelChange
>
> Impossibly level change request for RAID1
> -----------------------------------------------------------------
> I tried it with various permutations (+/- raid-devices, c, ...).
>
> Is something in my command wrong or is the level change not
> possible?
> This is on Debian 7, mdadm - v3.3-65-g2167de7 - 11th March 2014
>
I think you'll need one of the disks to be spare first. Try reducing the
RAID1 to a 2-disk plus spare configuration, then rerunning the grow.
Alternately you'd need to add a new disk as a spare first, then grow to
a 4 disk RAID5.
HTH,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox