* RE: best base / worst case RAID 5,6 write speeds
From: Robert Kierski @ 2015-12-15 14:09 UTC (permalink / raw)
To: Dallas Clement, Mark Knecht; +Cc: Phil Turmel, John Stoffel, Linux-RAID
In-Reply-To: <CAE9DZUQxT_5L0bW5m9SZ_d2GU6sZS8k0qD=g+o112qM4V=cJkw@mail.gmail.com>
Dallas,
The threshold between iodepth=32 and iodepth=64 might be caused by exceeding the stripe cache size.
In my opinion, if you're writing chunk aligned data, you shouldn't be doing any RMW's. That you're doing small numbers of RMW's with small iodepth, indicates that you're using the stripe cache.
I've tried similar tests where I've set the stripe cache size to 17 (the smallest you can set it to), and then did perfectly aligned IO's. The results showed that I was doing massive amounts of RMW, and my performance was horrible.
Your FIO job file looks ordinary (and by that I mean.... "good"). While I wouldn't have picked bs=1408k, because it's aligned, I wouldn't expect that to be a cause for problem.
You should be able to set iodepth to whatever you want. You could set it to a billion. The OS should block additional requests until the underlying device's queue has available space. Iodepth shouldn't affect RMW when you're doing aligned writes. In my opnion, increasing iodepth should only help... not hurt. If it causes low memory, that could be an issue, but it shouldn't increase the amount of RMW.
Bob Kierski
Senior Storage Performance Engineer
Cray Inc.
380 Jackson Street
Suite 210
St. Paul, MN 55101
Tele: 651-967-9590
Fax: 651-605-9001
Cell: 651-890-7461
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Phil Turmel @ 2015-12-15 13:53 UTC (permalink / raw)
To: Dallas Clement, Mark Knecht; +Cc: John Stoffel, Linux-RAID
In-Reply-To: <CAE9DZUQxT_5L0bW5m9SZ_d2GU6sZS8k0qD=g+o112qM4V=cJkw@mail.gmail.com>
Hi Dallas,
On December 14, 2015 9:36:05 PM EST, Dallas Clement
>Hi Everyone. I have some very interesting news to report. I did a
>little bit more playing around with fio, doing sequential writes to a
>RAID 5 device with all 12 disks. I kept the block size at the 128K
>chunk aligned value of 1408K. But this time I varied the queue depth.
>These are my results for writing a 10 GB of data:
>
>iodepth=1 => 642 MB/s, # of RMWs = 11
>
>iodepth=4 => 1108 MB/s, # of RMWs = 6
>
>iodepth=8 => 895 MB/s, # of RMWs = 7
>
>iodepth=16 => 855 MB/s, # of RMWs = 11
>
>iodepth=32 => 936 MB/s, # of RMWs = 11
>
>iodepth=64 => 551 MB/s, # of RMWs = 5606
>
>iodepth=128 => 554 MB/s, # of RMWs = 6333
>
>As you can see, something goes terribly wrong with async i/o with
>iodepth >= 64. Btw, not to be contentious Phil, I have checked
>multiple fio man pages and they clearly indicate that iodepth is for
>async i/o which this is (libaio). I don't see any mention of
>sequential writes being prohibited with async i/o. See
>https://github.com/axboe/fio/blob/master/HOWTO.
Hmmm. I misread that part. But do note the comment that you might not
achieve as many in-flight I/Os as you expect.
>However, maybe I'm
>missing something and it sure looks from these results that there may
>be a connection.
>
>This is my fio job config:
>
>[job]
>ioengine=libaio
>iodepth=128
>prio=0
>rw=write
>bs=1408k
>filename=/dev/md10
>numjobs=1
>size=10g
>direct=1
>invalidate=1
>
>Incidentally, the very best write speed here (1108 MB/s with
>iodepth=4) comes out to about 100 MB/s per disk, which is pretty close
>to the worst case inner disk speed of 95.5 MB/s I had recorded
>earlier.
Very interesting indeed. I wonder if the extra I/O in flight at high
depths is consuming all available stripe cache space, possibly not
consistently. I'd raise and lower that in various combinations with
various combinations of iodepth. Running out of stripe cache will cause
premature RMWs.
Regards,
Phil
^ permalink raw reply
* Re: Mdadm with data offsets
From: Phil Turmel @ 2015-12-15 13:44 UTC (permalink / raw)
To: TheGerwazy .; +Cc: linux-raid
In-Reply-To: <CAH_hVdqMdrPTAz1LcDzZNDMdGKFOpwo32mh0E+mFZtcCBEBqJg@mail.gmail.com>
On 12/15/2015 06:30 AM, TheGerwazy . wrote:
> The main problem is the data offsets are not stored elsewhere than in
> superblock. I have some drives from debian 6.0.5 and some were changed
> in debian 7.5. In 6.0.5 default data offset was at 2048 sector while
> int 7.5 it is 262144.
There have been other defaults, and various adjustments to the
algorithms that handle special cases. The purpose of the superblock is
to store the data needed by the features supported. Where else
would/could you store this information?
This highlights why --create is such a terrible way to recover an array,
and highlights the danger posed by well-intentioned bloggers who
cavalierly suggest using it.
Pretty much everyone who comes here for advice *before* trying --create
has been successfully helped. The results after --create are decidedly
mixed.
> Thanks for Phil
You're welcome.
Phil
^ permalink raw reply
* Re: [PATCH 2/2] md: dm-crypt: Optimize the dm-crypt for XTS mode
From: Baolin Wang @ 2015-12-15 11:35 UTC (permalink / raw)
To: Milan Broz
Cc: Jens Axboe, Alasdair G Kergon, Mike Snitzer, dm-devel, neilb,
dan.j.williams, martin.petersen, sagig, Kent Overstreet,
keith.busch, tj, Mark Brown, Arnd Bergmann, linux-block,
linux-raid, LKML
In-Reply-To: <566FCD57.3000507@gmail.com>
On 15 December 2015 at 16:20, Milan Broz <gmazyland@gmail.com> wrote:
> On 12/15/2015 03:56 AM, Baolin Wang wrote:
>>>> + /*
>>>> + * Here we need to check if it can be encrypted or decrypted with
>>>> + * bulk block, which means these encryption modes don't need IV or
>>>> + * just need one initial IV. For bulk mode, we can expand the
>>>> + * scatterlist entries to map the bio, then send all the scatterlists
>>>> + * to the hardware engine at one time to improve the crypto engine
>>>> + * efficiency. But it does not fit for other encryption modes, it has
>>>> + * to do encryption and decryption sector by sector because every
>>>> + * sector has different IV.
>>>> + */
>>>> + if (!strcmp(chainmode, "ecb") || !strcmp(chainmode, "xts"))
>>>> + cc->bulk_crypto = 1;
>>>> + else
>>>> + cc->bulk_crypto = 0;n
>>>
>>> It is perfectly fine to use another IV even for XTS mode (despite it is not needed).
>>> You can use ESSIV for example, or benbi (big-endian variant of plain IV).
>>> (And it is really used for some LUKS devices.)
>>>
>>> How it is supposed to work in this case?
>>> If I read this patch correctly, it will completely corrupt data in this case because
>>> it expects plain (consecutive) IV...
>>
>> The XTS mode can limit maximum size of each encrypted data unit
>> (typically a sector or disk block) to 2^20 AES blocks, so we can use
>> one bio as one encrypted data unit (we don't do it sector by sector).
>> It can generate one IV for each separate encrypted data unit. Please
>> correct me if I misunderstand something. Thanks.
>
> How this will help XTS-ESSIV, if you have to recalculate IV on every fixed
> encrypted block?
>
> TBH I think the whole patch here is doing something more than optimization
> and seriously touches cryptography part.
>
> Isn't in de-facto reinventing how the full disk encryption works today?
>
> All currently used systems use always disk encrypted block size (usually fixed
> to 512 bytes sectors like in dmcrypt) and these are encrypted independently with own IV.
> (We can talk about supporting larger disk encrypted block sizes but the principle
> it is still the same.
> And we played with it before already http://www.saout.de/pipermail/dm-crypt/2013-January/003125.html)
>
> Every encrypted disk block here has independent initialization vector - IV (or tweak).
> For some modes (like XTS, LRW) this vector can produce just predictable linear offset,
> usually just block number (it must not repeat though; exceptions are legacy/compatible IVs).
> But it can be also something else - pseudorandom sequence like in ESSIV or so.
>
> If I understand your patch correctly, you are trying to use the XTS mode for
> the whole bulk request (there is only one IV for the bulk request and this bulk request
> is larger than currently fixed disk block size).
That's right.
>
> Isn't it even incompatible with the current XTS (aes-xts-plain64) encrypted disk
> per-sector where for every sectors new IV initialization is used?
> In fact I think that your approach would need to implement some different
> IV name (aes-xts-bulk) or something like that so userspace (cryptsetup)
> can be backward compatible.
Yes, that sounds reasonable.
>
> BTW the 2^20 block limit requirement (the XTS block size) is strict limit in some specifications.
> For example, NIST SP800-38E (XTS-AES) says:
> "The length of the data unit for any instance of an implementation of XTS-AES shall not exceed
> 2^20 AES blocks."
OK.
>
> There are a lot of badly implemented hw crypto accelerators where initialization cost
> is quite big and that performs better if it encrypts large blocks of data
> (it applies even for CBC mode). These are simply not designed for FDE use case...
>
> But IMHO this is not correct reason to patch it in kernel, moreover on dmcrypt layer
> with side effect of hardcoding another kind of crypto logic into dmcrypt.
You're right, this patch is used for hw crypto acceleration. As you
know some hardware engine may handle the IV for one bulk data, so the
software just generate one initial IV. Qualcomm's implementation for
xts mode just use one initial IV for one bulk data to improve the hw
engine efficiency.
(https://github.com/major91/Zeta-Chromium-N5/blob/master/drivers/md/dm-req-crypt.c)
So like you said I can change the IV name as 'aes-xts-bulk' or
something else to avoid the ESSIV, cause this can really gain big
improvements for hw engine.
>
> DMcrypt should be completely independent of used symmetric block encryption mode
> (that's the crypto API job). There will be new modes in future, there can be requests
> to switch to different IVs (similar to preventing CBC watermarking attack with
> predictable IV) etc.
>
> Milan
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: Mdadm with data offsets
From: TheGerwazy . @ 2015-12-15 11:30 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <566EBC38.7060205@turmel.org>
2015-12-14 13:55 GMT+01:00 Phil Turmel <philip@turmel.org>:
> On 12/14/2015 07:03 AM, TheGerwazy . wrote:
>
>> Hi Phil
>> Thanks for the message .... but,
>> In the last version of mdadm I get:
>>
>> ./mdadm --create --raid-devices=8 --level=6 --assume-clean /dev/md7
>> /dev/mapper/ovsdb1:1024 /dev/mapper/ovsdc1:4096
>> /dev/mapper/ovsdd1:1024 /dev/mapper/ovsde1:1024
>> /dev/mapper/ovsdf1:1024 /dev/mapper/ovsdg1:1024
>> /dev/mapper/ovsdh1:1024 /dev/mapper/ovsdi1:4096
>> mdadm: cannot open /dev/mapper/ovsdb1:1024: No such file or directory
>
> The final version of that feature requires --data-offset=variable before
> the :offset syntax is recognized on the devices.
>
>> sudo ./mdadm --create --raid-devices=8 --level=6 --assume-clean
>
>> mdadm: array /dev/md7 started.
>
> You're committed now. I hope it works out for you.
>
>> BTW is it possible to find right data offset without superblock on drive ?
>
> Only if the content of that member device has searchable content in a
> predictable location. In practice that means finding the offset of the
> first member by locating filesystem magic, or otherwise locating the
> transition from all zeros to randomish data.
>
> Good luck.
>
> Phil
>
"Good Luck" seems to be working ... ;)
I finally recovered files with magic cmd:
sudo ./mdadm --create --raid-device=8 --level=6 --assume-clean
--size=974508544 /dev/md7 /dev/mapper/ovsde2:1024
/dev/mapper/ovsdh1:100352 /dev/mapper/ovsdg1:100352
/dev/mapper/ovsdc1:131072 /dev/mapper/ovsda1:1024
/dev/mapper/ovsdb1:1024 /dev/mapper/ovsdf1:131072
/dev/mapper/ovsdd1:100352
The main problem is the data offsets are not stored elsewhere than in
superblock. I have some drives from debian 6.0.5 and some were changed
in debian 7.5. In 6.0.5 default data offset was at 2048 sector while
int 7.5 it is 262144.
But - when add new drive to existing array and partition (or drive) is
smaller than 262144 plus array data (array device size) when mdadm
calculates something between 2048 and 262144. -> so as above magic
cmd's in my case it was 200704 sector.
I recovered it by doing all steps regarding mdadm in virtual
enviroment and I am surprised by the non standard data offsets.
Thanks for Phil
regards
Gerard
^ permalink raw reply
* Re: [PATCH 2/2] md: dm-crypt: Optimize the dm-crypt for XTS mode
From: Milan Broz @ 2015-12-15 8:20 UTC (permalink / raw)
To: Baolin Wang
Cc: Jens Axboe, Alasdair G Kergon, Mike Snitzer, dm-devel, neilb,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, Mark Brown, Arnd Bergmann, linux-block,
linux-raid, LKML
In-Reply-To: <CAMz4kuJn=2xE7XW79=fCa9qKRT5df9q6H0xNBnhYQveAZrF4ww@mail.gmail.com>
On 12/15/2015 03:56 AM, Baolin Wang wrote:
>>> + /*
>>> + * Here we need to check if it can be encrypted or decrypted with
>>> + * bulk block, which means these encryption modes don't need IV or
>>> + * just need one initial IV. For bulk mode, we can expand the
>>> + * scatterlist entries to map the bio, then send all the scatterlists
>>> + * to the hardware engine at one time to improve the crypto engine
>>> + * efficiency. But it does not fit for other encryption modes, it has
>>> + * to do encryption and decryption sector by sector because every
>>> + * sector has different IV.
>>> + */
>>> + if (!strcmp(chainmode, "ecb") || !strcmp(chainmode, "xts"))
>>> + cc->bulk_crypto = 1;
>>> + else
>>> + cc->bulk_crypto = 0;n
>>
>> It is perfectly fine to use another IV even for XTS mode (despite it is not needed).
>> You can use ESSIV for example, or benbi (big-endian variant of plain IV).
>> (And it is really used for some LUKS devices.)
>>
>> How it is supposed to work in this case?
>> If I read this patch correctly, it will completely corrupt data in this case because
>> it expects plain (consecutive) IV...
>
> The XTS mode can limit maximum size of each encrypted data unit
> (typically a sector or disk block) to 2^20 AES blocks, so we can use
> one bio as one encrypted data unit (we don't do it sector by sector).
> It can generate one IV for each separate encrypted data unit. Please
> correct me if I misunderstand something. Thanks.
How this will help XTS-ESSIV, if you have to recalculate IV on every fixed
encrypted block?
TBH I think the whole patch here is doing something more than optimization
and seriously touches cryptography part.
Isn't in de-facto reinventing how the full disk encryption works today?
All currently used systems use always disk encrypted block size (usually fixed
to 512 bytes sectors like in dmcrypt) and these are encrypted independently with own IV.
(We can talk about supporting larger disk encrypted block sizes but the principle
it is still the same.
And we played with it before already http://www.saout.de/pipermail/dm-crypt/2013-January/003125.html)
Every encrypted disk block here has independent initialization vector - IV (or tweak).
For some modes (like XTS, LRW) this vector can produce just predictable linear offset,
usually just block number (it must not repeat though; exceptions are legacy/compatible IVs).
But it can be also something else - pseudorandom sequence like in ESSIV or so.
If I understand your patch correctly, you are trying to use the XTS mode for
the whole bulk request (there is only one IV for the bulk request and this bulk request
is larger than currently fixed disk block size).
Isn't it even incompatible with the current XTS (aes-xts-plain64) encrypted disk
per-sector where for every sectors new IV initialization is used?
In fact I think that your approach would need to implement some different
IV name (aes-xts-bulk) or something like that so userspace (cryptsetup)
can be backward compatible.
BTW the 2^20 block limit requirement (the XTS block size) is strict limit in some specifications.
For example, NIST SP800-38E (XTS-AES) says:
"The length of the data unit for any instance of an implementation of XTS-AES shall not exceed
2^20 AES blocks."
There are a lot of badly implemented hw crypto accelerators where initialization cost
is quite big and that performs better if it encrypts large blocks of data
(it applies even for CBC mode). These are simply not designed for FDE use case...
But IMHO this is not correct reason to patch it in kernel, moreover on dmcrypt layer
with side effect of hardcoding another kind of crypto logic into dmcrypt.
DMcrypt should be completely independent of used symmetric block encryption mode
(that's the crypto API job). There will be new modes in future, there can be requests
to switch to different IVs (similar to preventing CBC watermarking attack with
predictable IV) etc.
Milan
^ permalink raw reply
* [PATCH] raid5-cache: add journal hot add/remove support
From: Shaohua Li @ 2015-12-15 6:22 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, neilb
Add support for journal disk hot add/remove. Mostly trival checks in md
part. The raid5 part is a little tricky. For hot-remove, we can't wait
pending write as it's called from raid5d. The wait will cause deadlock.
We simplily fail the hot-remove. A hot-remove retry can success
eventually since if journal disk is faulty all pending write will be
failed and finish. For hot-add, since an array supporting journal but
without journal disk will be marked read-only, we are safe to hot add
journal without stopping IO (should be read IO, while journal only
handles write IO).
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/md.c | 44 +++++++++++++++++++++++++++++++++-----------
drivers/md/raid5.c | 31 +++++++++++++++++++++++--------
2 files changed, 56 insertions(+), 19 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 807095f..de9e3a1 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2053,7 +2053,9 @@ static int bind_rdev_to_array(struct md_rdev *rdev, struct mddev *mddev)
/* make sure rdev->sectors exceeds mddev->dev_sectors */
if (rdev->sectors && (mddev->dev_sectors == 0 ||
- rdev->sectors < mddev->dev_sectors)) {
+ rdev->sectors < mddev->dev_sectors) &&
+ !(test_bit(MD_HAS_JOURNAL, &mddev->flags) &&
+ test_bit(Journal, &rdev->flags))) {
if (mddev->pers) {
/* Cannot change size, so fail
* If mddev->level <= 0, then we don't care
@@ -2084,7 +2086,9 @@ static int bind_rdev_to_array(struct md_rdev *rdev, struct mddev *mddev)
}
}
rcu_read_unlock();
- if (mddev->max_disks && rdev->desc_nr >= mddev->max_disks) {
+ if (mddev->max_disks && rdev->desc_nr >= mddev->max_disks &&
+ !(test_bit(MD_HAS_JOURNAL, &mddev->flags) &&
+ test_bit(Journal, &rdev->flags))) {
printk(KERN_WARNING "md: %s: array is limited to %d devices\n",
mdname(mddev), mddev->max_disks);
return -EBUSY;
@@ -6031,8 +6035,25 @@ static int add_new_disk(struct mddev *mddev, mdu_disk_info_t *info)
else
clear_bit(WriteMostly, &rdev->flags);
- if (info->state & (1<<MD_DISK_JOURNAL))
+ if (info->state & (1<<MD_DISK_JOURNAL)) {
+ struct md_rdev *rdev2;
+ bool has_journal = false;
+
+ /* make sure no existing journal disk */
+ rcu_read_lock();
+ rdev_for_each(rdev2, mddev) {
+ if (test_bit(Journal, &rdev2->flags)) {
+ has_journal = true;
+ break;
+ }
+ }
+ rcu_read_unlock();
+ if (has_journal) {
+ export_rdev(rdev);
+ return -EBUSY;
+ }
set_bit(Journal, &rdev->flags);
+ }
/*
* check whether the device shows up in other nodes
*/
@@ -8162,19 +8183,20 @@ static int remove_and_add_spares(struct mddev *mddev,
continue;
if (test_bit(Faulty, &rdev->flags))
continue;
- if (test_bit(Journal, &rdev->flags))
- continue;
- if (mddev->ro &&
- ! (rdev->saved_raid_disk >= 0 &&
- !test_bit(Bitmap_sync, &rdev->flags)))
- continue;
+ if (!test_bit(Journal, &rdev->flags)) {
+ if (mddev->ro &&
+ ! (rdev->saved_raid_disk >= 0 &&
+ !test_bit(Bitmap_sync, &rdev->flags)))
+ continue;
- rdev->recovery_offset = 0;
+ rdev->recovery_offset = 0;
+ }
if (mddev->pers->
hot_add_disk(mddev, rdev) == 0) {
if (sysfs_link_rdev(mddev, rdev))
/* failure here is OK */;
- spares++;
+ if (!test_bit(Journal, &rdev->flags))
+ spares++;
md_new_event(mddev);
set_bit(MD_CHANGE_DEVS, &mddev->flags);
}
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 704ef7f..f8f3d52 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7141,14 +7141,16 @@ static int raid5_remove_disk(struct mddev *mddev, struct md_rdev *rdev)
struct disk_info *p = conf->disks + number;
print_raid5_conf(conf);
- if (test_bit(Journal, &rdev->flags)) {
+ if (test_bit(Journal, &rdev->flags) && conf->log) {
/*
- * journal disk is not removable, but we need give a chance to
- * update superblock of other disks. Otherwise journal disk
- * will be considered as 'fresh'
+ * we can't wait pending write here, as this is called in
+ * raid5d, wait will deadlock.
*/
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
- return -EINVAL;
+ if (atomic_read(&mddev->writes_pending))
+ return -EBUSY;
+ r5l_exit_log(conf->log);
+ conf->log = NULL;
+ return 0;
}
if (rdev == p->rdev)
rdevp = &p->rdev;
@@ -7212,8 +7214,21 @@ static int raid5_add_disk(struct mddev *mddev, struct md_rdev *rdev)
int first = 0;
int last = conf->raid_disks - 1;
- if (test_bit(Journal, &rdev->flags))
- return -EINVAL;
+ if (test_bit(Journal, &rdev->flags)) {
+ char b[BDEVNAME_SIZE];
+ if (conf->log)
+ return -EBUSY;
+
+ rdev->raid_disk = mddev->raid_disks;
+ /*
+ * The array is in readonly mode if journal is missing, so no
+ * write requests running. We should be safe
+ */
+ r5l_init_log(conf, rdev);
+ printk(KERN_INFO"md/raid:%s: using device %s as journal\n",
+ mdname(mddev), bdevname(rdev->bdev, b));
+ return 0;
+ }
if (mddev->recovery_disabled == conf->recovery_disabled)
return -EBUSY;
--
2.4.6
^ permalink raw reply related
* Re: [PATCH 2/2] md: dm-crypt: Optimize the dm-crypt for XTS mode
From: Baolin Wang @ 2015-12-15 2:56 UTC (permalink / raw)
To: Milan Broz
Cc: Jens Axboe, Alasdair G Kergon, Mike Snitzer, dm-devel, neilb,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, Mark Brown, Arnd Bergmann, linux-block,
linux-raid, LKML
In-Reply-To: <566F5168.7010905@gmail.com>
>> + /*
>> + * Here we need to check if it can be encrypted or decrypted with
>> + * bulk block, which means these encryption modes don't need IV or
>> + * just need one initial IV. For bulk mode, we can expand the
>> + * scatterlist entries to map the bio, then send all the scatterlists
>> + * to the hardware engine at one time to improve the crypto engine
>> + * efficiency. But it does not fit for other encryption modes, it has
>> + * to do encryption and decryption sector by sector because every
>> + * sector has different IV.
>> + */
>> + if (!strcmp(chainmode, "ecb") || !strcmp(chainmode, "xts"))
>> + cc->bulk_crypto = 1;
>> + else
>> + cc->bulk_crypto = 0;n
>
> It is perfectly fine to use another IV even for XTS mode (despite it is not needed).
> You can use ESSIV for example, or benbi (big-endian variant of plain IV).
> (And it is really used for some LUKS devices.)
>
> How it is supposed to work in this case?
> If I read this patch correctly, it will completely corrupt data in this case because
> it expects plain (consecutive) IV...
The XTS mode can limit maximum size of each encrypted data unit
(typically a sector or disk block) to 2^20 AES blocks, so we can use
one bio as one encrypted data unit (we don't do it sector by sector).
It can generate one IV for each separate encrypted data unit. Please
correct me if I misunderstand something. Thanks.
>
> Also how it handles 32bit plain IV (that restart after 2TB)? (IOW plain IV, not plain64).
>
> Milan
>
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-15 2:36 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, John Stoffel, Linux-RAID
In-Reply-To: <CAE9DZUSbt7Kfwd9S3K_SXY7fVRk-vq5RhrPzKs4XO8uhyfPh3Q@mail.gmail.com>
On Mon, Dec 14, 2015 at 5:25 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Mon, Dec 14, 2015 at 4:17 PM, Mark Knecht <markknecht@gmail.com> wrote:
>>
>>
>> On Mon, Dec 14, 2015 at 2:05 PM, Dallas Clement <dallas.a.clement@gmail.com>
>> wrote:
>>>
>>> <SNIP>
>>>
>>> The speeds I am seeing with dd are definitely faster. I was getting
>>> about 333 MB/s when writing bs=2048k which was not chunk aligned.
>>> When writing bs=1408k I am getting at least 750 MB/s. Reducing the
>>> RMWs certainly did help. But this write speed is still far short of
>>> the (12 - 1) * 150 MB/s = 1650 MB/s I am expecting for minimal to no
>>> RMWs. I probably am not able to saturate the RAID device with dd
>>> though.
>>
>> But then you get back to all the questions about where you are on the drives
>> physically (inside vs outside) and all the potential bottlenecks in the
>> hardware. It
>> might not be 'far short' if you're on the inside of the drive.
>>
>> I have no idea about what vintage Cougar Point machine you have but there
>> are some reports about bugs that caused issues with a couple of the
>> higher hard drive interface ports on some earlier machines. Your nature
>> seems to be to generally build the largest configurations you can but Phil
>> suggested earlier and it might be appropriate here to disconnect a bunch of
>> drives and then do 1 drive, 2 drives, 3 drives and measure speeds. I seem
>> to remember you saying something about it working well until you added the
>> last drive so if you go this way I'd suggest physically disconnecting drives
>> you are not testing, booting up, testing, powering down, adding another
>> drive, etc.
>
> Hi Mark
>
>> But then you get back to all the questions about where you are on the drives
>> physically (inside vs outside) and all the potential bottlenecks in the
>> hardware. It
>> might not be 'far short' if you're on the inside of the drive.
>
> Perhaps. But I was getting about 95 MB/s on the inside when I
> measured earlier. Even with this number the write speed for RAID 5
> should be around 11 * 95 = 1045 MB/s. Also, when I was running fio on
> individual disks concurrently, adding one in at a time, iostat was
> showing wMB/s to be around 160-170 MB/s.
>
>> I have no idea about what vintage Cougar Point machine you have but there
>> are some reports about bugs that caused issues with a couple of the
>> higher hard drive interface ports on some earlier machines.
>
> Hmm, I will need to look into that some more.
>
>> I'd suggest physically disconnecting drives you are not testing, booting up, testing, powering down, adding another drive, etc.
>
> Yes, I haven't tried that yet with RAID 5 or 6. I'll give it a shot
> maybe starting with 4 disks, adding one at a time and measure the
> write speed.
>
> On another point, this blktrace program sure is neat! A wealth of info here.
Hi Everyone. I have some very interesting news to report. I did a
little bit more playing around with fio, doing sequential writes to a
RAID 5 device with all 12 disks. I kept the block size at the 128K
chunk aligned value of 1408K. But this time I varied the queue depth.
These are my results for writing a 10 GB of data:
iodepth=1 => 642 MB/s, # of RMWs = 11
iodepth=4 => 1108 MB/s, # of RMWs = 6
iodepth=8 => 895 MB/s, # of RMWs = 7
iodepth=16 => 855 MB/s, # of RMWs = 11
iodepth=32 => 936 MB/s, # of RMWs = 11
iodepth=64 => 551 MB/s, # of RMWs = 5606
iodepth=128 => 554 MB/s, # of RMWs = 6333
As you can see, something goes terribly wrong with async i/o with
iodepth >= 64. Btw, not to be contentious Phil, I have checked
multiple fio man pages and they clearly indicate that iodepth is for
async i/o which this is (libaio). I don't see any mention of
sequential writes being prohibited with async i/o. See
https://github.com/axboe/fio/blob/master/HOWTO. However, maybe I'm
missing something and it sure looks from these results that there may
be a connection.
This is my fio job config:
[job]
ioengine=libaio
iodepth=128
prio=0
rw=write
bs=1408k
filename=/dev/md10
numjobs=1
size=10g
direct=1
invalidate=1
Incidentally, the very best write speed here (1108 MB/s with
iodepth=4) comes out to about 100 MB/s per disk, which is pretty close
to the worst case inner disk speed of 95.5 MB/s I had recorded
earlier.
^ permalink raw reply
* [PATCH v2 2/2] [mdadm] recreate journal in mdadm
From: Song Liu @ 2015-12-15 1:43 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1450143823-1145846-1-git-send-email-songliubraving@fb.com>
This patch tries recreates missing/faulty journal in mdadm.
Example:
./mdadm --fail /dev/md1 /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md1
./mdadm --stop /dev/md1
mdadm: stopped /dev/md1
./mdadm -A --scan --force
mdadm: Journal is missing or stale, starting array read only.
mdadm: /dev/md/1 has been started with 15 drives.
./mdadm --add-journal /dev/md1 /dev/sdb2
mdadm: added /dev/sdb2
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
Manage.c | 42 +++++++++++++++++++++++++++++++++++++++---
ReadMe.c | 1 +
mdadm.c | 8 ++++++++
mdadm.h | 1 +
super1.c | 3 ++-
5 files changed, 51 insertions(+), 4 deletions(-)
diff --git a/Manage.c b/Manage.c
index 2df303d..4540fac 100644
--- a/Manage.c
+++ b/Manage.c
@@ -825,7 +825,8 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
}
/* Make sure device is large enough */
- if (tst->sb &&
+ if (dv->disposition != 'j' && /* skip size check for Journal */
+ tst->sb &&
tst->ss->avail_size(tst, ldsize/512, INVALID_SECTORS) <
array_size) {
if (dv->disposition == 'M')
@@ -929,8 +930,31 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
else
disc.number = raid_slot;
disc.state = 0;
+
+ /* only add journal to array that supports journaling */
+ if (dv->disposition == 'j') {
+ struct mdinfo mdi;
+ struct mdinfo *mdp;
+
+ mdp = sysfs_read(fd, NULL, GET_ARRAY_STATE);
+
+ if (strncmp(mdp->sysfs_array_state, "readonly", 8) != 0) {
+ pr_err("%s is not readonly, cannot add journal.\n", devname);
+ return -1;
+ }
+
+ tst->ss->getinfo_super(tst, &mdi, NULL);
+ if (mdi.journal_device_required == 0) {
+ pr_err("%s does not support journal device.\n", devname);
+ return -1;
+ }
+ disc.raid_disk = array->raid_disks;
+ }
+
if (array->not_persistent==0) {
int dfd;
+ if (dv->disposition == 'j')
+ disc.state |= (1 << MD_DISK_JOURNAL) | (1 << MD_DISK_SYNC);
if (dv->writemostly == 1)
disc.state |= 1 << MD_DISK_WRITEMOSTLY;
dfd = dev_open(dv->devname, O_RDWR | O_EXCL|O_DIRECT);
@@ -1041,10 +1065,20 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
} else {
tst->ss->free_super(tst);
if (ioctl(fd, ADD_NEW_DISK, &disc)) {
- pr_err("add new device failed for %s as %d: %s\n",
- dv->devname, j, strerror(errno));
+ if (dv->disposition == 'j')
+ pr_err("Failed to hot add %s as journal, "
+ "please try restart %s.\n", dv->devname, devname);
+ else
+ pr_err("add new device failed for %s as %d: %s\n",
+ dv->devname, j, strerror(errno));
return -1;
}
+ if (dv->disposition == 'j') {
+ pr_err("Journal added successfully, making %s read-write\n", devname);
+ if (Manage_ro(devname, fd, -1))
+ pr_err("Failed to make %s read-write\n", devname);
+ }
+
}
if (verbose >= 0)
pr_err("added %s\n", dv->devname);
@@ -1277,6 +1311,7 @@ int Manage_subdevs(char *devname, int fd,
* try HOT_ADD_DISK
* If that fails EINVAL, try ADD_NEW_DISK
* 'S' - add the device as a spare - don't try re-add
+ * 'j' - add the device as a journal device
* 'A' - re-add the device
* 'r' - remove the device: HOT_REMOVE_DISK
* device can be 'faulty' or 'detached' in which case all
@@ -1509,6 +1544,7 @@ int Manage_subdevs(char *devname, int fd,
goto abort;
case 'a':
case 'S': /* --add-spare */
+ case 'j': /* --add-journal */
case 'A':
case 'M': /* --re-add missing */
case 'F': /* --re-add faulty */
diff --git a/ReadMe.c b/ReadMe.c
index fb5a671..566fc8d 100644
--- a/ReadMe.c
+++ b/ReadMe.c
@@ -157,6 +157,7 @@ struct option long_options[] = {
/* Management */
{"add", 0, 0, Add},
{"add-spare", 0, 0, AddSpare},
+ {"add-journal", 0, 0, AddJournal},
{"remove", 0, 0, Remove},
{"fail", 0, 0, Fail},
{"set-faulty",0, 0, Fail},
diff --git a/mdadm.c b/mdadm.c
index f56a8cf..feec3b7 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -190,6 +190,7 @@ int main(int argc, char *argv[])
case 'a':
case Add:
case AddSpare:
+ case AddJournal:
case 'r':
case Remove:
case Replace:
@@ -925,6 +926,13 @@ int main(int argc, char *argv[])
case O(MANAGE,AddSpare): /* add drive - never re-add */
devmode = 'S';
continue;
+ case O(MANAGE,AddJournal): /* add journal */
+ if (s.journaldisks && (s.level < 4 || s.level > 6)) {
+ pr_err("--add-journal is only supported for RAID level 4/5/6.\n");
+ exit(2);
+ }
+ devmode = 'j';
+ continue;
case O(MANAGE,ReAdd):
devmode = 'A';
continue;
diff --git a/mdadm.h b/mdadm.h
index 21fe789..477ef18 100755
--- a/mdadm.h
+++ b/mdadm.h
@@ -370,6 +370,7 @@ enum special_options {
ManageOpt,
Add,
AddSpare,
+ AddJournal,
Remove,
Fail,
Replace,
diff --git a/super1.c b/super1.c
index 1735c2d..893f9bf 100644
--- a/super1.c
+++ b/super1.c
@@ -1713,7 +1713,8 @@ static int write_init_super1(struct supertype *st)
if (rfd >= 0)
close(rfd);
- sb->events = 0;
+ if (!(di->disk.state & (1<<MD_DISK_JOURNAL)))
+ sb->events = 0;
refst = dup_super(st);
if (load_super1(refst, di->fd, NULL)==0) {
--
2.4.6
^ permalink raw reply related
* [PATCH v2 1/2] add sysfs_array_state to struct mdinfo
From: Song Liu @ 2015-12-15 1:43 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1450143823-1145846-1-git-send-email-songliubraving@fb.com>
Add sysfs_array_state to struct mdinfo, and add GET_ARRAY_STATE to
options of sysfs_read.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
mdadm.h | 4 ++++
sysfs.c | 7 +++++++
2 files changed, 11 insertions(+)
diff --git a/mdadm.h b/mdadm.h
index 840a359..21fe789 100755
--- a/mdadm.h
+++ b/mdadm.h
@@ -305,6 +305,9 @@ struct mdinfo {
#define DS_REMOVE 1024
#define DS_UNBLOCK 2048
int prev_state, curr_state, next_state;
+
+ /* info read from sysfs */
+ char sysfs_array_state[20];
};
struct createinfo {
@@ -588,6 +591,7 @@ enum sysfs_read_flags {
GET_SIZE = (1 << 22),
GET_STATE = (1 << 23),
GET_ERROR = (1 << 24),
+ GET_ARRAY_STATE = (1 << 25),
};
/* If fd >= 0, get the array it is open on,
diff --git a/sysfs.c b/sysfs.c
index f1fd610..2600343 100644
--- a/sysfs.c
+++ b/sysfs.c
@@ -230,6 +230,13 @@ struct mdinfo *sysfs_read(int fd, char *devnm, unsigned long options)
goto abort;
}
+ if (options & GET_ARRAY_STATE) {
+ strcpy(base, "array_state");
+ if (load_sys(fname, sra->sysfs_array_state))
+ goto abort;
+ } else
+ sra->sysfs_array_state[0] = 0;
+
if (! (options & GET_DEVS))
return sra;
--
2.4.6
^ permalink raw reply related
* [PATCH v2 0/2] mdadm: re-add journal to array with bad journal
From: Song Liu @ 2015-12-15 1:43 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
These patches enable recreation of missing/faulty journal. This requires
kernel changes as well, which Shaohua will send out later.
Song Liu (2):
add sysfs_array_state to struct mdinfo
[mdadm] recreate journal in mdadm
Manage.c | 42 +++++++++++++++++++++++++++++++++++++++---
ReadMe.c | 1 +
mdadm.c | 8 ++++++++
mdadm.h | 5 +++++
super1.c | 3 ++-
sysfs.c | 7 +++++++
6 files changed, 62 insertions(+), 4 deletions(-)
--
2.4.6
^ permalink raw reply
* Re: [PATCH 2/2] md: dm-crypt: Optimize the dm-crypt for XTS mode
From: Milan Broz @ 2015-12-14 23:31 UTC (permalink / raw)
To: Baolin Wang, axboe, agk, snitzer, dm-devel
Cc: neilb, dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, broonie, arnd, linux-block, linux-raid,
linux-kernel
In-Reply-To: <02be0f42bf2d3c3d27b43bc050a783582b7af733.1450080755.git.baolin.wang@linaro.org>
On 12/14/2015 09:23 AM, Baolin Wang wrote:
> In now dm-crypt code, it is ineffective to map one bio with just only
> one scatterlist at one time for XTS mode. We can use multiple scatterlists
> to map the whole bio and send all scatterlists of one bio to crypto engine
> to encrypt or decrypt, which can improve the hardware engine's efficiency.
...
> + /*
> + * Here we need to check if it can be encrypted or decrypted with
> + * bulk block, which means these encryption modes don't need IV or
> + * just need one initial IV. For bulk mode, we can expand the
> + * scatterlist entries to map the bio, then send all the scatterlists
> + * to the hardware engine at one time to improve the crypto engine
> + * efficiency. But it does not fit for other encryption modes, it has
> + * to do encryption and decryption sector by sector because every
> + * sector has different IV.
> + */
> + if (!strcmp(chainmode, "ecb") || !strcmp(chainmode, "xts"))
> + cc->bulk_crypto = 1;
> + else
> + cc->bulk_crypto = 0;n
It is perfectly fine to use another IV even for XTS mode (despite it is not needed).
You can use ESSIV for example, or benbi (big-endian variant of plain IV).
(And it is really used for some LUKS devices.)
How it is supposed to work in this case?
If I read this patch correctly, it will completely corrupt data in this case because
it expects plain (consecutive) IV...
Also how it handles 32bit plain IV (that restart after 2TB)? (IOW plain IV, not plain64).
Milan
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-14 23:25 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, John Stoffel, Linux-RAID
In-Reply-To: <CAK2H+ecMvDLdYLhMtMQbP7Ygw-VohG7LGZ2n7H+LAXQ1waJK3A@mail.gmail.com>
On Mon, Dec 14, 2015 at 4:17 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Mon, Dec 14, 2015 at 2:05 PM, Dallas Clement <dallas.a.clement@gmail.com>
> wrote:
>>
>> <SNIP>
>>
>> The speeds I am seeing with dd are definitely faster. I was getting
>> about 333 MB/s when writing bs=2048k which was not chunk aligned.
>> When writing bs=1408k I am getting at least 750 MB/s. Reducing the
>> RMWs certainly did help. But this write speed is still far short of
>> the (12 - 1) * 150 MB/s = 1650 MB/s I am expecting for minimal to no
>> RMWs. I probably am not able to saturate the RAID device with dd
>> though.
>
> But then you get back to all the questions about where you are on the drives
> physically (inside vs outside) and all the potential bottlenecks in the
> hardware. It
> might not be 'far short' if you're on the inside of the drive.
>
> I have no idea about what vintage Cougar Point machine you have but there
> are some reports about bugs that caused issues with a couple of the
> higher hard drive interface ports on some earlier machines. Your nature
> seems to be to generally build the largest configurations you can but Phil
> suggested earlier and it might be appropriate here to disconnect a bunch of
> drives and then do 1 drive, 2 drives, 3 drives and measure speeds. I seem
> to remember you saying something about it working well until you added the
> last drive so if you go this way I'd suggest physically disconnecting drives
> you are not testing, booting up, testing, powering down, adding another
> drive, etc.
Hi Mark
> But then you get back to all the questions about where you are on the drives
> physically (inside vs outside) and all the potential bottlenecks in the
> hardware. It
> might not be 'far short' if you're on the inside of the drive.
Perhaps. But I was getting about 95 MB/s on the inside when I
measured earlier. Even with this number the write speed for RAID 5
should be around 11 * 95 = 1045 MB/s. Also, when I was running fio on
individual disks concurrently, adding one in at a time, iostat was
showing wMB/s to be around 160-170 MB/s.
> I have no idea about what vintage Cougar Point machine you have but there
> are some reports about bugs that caused issues with a couple of the
> higher hard drive interface ports on some earlier machines.
Hmm, I will need to look into that some more.
> I'd suggest physically disconnecting drives you are not testing, booting up, testing, powering down, adding another drive, etc.
Yes, I haven't tried that yet with RAID 5 or 6. I'll give it a shot
maybe starting with 4 disks, adding one at a time and measure the
write speed.
On another point, this blktrace program sure is neat! A wealth of info here.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Tommy Apel @ 2015-12-14 22:31 UTC (permalink / raw)
To: Dallas Clement; +Cc: Linux-RAID
In-Reply-To: <CAE9DZUT1v+CFZOs33CC+JrWcX_WHBu+WW78AynkWqJN+LLoqDA@mail.gmail.com>
On Mon, 2015-12-14 at 16:05 -0600, Dallas Clement wrote:
> On Mon, Dec 14, 2015 at 3:20 PM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
> > On Mon, Dec 14, 2015 at 3:02 PM, Mark Knecht <markknecht@gmail.com> wrote:
> > >
> > >
> > > On Mon, Dec 14, 2015 at 12:55 PM, Dallas Clement
> > > <dallas.a.clement@gmail.com> wrote:
> > > >
> > > > On Mon, Dec 14, 2015 at 2:40 PM, Mark Knecht <markknecht@gmail.com> wrote:
> > > > >
> > > > >
> > > > > On Mon, Dec 14, 2015 at 12:14 PM, Dallas Clement
> > > > > <dallas.a.clement@gmail.com> wrote:
> > > > > >
> > > > > > <SNIP>
> > > > > >
> > > > > > Hi Phil, I ran blktrace while writing with dd to a RAID 5 device with
> > > > > > 12 disks. My chunk size is 128K. So I set my block size to 128K *
> > > > > > (12-2) = 1280k. Here is the dd command I ran.
> > > > >
> > > > > Just curious but for my own knowledge if it's RAID5 why is it 12-2?
> > > > >
> > > > > - Mark
> > > >
> > > > > Just curious but for my own knowledge if it's RAID5 why is it 12-2?
> > > >
> > > > Shouldn't be. It should have been 12-1 or writing 1408k. Boy do I
> > > > feel dumb. Anyhow, when writing this value, no more RMWs. Yay!
> > >
> > > I wasn't going to be so bold as to suggest the RMW's would go away but I'm
> > > glad they did.
> > >
> > > So, now you can presumably gather new data looking at speed and post that,
> > > correct?
> > >
> > > Cheers,
> > > Mark
> >
> > Hmm, I think I may have spoke to soon. I did a speed test using fio
> > this time, same bs=1408k. I see lots of RMWs in the trace this time.
> > I did another larger dd transfer too, and I see some RMWs but not very
> > many - maybe 4 or 5 for a 20GB transfer.
> >
> > It looks like the LBAs are increasing for the writes to the disks.
> >
> > 9,10 2 2816 0.737523948 27410 Q WS 965888 + 256 [dd]
> > 9,10 2 2817 0.737620583 27410 Q WS 966144 + 256 [dd]
> > 9,10 2 2818 0.737630651 27410 Q WS 966400 + 256 [dd]
> > 9,10 2 2819 0.737641625 27410 Q WS 966656 + 256 [dd]
> > 9,10 2 2820 0.737651603 27410 Q WS 966912 + 256 [dd]
> > 9,10 2 2821 0.737662735 27410 Q WS 967168 + 256 [dd]
> > 9,10 2 2822 0.737672709 27410 Q WS 967424 + 256 [dd]
> > 9,10 2 2823 0.737683881 27410 Q WS 967680 + 256 [dd]
> > 9,10 2 2824 0.737693896 27410 Q WS 967936 + 256 [dd]
> > 9,10 2 2825 0.737704484 27410 Q WS 968192 + 256 [dd]
> > 9,10 2 2826 0.737714348 27410 Q WS 968448 + 256 [dd]
> >
> > The dd transfers do seem faster when using bs=1408k. But need to
> > collect some more data.
>
> The speeds I am seeing with dd are definitely faster. I was getting
> about 333 MB/s when writing bs=2048k which was not chunk aligned.
> When writing bs=1408k I am getting at least 750 MB/s. Reducing the
> RMWs certainly did help. But this write speed is still far short of
> the (12 - 1) * 150 MB/s = 1650 MB/s I am expecting for minimal to no
> RMWs. I probably am not able to saturate the RAID device with dd
> though.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
750MB/s ~ 6000Mbit/s which is most likely the limitation of your expander chip
--
/Tommy
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-14 22:05 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, John Stoffel, Linux-RAID
In-Reply-To: <CAE9DZUQHBycc5+Z2YrJtWZRYxOUMu3pgnaEQSrsyeCZEv8vndA@mail.gmail.com>
On Mon, Dec 14, 2015 at 3:20 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Mon, Dec 14, 2015 at 3:02 PM, Mark Knecht <markknecht@gmail.com> wrote:
>>
>>
>> On Mon, Dec 14, 2015 at 12:55 PM, Dallas Clement
>> <dallas.a.clement@gmail.com> wrote:
>>>
>>> On Mon, Dec 14, 2015 at 2:40 PM, Mark Knecht <markknecht@gmail.com> wrote:
>>> >
>>> >
>>> > On Mon, Dec 14, 2015 at 12:14 PM, Dallas Clement
>>> > <dallas.a.clement@gmail.com> wrote:
>>> >>
>>> >> <SNIP>
>>> >>
>>> >> Hi Phil, I ran blktrace while writing with dd to a RAID 5 device with
>>> >> 12 disks. My chunk size is 128K. So I set my block size to 128K *
>>> >> (12-2) = 1280k. Here is the dd command I ran.
>>> >
>>> > Just curious but for my own knowledge if it's RAID5 why is it 12-2?
>>> >
>>> > - Mark
>>>
>>> > Just curious but for my own knowledge if it's RAID5 why is it 12-2?
>>>
>>> Shouldn't be. It should have been 12-1 or writing 1408k. Boy do I
>>> feel dumb. Anyhow, when writing this value, no more RMWs. Yay!
>>
>> I wasn't going to be so bold as to suggest the RMW's would go away but I'm
>> glad they did.
>>
>> So, now you can presumably gather new data looking at speed and post that,
>> correct?
>>
>> Cheers,
>> Mark
>
> Hmm, I think I may have spoke to soon. I did a speed test using fio
> this time, same bs=1408k. I see lots of RMWs in the trace this time.
> I did another larger dd transfer too, and I see some RMWs but not very
> many - maybe 4 or 5 for a 20GB transfer.
>
> It looks like the LBAs are increasing for the writes to the disks.
>
> 9,10 2 2816 0.737523948 27410 Q WS 965888 + 256 [dd]
> 9,10 2 2817 0.737620583 27410 Q WS 966144 + 256 [dd]
> 9,10 2 2818 0.737630651 27410 Q WS 966400 + 256 [dd]
> 9,10 2 2819 0.737641625 27410 Q WS 966656 + 256 [dd]
> 9,10 2 2820 0.737651603 27410 Q WS 966912 + 256 [dd]
> 9,10 2 2821 0.737662735 27410 Q WS 967168 + 256 [dd]
> 9,10 2 2822 0.737672709 27410 Q WS 967424 + 256 [dd]
> 9,10 2 2823 0.737683881 27410 Q WS 967680 + 256 [dd]
> 9,10 2 2824 0.737693896 27410 Q WS 967936 + 256 [dd]
> 9,10 2 2825 0.737704484 27410 Q WS 968192 + 256 [dd]
> 9,10 2 2826 0.737714348 27410 Q WS 968448 + 256 [dd]
>
> The dd transfers do seem faster when using bs=1408k. But need to
> collect some more data.
The speeds I am seeing with dd are definitely faster. I was getting
about 333 MB/s when writing bs=2048k which was not chunk aligned.
When writing bs=1408k I am getting at least 750 MB/s. Reducing the
RMWs certainly did help. But this write speed is still far short of
the (12 - 1) * 150 MB/s = 1650 MB/s I am expecting for minimal to no
RMWs. I probably am not able to saturate the RAID device with dd
though.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-14 21:20 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, John Stoffel, Linux-RAID
In-Reply-To: <CAK2H+ed-3Z8SR20t8rpt3Fb48c3X2Jft=qZoiY9emC2nQww1xQ@mail.gmail.com>
On Mon, Dec 14, 2015 at 3:02 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Mon, Dec 14, 2015 at 12:55 PM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
>>
>> On Mon, Dec 14, 2015 at 2:40 PM, Mark Knecht <markknecht@gmail.com> wrote:
>> >
>> >
>> > On Mon, Dec 14, 2015 at 12:14 PM, Dallas Clement
>> > <dallas.a.clement@gmail.com> wrote:
>> >>
>> >> <SNIP>
>> >>
>> >> Hi Phil, I ran blktrace while writing with dd to a RAID 5 device with
>> >> 12 disks. My chunk size is 128K. So I set my block size to 128K *
>> >> (12-2) = 1280k. Here is the dd command I ran.
>> >
>> > Just curious but for my own knowledge if it's RAID5 why is it 12-2?
>> >
>> > - Mark
>>
>> > Just curious but for my own knowledge if it's RAID5 why is it 12-2?
>>
>> Shouldn't be. It should have been 12-1 or writing 1408k. Boy do I
>> feel dumb. Anyhow, when writing this value, no more RMWs. Yay!
>
> I wasn't going to be so bold as to suggest the RMW's would go away but I'm
> glad they did.
>
> So, now you can presumably gather new data looking at speed and post that,
> correct?
>
> Cheers,
> Mark
Hmm, I think I may have spoke to soon. I did a speed test using fio
this time, same bs=1408k. I see lots of RMWs in the trace this time.
I did another larger dd transfer too, and I see some RMWs but not very
many - maybe 4 or 5 for a 20GB transfer.
It looks like the LBAs are increasing for the writes to the disks.
9,10 2 2816 0.737523948 27410 Q WS 965888 + 256 [dd]
9,10 2 2817 0.737620583 27410 Q WS 966144 + 256 [dd]
9,10 2 2818 0.737630651 27410 Q WS 966400 + 256 [dd]
9,10 2 2819 0.737641625 27410 Q WS 966656 + 256 [dd]
9,10 2 2820 0.737651603 27410 Q WS 966912 + 256 [dd]
9,10 2 2821 0.737662735 27410 Q WS 967168 + 256 [dd]
9,10 2 2822 0.737672709 27410 Q WS 967424 + 256 [dd]
9,10 2 2823 0.737683881 27410 Q WS 967680 + 256 [dd]
9,10 2 2824 0.737693896 27410 Q WS 967936 + 256 [dd]
9,10 2 2825 0.737704484 27410 Q WS 968192 + 256 [dd]
9,10 2 2826 0.737714348 27410 Q WS 968448 + 256 [dd]
The dd transfers do seem faster when using bs=1408k. But need to
collect some more data.
^ permalink raw reply
* Re: [PATCH 1/2] raid5-cache: use a bio_set
From: Christoph Hellwig @ 2015-12-14 21:12 UTC (permalink / raw)
To: NeilBrown; +Cc: shli, linux-raid
In-Reply-To: <87bna0r0ez.fsf@notabene.neil.brown.name>
On Wed, Dec 09, 2015 at 10:22:12AM +1100, NeilBrown wrote:
> > +/*
> > + * We only need 2 bios per I/O unit to make progress, but ensure we
> > + * have a few more available to not get too tight.
> > + */
> > +#define R5L_POOL_SIZE 1024
> > +
>
> I'm really suspicious of big pool sizes.
> The memory allocated to the pool is almost never used - only where no
> other memory is available - so large pools are largely wasted.
I originally had 16 here, andd incremented it for some debugging that
stuck. I retested with 4 and that works as well. That will be included
in the next resend.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-14 20:55 UTC (permalink / raw)
To: Mark Knecht; +Cc: Phil Turmel, John Stoffel, Linux-RAID
In-Reply-To: <CAK2H+edazVORrVovWDeTA8DmqUL+5HRH-AcRwg8KkMas=o+Cog@mail.gmail.com>
On Mon, Dec 14, 2015 at 2:40 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
>
> On Mon, Dec 14, 2015 at 12:14 PM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
>>
>> <SNIP>
>>
>> Hi Phil, I ran blktrace while writing with dd to a RAID 5 device with
>> 12 disks. My chunk size is 128K. So I set my block size to 128K *
>> (12-2) = 1280k. Here is the dd command I ran.
>
> Just curious but for my own knowledge if it's RAID5 why is it 12-2?
>
> - Mark
> Just curious but for my own knowledge if it's RAID5 why is it 12-2?
Shouldn't be. It should have been 12-1 or writing 1408k. Boy do I
feel dumb. Anyhow, when writing this value, no more RMWs. Yay!
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-14 20:14 UTC (permalink / raw)
To: Phil Turmel; +Cc: John Stoffel, Mark Knecht, Linux-RAID
In-Reply-To: <566BA6E5.6030008@turmel.org>
On Fri, Dec 11, 2015 at 10:47 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/11/2015 09:55 PM, Dallas Clement wrote:
>
>> Right. I understand the fio iodepth is different than the hardware
>> queue depth. But the fio man page seems to only mention limitation on
>> synchronous operations which mine are not. I'm using direct=1 and
>> sync=0.
>
> You are confusing sequential and synchronous. The man page says it is
> ineffective for *sequential* operations, especially when direct=1.
>
>> I guess what I would really like to know is how I can achieve at or
>> near 100% utilization on the raid device and its member disks with
>> fio. Do I need to increase /sys/block/sd*/device/queue_depth and
>> /sys/block/sd*/queue/nr_requests to get more utilization?
>
> I don't know specifically. It seems to me that increasing queue depth
> adds resiliency in the face of data transfer timing jitter, but at the
> cost of more CPU overhead.
>
> I'm not convinced fio is the right workload, either. It's options are
> much more flexible for random I/O workloads. dd isn't perfect either,
> especially when writing zeroes -- it actually reads zeros over and over
> from the special device. For sequential operations I like dc3dd with
> its pat= wipe= mode. That'll only generate writes.
>
>>> That's why I suggested blktrace. Collect a trace while a single dd is
>>> writing to your raw array device. Compare the large writes submitted to
>>> the md device against the broken down writes submitted to the member
>>> devices.
>>
>> Sounds good. Will do. What signs of trouble should I be looking for?
>
> Look for strictly increasing logical block addresses in requests to the
> member devices. Any disruption in that will break optimum positioning
> for streaming throughput. Per device. Requests to the device have to be
> large enough and paced quickly enough to avoid starving the write head.
>
> Of course, any reads mixed in mean RMW cycles you didn't avoid. You
> shouldn't have any of those for sequential writes in chunk * (n-2)
> multiples.
>
> I know it's a bit hand-wavy, but you have more hardware to play with
> than I do :-)
>
> Phil
Hi Phil, I ran blktrace while writing with dd to a RAID 5 device with
12 disks. My chunk size is 128K. So I set my block size to 128K *
(12-2) = 1280k. Here is the dd command I ran.
# /usr/local/bin/dd if=/dev/zero of=/dev/md10 bs=1280k count=1000 oflag=direct
> Look for strictly increasing logical block addresses in requests to the
> member devices. Any disruption in that will break optimum positioning
> for streaming throughput. Per device. Requests to the device have to be
> large enough and paced quickly enough to avoid starving the write head.
I just ran blktrace and then blkparse after the write finished. I'm
new to blktrace so not really sure what I'm looking at. I wasn't able
to see the writes to individual disks.
> Of course, any reads mixed in mean RMW cycles you didn't avoid. You
> shouldn't have any of those for sequential writes in chunk * (n-2)
> multiples.
I did see lots of rmw's which I am assuming I should not be seeing if
everything is correctly aligned!
9,10 1 0 15.016034153 0 m N raid5 rmw 1536 5
9,10 1 0 15.016039816 0 m N raid5 rmw 1544 5
9,10 1 0 15.016042200 0 m N raid5 rmw 1552 5
9,10 1 0 15.016044241 0 m N raid5 rmw 1560 5
9,10 1 0 15.016046200 0 m N raid5 rmw 1568 5
9,10 1 0 15.016048096 0 m N raid5 rmw 1576 5
9,10 1 0 15.016049977 0 m N raid5 rmw 1584 5
9,10 1 0 15.016051851 0 m N raid5 rmw 1592 5
9,10 1 0 15.016054075 0 m N raid5 rmw 1600 5
9,10 1 0 15.016056042 0 m N raid5 rmw 1608 5
9,10 1 0 15.016057916 0 m N raid5 rmw 1616 5
9,10 1 0 15.016059809 0 m N raid5 rmw 1624 5
9,10 1 0 15.016061670 0 m N raid5 rmw 1632 5
9,10 1 0 15.016063578 0 m N raid5 rmw 1640 5
^ permalink raw reply
* Re: core dump on initial sync of raid10 please help
From: Micheal Blue @ 2015-12-14 19:31 UTC (permalink / raw)
To: Micheal Blue; +Cc: linux-raid
In-Reply-To: <trinity-c6018f35-ed0d-46be-b418-7799c4874238-1450100929825@3capp-mailcom-bs06>
> Sent: Monday, December 14, 2015 at 8:48 AM
> From: "Micheal Blue" <>
> To: linux-raid@vger.kernel.org
> Subject: core dump on initial sync of raid10 please help
>
> I noticed my initial sync when making a 4 disk raid10 is really slow. I looked in dmesg and found this.
> I do not know what it means. Suggestions please :)
>
> [ 600.959945] INFO: task md0_resync:676 blocked for more than 120 seconds.
> [ 600.959976] Not tainted 4.3.2-1-ARCH #1
> [ 600.959994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 600.960024] md0_resync D ffff88021fa956c0 0 676 2 0x00000000
> [ 600.960028] ffff8800ca143b40 0000000000000046 ffff880216659b80 ffff8800377244c0
> [ 600.960030] ffff8800ca144000 ffff88003765a000 ffff8800ca143b68 ffff88003765a100
> [ 600.960032] ffff88003765a000 ffff8800ca143b58 ffffffff8157f92a ffff88003765a0dc
> [ 600.960034] Call Trace:
> [ 600.960040] [<ffffffff8157f92a>] schedule+0x3a/0x90
> [ 600.960044] [<ffffffffa0548f52>] raise_barrier+0x122/0x190 [raid10]
> [ 600.960049] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
> [ 600.960051] [<ffffffffa054eea1>] sync_request+0x731/0x19b0 [raid10]
...
It seems that the kernel itself is to blame for this.
If I boot into kernel version 4.1.14 everything seems to be running fine as I create the array.
^ permalink raw reply
* Re: [PATCH] md/raid10: fix data corruption and crash during resync
From: Baldysiak, Pawel @ 2015-12-14 14:22 UTC (permalink / raw)
To: neilb@suse.de; +Cc: linux-raid@vger.kernel.org, Paszkiewicz, Artur
In-Reply-To: <20151104223353.GA99478@kernel.org>
Hi Neil,
Please look at the patch below.
Thanks,
Pawel Baldysiak
On Wed, 2015-11-04 at 14:33 -0800, Shaohua Li wrote:
> On Wed, Nov 04, 2015 at 05:30:30PM +0100, Artur Paszkiewicz wrote:
> > The commit c31df25f20e3 ("md/raid10: make sync_request_write() call
> > bio_copy_data()") replaced manual data copying with bio_copy_data()
> > but
> > it doesn't work as intended. The source bio (fbio) is already
> > processed,
> > so its bvec_iter has bi_size == 0 and bi_idx == bi_vcnt. Because
> > of
> > this, bio_copy_data() either does not copy anything, or worse,
> > copies
> > data from the ->bi_next bio if it is set. This causes wrong data
> > to be
> > written to drives during resync and sometimes lockups/crashes in
> > bio_copy_data():
> >
> > [ 517.338478] NMI watchdog: BUG: soft lockup - CPU#0 stuck for
> > 22s! [md126_raid10:3319]
> > [ 517.347324] Modules linked in: raid10 xt_CHECKSUM ipt_MASQUERADE
> > nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6
> > ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute
> > bridge stp llc ebtable_filter ebtables ip6table_nat
> > nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle
> > ip6table_security ip6table_raw ip6table_filter ip6_tables
> > iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat
> > nf_conntrack iptable_mangle iptable_security iptable_raw
> > iptable_filter ip_tables x86_pkg_temp_thermal coretemp kvm_intel
> > kvm crct10dif_pclmul crc32_pclmul cryptd shpchp pcspkr ipmi_si
> > ipmi_msghandler tpm_crb acpi_power_meter acpi_cpufreq ext4 mbcache
> > jbd2 sr_mod cdrom sd_mod e1000e ax88179_178a usbnet mii ahci
> > ata_generic crc32c_intel libahci ptp pata_acpi libata pps_core wmi
> > sunrpc dm_mirror dm_region_hash dm_log dm_mod
> > [ 517.440555] CPU: 0 PID: 3319 Comm: md126_raid10 Not tainted
> > 4.3.0-rc6+ #1
> > [ 517.448384] Hardware name: Intel Corporation PURLEY/PURLEY, BIOS
> > PLYDCRB1.86B.0055.D14.1509221924 09/22/2015
> > [ 517.459768] task: ffff880153773980 ti: ffff880150df8000 task.ti:
> > ffff880150df8000
> > [ 517.468529] RIP: 0010:[<ffffffff812e1888>] [<ffffffff812e1888>]
> > bio_copy_data+0xc8/0x3c0
> > [ 517.478164] RSP: 0018:ffff880150dfbc98 EFLAGS: 00000246
> > [ 517.484341] RAX: ffff880169356688 RBX: 0000000000001000 RCX:
> > 0000000000000000
> > [ 517.492558] RDX: 0000000000000000 RSI: ffffea0001ac2980 RDI:
> > ffffea0000d835c0
> > [ 517.500773] RBP: ffff880150dfbd08 R08: 0000000000000001 R09:
> > ffff880153773980
> > [ 517.508987] R10: ffff880169356600 R11: 0000000000001000 R12:
> > 0000000000010000
> > [ 517.517199] R13: 000000000000e000 R14: 0000000000000000 R15:
> > 0000000000001000
> > [ 517.525412] FS: 0000000000000000(0000)
> > GS:ffff880174a00000(0000) knlGS:0000000000000000
> > [ 517.534844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 517.541507] CR2: 00007f8a044d5fed CR3: 0000000169504000 CR4:
> > 00000000001406f0
> > [ 517.549722] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 517.557929] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
> > 0000000000000400
> > [ 517.566144] Stack:
> > [ 517.568626] ffff880174a16bc0 ffff880153773980 ffff880169356600
> > 0000000000000000
> > [ 517.577659] 0000000000000001 0000000000000001 ffff880153773980
> > ffff88016a61a800
> > [ 517.586715] ffff880150dfbcf8 0000000000000001 ffff88016dd209e0
> > 0000000000001000
> > [ 517.595773] Call Trace:
> > [ 517.598747] [<ffffffffa043ef95>] raid10d+0xfc5/0x1690 [raid10]
> > [ 517.605610] [<ffffffff816697ae>] ? __schedule+0x29e/0x8e2
> > [ 517.611987] [<ffffffff814ff206>] md_thread+0x106/0x140
> > [ 517.618072] [<ffffffff810c1d80>] ? wait_woken+0x80/0x80
> > [ 517.624252] [<ffffffff814ff100>] ? super_1_load+0x520/0x520
> > [ 517.630817] [<ffffffff8109ef89>] kthread+0xc9/0xe0
> > [ 517.636506] [<ffffffff8109eec0>] ?
> > flush_kthread_worker+0x70/0x70
> > [ 517.643653] [<ffffffff8166d99f>] ret_from_fork+0x3f/0x70
> > [ 517.649929] [<ffffffff8109eec0>] ?
> > flush_kthread_worker+0x70/0x70
> >
> > Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
> > ---
> > drivers/md/raid10.c | 4 +++-
> > 1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> > index 96f3659..23bbe61 100644
> > --- a/drivers/md/raid10.c
> > +++ b/drivers/md/raid10.c
> > @@ -1944,6 +1944,8 @@ static void sync_request_write(struct mddev
> > *mddev, struct r10bio *r10_bio)
> >
> > first = i;
> > fbio = r10_bio->devs[i].bio;
> > + fbio->bi_iter.bi_size = r10_bio->sectors << 9;
> > + fbio->bi_iter.bi_idx = 0;
> >
> > vcnt = (r10_bio->sectors + (PAGE_SIZE >> 9) - 1) >>
> > (PAGE_SHIFT - 9);
> > /* now find blocks with errors */
> > @@ -1987,7 +1989,7 @@ static void sync_request_write(struct mddev
> > *mddev, struct r10bio *r10_bio)
> > bio_reset(tbio);
> >
> > tbio->bi_vcnt = vcnt;
> > - tbio->bi_iter.bi_size = r10_bio->sectors << 9;
> > + tbio->bi_iter.bi_size = fbio->bi_iter.bi_size;
> > tbio->bi_rw = WRITE;
> > tbio->bi_private = r10_bio;
> > tbio->bi_iter.bi_sector = r10_bio->devs[i].addr;
>
> Looks good. Reviewed-by: Shaohua Li <shli@kernel.org>
>
> A nitpick, I'm wondering if we should do a full reset like raid1 does
> to make this more clear.
^ permalink raw reply
* core dump on initial sync of raid10 please help
From: Micheal Blue @ 2015-12-14 13:48 UTC (permalink / raw)
To: linux-raid
I noticed my initial sync when making a 4 disk raid10 is really slow. I looked in dmesg and found this.
I do not know what it means. Suggestions please :)
[ 600.959945] INFO: task md0_resync:676 blocked for more than 120 seconds.
[ 600.959976] Not tainted 4.3.2-1-ARCH #1
[ 600.959994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 600.960024] md0_resync D ffff88021fa956c0 0 676 2 0x00000000
[ 600.960028] ffff8800ca143b40 0000000000000046 ffff880216659b80 ffff8800377244c0
[ 600.960030] ffff8800ca144000 ffff88003765a000 ffff8800ca143b68 ffff88003765a100
[ 600.960032] ffff88003765a000 ffff8800ca143b58 ffffffff8157f92a ffff88003765a0dc
[ 600.960034] Call Trace:
[ 600.960040] [<ffffffff8157f92a>] schedule+0x3a/0x90
[ 600.960044] [<ffffffffa0548f52>] raise_barrier+0x122/0x190 [raid10]
[ 600.960049] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
[ 600.960051] [<ffffffffa054eea1>] sync_request+0x731/0x19b0 [raid10]
[ 600.960055] [<ffffffffa0390637>] ? is_mddev_idle+0x118/0x13e [md_mod]
[ 600.960058] [<ffffffffa0384ee7>] md_do_sync+0x897/0xf00 [md_mod]
[ 600.960061] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
[ 600.960065] [<ffffffff810809fe>] ? kernel_sigaction+0x7e/0xe0
[ 600.960068] [<ffffffffa0380fe0>] md_thread+0x130/0x140 [md_mod]
[ 600.960069] [<ffffffff8157f1d0>] ? __schedule+0x340/0xa60
[ 600.960072] [<ffffffffa0380eb0>] ? md_wait_for_blocked_rdev+0x130/0x130 [md_mod]
[ 600.960075] [<ffffffff81092e68>] kthread+0xd8/0xf0
[ 600.960077] [<ffffffff81092d90>] ? kthread_worker_fn+0x170/0x170
[ 600.960078] [<ffffffff8158375f>] ret_from_fork+0x3f/0x70
[ 600.960080] [<ffffffff81092d90>] ? kthread_worker_fn+0x170/0x170
[ 721.067472] INFO: task md0_resync:676 blocked for more than 120 seconds.
[ 721.067500] Not tainted 4.3.2-1-ARCH #1
[ 721.067517] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 721.067545] md0_resync D ffff88021fa956c0 0 676 2 0x00000000
[ 721.067548] ffff8800ca143b40 0000000000000046 ffff880216659b80 ffff8800377244c0
[ 721.067551] ffff8800ca144000 ffff88003765a000 ffff8800ca143b68 ffff88003765a100
[ 721.067553] ffff88003765a000 ffff8800ca143b58 ffffffff8157f92a ffff88003765a0dc
[ 721.067555] Call Trace:
[ 721.067561] [<ffffffff8157f92a>] schedule+0x3a/0x90
[ 721.067565] [<ffffffffa0548f52>] raise_barrier+0x122/0x190 [raid10]
[ 721.067569] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
[ 721.067572] [<ffffffffa054eea1>] sync_request+0x731/0x19b0 [raid10]
[ 721.067575] [<ffffffffa0390637>] ? is_mddev_idle+0x118/0x13e [md_mod]
[ 721.067578] [<ffffffffa0384ee7>] md_do_sync+0x897/0xf00 [md_mod]
[ 721.067581] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
[ 721.067585] [<ffffffff810809fe>] ? kernel_sigaction+0x7e/0xe0
[ 721.067588] [<ffffffffa0380fe0>] md_thread+0x130/0x140 [md_mod]
[ 721.067590] [<ffffffff8157f1d0>] ? __schedule+0x340/0xa60
[ 721.067592] [<ffffffffa0380eb0>] ? md_wait_for_blocked_rdev+0x130/0x130 [md_mod]
[ 721.067595] [<ffffffff81092e68>] kthread+0xd8/0xf0
[ 721.067597] [<ffffffff81092d90>] ? kthread_worker_fn+0x170/0x170
[ 721.067599] [<ffffffff8158375f>] ret_from_fork+0x3f/0x70
[ 721.067600] [<ffffffff81092d90>] ? kthread_worker_fn+0x170/0x170
[ 841.176298] INFO: task md0_resync:676 blocked for more than 120 seconds.
[ 841.176326] Not tainted 4.3.2-1-ARCH #1
[ 841.176342] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 841.176370] md0_resync D ffff88021fa956c0 0 676 2 0x00000000
[ 841.176374] ffff8800ca143b40 0000000000000046 ffff880216659b80 ffff8800377244c0
[ 841.176376] ffff8800ca144000 ffff88003765a000 ffff8800ca143b68 ffff88003765a100
[ 841.176378] ffff88003765a000 ffff8800ca143b58 ffffffff8157f92a ffff88003765a0dc
[ 841.176380] Call Trace:
[ 841.176386] [<ffffffff8157f92a>] schedule+0x3a/0x90
[ 841.176390] [<ffffffffa0548f52>] raise_barrier+0x122/0x190 [raid10]
[ 841.176395] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
[ 841.176397] [<ffffffffa054eea1>] sync_request+0x731/0x19b0 [raid10]
[ 841.176400] [<ffffffffa0390637>] ? is_mddev_idle+0x118/0x13e [md_mod]
[ 841.176403] [<ffffffffa0384ee7>] md_do_sync+0x897/0xf00 [md_mod]
[ 841.176406] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
[ 841.176409] [<ffffffff810809fe>] ? kernel_sigaction+0x7e/0xe0
[ 841.176412] [<ffffffffa0380fe0>] md_thread+0x130/0x140 [md_mod]
[ 841.176414] [<ffffffff8157f1d0>] ? __schedule+0x340/0xa60
[ 841.176416] [<ffffffffa0380eb0>] ? md_wait_for_blocked_rdev+0x130/0x130 [md_mod]
[ 841.176419] [<ffffffff81092e68>] kthread+0xd8/0xf0
[ 841.176421] [<ffffffff81092d90>] ? kthread_worker_fn+0x170/0x170
[ 841.176423] [<ffffffff8158375f>] ret_from_fork+0x3f/0x70
[ 841.176424] [<ffffffff81092d90>] ? kthread_worker_fn+0x170/0x170
[ 961.285635] INFO: task md0_resync:676 blocked for more than 120 seconds.
[ 961.285653] Not tainted 4.3.2-1-ARCH #1
[ 961.285662] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 961.285678] md0_resync D ffff88021fa956c0 0 676 2 0x00000000
[ 961.285681] ffff8800ca143b40 0000000000000046 ffff880216659b80 ffff8800377244c0
[ 961.285682] ffff8800ca144000 ffff88003765a000 ffff8800ca143b68 ffff88003765a100
[ 961.285684] ffff88003765a000 ffff8800ca143b58 ffffffff8157f92a ffff88003765a0dc
[ 961.285685] Call Trace:
[ 961.285690] [<ffffffff8157f92a>] schedule+0x3a/0x90
[ 961.285694] [<ffffffffa0548f52>] raise_barrier+0x122/0x190 [raid10]
[ 961.285697] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
[ 961.285699] [<ffffffffa054eea1>] sync_request+0x731/0x19b0 [raid10]
[ 961.285702] [<ffffffffa0390637>] ? is_mddev_idle+0x118/0x13e [md_mod]
[ 961.285704] [<ffffffffa0384ee7>] md_do_sync+0x897/0xf00 [md_mod]
[ 961.285706] [<ffffffff810b5dc0>] ? wake_atomic_t_function+0x60/0x60
[ 961.285709] [<ffffffff810809fe>] ? kernel_sigaction+0x7e/0xe0
[ 961.285711] [<ffffffffa0380fe0>] md_thread+0x130/0x140 [md_mod]
[ 961.285712] [<ffffffff8157f1d0>] ? __schedule+0x340/0xa60
[ 961.285714] [<ffffffffa0380eb0>] ? md_wait_for_blocked_rdev+0x130/0x130 [md_mod]
[ 961.285716] [<ffffffff81092e68>] kthread+0xd8/0xf0
[ 961.285717] [<ffffffff81092d90>] ? kthread_worker_fn+0x170/0x170
[ 961.285719] [<ffffffff8158375f>] ret_from_fork+0x3f/0x70
[ 961.285720] [<ffffffff81092d90>] ? kthread_worker_fn+0x170/0x170
^ permalink raw reply
* Re: Mdadm with data offsets
From: Phil Turmel @ 2015-12-14 12:55 UTC (permalink / raw)
To: TheGerwazy .; +Cc: linux-raid
In-Reply-To: <CAH_hVdo43dSuaeJT-QGN14LhiRnR1zYVFwqvueV6vxthHYYU9g@mail.gmail.com>
On 12/14/2015 07:03 AM, TheGerwazy . wrote:
> Hi Phil
> Thanks for the message .... but,
> In the last version of mdadm I get:
>
> ./mdadm --create --raid-devices=8 --level=6 --assume-clean /dev/md7
> /dev/mapper/ovsdb1:1024 /dev/mapper/ovsdc1:4096
> /dev/mapper/ovsdd1:1024 /dev/mapper/ovsde1:1024
> /dev/mapper/ovsdf1:1024 /dev/mapper/ovsdg1:1024
> /dev/mapper/ovsdh1:1024 /dev/mapper/ovsdi1:4096
> mdadm: cannot open /dev/mapper/ovsdb1:1024: No such file or directory
The final version of that feature requires --data-offset=variable before
the :offset syntax is recognized on the devices.
> sudo ./mdadm --create --raid-devices=8 --level=6 --assume-clean
> mdadm: array /dev/md7 started.
You're committed now. I hope it works out for you.
> BTW is it possible to find right data offset without superblock on drive ?
Only if the content of that member device has searchable content in a
predictable location. In practice that means finding the offset of the
first member by locating filesystem magic, or otherwise locating the
transition from all zeros to randomish data.
Good luck.
Phil
^ permalink raw reply
* Re: Mdadm with data offsets
From: TheGerwazy . @ 2015-12-14 12:03 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <566E05B2.20803@turmel.org>
2015-12-14 0:56 GMT+01:00 Phil Turmel <philip@turmel.org>:
> On 12/13/2015 06:50 PM, Phil Turmel wrote:
>> On 12/13/2015 01:18 AM, TheGerwazy . wrote:
>>> Hello
>>> I have to recreate mdadm raid 6 made with different data offsets. Some
>>> drives have 2048 and others 262144.
>>> I am looking for mdadm version with data offsets option for each drive.
>>> Could somebody help me. Please ?
>>
>> version 3.3 and above, I believe. If in doubt, just compile the latest
>> version yourself -- no root privileges are required to compile it. Then
>> run it from the current directory.
>>
>> git clone git://neil.brown.name/mdadm
>>
>> make
>> [sudo] ./mdadm ......
>
> I should add that using --create to reconstruct an array is incredibly
> dangerous (to your data) and should only be used as a last resort and
> when you are 100% confident you know how to do it. Your question
> suggests that might not be true.... in which case sharing your problem
> here might be wise.
>
> Phil
Hi Phil
Thanks for the message .... but,
In the last version of mdadm I get:
./mdadm --create --raid-devices=8 --level=6 --assume-clean /dev/md7
/dev/mapper/ovsdb1:1024 /dev/mapper/ovsdc1:4096
/dev/mapper/ovsdd1:1024 /dev/mapper/ovsde1:1024
/dev/mapper/ovsdf1:1024 /dev/mapper/ovsdg1:1024
/dev/mapper/ovsdh1:1024 /dev/mapper/ovsdi1:4096
mdadm: cannot open /dev/mapper/ovsdb1:1024: No such file or directory
but, There is branch data_offset of mdadm 3.2.5:
sudo ./mdadm --create --raid-devices=8 --level=6 --assume-clean
/dev/md7 /dev/mapper/ovsdb1 /dev/mapper/ovsdc1:4096
/dev/mapper/ovsdd1:1024 /dev/mapper/ovsde1:1024
/dev/mapper/ovsdf1:1024 /dev/mapper/ovsdg1:1024
/dev/mapper/ovsdh1:1024 /dev/mapper/ovsdi1:4096
dadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md7 started.
so it is exactly what I need :)
BTW is it possible to find right data offset without superblock on drive ?
Regards, Gerard
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox