* Re: Mdadm with data offsets
From: TheGerwazy . @ 2015-12-14 12:03 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <566E05B2.20803@turmel.org>
2015-12-14 0:56 GMT+01:00 Phil Turmel <philip@turmel.org>:
> On 12/13/2015 06:50 PM, Phil Turmel wrote:
>> On 12/13/2015 01:18 AM, TheGerwazy . wrote:
>>> Hello
>>> I have to recreate mdadm raid 6 made with different data offsets. Some
>>> drives have 2048 and others 262144.
>>> I am looking for mdadm version with data offsets option for each drive.
>>> Could somebody help me. Please ?
>>
>> version 3.3 and above, I believe. If in doubt, just compile the latest
>> version yourself -- no root privileges are required to compile it. Then
>> run it from the current directory.
>>
>> git clone git://neil.brown.name/mdadm
>>
>> make
>> [sudo] ./mdadm ......
>
> I should add that using --create to reconstruct an array is incredibly
> dangerous (to your data) and should only be used as a last resort and
> when you are 100% confident you know how to do it. Your question
> suggests that might not be true.... in which case sharing your problem
> here might be wise.
>
> Phil
Hi Phil
Thanks for the message .... but,
In the last version of mdadm I get:
./mdadm --create --raid-devices=8 --level=6 --assume-clean /dev/md7
/dev/mapper/ovsdb1:1024 /dev/mapper/ovsdc1:4096
/dev/mapper/ovsdd1:1024 /dev/mapper/ovsde1:1024
/dev/mapper/ovsdf1:1024 /dev/mapper/ovsdg1:1024
/dev/mapper/ovsdh1:1024 /dev/mapper/ovsdi1:4096
mdadm: cannot open /dev/mapper/ovsdb1:1024: No such file or directory
but, There is branch data_offset of mdadm 3.2.5:
sudo ./mdadm --create --raid-devices=8 --level=6 --assume-clean
/dev/md7 /dev/mapper/ovsdb1 /dev/mapper/ovsdc1:4096
/dev/mapper/ovsdd1:1024 /dev/mapper/ovsde1:1024
/dev/mapper/ovsdf1:1024 /dev/mapper/ovsdg1:1024
/dev/mapper/ovsdh1:1024 /dev/mapper/ovsdi1:4096
dadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md7 started.
so it is exactly what I need :)
BTW is it possible to find right data offset without superblock on drive ?
Regards, Gerard
^ permalink raw reply
* [PATCH 2/2] md: dm-crypt: Optimize the dm-crypt for XTS mode
From: Baolin Wang @ 2015-12-14 8:23 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel
Cc: neilb, dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, broonie, arnd, linux-block, linux-raid,
linux-kernel, baolin.wang
In-Reply-To: <cover.1450080755.git.baolin.wang@linaro.org>
In now dm-crypt code, it is ineffective to map one bio with just only
one scatterlist at one time for XTS mode. We can use multiple scatterlists
to map the whole bio and send all scatterlists of one bio to crypto engine
to encrypt or decrypt, which can improve the hardware engine's efficiency.
With this optimization, On my test setup (beaglebone black board) using 64KB
I/Os on an eMMC storage device I saw about 60% improvement in throughput for
encrypted writes, and about 100% improvement for encrypted reads. But this
is not fit for other modes which need different IV for each sector.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/md/dm-crypt.c | 315 ++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 309 insertions(+), 6 deletions(-)
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 917d47e..9f6f131 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -32,6 +32,7 @@
#include <linux/device-mapper.h>
#define DM_MSG_PREFIX "crypt"
+#define DM_MAX_SG_LIST 1024
/*
* context holding the current state of a multi-part conversion
@@ -68,6 +69,8 @@ struct dm_crypt_request {
struct convert_context *ctx;
struct scatterlist sg_in;
struct scatterlist sg_out;
+ struct sg_table sgt_in;
+ struct sg_table sgt_out;
sector_t iv_sector;
};
@@ -140,6 +143,7 @@ struct crypt_config {
char *cipher;
char *cipher_string;
+ int bulk_crypto;
struct crypt_iv_operations *iv_gen_ops;
union {
struct iv_essiv_private essiv;
@@ -833,6 +837,11 @@ static u8 *iv_of_dmreq(struct crypt_config *cc,
crypto_ablkcipher_alignmask(any_tfm(cc)) + 1);
}
+static int crypt_is_bulk_mode(struct crypt_config *cc)
+{
+ return cc->bulk_crypto;
+}
+
static int crypt_convert_block(struct crypt_config *cc,
struct convert_context *ctx,
struct ablkcipher_request *req)
@@ -881,24 +890,40 @@ static int crypt_convert_block(struct crypt_config *cc,
static void kcryptd_async_done(struct crypto_async_request *async_req,
int error);
+static void kcryptd_async_all_done(struct crypto_async_request *async_req,
+ int error);
static void crypt_alloc_req(struct crypt_config *cc,
struct convert_context *ctx)
{
unsigned key_index = ctx->cc_sector & (cc->tfms_count - 1);
+ struct dm_crypt_request *dmreq;
if (!ctx->req)
ctx->req = mempool_alloc(cc->req_pool, GFP_NOIO);
+ dmreq = dmreq_of_req(cc, ctx->req);
+ dmreq->sgt_in.orig_nents = 0;
+ dmreq->sgt_out.orig_nents = 0;
+
ablkcipher_request_set_tfm(ctx->req, cc->tfms[key_index]);
/*
* Use REQ_MAY_BACKLOG so a cipher driver internally backlogs
* requests if driver request queue is full.
*/
- ablkcipher_request_set_callback(ctx->req,
- CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP,
- kcryptd_async_done, dmreq_of_req(cc, ctx->req));
+ if (crypt_is_bulk_mode(cc))
+ ablkcipher_request_set_callback(ctx->req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG
+ | CRYPTO_TFM_REQ_MAY_SLEEP,
+ kcryptd_async_all_done,
+ dmreq_of_req(cc, ctx->req));
+ else
+ ablkcipher_request_set_callback(ctx->req,
+ CRYPTO_TFM_REQ_MAY_BACKLOG
+ | CRYPTO_TFM_REQ_MAY_SLEEP,
+ kcryptd_async_done,
+ dmreq_of_req(cc, ctx->req));
}
static void crypt_free_req(struct crypt_config *cc,
@@ -911,6 +936,221 @@ static void crypt_free_req(struct crypt_config *cc,
}
/*
+ * Check how many sg entry numbers are needed when map one bio
+ * with scatterlist in advance.
+ */
+static unsigned int crypt_sg_entry(struct bio *bio_t)
+{
+ struct request_queue *q = bdev_get_queue(bio_t->bi_bdev);
+ int cluster = blk_queue_cluster(q);
+ struct bio_vec bvec, bvprv = { NULL };
+ struct bvec_iter biter;
+ unsigned long nbytes = 0, sg_length = 0;
+ unsigned int sg_cnt = 0;
+
+ if (bio_t->bi_rw & REQ_DISCARD) {
+ if (bio_t->bi_vcnt)
+ return 1;
+ return 0;
+ }
+
+ if (bio_t->bi_rw & REQ_WRITE_SAME)
+ return 1;
+
+ bio_for_each_segment(bvec, bio_t, biter) {
+ nbytes = bvec.bv_len;
+
+ if (!cluster) {
+ sg_cnt++;
+ continue;
+ }
+
+ if (sg_length + nbytes > queue_max_segment_size(q)) {
+ sg_length = nbytes;
+ sg_cnt++;
+ goto next;
+ }
+
+ if (!BIOVEC_PHYS_MERGEABLE(&bvprv, &bvec)) {
+ sg_length = nbytes;
+ sg_cnt++;
+ goto next;
+ }
+
+ if (!BIOVEC_SEG_BOUNDARY(q, &bvprv, &bvec)) {
+ sg_length = nbytes;
+ sg_cnt++;
+ goto next;
+ }
+
+ sg_length += nbytes;
+next:
+ memcpy(&bvprv, &bvec, sizeof(struct bio_vec));
+ }
+
+ return sg_cnt;
+}
+
+static int crypt_convert_all_blocks(struct crypt_config *cc,
+ struct convert_context *ctx,
+ struct ablkcipher_request *req)
+{
+ struct dm_crypt_io *io =
+ container_of(ctx, struct dm_crypt_io, ctx);
+ struct dm_crypt_request *dmreq = dmreq_of_req(cc, req);
+ u8 *iv = iv_of_dmreq(cc, dmreq);
+ struct bio *orig_bio = io->base_bio;
+ struct bio *bio_in = ctx->bio_in;
+ struct bio *bio_out = ctx->bio_out;
+ unsigned int total_bytes = orig_bio->bi_iter.bi_size;
+ struct scatterlist *sg_in = NULL;
+ struct scatterlist *sg_out = NULL;
+ struct scatterlist *sg = NULL;
+ unsigned int total_sg_len_in = 0;
+ unsigned int total_sg_len_out = 0;
+ unsigned int sg_in_max = 0, sg_out_max = 0;
+ int ret;
+
+ dmreq->iv_sector = ctx->cc_sector;
+ dmreq->ctx = ctx;
+
+ /*
+ * Need to calculate how many sg entry need to be used
+ * for this bio.
+ */
+ sg_in_max = crypt_sg_entry(bio_in) + 1;
+ if (sg_in_max > DM_MAX_SG_LIST || sg_in_max <= 0) {
+ DMERR("%s sg entry too large or none %d\n",
+ __func__, sg_in_max);
+ return -EINVAL;
+ } else if (sg_in_max == 2) {
+ sg_in = &dmreq->sg_in;
+ }
+
+ if (!sg_in) {
+ ret = sg_alloc_table(&dmreq->sgt_in, sg_in_max, GFP_KERNEL);
+ if (ret) {
+ DMERR("%s sg in allocation failed\n", __func__);
+ return -ENOMEM;
+ }
+
+ sg_in = dmreq->sgt_in.sgl;
+ }
+
+ total_sg_len_in = __blk_bios_map_sg(bdev_get_queue(bio_in->bi_bdev),
+ bio_in, sg_in, &sg);
+ if ((total_sg_len_in <= 0)
+ || (total_sg_len_in > sg_in_max)) {
+ DMERR("%s in sg map error %d, sg_in_max[%d]\n",
+ __func__, total_sg_len_in, sg_in_max);
+ return -EINVAL;
+ }
+
+ if (sg)
+ sg_mark_end(sg);
+
+ ctx->iter_in.bi_size -= total_bytes;
+
+ if (bio_data_dir(orig_bio) == READ)
+ goto set_crypt;
+
+ sg_out_max = crypt_sg_entry(bio_out) + 1;
+ if (sg_out_max > DM_MAX_SG_LIST || sg_out_max <= 0) {
+ DMERR("%s sg entry too large or none %d\n",
+ __func__, sg_out_max);
+ return -EINVAL;
+ } else if (sg_out_max == 2) {
+ sg_out = &dmreq->sg_out;
+ }
+
+ if (!sg_out) {
+ ret = sg_alloc_table(&dmreq->sgt_out, sg_out_max, GFP_KERNEL);
+ if (ret) {
+ DMERR("%s sg out allocation failed\n", __func__);
+ return -ENOMEM;
+ }
+
+ sg_out = dmreq->sgt_out.sgl;
+ }
+
+ sg = NULL;
+ total_sg_len_out = __blk_bios_map_sg(bdev_get_queue(bio_out->bi_bdev),
+ bio_out, sg_out, &sg);
+ if ((total_sg_len_out <= 0) ||
+ (total_sg_len_out > sg_out_max)) {
+ DMERR("%s out sg map error %d, sg_out_max[%d]\n",
+ __func__, total_sg_len_out, sg_out_max);
+ return -EINVAL;
+ }
+
+ if (sg)
+ sg_mark_end(sg);
+
+ ctx->iter_out.bi_size -= total_bytes;
+set_crypt:
+ if (cc->iv_gen_ops) {
+ ret = cc->iv_gen_ops->generator(cc, iv, dmreq);
+ if (ret < 0) {
+ DMERR("%s generator iv error %d\n", __func__, ret);
+ return ret;
+ }
+ }
+
+ if (bio_data_dir(orig_bio) == WRITE) {
+ ablkcipher_request_set_crypt(req, sg_in,
+ sg_out, total_bytes, iv);
+
+ ret = crypto_ablkcipher_encrypt(req);
+ } else {
+ ablkcipher_request_set_crypt(req, sg_in,
+ sg_in, total_bytes, iv);
+
+ ret = crypto_ablkcipher_decrypt(req);
+ }
+
+ if (!ret && cc->iv_gen_ops && cc->iv_gen_ops->post)
+ ret = cc->iv_gen_ops->post(cc, iv, dmreq);
+
+ return ret;
+}
+
+/*
+ * Encrypt / decrypt data from one whole bio at one time.
+ */
+static int crypt_convert_io(struct crypt_config *cc,
+ struct convert_context *ctx)
+{
+ int r;
+
+ atomic_set(&ctx->cc_pending, 1);
+ crypt_alloc_req(cc, ctx);
+ atomic_inc(&ctx->cc_pending);
+
+ r = crypt_convert_all_blocks(cc, ctx, ctx->req);
+ switch (r) {
+ case -EBUSY:
+ /*
+ * Lets make this synchronous bio by waiting on
+ * in progress as well.
+ */
+ case -EINPROGRESS:
+ wait_for_completion(&ctx->restart);
+ ctx->req = NULL;
+ break;
+ case 0:
+ atomic_dec(&ctx->cc_pending);
+ cond_resched();
+ break;
+ /* There was an error while processing the request. */
+ default:
+ atomic_dec(&ctx->cc_pending);
+ return r;
+ }
+
+ return 0;
+}
+
+/*
* Encrypt / decrypt data from one bio to another one (can be the same one)
*/
static int crypt_convert(struct crypt_config *cc,
@@ -1070,12 +1310,18 @@ static void crypt_dec_pending(struct dm_crypt_io *io)
struct crypt_config *cc = io->cc;
struct bio *base_bio = io->base_bio;
int error = io->error;
+ struct dm_crypt_request *dmreq;
if (!atomic_dec_and_test(&io->io_pending))
return;
- if (io->ctx.req)
+ if (io->ctx.req) {
+ dmreq = dmreq_of_req(cc, io->ctx.req);
+ sg_free_table(&dmreq->sgt_out);
+ sg_free_table(&dmreq->sgt_in);
+
crypt_free_req(cc, io->ctx.req, base_bio);
+ }
base_bio->bi_error = error;
bio_endio(base_bio);
@@ -1312,7 +1558,11 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
sector += bio_sectors(clone);
crypt_inc_pending(io);
- r = crypt_convert(cc, &io->ctx);
+ if (crypt_is_bulk_mode(cc))
+ r = crypt_convert_io(cc, &io->ctx);
+ else
+ r = crypt_convert(cc, &io->ctx);
+
if (r)
io->error = -EIO;
crypt_finished = atomic_dec_and_test(&io->ctx.cc_pending);
@@ -1342,7 +1592,11 @@ static void kcryptd_crypt_read_convert(struct dm_crypt_io *io)
crypt_convert_init(cc, &io->ctx, io->base_bio, io->base_bio,
io->sector);
- r = crypt_convert(cc, &io->ctx);
+ if (crypt_is_bulk_mode(cc))
+ r = crypt_convert_io(cc, &io->ctx);
+ else
+ r = crypt_convert(cc, &io->ctx);
+
if (r < 0)
io->error = -EIO;
@@ -1387,6 +1641,40 @@ static void kcryptd_async_done(struct crypto_async_request *async_req,
kcryptd_crypt_write_io_submit(io, 1);
}
+static void kcryptd_async_all_done(struct crypto_async_request *async_req,
+ int error)
+{
+ struct dm_crypt_request *dmreq = async_req->data;
+ struct convert_context *ctx = dmreq->ctx;
+ struct dm_crypt_io *io = container_of(ctx, struct dm_crypt_io, ctx);
+ struct crypt_config *cc = io->cc;
+
+ if (error == -EINPROGRESS)
+ return;
+
+ if (!error && cc->iv_gen_ops && cc->iv_gen_ops->post)
+ error = cc->iv_gen_ops->post(cc, iv_of_dmreq(cc, dmreq), dmreq);
+
+ if (error < 0)
+ io->error = error;
+
+ sg_free_table(&dmreq->sgt_out);
+ sg_free_table(&dmreq->sgt_in);
+
+ crypt_free_req(cc, req_of_dmreq(cc, dmreq), io->base_bio);
+
+ if (!atomic_dec_and_test(&ctx->cc_pending)) {
+ complete(&io->ctx.restart);
+ return;
+ }
+
+ complete(&io->ctx.restart);
+ if (bio_data_dir(io->base_bio) == READ)
+ kcryptd_crypt_read_done(io);
+ else
+ kcryptd_crypt_write_io_submit(io, 1);
+}
+
static void kcryptd_crypt(struct work_struct *work)
{
struct dm_crypt_io *io = container_of(work, struct dm_crypt_io, work);
@@ -1633,6 +1921,21 @@ static int crypt_ctr_cipher(struct dm_target *ti,
goto bad_mem;
}
+ /*
+ * Here we need to check if it can be encrypted or decrypted with
+ * bulk block, which means these encryption modes don't need IV or
+ * just need one initial IV. For bulk mode, we can expand the
+ * scatterlist entries to map the bio, then send all the scatterlists
+ * to the hardware engine at one time to improve the crypto engine
+ * efficiency. But it does not fit for other encryption modes, it has
+ * to do encryption and decryption sector by sector because every
+ * sector has different IV.
+ */
+ if (!strcmp(chainmode, "ecb") || !strcmp(chainmode, "xts"))
+ cc->bulk_crypto = 1;
+ else
+ cc->bulk_crypto = 0;
+
/* Allocate cipher */
ret = crypt_alloc_tfms(cc, cipher_api);
if (ret < 0) {
--
1.7.9.5
^ permalink raw reply related
* [PATCH 1/2] block: Export the __blk_bios_map_sg() to map one bio
From: Baolin Wang @ 2015-12-14 8:23 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel
Cc: neilb, dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, broonie, arnd, linux-block, linux-raid,
linux-kernel, baolin.wang
In-Reply-To: <cover.1450080755.git.baolin.wang@linaro.org>
In dm-crypt, it need to map one bio to scatterlist for improving the
encryption efficiency. Thus this patch exports the __blk_bios_map_sg()
function to map one bio with scatterlists.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
block/blk-merge.c | 7 ++++---
include/linux/blkdev.h | 3 +++
2 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index de5716d8..09cc7c4 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -331,9 +331,9 @@ new_segment:
*bvprv = *bvec;
}
-static int __blk_bios_map_sg(struct request_queue *q, struct bio *bio,
- struct scatterlist *sglist,
- struct scatterlist **sg)
+int __blk_bios_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist,
+ struct scatterlist **sg)
{
struct bio_vec bvec, bvprv = { NULL };
struct bvec_iter iter;
@@ -372,6 +372,7 @@ single_segment:
return nsegs;
}
+EXPORT_SYMBOL(__blk_bios_map_sg);
/*
* map a request to scatterlist, return number of sg entries setup. Caller
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 3fe27f8..dd8d10f 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1004,6 +1004,9 @@ extern void blk_queue_flush_queueable(struct request_queue *q, bool queueable);
extern struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev);
extern int blk_rq_map_sg(struct request_queue *, struct request *, struct scatterlist *);
+extern int __blk_bios_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist,
+ struct scatterlist **sg);
extern void blk_dump_rq_flags(struct request *, char *);
extern long nr_blockdev_pages(void);
--
1.7.9.5
^ permalink raw reply related
* [PATCH 0/2] Optimize the dm-crypt for XTS mode
From: Baolin Wang @ 2015-12-14 8:23 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel
Cc: neilb, dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, broonie, arnd, linux-block, linux-raid,
linux-kernel, baolin.wang
In now dm-crypt code, we find it only send one scatterlist which maps
one segment of one bio to the crypto engine at one time. But that will
be more time-consuming and ineffective for the crypto engine.
For XTS mode, it can map the whole bio with multiple scatterlists,
and send all the scatterlists to the crypto engine which can improve
the performance.
Baolin Wang (2):
block: Export the __blk_bios_map_sg() to map one bio
md: dm-crypt: Optimize the dm-crypt for XTS mode
block/blk-merge.c | 7 +-
drivers/md/dm-crypt.c | 315 +++++++++++++++++++++++++++++++++++++++++++++++-
include/linux/blkdev.h | 3 +
3 files changed, 316 insertions(+), 9 deletions(-)
--
1.7.9.5
^ permalink raw reply
* Re: Mdadm with data offsets
From: Phil Turmel @ 2015-12-13 23:56 UTC (permalink / raw)
To: TheGerwazy ., linux-raid
In-Reply-To: <566E0440.70801@turmel.org>
On 12/13/2015 06:50 PM, Phil Turmel wrote:
> On 12/13/2015 01:18 AM, TheGerwazy . wrote:
>> Hello
>> I have to recreate mdadm raid 6 made with different data offsets. Some
>> drives have 2048 and others 262144.
>> I am looking for mdadm version with data offsets option for each drive.
>> Could somebody help me. Please ?
>
> version 3.3 and above, I believe. If in doubt, just compile the latest
> version yourself -- no root privileges are required to compile it. Then
> run it from the current directory.
>
> git clone git://neil.brown.name/mdadm
>
> make
> [sudo] ./mdadm ......
I should add that using --create to reconstruct an array is incredibly
dangerous (to your data) and should only be used as a last resort and
when you are 100% confident you know how to do it. Your question
suggests that might not be true.... in which case sharing your problem
here might be wise.
Phil
^ permalink raw reply
* Re: Mdadm with data offsets
From: Phil Turmel @ 2015-12-13 23:50 UTC (permalink / raw)
To: TheGerwazy ., linux-raid
In-Reply-To: <CAH_hVdp4mGE-AZ1ww81u9vS0xri4xR+LrpaRSbaUb6VRohZ_Rg@mail.gmail.com>
On 12/13/2015 01:18 AM, TheGerwazy . wrote:
> Hello
> I have to recreate mdadm raid 6 made with different data offsets. Some
> drives have 2048 and others 262144.
> I am looking for mdadm version with data offsets option for each drive.
> Could somebody help me. Please ?
version 3.3 and above, I believe. If in doubt, just compile the latest
version yourself -- no root privileges are required to compile it. Then
run it from the current directory.
git clone git://neil.brown.name/mdadm
make
[sudo] ./mdadm ......
^ permalink raw reply
* Seeking advice to convert RAID5 to RAID10 adding a drive in the process
From: Micheal Blue @ 2015-12-13 14:47 UTC (permalink / raw)
To: linux-raid
I have a 3 disk RAID5 currently that uses LUKS. I would like to add a new disk and covert the array to RAID10 but cannot find any walk-through of this process. I am grateful for any input from the list be it links or a personal summary.
Below is a graphic of how my current 3 drive array is setup:
---------------------------------------
[ ext4 file system ]
---------------------------------------
[ LUKS cryptdevice ]
---------------------------------------
[ /dev/dm0 ]
---------------------------------------
[ /dev/sdb1 ] [ /dev/sdc1 ] [ /dev/sdd1 ]
---------------------------------------
^ permalink raw reply
* Mdadm with data offsets
From: TheGerwazy . @ 2015-12-13 6:18 UTC (permalink / raw)
To: linux-raid
Hello
I have to recreate mdadm raid 6 made with different data offsets. Some
drives have 2048 and others 262144.
I am looking for mdadm version with data offsets option for each drive.
Could somebody help me. Please ?
Regards. Gerard
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-12 19:17 UTC (permalink / raw)
To: John Stoffel; +Cc: Phil Turmel, linux-raid
In-Reply-To: <22123.9272.750357.130814@quad.stoffel.home>
>
> Do you have everything setup as a module or a mix? That can certainly
> change things. It might make sense to re-build the kernel so that the
> MDADM stuff is modules, but the rest is compiled in, or visa versa.
Here is all the information I collected: http://pastebin.com/VF4Sfrq5
Regarding the testing setup, we can safely consider whole system as a
test, we will rebuild the system once we figure final config anyway.
So we can play around with the current raid with no risk of data loss.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Phil Turmel @ 2015-12-12 4:47 UTC (permalink / raw)
To: Dallas Clement; +Cc: John Stoffel, Mark Knecht, Linux-RAID
In-Reply-To: <CAE9DZURoHBRHq2M0spkTrBGoXmw9QjoARb_Gc6C6OvM9940aMA@mail.gmail.com>
On 12/11/2015 09:55 PM, Dallas Clement wrote:
> Right. I understand the fio iodepth is different than the hardware
> queue depth. But the fio man page seems to only mention limitation on
> synchronous operations which mine are not. I'm using direct=1 and
> sync=0.
You are confusing sequential and synchronous. The man page says it is
ineffective for *sequential* operations, especially when direct=1.
> I guess what I would really like to know is how I can achieve at or
> near 100% utilization on the raid device and its member disks with
> fio. Do I need to increase /sys/block/sd*/device/queue_depth and
> /sys/block/sd*/queue/nr_requests to get more utilization?
I don't know specifically. It seems to me that increasing queue depth
adds resiliency in the face of data transfer timing jitter, but at the
cost of more CPU overhead.
I'm not convinced fio is the right workload, either. It's options are
much more flexible for random I/O workloads. dd isn't perfect either,
especially when writing zeroes -- it actually reads zeros over and over
from the special device. For sequential operations I like dc3dd with
its pat= wipe= mode. That'll only generate writes.
>> That's why I suggested blktrace. Collect a trace while a single dd is
>> writing to your raw array device. Compare the large writes submitted to
>> the md device against the broken down writes submitted to the member
>> devices.
>
> Sounds good. Will do. What signs of trouble should I be looking for?
Look for strictly increasing logical block addresses in requests to the
member devices. Any disruption in that will break optimum positioning
for streaming throughput. Per device. Requests to the device have to be
large enough and paced quickly enough to avoid starving the write head.
Of course, any reads mixed in mean RMW cycles you didn't avoid. You
shouldn't have any of those for sequential writes in chunk * (n-2)
multiples.
I know it's a bit hand-wavy, but you have more hardware to play with
than I do :-)
Phil
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-12 2:55 UTC (permalink / raw)
To: Phil Turmel; +Cc: John Stoffel, Mark Knecht, Linux-RAID
In-Reply-To: <566B6C8F.7020201@turmel.org>
On Fri, Dec 11, 2015 at 6:38 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/11/2015 07:00 PM, Dallas Clement wrote:
>
>>> So is my workload of 12 fio jobs writing sequential 2 MB blocks with
>>> direct I/O just too abusive? Seems so with high queue depth.
>
> I don't think you are adjusting any hardware queue depth here. The fio
> man page is quite explicit that iodepth=N is ineffective for sequential
> operations. But you are using the libaio engine, so you are piling up
> many *software* queued operations for the kernel to execute, not
> operations in flight to the disks. From the histograms in your results,
> the vast majority of ops are completing at depth=4. Further queuing is
> just adding kernel overhead.
>
> The queuing differences from one kernel to another is a driver and
> hardware property, not an application property.
>
>>> I started this discussion because my RAID 5 and RAID 6 write
>>> performance is really bad. If my system is able to write to all 12
>>> disks at 170 MB/s in JBOD mode, I am expecting that one fio job should
>>> be able to write at a speed of (N - 1) * X = 11 * 170 MB/s = 1870
>>> MB/s. However, I am getting < 700 MB/s for queue depth = 32 and < 600
>>> MB/s for queue depth = 256. I get similarly disappointing results for
>>> RAID 6 writes.
>
> That's why I suggested blktrace. Collect a trace while a single dd is
> writing to your raw array device. Compare the large writes submitted to
> the md device against the broken down writes submitted to the member
> devices.
>
> Compare the patterns and sizes from older kernels against newer kernels,
> possibly varying which controllers and data paths are involved.
>
> Phil
Hi Phil,
> I don't think you are adjusting any hardware queue depth here.
Right, that was my understanding as well. The fio iodepth setting
just controls how many I/Os can be in flight from the application
perspective. I have not modified the hardware queue depth on my disks
at all yet. Was saving that for later.
> The fio man page is quite explicit that iodepth=N is ineffective for sequential
> operations. But you are using the libaio engine, so you are piling up
> many *software* queued operations for the kernel to execute, not
> operations in flight to the disks.
Right. I understand the fio iodepth is different than the hardware
queue depth. But the fio man page seems to only mention limitation on
synchronous operations which mine are not. I'm using direct=1 and
sync=0.
I guess what I would really like to know is how I can achieve at or
near 100% utilization on the raid device and its member disks with
fio. Do I need to increase /sys/block/sd*/device/queue_depth and
/sys/block/sd*/queue/nr_requests to get more utilization?
> That's why I suggested blktrace. Collect a trace while a single dd is
> writing to your raw array device. Compare the large writes submitted to
> the md device against the broken down writes submitted to the member
> devices.
Sounds good. Will do. What signs of trouble should I be looking for?
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Phil Turmel @ 2015-12-12 0:38 UTC (permalink / raw)
To: Dallas Clement, John Stoffel; +Cc: Mark Knecht, Linux-RAID
In-Reply-To: <CAE9DZURuPGEL4bG=44ntbjp+51jktn36LFGfn11xFR-X9O9POw@mail.gmail.com>
On 12/11/2015 07:00 PM, Dallas Clement wrote:
>> So is my workload of 12 fio jobs writing sequential 2 MB blocks with
>> direct I/O just too abusive? Seems so with high queue depth.
I don't think you are adjusting any hardware queue depth here. The fio
man page is quite explicit that iodepth=N is ineffective for sequential
operations. But you are using the libaio engine, so you are piling up
many *software* queued operations for the kernel to execute, not
operations in flight to the disks. From the histograms in your results,
the vast majority of ops are completing at depth=4. Further queuing is
just adding kernel overhead.
The queuing differences from one kernel to another is a driver and
hardware property, not an application property.
>> I started this discussion because my RAID 5 and RAID 6 write
>> performance is really bad. If my system is able to write to all 12
>> disks at 170 MB/s in JBOD mode, I am expecting that one fio job should
>> be able to write at a speed of (N - 1) * X = 11 * 170 MB/s = 1870
>> MB/s. However, I am getting < 700 MB/s for queue depth = 32 and < 600
>> MB/s for queue depth = 256. I get similarly disappointing results for
>> RAID 6 writes.
That's why I suggested blktrace. Collect a trace while a single dd is
writing to your raw array device. Compare the large writes submitted to
the md device against the broken down writes submitted to the member
devices.
Compare the patterns and sizes from older kernels against newer kernels,
possibly varying which controllers and data paths are involved.
Phil
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-12 0:00 UTC (permalink / raw)
To: John Stoffel; +Cc: Mark Knecht, Phil Turmel, Linux-RAID
In-Reply-To: <CAE9DZUTTP1VhVgT56dyv6aLaM2V8peWSHaBg4xvXzGGUZcJ_hw@mail.gmail.com>
On Fri, Dec 11, 2015 at 5:30 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Fri, Dec 11, 2015 at 3:24 PM, Dallas Clement
> <dallas.a.clement@gmail.com> wrote:
>> On Fri, Dec 11, 2015 at 1:34 PM, John Stoffel <john@stoffel.org> wrote:
>>>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>>>
>>> Dallas> On Fri, Dec 11, 2015 at 10:32 AM, John Stoffel <john@stoffel.org> wrote:
>>>>>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>>>>>
>>> Dallas> Hi Mark. I have three different controllers on this
>>> Dallas> motherboard. A Marvell 9485 controls 8 of the disks. And an
>>> Dallas> Intel Cougar Point controls the 4 remaining disks.
>>>>>
>>>>> What type of PCIe slots are the controllers in? And how fast are the
>>>>> controllers/drives? Are they SATA1/2/3 drives?
>>>>>
>>>>>>> If you're spinning in IO loops then it could be a driver issue.
>>>>>
>>> Dallas> It sure is looking like that. I will try to profile the
>>> Dallas> kernel threads today and maybe use blktrace as Phil
>>> Dallas> recommended to see what is going on there.
>>>>>
>>>>> what kernel aer you running?
>>>>>
>>> Dallas> This is pretty sad that 12 single threaded fio jobs can bring
>>> Dallas> this system to its knees.
>>>>>
>>>>> I think it might be better to lower the queue depth, you might be just
>>>>> blowing out the controller caches... hard to know.
>>>
>>> Dallas> Hi John.
>>>
>>>>> What type of PCIe slots are the controllers in? And how fast are the
>>>>> controllers/drives? Are they SATA1/2/3 drives?
>>>
>>> Dallas> The MV 9485 controller is attached to an Intel Sandy Bridge
>>> Dallas> via PCIe GEN2 x 8. This one controls 8 of the disks. The
>>> Dallas> Intel Cougar Point is connected to the Intel Sandy Bridge via
>>> Dallas> DMI bus.
>>>
>>> So that should all be nice and fast.
>>>
>>> Dallas> All of the drives are SATA III, however I do have two of the
>>> Dallas> drives connected to SATA II ports on the Cougar Point. These
>>> Dallas> two drives used to be connected to SATA III ports on a MV
>>> Dallas> 9125/9120 controller. But it had truly horrible write
>>> Dallas> performance. Moving to the SATA II ports on the Cougar Point
>>> Dallas> boosted the performance close to the same as the other drives.
>>> Dallas> The remaining 10 drives are all connected to SATA III ports.
>>>
>>>>> what kernel aer you running?
>>>
>>> Dallas> Right now, I'm using 3.10.69. But I have tried the 4.2 kernel
>>> Dallas> in Fedora 23 with similar results.
>>>
>>> Hmm... maybe if your feeling adventerous you could try v4.4-rc4 and
>>> see how it works. You don't want anything between 4.2.6 and that
>>> because of problems with blk req management. I'm hazy on the details.
>>>
>>>>> I think it might be better to lower the queue depth, you might be just
>>>>> blowing out the controller caches... hard to know.
>>>
>>> Dallas> Good idea. I'll trying lowering to see what effect.
>>>
>>> It might also make sense to try your tests starting with just 1 disk,
>>> and then adding one more disk, re-running the tests, then another
>>> disk, re-running the tests, etc.
>>>
>>> Try with one on the MV, then one on the Cougar, then one on MV and one
>>> on Cougar, etc.
>>>
>>> Try to see if you can spot where the performance falls off the cliff.
>>>
>>> Also, which disk scheduler are you using? Instead of CFQ, you might
>>> try deadline instead.
>>>
>>> As you can see, there's a TON of knobs to twiddle with, it's not a
>>> simple thing to do at times.
>>>
>>> John
>>
>>> It might also make sense to try your tests starting with just 1 disk,
>>> and then adding one more disk, re-running the tests, then another
>>> disk, re-running the tests, etc
>>
>>> Try to see if you can spot where the performance falls off the cliff.
>>
>> Okay, did this. Interestingly, things did not fall of the cliff until
>> adding in the 12th disk. I started adding disks one at a time
>> beginning with the Cougar Point. The %iowait jumped up right away
>> with this guy also.
>>
>>> Also, which disk scheduler are you using? Instead of CFQ, you might
>>> try deadline instead.
>>
>> I'm using deadline. I have definitely observed better performance
>> with this vs cfq.
>>
>> At this point I think I need to probably use a tool like blktrace to
>> get more visibility than what I have with ps and iostat.
>
> I have one more observation. I tried varying the queue depth from 1,
> 4, 16, 32, 64, 128, 256. Surprisingly, all 12 disks are able to
> handle this load with queue depth <= 128. Each disk is at 100%
> utilization and writing 170-180 MB/s. Things start to fall apart with
> queue depth = 256 after adding in the 12th disk. The inflection point
> on load average seems to be around queue depth = 32. The load average
> for this 8 core system goes up to about 13 when I increase the queue
> depth to 64.
>
> So is my workload of 12 fio jobs writing sequential 2 MB blocks with
> direct I/O just too abusive? Seems so with high queue depth.
>
> I started this discussion because my RAID 5 and RAID 6 write
> performance is really bad. If my system is able to write to all 12
> disks at 170 MB/s in JBOD mode, I am expecting that one fio job should
> be able to write at a speed of (N - 1) * X = 11 * 170 MB/s = 1870
> MB/s. However, I am getting < 700 MB/s for queue depth = 32 and < 600
> MB/s for queue depth = 256. I get similarly disappointing results for
> RAID 6 writes.
One other thing I failed to mention is that I seem to be unable to
saturate my RAID device using fio. I have tried increasing the number
of jobs and that has actually resulted in worse performance. Here's
what I get with just one job thread.
# fio ../job.fio
job: (g=0): rw=write, bs=2M-2M/2M-2M/2M-2M, ioengine=libaio, iodepth=256
fio-2.2.7
Starting 1 process
Jobs: 1 (f=1): [W(1)] [90.5% done] [0KB/725.3MB/0KB /s] [0/362/0 iops]
[eta 00m:02s]
job: (groupid=0, jobs=1): err= 0: pid=30569: Sat Dec 12 08:22:54 2015
write: io=10240MB, bw=561727KB/s, iops=274, runt= 18667msec
slat (usec): min=316, max=554160, avg=3623.16, stdev=20560.63
clat (msec): min=25, max=2744, avg=913.26, stdev=508.27
lat (msec): min=26, max=2789, avg=916.88, stdev=510.13
clat percentiles (msec):
| 1.00th=[ 221], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 635],
| 30.00th=[ 660], 40.00th=[ 685], 50.00th=[ 709], 60.00th=[ 742],
| 70.00th=[ 791], 80.00th=[ 947], 90.00th=[ 1827], 95.00th=[ 2114],
| 99.00th=[ 2442], 99.50th=[ 2474], 99.90th=[ 2540], 99.95th=[ 2737],
| 99.99th=[ 2737]
bw (KB /s): min= 3093, max=934603, per=97.80%, avg=549364.82,
stdev=269856.22
lat (msec) : 50=0.14%, 100=0.39%, 250=0.78%, 500=2.03%, 750=58.67%
lat (msec) : 1000=18.18%, 2000=11.41%, >=2000=8.40%
cpu : usr=5.30%, sys=8.89%, ctx=2219, majf=0, minf=32
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.6%, >=64=98.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=0/w=5120/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: io=10240MB, aggrb=561727KB/s, minb=561727KB/s,
maxb=561727KB/s, mint=18667msec, maxt=18667msec
Disk stats (read/write):
md10: ios=1/81360, merge=0/0, ticks=0/0, in_queue=0, util=0.00%,
aggrios=660/4402, aggrmerge=9848/234056, aggrticks=23282/123890,
aggrin_queue=147976, aggrutil=66.50%
sda: ios=712/4387, merge=10727/233944, ticks=24150/130830,
in_queue=155810, util=61.32%
sdb: ios=697/4441, merge=10246/234331, ticks=19820/108830,
in_queue=129430, util=59.58%
sdc: ios=636/4384, merge=9273/233886, ticks=21380/123780,
in_queue=146070, util=62.17%
sdd: ios=656/4399, merge=9731/234030, ticks=23050/135000,
in_queue=158880, util=63.91%
sdf: ios=672/4427, merge=9862/234117, ticks=20110/101910,
in_queue=122790, util=58.53%
sdg: ios=656/4414, merge=9801/234081, ticks=20820/110860,
in_queue=132390, util=61.38%
sdh: ios=644/4385, merge=9526/234047, ticks=25120/131670,
in_queue=157630, util=62.80%
sdi: ios=739/4369, merge=10757/233876, ticks=32430/160810,
in_queue=194080, util=66.50%
sdj: ios=687/4386, merge=10525/234033, ticks=25770/131950,
in_queue=158530, util=64.18%
sdk: ios=620/4454, merge=9572/234495, ticks=22010/117190,
in_queue=139960, util=60.80%
sdl: ios=610/4393, merge=9090/233924, ticks=23800/118340,
in_queue=142910, util=62.12%
sdm: ios=602/4394, merge=9066/233915, ticks=20930/115520,
in_queue=137240, util=60.96%
As you can see, the array utilization is only 66.5% and the disk
utilization is about the same. Perhaps I am just using the wrong tool
or using fio incorrectly. On the other hand, I suppose it still could
be a problem with RAID 5, 6 implementation.
This is my fio job config:
# cat ../job.fio
[job]
ioengine=libaio
iodepth=256
prio=0
rw=write
bs=2048k
filename=/dev/md10
numjobs=1
size=10g
direct=1
invalidate=1
ramp_time=15
runtime=120
time_based
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-11 23:30 UTC (permalink / raw)
To: John Stoffel; +Cc: Mark Knecht, Phil Turmel, Linux-RAID
In-Reply-To: <CAE9DZUTMnwpUX1e95c_i04uWREHd+aR8P2yCE_W-WmEbL6YRkw@mail.gmail.com>
On Fri, Dec 11, 2015 at 3:24 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Fri, Dec 11, 2015 at 1:34 PM, John Stoffel <john@stoffel.org> wrote:
>>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>>
>> Dallas> On Fri, Dec 11, 2015 at 10:32 AM, John Stoffel <john@stoffel.org> wrote:
>>>>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>>>>
>> Dallas> Hi Mark. I have three different controllers on this
>> Dallas> motherboard. A Marvell 9485 controls 8 of the disks. And an
>> Dallas> Intel Cougar Point controls the 4 remaining disks.
>>>>
>>>> What type of PCIe slots are the controllers in? And how fast are the
>>>> controllers/drives? Are they SATA1/2/3 drives?
>>>>
>>>>>> If you're spinning in IO loops then it could be a driver issue.
>>>>
>> Dallas> It sure is looking like that. I will try to profile the
>> Dallas> kernel threads today and maybe use blktrace as Phil
>> Dallas> recommended to see what is going on there.
>>>>
>>>> what kernel aer you running?
>>>>
>> Dallas> This is pretty sad that 12 single threaded fio jobs can bring
>> Dallas> this system to its knees.
>>>>
>>>> I think it might be better to lower the queue depth, you might be just
>>>> blowing out the controller caches... hard to know.
>>
>> Dallas> Hi John.
>>
>>>> What type of PCIe slots are the controllers in? And how fast are the
>>>> controllers/drives? Are they SATA1/2/3 drives?
>>
>> Dallas> The MV 9485 controller is attached to an Intel Sandy Bridge
>> Dallas> via PCIe GEN2 x 8. This one controls 8 of the disks. The
>> Dallas> Intel Cougar Point is connected to the Intel Sandy Bridge via
>> Dallas> DMI bus.
>>
>> So that should all be nice and fast.
>>
>> Dallas> All of the drives are SATA III, however I do have two of the
>> Dallas> drives connected to SATA II ports on the Cougar Point. These
>> Dallas> two drives used to be connected to SATA III ports on a MV
>> Dallas> 9125/9120 controller. But it had truly horrible write
>> Dallas> performance. Moving to the SATA II ports on the Cougar Point
>> Dallas> boosted the performance close to the same as the other drives.
>> Dallas> The remaining 10 drives are all connected to SATA III ports.
>>
>>>> what kernel aer you running?
>>
>> Dallas> Right now, I'm using 3.10.69. But I have tried the 4.2 kernel
>> Dallas> in Fedora 23 with similar results.
>>
>> Hmm... maybe if your feeling adventerous you could try v4.4-rc4 and
>> see how it works. You don't want anything between 4.2.6 and that
>> because of problems with blk req management. I'm hazy on the details.
>>
>>>> I think it might be better to lower the queue depth, you might be just
>>>> blowing out the controller caches... hard to know.
>>
>> Dallas> Good idea. I'll trying lowering to see what effect.
>>
>> It might also make sense to try your tests starting with just 1 disk,
>> and then adding one more disk, re-running the tests, then another
>> disk, re-running the tests, etc.
>>
>> Try with one on the MV, then one on the Cougar, then one on MV and one
>> on Cougar, etc.
>>
>> Try to see if you can spot where the performance falls off the cliff.
>>
>> Also, which disk scheduler are you using? Instead of CFQ, you might
>> try deadline instead.
>>
>> As you can see, there's a TON of knobs to twiddle with, it's not a
>> simple thing to do at times.
>>
>> John
>
>> It might also make sense to try your tests starting with just 1 disk,
>> and then adding one more disk, re-running the tests, then another
>> disk, re-running the tests, etc
>
>> Try to see if you can spot where the performance falls off the cliff.
>
> Okay, did this. Interestingly, things did not fall of the cliff until
> adding in the 12th disk. I started adding disks one at a time
> beginning with the Cougar Point. The %iowait jumped up right away
> with this guy also.
>
>> Also, which disk scheduler are you using? Instead of CFQ, you might
>> try deadline instead.
>
> I'm using deadline. I have definitely observed better performance
> with this vs cfq.
>
> At this point I think I need to probably use a tool like blktrace to
> get more visibility than what I have with ps and iostat.
I have one more observation. I tried varying the queue depth from 1,
4, 16, 32, 64, 128, 256. Surprisingly, all 12 disks are able to
handle this load with queue depth <= 128. Each disk is at 100%
utilization and writing 170-180 MB/s. Things start to fall apart with
queue depth = 256 after adding in the 12th disk. The inflection point
on load average seems to be around queue depth = 32. The load average
for this 8 core system goes up to about 13 when I increase the queue
depth to 64.
So is my workload of 12 fio jobs writing sequential 2 MB blocks with
direct I/O just too abusive? Seems so with high queue depth.
I started this discussion because my RAID 5 and RAID 6 write
performance is really bad. If my system is able to write to all 12
disks at 170 MB/s in JBOD mode, I am expecting that one fio job should
be able to write at a speed of (N - 1) * X = 11 * 170 MB/s = 1870
MB/s. However, I am getting < 700 MB/s for queue depth = 32 and < 600
MB/s for queue depth = 256. I get similarly disappointing results for
RAID 6 writes.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-11 21:24 UTC (permalink / raw)
To: John Stoffel; +Cc: Mark Knecht, Phil Turmel, Linux-RAID
In-Reply-To: <22123.9525.433754.283927@quad.stoffel.home>
On Fri, Dec 11, 2015 at 1:34 PM, John Stoffel <john@stoffel.org> wrote:
>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>
> Dallas> On Fri, Dec 11, 2015 at 10:32 AM, John Stoffel <john@stoffel.org> wrote:
>>>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>>>
> Dallas> Hi Mark. I have three different controllers on this
> Dallas> motherboard. A Marvell 9485 controls 8 of the disks. And an
> Dallas> Intel Cougar Point controls the 4 remaining disks.
>>>
>>> What type of PCIe slots are the controllers in? And how fast are the
>>> controllers/drives? Are they SATA1/2/3 drives?
>>>
>>>>> If you're spinning in IO loops then it could be a driver issue.
>>>
> Dallas> It sure is looking like that. I will try to profile the
> Dallas> kernel threads today and maybe use blktrace as Phil
> Dallas> recommended to see what is going on there.
>>>
>>> what kernel aer you running?
>>>
> Dallas> This is pretty sad that 12 single threaded fio jobs can bring
> Dallas> this system to its knees.
>>>
>>> I think it might be better to lower the queue depth, you might be just
>>> blowing out the controller caches... hard to know.
>
> Dallas> Hi John.
>
>>> What type of PCIe slots are the controllers in? And how fast are the
>>> controllers/drives? Are they SATA1/2/3 drives?
>
> Dallas> The MV 9485 controller is attached to an Intel Sandy Bridge
> Dallas> via PCIe GEN2 x 8. This one controls 8 of the disks. The
> Dallas> Intel Cougar Point is connected to the Intel Sandy Bridge via
> Dallas> DMI bus.
>
> So that should all be nice and fast.
>
> Dallas> All of the drives are SATA III, however I do have two of the
> Dallas> drives connected to SATA II ports on the Cougar Point. These
> Dallas> two drives used to be connected to SATA III ports on a MV
> Dallas> 9125/9120 controller. But it had truly horrible write
> Dallas> performance. Moving to the SATA II ports on the Cougar Point
> Dallas> boosted the performance close to the same as the other drives.
> Dallas> The remaining 10 drives are all connected to SATA III ports.
>
>>> what kernel aer you running?
>
> Dallas> Right now, I'm using 3.10.69. But I have tried the 4.2 kernel
> Dallas> in Fedora 23 with similar results.
>
> Hmm... maybe if your feeling adventerous you could try v4.4-rc4 and
> see how it works. You don't want anything between 4.2.6 and that
> because of problems with blk req management. I'm hazy on the details.
>
>>> I think it might be better to lower the queue depth, you might be just
>>> blowing out the controller caches... hard to know.
>
> Dallas> Good idea. I'll trying lowering to see what effect.
>
> It might also make sense to try your tests starting with just 1 disk,
> and then adding one more disk, re-running the tests, then another
> disk, re-running the tests, etc.
>
> Try with one on the MV, then one on the Cougar, then one on MV and one
> on Cougar, etc.
>
> Try to see if you can spot where the performance falls off the cliff.
>
> Also, which disk scheduler are you using? Instead of CFQ, you might
> try deadline instead.
>
> As you can see, there's a TON of knobs to twiddle with, it's not a
> simple thing to do at times.
>
> John
> It might also make sense to try your tests starting with just 1 disk,
> and then adding one more disk, re-running the tests, then another
> disk, re-running the tests, etc
> Try to see if you can spot where the performance falls off the cliff.
Okay, did this. Interestingly, things did not fall of the cliff until
adding in the 12th disk. I started adding disks one at a time
beginning with the Cougar Point. The %iowait jumped up right away
with this guy also.
> Also, which disk scheduler are you using? Instead of CFQ, you might
> try deadline instead.
I'm using deadline. I have definitely observed better performance
with this vs cfq.
At this point I think I need to probably use a tool like blktrace to
get more visibility than what I have with ps and iostat.
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: John Stoffel @ 2015-12-11 19:34 UTC (permalink / raw)
To: Dallas Clement; +Cc: John Stoffel, Mark Knecht, Phil Turmel, Linux-RAID
In-Reply-To: <CAE9DZUQ=QynBKYJvq2JSnMaACKNpm+5yrhz+5x9Tx6_TK78mCg@mail.gmail.com>
>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
Dallas> On Fri, Dec 11, 2015 at 10:32 AM, John Stoffel <john@stoffel.org> wrote:
>>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>>
Dallas> Hi Mark. I have three different controllers on this
Dallas> motherboard. A Marvell 9485 controls 8 of the disks. And an
Dallas> Intel Cougar Point controls the 4 remaining disks.
>>
>> What type of PCIe slots are the controllers in? And how fast are the
>> controllers/drives? Are they SATA1/2/3 drives?
>>
>>>> If you're spinning in IO loops then it could be a driver issue.
>>
Dallas> It sure is looking like that. I will try to profile the
Dallas> kernel threads today and maybe use blktrace as Phil
Dallas> recommended to see what is going on there.
>>
>> what kernel aer you running?
>>
Dallas> This is pretty sad that 12 single threaded fio jobs can bring
Dallas> this system to its knees.
>>
>> I think it might be better to lower the queue depth, you might be just
>> blowing out the controller caches... hard to know.
Dallas> Hi John.
>> What type of PCIe slots are the controllers in? And how fast are the
>> controllers/drives? Are they SATA1/2/3 drives?
Dallas> The MV 9485 controller is attached to an Intel Sandy Bridge
Dallas> via PCIe GEN2 x 8. This one controls 8 of the disks. The
Dallas> Intel Cougar Point is connected to the Intel Sandy Bridge via
Dallas> DMI bus.
So that should all be nice and fast.
Dallas> All of the drives are SATA III, however I do have two of the
Dallas> drives connected to SATA II ports on the Cougar Point. These
Dallas> two drives used to be connected to SATA III ports on a MV
Dallas> 9125/9120 controller. But it had truly horrible write
Dallas> performance. Moving to the SATA II ports on the Cougar Point
Dallas> boosted the performance close to the same as the other drives.
Dallas> The remaining 10 drives are all connected to SATA III ports.
>> what kernel aer you running?
Dallas> Right now, I'm using 3.10.69. But I have tried the 4.2 kernel
Dallas> in Fedora 23 with similar results.
Hmm... maybe if your feeling adventerous you could try v4.4-rc4 and
see how it works. You don't want anything between 4.2.6 and that
because of problems with blk req management. I'm hazy on the details.
>> I think it might be better to lower the queue depth, you might be just
>> blowing out the controller caches... hard to know.
Dallas> Good idea. I'll trying lowering to see what effect.
It might also make sense to try your tests starting with just 1 disk,
and then adding one more disk, re-running the tests, then another
disk, re-running the tests, etc.
Try with one on the MV, then one on the Cougar, then one on MV and one
on Cougar, etc.
Try to see if you can spot where the performance falls off the cliff.
Also, which disk scheduler are you using? Instead of CFQ, you might
try deadline instead.
As you can see, there's a TON of knobs to twiddle with, it's not a
simple thing to do at times.
John
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: John Stoffel @ 2015-12-11 19:30 UTC (permalink / raw)
To: P. Remek; +Cc: Phil Turmel, linux-raid
In-Reply-To: <CABdHLQ4++kTu22YGkdDW3r1UAxf_7E_+fcn9SfFnTSg6MgSm9w@mail.gmail.com>
>>>>> "P" == P Remek <p.remek1@googlemail.com> writes:
>> It's likely that the udev rules that drive incremental assembly don't
>> have the mapper aliases at the moment they run. Try:
>>
>> DEVICE /dev/dm*
>>
P> That crossed my mind too, but no, this doesn't help, the array still
P> doesn't show up
What distro are you based on? Have you re-built your initramfs and
then looked inside it to check that it made the changes?
Do you have everything setup as a module or a mix? That can certainly
change things. It might make sense to re-build the kernel so that the
MDADM stuff is modules, but the rest is compiled in, or visa versa.
John
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-11 18:10 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <22121.43026.985668.789870@quad.stoffel.home>
>
> But really, you need to provide details. Can you setup a small test
> MD RAID and show all the details?
>
I've provided all the details in the other response in this thread.
What do you want me to set up and what other details do you need?
Regards,
Remek
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-11 17:57 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <566AFC2A.7040009@turmel.org>
Not sure what to look for, the only relevant thing I see is:
>
> Next step would be to run udevadm trigger while running udevadm monitor
> in another session right after boot to see what events run and what
> rules are executed. That should help narrow this down. Also compare
> dmesg from boot with dmesg from udevadm trigger.
>
> I'm not familiar enough with ubuntu to be much more help.
UDEV [5341.078565] change /devices/virtual/block/dm-1 (block)
UDEV [5341.079255] change /devices/virtual/block/dm-2 (block)
UDEV [5341.082457] change /devices/virtual/block/dm-0 (block)
UDEV [5341.085049] change /devices/virtual/block/dm-6 (block)
UDEV [5341.087382] change /devices/virtual/block/dm-8 (block)
UDEV [5341.088389] change /devices/virtual/block/dm-7 (block)
UDEV [5341.089766] change /devices/virtual/block/dm-5 (block)
UDEV [5341.090079] change /devices/virtual/block/dm-4 (block)
UDEV [5341.090105] change /devices/virtual/block/dm-3 (block)
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: Dallas Clement @ 2015-12-11 16:47 UTC (permalink / raw)
To: John Stoffel; +Cc: Mark Knecht, Phil Turmel, Linux-RAID
In-Reply-To: <22122.64143.522908.45940@quad.stoffel.home>
On Fri, Dec 11, 2015 at 10:32 AM, John Stoffel <john@stoffel.org> wrote:
>>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
>
> Dallas> Hi Mark. I have three different controllers on this
> Dallas> motherboard. A Marvell 9485 controls 8 of the disks. And an
> Dallas> Intel Cougar Point controls the 4 remaining disks.
>
> What type of PCIe slots are the controllers in? And how fast are the
> controllers/drives? Are they SATA1/2/3 drives?
>
>>> If you're spinning in IO loops then it could be a driver issue.
>
> Dallas> It sure is looking like that. I will try to profile the
> Dallas> kernel threads today and maybe use blktrace as Phil
> Dallas> recommended to see what is going on there.
>
> what kernel aer you running?
>
> Dallas> This is pretty sad that 12 single threaded fio jobs can bring
> Dallas> this system to its knees.
>
> I think it might be better to lower the queue depth, you might be just
> blowing out the controller caches... hard to know.
Hi John.
> What type of PCIe slots are the controllers in? And how fast are the
> controllers/drives? Are they SATA1/2/3 drives?
The MV 9485 controller is attached to an Intel Sandy Bridge via PCIe
GEN2 x 8. This one controls 8 of the disks.
The Intel Cougar Point is connected to the Intel Sandy Bridge via DMI bus.
All of the drives are SATA III, however I do have two of the drives
connected to SATA II ports on the Cougar Point. These two drives used
to be connected to SATA III ports on a MV 9125/9120 controller. But
it had truly horrible write performance. Moving to the SATA II ports
on the Cougar Point boosted the performance close to the same as the
other drives. The remaining 10 drives are all connected to SATA III
ports.
> what kernel aer you running?
Right now, I'm using 3.10.69. But I have tried the 4.2 kernel in
Fedora 23 with similar results.
> I think it might be better to lower the queue depth, you might be just
> blowing out the controller caches... hard to know.
Good idea. I'll trying lowering to see what effect.
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: Phil Turmel @ 2015-12-11 16:39 UTC (permalink / raw)
To: P. Remek; +Cc: linux-raid
In-Reply-To: <CABdHLQ4++kTu22YGkdDW3r1UAxf_7E_+fcn9SfFnTSg6MgSm9w@mail.gmail.com>
On 12/11/2015 11:32 AM, P. Remek wrote:
>> It's likely that the udev rules that drive incremental assembly don't
>> have the mapper aliases at the moment they run. Try:
>>
>> DEVICE /dev/dm*
>
> That crossed my mind too, but no, this doesn't help, the array still
> doesn't show up
Next step would be to run udevadm trigger while running udevadm monitor
in another session right after boot to see what events run and what
rules are executed. That should help narrow this down. Also compare
dmesg from boot with dmesg from udevadm trigger.
I'm not familiar enough with ubuntu to be much more help.
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-11 16:32 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <566AF4E8.5030801@turmel.org>
> It's likely that the udev rules that drive incremental assembly don't
> have the mapper aliases at the moment they run. Try:
>
> DEVICE /dev/dm*
>
That crossed my mind too, but no, this doesn't help, the array still
doesn't show up
^ permalink raw reply
* Re: best base / worst case RAID 5,6 write speeds
From: John Stoffel @ 2015-12-11 16:32 UTC (permalink / raw)
To: Dallas Clement; +Cc: Mark Knecht, Phil Turmel, Linux-RAID
In-Reply-To: <CAE9DZURK+bZ=4czbGojzW815Du1ascr5vzAPtQBw4ZDGyq0MAQ@mail.gmail.com>
>>>>> "Dallas" == Dallas Clement <dallas.a.clement@gmail.com> writes:
Dallas> Hi Mark. I have three different controllers on this
Dallas> motherboard. A Marvell 9485 controls 8 of the disks. And an
Dallas> Intel Cougar Point controls the 4 remaining disks.
What type of PCIe slots are the controllers in? And how fast are the
controllers/drives? Are they SATA1/2/3 drives?
>> If you're spinning in IO loops then it could be a driver issue.
Dallas> It sure is looking like that. I will try to profile the
Dallas> kernel threads today and maybe use blktrace as Phil
Dallas> recommended to see what is going on there.
what kernel aer you running?
Dallas> This is pretty sad that 12 single threaded fio jobs can bring
Dallas> this system to its knees.
I think it might be better to lower the queue depth, you might be just
blowing out the controller caches... hard to know.
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: Phil Turmel @ 2015-12-11 16:08 UTC (permalink / raw)
To: P. Remek; +Cc: linux-raid
In-Reply-To: <CABdHLQ7Wdg1sUdC-Ryamdy+et6SNJRJnfdnRXSPOrWJYX9sT_g@mail.gmail.com>
{Convention on kernel.org is to trim and either interleave the reply or
bottom post. Please do.}
On 12/11/2015 11:01 AM, P. Remek wrote:
> Here is my config:
> root@os-node1:~# cat /etc/mdadm/mdadm.conf
>
> DEVICE /dev/mapper/hgst-ssd*
> CREATE owner=root group=disk mode=0660 auto=yes
> HOMEHOST <system>
> MAILADDR root
> ARRAY /dev/md0 metadata=0.90 UUID=4472a74f:ca206897:3b82bcad:daeae9f9
It's likely that the udev rules that drive incremental assembly don't
have the mapper aliases at the moment they run. Try:
DEVICE /dev/dm*
Phil
^ permalink raw reply
* Re: start dm-multipath before mdadm raid
From: P. Remek @ 2015-12-11 16:01 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <566AD7EA.4070903@turmel.org>
Here is my config:
root@os-node1:~# uname -a
Linux os-node1 3.16.0-48-generic #64~14.04.1-Ubuntu SMP Thu Aug 20
23:03:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@os-node1:~# cat /etc/mdadm/mdadm.conf
DEVICE /dev/mapper/hgst-ssd*
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR root
ARRAY /dev/md0 metadata=0.90 UUID=4472a74f:ca206897:3b82bcad:daeae9f9
root@os-node1:~# cat /etc/multipath.conf
blacklist {
wwid 3600508b1001cbb0c5b5a2fe0a85e0053
wwid 3600508b1001c33e8fe607dfd73043d03
wwid 32020030102060804
}
multipaths {
multipath {
wwid 35000cca04f1cbdfc
alias hgst-ssd1
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1cc050
alias hgst-ssd2
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1ce930
alias hgst-ssd3
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1ce1f4
alias hgst-ssd4
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1cbe18
alias hgst-ssd5
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
}
multipath {
wwid 35000cca04f1cdc58
alias hgst-ssd6
path_grouping_policy multibus
path_selector "round-robin 0"
failback 2
rr_weight uniform
no_path_retry 0
rr_min_io 1000
with this config I do:
root@os-node1:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-3.16.0-48-generic
Then I reboot the computer but the mdadm raid doesn't start up. The
multipathed devides does start up:
root@os-node1:~# multipath -ll
Error: : Inappropriate ioctl for device
cciss TUR failed in CCISS_GETLUNINFO: Inappropriate ioctl for device
Error: : Inappropriate ioctl for device
cciss TUR failed in CCISS_GETLUNINFO: Inappropriate ioctl for device
hgst-ssd6 (35000cca04f1cdc58) dm-5 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:2:0 sde 8:64 active ready running
`- 4:0:2:0 sdk 8:160 active ready running
hgst-ssd5 (35000cca04f1cbe18) dm-7 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:4:0 sdg 8:96 active ready running
`- 4:0:4:0 sdm 8:192 active ready running
hgst-ssd4 (35000cca04f1ce1f4) dm-3 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:0:0 sdc 8:32 active ready running
`- 4:0:0:0 sdi 8:128 active ready running
hgst-ssd3 (35000cca04f1ce930) dm-4 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:1:0 sdd 8:48 active ready running
`- 4:0:1:0 sdj 8:144 active ready running
hgst-ssd2 (35000cca04f1cc050) dm-8 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:5:0 sdh 8:112 active ready running
`- 4:0:5:0 sdn 8:208 active ready running
hgst-ssd1 (35000cca04f1cbdfc) dm-6 HGST ,HUSMM1680ASS200
size=745G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 3:0:3:0 sdf 8:80 active ready running
`- 4:0:3:0 sdl 8:176 active ready running
Regards,
Remek
On Fri, Dec 11, 2015 at 3:04 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/10/2015 05:44 PM, P. Remek wrote:
>>>
>>> Order of device discovery is not guaranteed, but base devices will
>>> almost always show up before multipath devices. You have to filter out
>>> the base devices from mdadm consideration:
>>>
>>> Add a DEVICES statement to your mdadm.conf that only matches your
>>> multipath device names, not the base names. Then update your initramfs
>>> so it applies to early boot as well.
>>>
>>
>>
>> This is something which I actually already tried. I specified in my
>> /etc/mdadm/mdadm.conf following:
>>
>> DEVICE /dev/mapper/hgst-ssd*
>>
>> In my /etc/multipath.conf I instruct dm-multipath to create
>> /dev/mapper/hgst-ssd1 /dev/mapper/hgst-ssd2 /dev/mapper/hgst-ssd3
>
> You haven't said that you've updated your initramfs with this info.
>
>> But the only result is that after this config, the md raid is not
>> started at all. My conclusion was that when md raid was starting, the
>> multipath devices did not yet exist so it did not start up the array.
>
> That would be an initramfs bug, most likely. Modern initramfs like
> dracut use incremental assembly to start arrays and subsystems as
> devices are started, regardless what order they show up.
>
> Phil
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox