* Re: [RFC 1/3] block: Introduce blk_bio_map_sg() to map one bio
From: Ming Lei @ 2016-05-25 8:52 UTC (permalink / raw)
To: Baolin Wang
Cc: Jens Axboe, Alasdair Kergon, Mike Snitzer,
open list:DEVICE-MAPPER (LVM), Herbert Xu, David Miller,
ebiggers3, js1304, tadeusz.struk, smueller, standby24x7,
Shaohua Li, Dan Williams, Martin K. Petersen, Sagi Grimberg,
Kent Overstreet, Keith Busch, Tejun Heo, Mark Brown,
Arnd Bergmann, linux-crypto, linux-block,
open list:SOFTWARE RAID (Multiple Disks) SUPPORT,
Linux Kernel Mailing List
In-Reply-To: <abe693fad3abb0adfc0cf9ca5b3d287fdb0f2d5b.1464144791.git.baolin.wang@linaro.org>
On Wed, May 25, 2016 at 2:12 PM, Baolin Wang <baolin.wang@linaro.org> wrote:
> In dm-crypt, it need to map one bio to scatterlist for improving the
> hardware engine encryption efficiency. Thus this patch introduces the
> blk_bio_map_sg() function to map one bio with scatterlists.
>
> Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
> ---
> block/blk-merge.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
> include/linux/blkdev.h | 3 +++
> 2 files changed, 48 insertions(+)
>
> diff --git a/block/blk-merge.c b/block/blk-merge.c
> index 2613531..9b92af4 100644
> --- a/block/blk-merge.c
> +++ b/block/blk-merge.c
> @@ -417,6 +417,51 @@ single_segment:
> }
>
> /*
> + * map a bio to scatterlist, return number of sg entries setup.
> + */
> +int blk_bio_map_sg(struct request_queue *q, struct bio *bio,
> + struct scatterlist *sglist,
> + struct scatterlist **sg)
> +{
> + struct bio_vec bvec, bvprv = { NULL };
> + struct bvec_iter iter;
> + int nsegs, cluster;
> +
> + nsegs = 0;
> + cluster = blk_queue_cluster(q);
> +
> + if (bio->bi_rw & REQ_DISCARD) {
> + /*
> + * This is a hack - drivers should be neither modifying the
> + * biovec, nor relying on bi_vcnt - but because of
> + * blk_add_request_payload(), a discard bio may or may not have
> + * a payload we need to set up here (thank you Christoph) and
> + * bi_vcnt is really the only way of telling if we need to.
> + */
> +
> + if (bio->bi_vcnt)
> + goto single_segment;
> +
> + return 0;
> + }
> +
> + if (bio->bi_rw & REQ_WRITE_SAME) {
> +single_segment:
> + *sg = sglist;
> + bvec = bio_iovec(bio);
> + sg_set_page(*sg, bvec.bv_page, bvec.bv_len, bvec.bv_offset);
> + return 1;
> + }
> +
> + bio_for_each_segment(bvec, bio, iter)
> + __blk_segment_map_sg(q, &bvec, sglist, &bvprv, sg,
> + &nsegs, &cluster);
> +
> + return nsegs;
> +}
> +EXPORT_SYMBOL(blk_bio_map_sg);
You can use __blk_bios_map_sg() to implement blk_bio_map_sg(),
then code duplication may be avoided.
> +
> +/*
> * map a request to scatterlist, return number of sg entries setup. Caller
> * must make sure sg can hold rq->nr_phys_segments entries
> */
> diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
> index 1fd8fdf..e5de4f8 100644
> --- a/include/linux/blkdev.h
> +++ b/include/linux/blkdev.h
> @@ -1013,6 +1013,9 @@ extern void blk_queue_write_cache(struct request_queue *q, bool enabled, bool fu
> extern struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev);
>
> extern int blk_rq_map_sg(struct request_queue *, struct request *, struct scatterlist *);
> +extern int blk_bio_map_sg(struct request_queue *q, struct bio *bio,
> + struct scatterlist *sglist,
> + struct scatterlist **sg);
> extern void blk_dump_rq_flags(struct request *, char *);
> extern long nr_blockdev_pages(void);
>
> --
> 1.7.9.5
>
^ permalink raw reply
* [RFC 3/3] md: dm-crypt: Introduce the bulk mode method when sending request
From: Baolin Wang @ 2016-05-25 6:12 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel, herbert, davem
Cc: ebiggers3, js1304, tadeusz.struk, smueller, standby24x7, shli,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, ming.lei, broonie, arnd, linux-crypto,
linux-block, linux-raid, linux-kernel, baolin.wang
In-Reply-To: <cover.1464144790.git.baolin.wang@linaro.org>
In now dm-crypt code, it is ineffective to map one segment (always one
sector) of one bio with just only one scatterlist at one time for hardware
crypto engine. Especially for some encryption mode (like ecb or xts mode)
cooperating with the crypto engine, they just need one initial IV or null
IV instead of different IV for each sector. In this situation We can consider
to use multiple scatterlists to map the whole bio and send all scatterlists
of one bio to crypto engine to encrypt or decrypt, which can improve the
hardware engine's efficiency.
With this optimization, On my test setup (beaglebone black board) using 64KB
I/Os on an eMMC storage device I saw about 60% improvement in throughput for
encrypted writes, and about 100% improvement for encrypted reads. But this
is not fit for other modes which need different IV for each sector.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/md/dm-crypt.c | 188 +++++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 176 insertions(+), 12 deletions(-)
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 4f3cb35..1c86ea7 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -33,6 +33,7 @@
#include <linux/device-mapper.h>
#define DM_MSG_PREFIX "crypt"
+#define DM_MAX_SG_LIST 1024
/*
* context holding the current state of a multi-part conversion
@@ -46,6 +47,8 @@ struct convert_context {
sector_t cc_sector;
atomic_t cc_pending;
struct skcipher_request *req;
+ struct sg_table sgt_in;
+ struct sg_table sgt_out;
};
/*
@@ -803,6 +806,108 @@ static struct crypt_iv_operations crypt_iv_tcw_ops = {
.post = crypt_iv_tcw_post
};
+/*
+ * Check how many sg entry numbers are needed when map one bio
+ * with scatterlists in advance.
+ */
+static unsigned int crypt_sg_entry(struct bio *bio_t)
+{
+ struct request_queue *q = bdev_get_queue(bio_t->bi_bdev);
+ int cluster = blk_queue_cluster(q);
+ struct bio_vec bvec, bvprv = { NULL };
+ struct bvec_iter biter;
+ unsigned long nbytes = 0, sg_length = 0;
+ unsigned int sg_cnt = 0, first_bvec = 0;
+
+ if (bio_t->bi_rw & REQ_DISCARD) {
+ if (bio_t->bi_vcnt)
+ return 1;
+ return 0;
+ }
+
+ if (bio_t->bi_rw & REQ_WRITE_SAME)
+ return 1;
+
+ bio_for_each_segment(bvec, bio_t, biter) {
+ nbytes = bvec.bv_len;
+
+ if (!cluster) {
+ sg_cnt++;
+ continue;
+ }
+
+ if (!first_bvec) {
+ first_bvec = 1;
+ goto new_segment;
+ }
+
+ if (sg_length + nbytes > queue_max_segment_size(q))
+ goto new_segment;
+
+ if (!BIOVEC_PHYS_MERGEABLE(&bvprv, &bvec))
+ goto new_segment;
+
+ if (!BIOVEC_SEG_BOUNDARY(q, &bvprv, &bvec))
+ goto new_segment;
+
+ sg_length += nbytes;
+ continue;
+
+new_segment:
+ memcpy(&bvprv, &bvec, sizeof(struct bio_vec));
+ sg_length = nbytes;
+ sg_cnt++;
+ }
+
+ return sg_cnt;
+}
+
+static int crypt_convert_alloc_table(struct crypt_config *cc,
+ struct convert_context *ctx)
+{
+ struct bio *bio_in = ctx->bio_in;
+ struct bio *bio_out = ctx->bio_out;
+ unsigned int mode = skcipher_is_bulk_mode(any_tfm(cc));
+ unsigned int sg_in_max, sg_out_max;
+ int ret = 0;
+
+ if (!mode)
+ goto out2;
+
+ /*
+ * Need to calculate how many sg entry need to be used
+ * for this bio.
+ */
+ sg_in_max = crypt_sg_entry(bio_in) + 1;
+ if (sg_in_max > DM_MAX_SG_LIST || sg_in_max <= 2)
+ goto out2;
+
+ ret = sg_alloc_table(&ctx->sgt_in, sg_in_max, GFP_KERNEL);
+ if (ret)
+ goto out2;
+
+ if (bio_data_dir(bio_in) == READ)
+ goto out1;
+
+ sg_out_max = crypt_sg_entry(bio_out) + 1;
+ if (sg_out_max > DM_MAX_SG_LIST || sg_out_max <= 2)
+ goto out3;
+
+ ret = sg_alloc_table(&ctx->sgt_out, sg_out_max, GFP_KERNEL);
+ if (ret)
+ goto out3;
+
+ return 0;
+
+out3:
+ sg_free_table(&ctx->sgt_in);
+out2:
+ ctx->sgt_in.orig_nents = 0;
+out1:
+ ctx->sgt_out.orig_nents = 0;
+ return ret;
+}
+
static void crypt_convert_init(struct crypt_config *cc,
struct convert_context *ctx,
struct bio *bio_out, struct bio *bio_in,
@@ -843,7 +948,13 @@ static int crypt_convert_block(struct crypt_config *cc,
{
struct bio_vec bv_in = bio_iter_iovec(ctx->bio_in, ctx->iter_in);
struct bio_vec bv_out = bio_iter_iovec(ctx->bio_out, ctx->iter_out);
+ unsigned int mode = skcipher_is_bulk_mode(any_tfm(cc));
+ struct bio *bio_in = ctx->bio_in;
+ struct bio *bio_out = ctx->bio_out;
+ unsigned int total_bytes = bio_in->bi_iter.bi_size;
struct dm_crypt_request *dmreq;
+ struct scatterlist *sg_in;
+ struct scatterlist *sg_out;
u8 *iv;
int r;
@@ -852,16 +963,6 @@ static int crypt_convert_block(struct crypt_config *cc,
dmreq->iv_sector = ctx->cc_sector;
dmreq->ctx = ctx;
- sg_init_table(&dmreq->sg_in, 1);
- sg_set_page(&dmreq->sg_in, bv_in.bv_page, 1 << SECTOR_SHIFT,
- bv_in.bv_offset);
-
- sg_init_table(&dmreq->sg_out, 1);
- sg_set_page(&dmreq->sg_out, bv_out.bv_page, 1 << SECTOR_SHIFT,
- bv_out.bv_offset);
-
- bio_advance_iter(ctx->bio_in, &ctx->iter_in, 1 << SECTOR_SHIFT);
- bio_advance_iter(ctx->bio_out, &ctx->iter_out, 1 << SECTOR_SHIFT);
if (cc->iv_gen_ops) {
r = cc->iv_gen_ops->generator(cc, iv, dmreq);
@@ -869,8 +970,63 @@ static int crypt_convert_block(struct crypt_config *cc,
return r;
}
- skcipher_request_set_crypt(req, &dmreq->sg_in, &dmreq->sg_out,
- 1 << SECTOR_SHIFT, iv);
+ if (mode && ctx->sgt_in.orig_nents > 0) {
+ struct scatterlist *sg = NULL;
+ unsigned int total_sg_in, total_sg_out;
+
+ total_sg_in = blk_bio_map_sg(bdev_get_queue(bio_in->bi_bdev),
+ bio_in, ctx->sgt_in.sgl, &sg);
+ if ((total_sg_in <= 0) ||
+ (total_sg_in > ctx->sgt_in.orig_nents)) {
+ DMERR("%s in sg map error %d, sg table nents[%d]\n",
+ __func__, total_sg_in, ctx->sgt_in.orig_nents);
+ return -EINVAL;
+ }
+
+ if (sg)
+ sg_mark_end(sg);
+
+ ctx->iter_in.bi_size -= total_bytes;
+ sg_in = ctx->sgt_in.sgl;
+ sg_out = ctx->sgt_in.sgl;
+
+ if (bio_data_dir(bio_in) == READ)
+ goto set_crypt;
+
+ sg = NULL;
+ total_sg_out = blk_bio_map_sg(bdev_get_queue(bio_out->bi_bdev),
+ bio_out, ctx->sgt_out.sgl, &sg);
+ if ((total_sg_out <= 0) ||
+ (total_sg_out > ctx->sgt_out.orig_nents)) {
+ DMERR("%s out sg map error %d, sg table nents[%d]\n",
+ __func__, total_sg_out, ctx->sgt_out.orig_nents);
+ return -EINVAL;
+ }
+
+ if (sg)
+ sg_mark_end(sg);
+
+ ctx->iter_out.bi_size -= total_bytes;
+ sg_out = ctx->sgt_out.sgl;
+ } else {
+ sg_init_table(&dmreq->sg_in, 1);
+ sg_set_page(&dmreq->sg_in, bv_in.bv_page, 1 << SECTOR_SHIFT,
+ bv_in.bv_offset);
+
+ sg_init_table(&dmreq->sg_out, 1);
+ sg_set_page(&dmreq->sg_out, bv_out.bv_page, 1 << SECTOR_SHIFT,
+ bv_out.bv_offset);
+
+ bio_advance_iter(ctx->bio_in, &ctx->iter_in, 1 << SECTOR_SHIFT);
+ bio_advance_iter(ctx->bio_out, &ctx->iter_out, 1 << SECTOR_SHIFT);
+
+ sg_in = &dmreq->sg_in;
+ sg_out = &dmreq->sg_out;
+ total_bytes = 1 << SECTOR_SHIFT;
+ }
+
+set_crypt:
+ skcipher_request_set_crypt(req, sg_in, sg_out, total_bytes, iv);
if (bio_data_dir(ctx->bio_in) == WRITE)
r = crypto_skcipher_encrypt(req);
@@ -1081,6 +1237,8 @@ static void crypt_dec_pending(struct dm_crypt_io *io)
if (io->ctx.req)
crypt_free_req(cc, io->ctx.req, base_bio);
+ sg_free_table(&io->ctx.sgt_in);
+ sg_free_table(&io->ctx.sgt_out);
base_bio->bi_error = error;
bio_endio(base_bio);
}
@@ -1312,6 +1470,9 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
io->ctx.iter_out = clone->bi_iter;
sector += bio_sectors(clone);
+ r = crypt_convert_alloc_table(cc, &io->ctx);
+ if (r < 0)
+ io->error = -EIO;
crypt_inc_pending(io);
r = crypt_convert(cc, &io->ctx);
@@ -1343,6 +1504,9 @@ static void kcryptd_crypt_read_convert(struct dm_crypt_io *io)
crypt_convert_init(cc, &io->ctx, io->base_bio, io->base_bio,
io->sector);
+ r = crypt_convert_alloc_table(cc, &io->ctx);
+ if (r < 0)
+ io->error = -EIO;
r = crypt_convert(cc, &io->ctx);
if (r < 0)
--
1.7.9.5
^ permalink raw reply related
* [RFC 2/3] crypto: Introduce CRYPTO_ALG_BULK flag
From: Baolin Wang @ 2016-05-25 6:12 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel, herbert, davem
Cc: ebiggers3, js1304, tadeusz.struk, smueller, standby24x7, shli,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, ming.lei, broonie, arnd, linux-crypto,
linux-block, linux-raid, linux-kernel, baolin.wang
In-Reply-To: <cover.1464144790.git.baolin.wang@linaro.org>
Now some cipher hardware engines prefer to handle bulk block rather than one
sector (512 bytes) created by dm-crypt, cause these cipher engines can handle
the intermediate values (IV) by themselves in one bulk block. This means we
can increase the size of the request by merging request rather than always 512
bytes and thus increase the hardware engine processing speed.
So introduce 'CRYPTO_ALG_BULK' flag to indicate this cipher can support bulk
mode.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
include/crypto/skcipher.h | 7 +++++++
include/linux/crypto.h | 6 ++++++
2 files changed, 13 insertions(+)
diff --git a/include/crypto/skcipher.h b/include/crypto/skcipher.h
index 0f987f5..d89d29a 100644
--- a/include/crypto/skcipher.h
+++ b/include/crypto/skcipher.h
@@ -519,5 +519,12 @@ static inline void skcipher_request_set_crypt(
req->iv = iv;
}
+static inline unsigned int skcipher_is_bulk_mode(struct crypto_skcipher *sk_tfm)
+{
+ struct crypto_tfm *tfm = crypto_skcipher_tfm(sk_tfm);
+
+ return crypto_tfm_alg_bulk(tfm);
+}
+
#endif /* _CRYPTO_SKCIPHER_H */
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 6e28c89..a315487 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -63,6 +63,7 @@
#define CRYPTO_ALG_DEAD 0x00000020
#define CRYPTO_ALG_DYING 0x00000040
#define CRYPTO_ALG_ASYNC 0x00000080
+#define CRYPTO_ALG_BULK 0x00000100
/*
* Set this bit if and only if the algorithm requires another algorithm of
@@ -623,6 +624,11 @@ static inline u32 crypto_tfm_alg_type(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_TYPE_MASK;
}
+static inline unsigned int crypto_tfm_alg_bulk(struct crypto_tfm *tfm)
+{
+ return tfm->__crt_alg->cra_flags & CRYPTO_ALG_BULK;
+}
+
static inline unsigned int crypto_tfm_alg_blocksize(struct crypto_tfm *tfm)
{
return tfm->__crt_alg->cra_blocksize;
--
1.7.9.5
^ permalink raw reply related
* [RFC 1/3] block: Introduce blk_bio_map_sg() to map one bio
From: Baolin Wang @ 2016-05-25 6:12 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel, herbert, davem
Cc: ebiggers3, js1304, tadeusz.struk, smueller, standby24x7, shli,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, ming.lei, broonie, arnd, linux-crypto,
linux-block, linux-raid, linux-kernel, baolin.wang
In-Reply-To: <cover.1464144790.git.baolin.wang@linaro.org>
In dm-crypt, it need to map one bio to scatterlist for improving the
hardware engine encryption efficiency. Thus this patch introduces the
blk_bio_map_sg() function to map one bio with scatterlists.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
block/blk-merge.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
include/linux/blkdev.h | 3 +++
2 files changed, 48 insertions(+)
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 2613531..9b92af4 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -417,6 +417,51 @@ single_segment:
}
/*
+ * map a bio to scatterlist, return number of sg entries setup.
+ */
+int blk_bio_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist,
+ struct scatterlist **sg)
+{
+ struct bio_vec bvec, bvprv = { NULL };
+ struct bvec_iter iter;
+ int nsegs, cluster;
+
+ nsegs = 0;
+ cluster = blk_queue_cluster(q);
+
+ if (bio->bi_rw & REQ_DISCARD) {
+ /*
+ * This is a hack - drivers should be neither modifying the
+ * biovec, nor relying on bi_vcnt - but because of
+ * blk_add_request_payload(), a discard bio may or may not have
+ * a payload we need to set up here (thank you Christoph) and
+ * bi_vcnt is really the only way of telling if we need to.
+ */
+
+ if (bio->bi_vcnt)
+ goto single_segment;
+
+ return 0;
+ }
+
+ if (bio->bi_rw & REQ_WRITE_SAME) {
+single_segment:
+ *sg = sglist;
+ bvec = bio_iovec(bio);
+ sg_set_page(*sg, bvec.bv_page, bvec.bv_len, bvec.bv_offset);
+ return 1;
+ }
+
+ bio_for_each_segment(bvec, bio, iter)
+ __blk_segment_map_sg(q, &bvec, sglist, &bvprv, sg,
+ &nsegs, &cluster);
+
+ return nsegs;
+}
+EXPORT_SYMBOL(blk_bio_map_sg);
+
+/*
* map a request to scatterlist, return number of sg entries setup. Caller
* must make sure sg can hold rq->nr_phys_segments entries
*/
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 1fd8fdf..e5de4f8 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1013,6 +1013,9 @@ extern void blk_queue_write_cache(struct request_queue *q, bool enabled, bool fu
extern struct backing_dev_info *blk_get_backing_dev_info(struct block_device *bdev);
extern int blk_rq_map_sg(struct request_queue *, struct request *, struct scatterlist *);
+extern int blk_bio_map_sg(struct request_queue *q, struct bio *bio,
+ struct scatterlist *sglist,
+ struct scatterlist **sg);
extern void blk_dump_rq_flags(struct request *, char *);
extern long nr_blockdev_pages(void);
--
1.7.9.5
^ permalink raw reply related
* [RFC 0/3] Introduce the bulk mode method when sending request to crypto layer
From: Baolin Wang @ 2016-05-25 6:12 UTC (permalink / raw)
To: axboe, agk, snitzer, dm-devel, herbert, davem
Cc: ebiggers3, js1304, tadeusz.struk, smueller, standby24x7, shli,
dan.j.williams, martin.petersen, sagig, kent.overstreet,
keith.busch, tj, ming.lei, broonie, arnd, linux-crypto,
linux-block, linux-raid, linux-kernel, baolin.wang
This patchset will check if the cipher can support bulk mode, then dm-crypt
will handle different ways to send requests to crypto layer according to
cipher mode.
Looking forward to any comments and suggestions. Thanks.
Baolin Wang (3):
block: Introduce blk_bio_map_sg() to map one bio
crypto: Introduce CRYPTO_ALG_BULK flag
md: dm-crypt: Introduce the bulk mode method when sending request
block/blk-merge.c | 45 +++++++++++
drivers/md/dm-crypt.c | 188 ++++++++++++++++++++++++++++++++++++++++++---
include/crypto/skcipher.h | 7 ++
include/linux/blkdev.h | 3 +
include/linux/crypto.h | 6 ++
5 files changed, 237 insertions(+), 12 deletions(-)
--
1.7.9.5
^ permalink raw reply
* [BISECT dbba42d8] dm: regression: large bio splits fail with dm-crypt under bcache in dm_make_request()
From: Eric Wheeler @ 2016-05-24 19:35 UTC (permalink / raw)
To: James Johnston
Cc: 'Tim Small', 'Kent Overstreet',
'Alasdair Kergon', 'Mike Snitzer', linux-bcache,
dm-devel, dm-crypt, 'Neil Brown', linux-raid,
'Mikulas Patocka'
[ New subject after bisect:
Was: bcache gets stuck flushing writeback cache when used in
combination with LUKS/dm-crypt and non-default bucket size ]
Mikulas, Mike, please see below when you have a minute. This bisect found
a commit with your signoffs. This affects all kernels >= v4.4.
Should the commit be reverted, or is there a better fix?
Intentional top post, more below:
On Sun, 22 May 2016, James Johnston wrote:
> > On Fri, 20 May 2016, James Johnston wrote:
> >
> > > > On Mon, 16 May 2016, Tim Small wrote:
> > > >
> > > > > On 08/05/16 19:39, James Johnston wrote:
> > > > > > I've run into a problem where the bcache writeback cache can't be flushed to
> > > > > > disk when the backing device is a LUKS / dm-crypt device and the cache set has
> > > > > > a non-default bucket size. Basically, only a few megabytes will be flushed to
> > > > > > disk, and then it gets stuck. Stuck means that the bcache writeback task
> > > > > > thrashes the disk by constantly reading hundreds of MB/second from the cache set
> > > > > > in an infinite loop, while not actually progressing (dirty_data never decreases
> > > > > > beyond a certain point).
> > > > >
> > > > > > [...]
> > > > >
> > > > > > The situation is basically unrecoverable as far as I can tell: if you attempt
> > > > > > to detach the cache set then the cache set disk gets thrashed extra-hard
> > > > > > forever, and it's impossible to actually get the cache set detached. The only
> > > > > > solution seems to be to back up the data and destroy the volume...
> > > > >
> > > > > You can boot an older kernel to flush the device without destroying it
> > > > > (I'm guessing that's because older kernels split down the big requests
> > > > > which are failing on the 4.4 kernel). Once flushed you could put the
> > > > > cache into writethrough mode, or use a smaller bucket size.
> > > >
> > > > Indeed, can someone test 4.1.y and see if the problem persists with a 2M
> > > > bucket size? (If someone has already tested 4.1, then appologies as I've
> > > > not yet seen that report.)
> > > >
> > > > If 4.1 works, then I think a bisect is in order. Such a bisect would at
> > > > least highlight the problem and might indicate a (hopefully trivial) fix.
> > >
> > > To help narrow this down, I tested the following generic pre-compiled mainline kernels
> > > on Ubuntu 15.10:
> > >
> > > * WORKS: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.6-wily/
> > > * DOES NOT WORK: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4-rc1+cod1-wily/
> > >
> > > I also tried the default & latest distribution-provided 4.2 kernel. It worked.
> > > This one also worked:
> > >
> > > * WORKS: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2.8-wily/
> > >
> > > So it seems to me that it is a regression from 4.3.6 kernel to any 4.4 kernel. That
> > > should help save time with bisection...
> >
> > Below is the patchlist for md and block that might help with a place to
> > start. Are there any other places in the Linux tree where we should watch
> > for changes?
> >
> > I'm wondering if it might be in dm-4.4-changes since this is dm-crypt
> > related, but it could be ac322de which was quite large.
> >
> > James or Tim,
> >
> > Can you try building ac322de? If that produces the problem, then there
> > are only 3 more to try (unless this was actually a problem in 4.3 which
> > was fixed in 4.3.y, but hopefully that isn't so).
> >
> > ccf21b6 is probably the next to test to rule out neil's big md patch,
> > which Linus abreviated in the commit log so it must be quite long. OTOH,
> > if dm-4.4-changes works, then I'm not sure what commit might produce the
> > problem because the rest are not obviously relevant to the issue that are
> > more recent.
>
> So I decided to go ahead and bisect it today. Looks like the bad commit is
> this one. The commit prior flushed the bcache writeback cache without
> incident; this one does not and I guess caused this bcache regression.
> (FWIW ac322de came up during bisection, and tested good.)
>
> johnstonj@kernel-build:~/linux$ git bisect bad
> dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7 is the first bad commit
> commit dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7
> Author: Mikulas Patocka <mpatocka@redhat.com>
> Date: Wed Oct 21 16:34:20 2015 -0400
>
> dm: eliminate unused "bioset" process for each bio-based DM device
>
> Commit 54efd50bfd873e2dbf784e0b21a8027ba4299a3e ("block: make
> generic_make_request handle arbitrarily sized bios") makes it possible
> for block devices to process large bios. In doing so that commit
> allocates a new queue->bio_split bioset for each block device, this
> bioset is used for allocating bios when the driver needs to split large
> bios.
>
> Each bioset allocates a workqueue process, thus the above commit
> increases the number of processes allocated per block device.
>
> DM doesn't need the queue->bio_split bioset, thus we can deallocate it.
> This reduces the number of allocated processes per bio-based DM device
> from 3 to 2. Also remove the call to blk_queue_split(), it is not
> needed because DM does its own splitting.
>
> Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> Signed-off-by: Mike Snitzer <snitzer@redhat.com>
>
> The patch for this commit is very brief; reproduced here:
>
> diff --git a/drivers/md/dm.c b/drivers/md/dm.c
> index 9555843..64b50b7 100644
> --- a/drivers/md/dm.c
> +++ b/drivers/md/dm.c
> @@ -1763,8 +1763,6 @@ static void dm_make_request(struct request_queue *q, struct bio *bio)
>
> map = dm_get_live_table(md, &srcu_idx);
>
> - blk_queue_split(q, &bio, q->bio_split);
> -
> generic_start_io_acct(rw, bio_sectors(bio), &dm_disk(md)->part0);
>
> /* if we're suspended, we have to queue this io for later */
> @@ -2792,6 +2790,12 @@ int dm_setup_md_queue(struct mapped_device *md)
> case DM_TYPE_BIO_BASED:
> dm_init_old_md_queue(md);
> blk_queue_make_request(md->queue, dm_make_request);
> + /*
> + * DM handles splitting bios as needed. Free the bio_split bioset
> + * since it won't be used (saves 1 process per bio-based DM device).
> + */
> + bioset_free(md->queue->bio_split);
> + md->queue->bio_split = NULL;
> break;
> }
>
> Here is the bisect log:
>
> johnstonj@kernel-build:~/linux$ git bisect log
> git bisect start
> # good: [6a13feb9c82803e2b815eca72fa7a9f5561d7861] Linux 4.3
> git bisect good 6a13feb9c82803e2b815eca72fa7a9f5561d7861
> # bad: [8005c49d9aea74d382f474ce11afbbc7d7130bec] Linux 4.4-rc1
> git bisect bad 8005c49d9aea74d382f474ce11afbbc7d7130bec
> # bad: [118c216e16c5ccb028cd03a0dcd56d17a07ff8d7] Merge tag 'staging-4.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> git bisect bad 118c216e16c5ccb028cd03a0dcd56d17a07ff8d7
> # good: [e627078a0cbdc0c391efeb5a2c4eb287328fd633] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
> git bisect good e627078a0cbdc0c391efeb5a2c4eb287328fd633
> # good: [c17c6da659571a115c7b4983da6c6ac464317c34] staging: wilc1000: rename pfScanResult of struct scan_attr
> git bisect good c17c6da659571a115c7b4983da6c6ac464317c34
> # good: [7bdb7d554e0e433b92b63f3472523cc3067f8ab4] Staging: rtl8192u: ieee80211: corrected indent
> git bisect good 7bdb7d554e0e433b92b63f3472523cc3067f8ab4
> # good: [ac322de6bf5416cb145b58599297b8be73cd86ac] Merge tag 'md/4.4' of git://neil.brown.name/md
> git bisect good ac322de6bf5416cb145b58599297b8be73cd86ac
> # good: [a4d8e93c3182a54d8d21a4d1cec6538ae1be9e16] Merge tag 'usb-for-v4.4' of git://git.kernel.org/pub/scm/linux/kernel/git/balbi/usb into usb-next
> git bisect good a4d8e93c3182a54d8d21a4d1cec6538ae1be9e16
> # good: [4f56f3fdca43c9a18339b6e0c3b1aa2f57f6d0b0] serial: 8250: Tolerate clock variance for max baud rate
> git bisect good 4f56f3fdca43c9a18339b6e0c3b1aa2f57f6d0b0
> # good: [e052c6d15c61cc4caff2f06cbca72b183da9f15e] tty: Use unbound workqueue for all input workers
> git bisect good e052c6d15c61cc4caff2f06cbca72b183da9f15e
> # good: [b9ca0c948c921e960006aaf319a29c004917cdf6] uwb: neh: Use setup_timer
> git bisect good b9ca0c948c921e960006aaf319a29c004917cdf6
> # bad: [aad9ae4550755edc020b5c511a8b54f0104b2f47] dm switch: simplify conditional in alloc_region_table()
> git bisect bad aad9ae4550755edc020b5c511a8b54f0104b2f47
> # good: [a3d939ae7b5f82688a6d3450f95286eaea338328] dm: convert ffs to __ffs
> git bisect good a3d939ae7b5f82688a6d3450f95286eaea338328
> # bad: [00272c854ee17b804ce81ef706f611dac17f4f89] dm linear: remove redundant target name from error messages
> git bisect bad 00272c854ee17b804ce81ef706f611dac17f4f89
> # bad: [4c7da06f5a780bbf44ebd7547789e48536d0a823] dm persistent data: eliminate unnecessary return values
> git bisect bad 4c7da06f5a780bbf44ebd7547789e48536d0a823
> # bad: [dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7] dm: eliminate unused "bioset" process for each bio-based DM device
> git bisect bad dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7
> # first bad commit: [dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7] dm: eliminate unused "bioset" process for each bio-based DM device
>
> Commands used for testing:
>
> # Make cache set
> make-bcache --bucket 2M -C /dev/sdb
> # Set up backing device crypto
> cryptsetup luksFormat /dev/sdc
> cryptsetup open --type luks /dev/sdc backCrypt
> # Make backing device & enable writeback
> make-bcache -B /dev/mapper/backCrypt
> bcache-super-show /dev/sdb | grep cset.uuid | cut -f 3 > /sys/block/bcache0/bcache/attach
> echo writeback > /sys/block/bcache0/bcache/cache_mode
>
> # KILL SEQUENCE
>
> cd /sys/block/bcache0/bcache
> echo 0 > sequential_cutoff
> # Verify that the cache is attached (i.e. does not say "no cache")
> cat state
> dd if=/dev/urandom of=/dev/bcache0 bs=1M count=250
> cat dirty_data
> cat state
> # Next line causes severe disk thrashing and failure to flush writeback cache
> # on bad commits.
> echo 1 > detach
> cat dirty_data
> cat state
>
> Hope this provides some insight into the problem...
>
> James
>
>
>
--
Eric Wheeler
^ permalink raw reply
* Re: [PATCH v4 10/21] fs: Check for invalid i_uid in may_follow_link()
From: Djalal Harouni @ 2016-05-24 15:55 UTC (permalink / raw)
To: Seth Forshee
Cc: Eric W. Biederman, Alexander Viro, Serge Hallyn,
Richard Weinberger, Austin S Hemmelgarn, Miklos Szeredi,
Pavel Tikhomirov, linux-kernel, linux-bcache, dm-devel,
linux-raid, linux-mtd, linux-fsdevel, fuse-devel,
linux-security-module, selinux, cgroups
In-Reply-To: <1461699396-33000-11-git-send-email-seth.forshee@canonical.com>
On Tue, Apr 26, 2016 at 02:36:23PM -0500, Seth Forshee wrote:
> Filesystem uids which don't map into a user namespace may result
> in inode->i_uid being INVALID_UID. A symlink and its parent
> could have different owners in the filesystem can both get
> mapped to INVALID_UID, which may result in following a symlink
> when this would not have otherwise been permitted when protected
> symlinks are enabled.
>
> Add a new helper function, uid_valid_eq(), and use this to
> validate that the ids in may_follow_link() are both equal and
> valid. Also add an equivalent helper for gids, which is
> currently unused.
>
> Signed-off-by: Seth Forshee <seth.forshee@canonical.com>
> Acked-by: Serge Hallyn <serge.hallyn@canonical.com>
Reviewed-by: Djalal Harouni <tixxdz@opendz.org>
> ---
> fs/namei.c | 2 +-
> include/linux/uidgid.h | 10 ++++++++++
> 2 files changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/fs/namei.c b/fs/namei.c
> index a29094c6f4a1..6fe8b0d8ca90 100644
> --- a/fs/namei.c
> +++ b/fs/namei.c
> @@ -915,7 +915,7 @@ static inline int may_follow_link(struct nameidata *nd)
> return 0;
>
> /* Allowed if parent directory and link owner match. */
> - if (uid_eq(parent->i_uid, inode->i_uid))
> + if (uid_valid_eq(parent->i_uid, inode->i_uid))
> return 0;
>
> if (nd->flags & LOOKUP_RCU)
> diff --git a/include/linux/uidgid.h b/include/linux/uidgid.h
> index 03835522dfcb..e09529fe2668 100644
> --- a/include/linux/uidgid.h
> +++ b/include/linux/uidgid.h
> @@ -117,6 +117,16 @@ static inline bool gid_valid(kgid_t gid)
> return __kgid_val(gid) != (gid_t) -1;
> }
>
> +static inline bool uid_valid_eq(kuid_t left, kuid_t right)
> +{
> + return uid_eq(left, right) && uid_valid(left);
> +}
> +
> +static inline bool gid_valid_eq(kgid_t left, kgid_t right)
> +{
> + return gid_eq(left, right) && gid_valid(left);
> +}
> +
> #ifdef CONFIG_USER_NS
>
> extern kuid_t make_kuid(struct user_namespace *from, uid_t uid);
> --
> 2.7.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Djalal Harouni
http://opendz.org
^ permalink raw reply
* [PATCH] right meaning of PARITY_ENABLE_RMW and PARITY_PREFER_RMW
From: Song Liu @ 2016-05-24 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: Song Liu, Shaohua Li
In current handle_stripe_dirtying, the code prefers rmw with
PARITY_ENABLE_RMW; while prefers rcw with PARITY_PREFER_RMW.
This patch reverses this behavior.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 8959e6d..ad9e15a 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3600,7 +3600,7 @@ static void handle_stripe_dirtying(struct r5conf *conf,
pr_debug("for sector %llu, rmw=%d rcw=%d\n",
(unsigned long long)sh->sector, rmw, rcw);
set_bit(STRIPE_HANDLE, &sh->state);
- if ((rmw < rcw || (rmw == rcw && conf->rmw_level == PARITY_ENABLE_RMW)) && rmw > 0) {
+ if ((rmw < rcw || (rmw == rcw && conf->rmw_level == PARITY_PREFER_RMW)) && rmw > 0) {
/* prefer read-modify-write, but need to get some data */
if (conf->mddev->queue)
blk_add_trace_msg(conf->mddev->queue,
@@ -3627,7 +3627,7 @@ static void handle_stripe_dirtying(struct r5conf *conf,
}
}
}
- if ((rcw < rmw || (rcw == rmw && conf->rmw_level != PARITY_ENABLE_RMW)) && rcw > 0) {
+ if ((rcw < rmw || (rcw == rmw && conf->rmw_level != PARITY_PREFER_RMW)) && rcw > 0) {
/* want reconstruct write, but need to get some data */
int qread =0;
rcw = 0;
--
2.8.0.rc2
^ permalink raw reply related
* Re: [PATCH] Need update superblock on time when deciding to do reshape
From: Shaohua Li @ 2016-05-23 23:02 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid, Jes Sorensen
In-Reply-To: <1653970376.53638422.1464038381331.JavaMail.zimbra@redhat.com>
On Mon, May 23, 2016 at 05:19:41PM -0400, Xiao Ni wrote:
>
>
> ----- Original Message -----
> > From: "Shaohua Li" <shli@kernel.org>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org, "Jes Sorensen" <Jes.Sorensen@redhat.com>
> > Sent: Tuesday, May 24, 2016 3:02:04 AM
> > Subject: Re: [PATCH] Need update superblock on time when deciding to do reshape
> >
> > On Sat, May 21, 2016 at 10:14:07PM -0400, Xiao Ni wrote:
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Shaohua Li" <shli@kernel.org>
> > > > To: "Xiao Ni" <xni@redhat.com>
> > > > Cc: linux-raid@vger.kernel.org, "Jes Sorensen" <Jes.Sorensen@redhat.com>
> > > > Sent: Saturday, May 21, 2016 1:59:46 AM
> > > > Subject: Re: [PATCH] Need update superblock on time when deciding to do
> > > > reshape
> > > >
> > > > On Tue, May 17, 2016 at 04:54:09PM +0800, Xiao Ni wrote:
> > > > > Hi all
> > > > >
> > > > > If the disks are not enough to have spaces for relocating the
> > > > > data_offset,
> > > > > it needs to run start_reshape and then run mdadm --grow --continue by
> > > > > systemd. But mdadm --grow --continue fails because it checkes that
> > > > > info->reshape_active is 0.
> > > > >
> > > > > The info->reshape_active is set to 1 when the superblock feature_map
> > > > > have the flag MD_FEATURE_RESHAPE_ACTIVE. Superblock feature_map is set
> > > > > MD_FEATURE_RESHAPE_ACTIVE as mddev->reshape_position != MaxSector.
> > > > >
> > > > > Function start_reshape calls raid5_start_reshape which changes
> > > > > mddev->reshape_position to 0. Then in md_check_recovery it updates the
> > > > > superblock to underlying devices. But there is a chance that the
> > > > > superblock
> > > > > haven't written to underlying devices, the mdadm reads the superblock
> > > > > data.
> > > > > So mdadm --grow --continue fails.
> > > > >
> > > > > The steps to reproduce this:
> > > > > mdadm -CR /dev/md0 -l5 -n3 /dev/loop[0-2] --bitmap=internal
> > > > > mdadm --wait /dev/md0
> > > > > mdadm /dev/md0 -a /dev/loop3
> > > > > mdadm --grow --raid-devices 4 /dev/md0
> > > > > The loop device size is 500MB
> > > > >
> > > > > [root@storageqe-09 ~]# cat /proc/mdstat
> > > > > Personalities : [raid6] [raid5] [raid4]
> > > > > md0 : active raid5 loop3[4] loop2[3] loop1[1] loop0[0]
> > > > > 1021952 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4]
> > > > > [UUUU]
> > > > > [>....................] reshape = 0.0% (1/510976) finish=0.0min
> > > > > speed=255488K/sec
> > > > > bitmap: 1/1 pages [4KB], 65536KB chunk
> > > >
> > > > what's the bad effect of the --continue failure? I think reshape will
> > > > still
> > > > continue. Doing a update super there is ok, but I'm wondering if it's the
> > > > good
> > > > way. Could mdadm wait for MD_FEATURE_RESHAPE_ACTIVE then let systemd run?
> > > > Because sounds like we are working around systemd bug, kernel itself will
> > > > write
> > > > superblock anyway soon, so we probably working around in userspace.
> > >
> > > There is no bad effect if --continue failure. It just miss one chance to
> > > go on reshaping. Yes, as you said we can fix this in userspace. I tried
> > > to start mdadm-grow-continue@.service 30 seconds later as
> > > mdadm-last-resort@.service
> > > does. It can fix this too.
> > >
> > > Sure it should be fixed this if mdadm try more times to check
> > > MD_FEATURE_RESHAPE_ACTIVE.
> > >
> > > >
> > > > > unused devices: <none>
> > > > >
> > > > > So if we update the superblock on time, mdadm can read the right
> > > > > superblock
> > > > > data.
> > > > >
> > > > > Signed-off-by <xni@redhat.com>
> > > > >
> > > > > ---
> > > > > drivers/md/md.c | 1 +
> > > > > 1 file changed, 1 insertion(+)
> > > > >
> > > > > diff --git a/drivers/md/md.c b/drivers/md/md.c
> > > > > index 14d3b37..7919606 100644
> > > > > --- a/drivers/md/md.c
> > > > > +++ b/drivers/md/md.c
> > > > > @@ -4350,6 +4350,7 @@ action_store(struct mddev *mddev, const char
> > > > > *page,
> > > > > size_t len)
> > > > > else {
> > > > > clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> > > > > err = mddev->pers->start_reshape(mddev);
> > > > > + md_update_sb(mddev, 1);
> > > >
> > > > write super even err != 0?
> > >
> > > Ah, sorry for this. It should update superblock only err is 0.
> > >
> > > At first I want to fix this in userspace, but I ask myself why shouldn't
> > > update
> > > the superblock once start_reshape returns. There are some guys waiting for
> > > the
> > > update. The function action_store should update the superblock in time to
> > > tell
> > > the colleagues what happens now. Then I checked the places where
> > > md_update_sb
> > > is called. I found it is indeed called in some places where the md device
> > > changes. So I sent the patch here.
> > >
> > > By the way, in size_store it update the superblock when err != 0. Is it
> > > right
> > > to check err there?
> >
> > I think we should. probably nobody noticed it before.
> >
>
> So does it mean you applied the way fixing the problem in kernel space? If
> you apply the method I'll re-send the patch.
No, I didn't. I thought you agree with fixing it in userspace. For the err != 0
checking, I mean we probably should do it for size_store if somebody wants to
fix it.
^ permalink raw reply
* Re: [PATCH] Need update superblock on time when deciding to do reshape
From: Xiao Ni @ 2016-05-23 21:19 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Jes Sorensen
In-Reply-To: <20160523190204.GA129477@kernel.org>
----- Original Message -----
> From: "Shaohua Li" <shli@kernel.org>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org, "Jes Sorensen" <Jes.Sorensen@redhat.com>
> Sent: Tuesday, May 24, 2016 3:02:04 AM
> Subject: Re: [PATCH] Need update superblock on time when deciding to do reshape
>
> On Sat, May 21, 2016 at 10:14:07PM -0400, Xiao Ni wrote:
> >
> >
> > ----- Original Message -----
> > > From: "Shaohua Li" <shli@kernel.org>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: linux-raid@vger.kernel.org, "Jes Sorensen" <Jes.Sorensen@redhat.com>
> > > Sent: Saturday, May 21, 2016 1:59:46 AM
> > > Subject: Re: [PATCH] Need update superblock on time when deciding to do
> > > reshape
> > >
> > > On Tue, May 17, 2016 at 04:54:09PM +0800, Xiao Ni wrote:
> > > > Hi all
> > > >
> > > > If the disks are not enough to have spaces for relocating the
> > > > data_offset,
> > > > it needs to run start_reshape and then run mdadm --grow --continue by
> > > > systemd. But mdadm --grow --continue fails because it checkes that
> > > > info->reshape_active is 0.
> > > >
> > > > The info->reshape_active is set to 1 when the superblock feature_map
> > > > have the flag MD_FEATURE_RESHAPE_ACTIVE. Superblock feature_map is set
> > > > MD_FEATURE_RESHAPE_ACTIVE as mddev->reshape_position != MaxSector.
> > > >
> > > > Function start_reshape calls raid5_start_reshape which changes
> > > > mddev->reshape_position to 0. Then in md_check_recovery it updates the
> > > > superblock to underlying devices. But there is a chance that the
> > > > superblock
> > > > haven't written to underlying devices, the mdadm reads the superblock
> > > > data.
> > > > So mdadm --grow --continue fails.
> > > >
> > > > The steps to reproduce this:
> > > > mdadm -CR /dev/md0 -l5 -n3 /dev/loop[0-2] --bitmap=internal
> > > > mdadm --wait /dev/md0
> > > > mdadm /dev/md0 -a /dev/loop3
> > > > mdadm --grow --raid-devices 4 /dev/md0
> > > > The loop device size is 500MB
> > > >
> > > > [root@storageqe-09 ~]# cat /proc/mdstat
> > > > Personalities : [raid6] [raid5] [raid4]
> > > > md0 : active raid5 loop3[4] loop2[3] loop1[1] loop0[0]
> > > > 1021952 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4]
> > > > [UUUU]
> > > > [>....................] reshape = 0.0% (1/510976) finish=0.0min
> > > > speed=255488K/sec
> > > > bitmap: 1/1 pages [4KB], 65536KB chunk
> > >
> > > what's the bad effect of the --continue failure? I think reshape will
> > > still
> > > continue. Doing a update super there is ok, but I'm wondering if it's the
> > > good
> > > way. Could mdadm wait for MD_FEATURE_RESHAPE_ACTIVE then let systemd run?
> > > Because sounds like we are working around systemd bug, kernel itself will
> > > write
> > > superblock anyway soon, so we probably working around in userspace.
> >
> > There is no bad effect if --continue failure. It just miss one chance to
> > go on reshaping. Yes, as you said we can fix this in userspace. I tried
> > to start mdadm-grow-continue@.service 30 seconds later as
> > mdadm-last-resort@.service
> > does. It can fix this too.
> >
> > Sure it should be fixed this if mdadm try more times to check
> > MD_FEATURE_RESHAPE_ACTIVE.
> >
> > >
> > > > unused devices: <none>
> > > >
> > > > So if we update the superblock on time, mdadm can read the right
> > > > superblock
> > > > data.
> > > >
> > > > Signed-off-by <xni@redhat.com>
> > > >
> > > > ---
> > > > drivers/md/md.c | 1 +
> > > > 1 file changed, 1 insertion(+)
> > > >
> > > > diff --git a/drivers/md/md.c b/drivers/md/md.c
> > > > index 14d3b37..7919606 100644
> > > > --- a/drivers/md/md.c
> > > > +++ b/drivers/md/md.c
> > > > @@ -4350,6 +4350,7 @@ action_store(struct mddev *mddev, const char
> > > > *page,
> > > > size_t len)
> > > > else {
> > > > clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> > > > err = mddev->pers->start_reshape(mddev);
> > > > + md_update_sb(mddev, 1);
> > >
> > > write super even err != 0?
> >
> > Ah, sorry for this. It should update superblock only err is 0.
> >
> > At first I want to fix this in userspace, but I ask myself why shouldn't
> > update
> > the superblock once start_reshape returns. There are some guys waiting for
> > the
> > update. The function action_store should update the superblock in time to
> > tell
> > the colleagues what happens now. Then I checked the places where
> > md_update_sb
> > is called. I found it is indeed called in some places where the md device
> > changes. So I sent the patch here.
> >
> > By the way, in size_store it update the superblock when err != 0. Is it
> > right
> > to check err there?
>
> I think we should. probably nobody noticed it before.
>
So does it mean you applied the way fixing the problem in kernel space? If
you apply the method I'll re-send the patch.
Best Regards
Xiao
^ permalink raw reply
* Re: [PATCH] Need update superblock on time when deciding to do reshape
From: Shaohua Li @ 2016-05-23 19:02 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid, Jes Sorensen
In-Reply-To: <953124641.53356599.1463883247414.JavaMail.zimbra@redhat.com>
On Sat, May 21, 2016 at 10:14:07PM -0400, Xiao Ni wrote:
>
>
> ----- Original Message -----
> > From: "Shaohua Li" <shli@kernel.org>
> > To: "Xiao Ni" <xni@redhat.com>
> > Cc: linux-raid@vger.kernel.org, "Jes Sorensen" <Jes.Sorensen@redhat.com>
> > Sent: Saturday, May 21, 2016 1:59:46 AM
> > Subject: Re: [PATCH] Need update superblock on time when deciding to do reshape
> >
> > On Tue, May 17, 2016 at 04:54:09PM +0800, Xiao Ni wrote:
> > > Hi all
> > >
> > > If the disks are not enough to have spaces for relocating the data_offset,
> > > it needs to run start_reshape and then run mdadm --grow --continue by
> > > systemd. But mdadm --grow --continue fails because it checkes that
> > > info->reshape_active is 0.
> > >
> > > The info->reshape_active is set to 1 when the superblock feature_map
> > > have the flag MD_FEATURE_RESHAPE_ACTIVE. Superblock feature_map is set
> > > MD_FEATURE_RESHAPE_ACTIVE as mddev->reshape_position != MaxSector.
> > >
> > > Function start_reshape calls raid5_start_reshape which changes
> > > mddev->reshape_position to 0. Then in md_check_recovery it updates the
> > > superblock to underlying devices. But there is a chance that the superblock
> > > haven't written to underlying devices, the mdadm reads the superblock data.
> > > So mdadm --grow --continue fails.
> > >
> > > The steps to reproduce this:
> > > mdadm -CR /dev/md0 -l5 -n3 /dev/loop[0-2] --bitmap=internal
> > > mdadm --wait /dev/md0
> > > mdadm /dev/md0 -a /dev/loop3
> > > mdadm --grow --raid-devices 4 /dev/md0
> > > The loop device size is 500MB
> > >
> > > [root@storageqe-09 ~]# cat /proc/mdstat
> > > Personalities : [raid6] [raid5] [raid4]
> > > md0 : active raid5 loop3[4] loop2[3] loop1[1] loop0[0]
> > > 1021952 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4]
> > > [UUUU]
> > > [>....................] reshape = 0.0% (1/510976) finish=0.0min
> > > speed=255488K/sec
> > > bitmap: 1/1 pages [4KB], 65536KB chunk
> >
> > what's the bad effect of the --continue failure? I think reshape will still
> > continue. Doing a update super there is ok, but I'm wondering if it's the
> > good
> > way. Could mdadm wait for MD_FEATURE_RESHAPE_ACTIVE then let systemd run?
> > Because sounds like we are working around systemd bug, kernel itself will
> > write
> > superblock anyway soon, so we probably working around in userspace.
>
> There is no bad effect if --continue failure. It just miss one chance to
> go on reshaping. Yes, as you said we can fix this in userspace. I tried
> to start mdadm-grow-continue@.service 30 seconds later as mdadm-last-resort@.service
> does. It can fix this too.
>
> Sure it should be fixed this if mdadm try more times to check MD_FEATURE_RESHAPE_ACTIVE.
>
> >
> > > unused devices: <none>
> > >
> > > So if we update the superblock on time, mdadm can read the right superblock
> > > data.
> > >
> > > Signed-off-by <xni@redhat.com>
> > >
> > > ---
> > > drivers/md/md.c | 1 +
> > > 1 file changed, 1 insertion(+)
> > >
> > > diff --git a/drivers/md/md.c b/drivers/md/md.c
> > > index 14d3b37..7919606 100644
> > > --- a/drivers/md/md.c
> > > +++ b/drivers/md/md.c
> > > @@ -4350,6 +4350,7 @@ action_store(struct mddev *mddev, const char *page,
> > > size_t len)
> > > else {
> > > clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> > > err = mddev->pers->start_reshape(mddev);
> > > + md_update_sb(mddev, 1);
> >
> > write super even err != 0?
>
> Ah, sorry for this. It should update superblock only err is 0.
>
> At first I want to fix this in userspace, but I ask myself why shouldn't update
> the superblock once start_reshape returns. There are some guys waiting for the
> update. The function action_store should update the superblock in time to tell
> the colleagues what happens now. Then I checked the places where md_update_sb
> is called. I found it is indeed called in some places where the md device
> changes. So I sent the patch here.
>
> By the way, in size_store it update the superblock when err != 0. Is it right
> to check err there?
I think we should. probably nobody noticed it before.
Thanks,
Shaohua
^ permalink raw reply
* [BUG] call trace in raid5 do_release_stripe() is triggered during I/O stress test in kernel 3.19-rc8
From: Joey Liao @ 2016-05-23 4:12 UTC (permalink / raw)
To: linux-raid
Hi,
We start to do the I/O stress test for raid5 in kernel 3.19.8. During
the stress test, the raid5 is clean (not
resyncing/rebuilding/reshaping) and it is not degraded as well.
However, we meet a machine reboot problem and we believe it hugely
related to the following call trace:
Ps.
1. The call trace shows suddenly and the previous messages were 1 hour ago.
2. It is not easy to reproduce but it has been taken place twice this week.
It is really a serious problem to us, would you please help to provide
us some suggestions to solve the problem or at least to figure out the
root cause? Thanks.
=== related source code ===
static void do_release_stripe(struct r5conf *conf, struct stripe_head *sh,
struct list_head *temp_inactive_list)
{
BUG_ON(!list_empty(&sh->lru));
....
}
=== related call trace ===
kernel BUG at drivers/md/raid5.c:299!
<4>[53230.958843] invalid opcode: 0000 [#1] SMP
<4>[53230.962991] Modules linked in: thunderbolt xt_mark
ipt_MASQUERADE iptable_nat nf_nat_masquerade_ipv4 nf_nat_ipv4 nf_nat
ppp_deflate bsd_comp ppp_mppe ppp_async ppp_generic slhc tun
iscsi_tcp(O) libiscsi_tcp(O) libiscsi(O) scsi_transport_iscsi(O)
iscsi_target_mod target_core_file target_core_iblock target_core_mod
fbdisk(O) bonding bridge stp ipv6 uvcvideo videobuf2_vmalloc
videobuf2_memops videobuf2_core snd_usb_caiaq snd_usb_audio
snd_usbmidi_lib snd_seq_midi snd_rawmidi fnotify(PO) udf isofs
iTCO_wdt psnap llc ufsd(PO) jnl(O) pl2303 usbserial intel_ips drbd(O)
flashcache(O) dm_thin_pool dm_bio_prison dm_persistent_data
hal_netlink(O) coretemp ixgbe mdio r8152 usbnet mii igb e1000e(O)
mpt3sas mpt2sas scsi_transport_sas raid_class uas usb_storage xhci_pci
xhci_hcd usblp uhci_hcd ehci_pci ehci_hcd
<4>[53231.035304] CPU: 2 PID: 4316 Comm: md1_raid5 Tainted: P U
O 3.19.8 #1
<4>[53231.042620] Hardware name: Default string Default string/SKYBAY,
BIOS QX80AT04 03/29/2016
<4>[53231.050818] task: ffff8802b383a0d0 ti: ffff88029e734000 task.ti:
ffff88029e734000
<4>[53231.058308] RIP: 0010:[] [] do_release_stripe+0x190/0x1a0
<4>[53231.066953] RSP: 0000:ffff88029e737c48 EFLAGS: 00210006
<4>[53231.072283] RAX: 000000000000d201 RBX: ffff8802b05aec98 RCX:
0000000000000001
<4>[53231.079426] RDX: ffff8802aed74b48 RSI: ffff8802b05aec98 RDI:
ffff8802aed74800
<4>[53231.086571] RBP: ffff88029e737c68 R08: 0000000000000000 R09:
0000000000000000
<4>[53231.093714] R10: ffff88029e737b28 R11: 0000000000000002 R12:
ffff8802aed74800
<4>[53231.100858] R13: ffff8802b05aeca8 R14: ffff8802aed74b48 R15:
ffff88029e737d08
<4>[53231.108001] FS: 0000000000000000(0000)
GS:ffff8802bdd00000(0000) knlGS:0000000000000000
<4>[53231.116099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[53231.121856] CR2: 00000000f76ade38 CR3: 0000000001c7d000 CR4:
00000000003407e0
<4>[53231.129017] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
<4>[53231.136161] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
<4>[53231.143304] Stack:
<4>[53231.145326] ffff88029e737cd0 0000000000000001 ffff8802aed74800
0000000000000008
<4>[53231.152818] ffff88029e737c78 ffffffff81698bbb ffff88029e737d38
ffffffff816a2318
<4>[53231.160312] ffff8802aed74800 ffff8802a903ece8 0000000000000000
ffff88029e737e58
<4>[53231.167835] Call Trace:
<4>[53231.170295] [] __release_stripe+0x1b/0x30
<4>[53231.175966] [] handle_active_stripes+0x3d8/0x450
<4>[53231.182243] [] raid5d+0x399/0x630
<4>[53231.187221] [] md_thread+0x7d/0x130
<4>[53231.192369] [] ? woken_wake_function+0x20/0x20
<4>[53231.198473] [] ? errors_store+0x70/0x70
<4>[53231.203985] [] kthread+0xe3/0xf0
<4>[53231.208874] [] ? kthreadd+0x160/0x160
<4>[53231.214198] [] ret_from_fork+0x58/0x90
<4>[53231.219608] [] ? kthreadd+0x160/0x160
<4>[53231.224931] Code: b8 49 8b 44 24 18 48 8b b8 48 01 00 00 e8 89
fe 00 00 eb a5 48 89 df e8 0f fd ff ff e9 14 ff ff ff 0f 0b eb fe 66
0f 1f 44 00 00 <0f> 0b eb fe 0f 0b eb fe 0f 1f 84 00 00 00 00 00 55 48
89 e5 e8
<1>[53231.245381] RIP [] do_release_stripe+0x190/0x1a0
<4>[53231.251681] RSP
^ permalink raw reply
* Re: Write intent bitmap algorithm
From: NeilBrown @ 2016-05-22 23:19 UTC (permalink / raw)
To: Ankur Bose, linux-raid
In-Reply-To: <cd14f4dd-a88a-4089-9b07-f16a221e003c@default>
[-- Attachment #1: Type: text/plain, Size: 1837 bytes --]
On Fri, May 20 2016, Ankur Bose wrote:
> Hi All,
>
> We are doing performance analysis of MD raid1 and wanted to know the flow/algorithm of the writes with write intent bitmap.
> As per "man md" below lines gives the algo
>
> "Before any write request is honored, md will make sure that the corresponding bit in the log is set. After a period of time with no writes to an area of the array, the corresponding bit will be cleared."
>
> Here we wanted to know after what period of time and what is the logic
> to calculate this period of time which MD does ?
When you create an array with a bitmap, or use --grow to add a bitmap,
you can provide the --delay option to give a delay in seconds (which
seems to not be documented in mdadm.8 :-( Patches welcomem.
Alternately when the array is running you can inspected and change
/sys/block/mdXXX/md/bitmap_time_base
Every this-many seconds a task runs in the kernel which marks any bits
with a count that recently reach zero as "should be written", updates
the on-disk bitmap to clear any bits that were marked "should be
written".
So a bit with an active-write count of zero will be cleared in the
non-disk bitmap between 1 and 2 delays after the count reaches zero.
>
> I see a function "void bitmap_daemon_work(struct mddev *mddev)" in bitmap.c file but trying to figure out the logic how it is used.
Yes, this is the task that is run every 'delay' or 'time_base' seconds.
The counter is offset bit 2 normally so when there are no active writes
bitmap_daemon_work will find a counter which is '2' and will change it
to '1'. If it finds a counter that is '1' it will change it to '0' and
immediately clear the bit in the in-memory image of the bitmap. Then it
will write out any pages of the bitmap which have changed.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* RE: bcache gets stuck flushing writeback cache when used in combination with LUKS/dm-crypt and non-default bucket size
From: James Johnston @ 2016-05-22 4:26 UTC (permalink / raw)
To: 'Eric Wheeler'
Cc: 'Tim Small', 'Kent Overstreet',
'Alasdair Kergon', 'Mike Snitzer', linux-bcache,
dm-devel, dm-crypt, 'Neil Brown', linux-raid,
'Mikulas Patocka'
In-Reply-To: <alpine.LRH.2.11.1605201322530.26328@mx.ewheeler.net>
> On Fri, 20 May 2016, James Johnston wrote:
>
> > > On Mon, 16 May 2016, Tim Small wrote:
> > >
> > > > On 08/05/16 19:39, James Johnston wrote:
> > > > > I've run into a problem where the bcache writeback cache can't be flushed to
> > > > > disk when the backing device is a LUKS / dm-crypt device and the cache set has
> > > > > a non-default bucket size. Basically, only a few megabytes will be flushed to
> > > > > disk, and then it gets stuck. Stuck means that the bcache writeback task
> > > > > thrashes the disk by constantly reading hundreds of MB/second from the cache set
> > > > > in an infinite loop, while not actually progressing (dirty_data never decreases
> > > > > beyond a certain point).
> > > >
> > > > > [...]
> > > >
> > > > > The situation is basically unrecoverable as far as I can tell: if you attempt
> > > > > to detach the cache set then the cache set disk gets thrashed extra-hard
> > > > > forever, and it's impossible to actually get the cache set detached. The only
> > > > > solution seems to be to back up the data and destroy the volume...
> > > >
> > > > You can boot an older kernel to flush the device without destroying it
> > > > (I'm guessing that's because older kernels split down the big requests
> > > > which are failing on the 4.4 kernel). Once flushed you could put the
> > > > cache into writethrough mode, or use a smaller bucket size.
> > >
> > > Indeed, can someone test 4.1.y and see if the problem persists with a 2M
> > > bucket size? (If someone has already tested 4.1, then appologies as I've
> > > not yet seen that report.)
> > >
> > > If 4.1 works, then I think a bisect is in order. Such a bisect would at
> > > least highlight the problem and might indicate a (hopefully trivial) fix.
> >
> > To help narrow this down, I tested the following generic pre-compiled mainline kernels
> > on Ubuntu 15.10:
> >
> > * WORKS: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.6-wily/
> > * DOES NOT WORK: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4-rc1+cod1-wily/
> >
> > I also tried the default & latest distribution-provided 4.2 kernel. It worked.
> > This one also worked:
> >
> > * WORKS: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.2.8-wily/
> >
> > So it seems to me that it is a regression from 4.3.6 kernel to any 4.4 kernel. That
> > should help save time with bisection...
>
> Below is the patchlist for md and block that might help with a place to
> start. Are there any other places in the Linux tree where we should watch
> for changes?
>
> I'm wondering if it might be in dm-4.4-changes since this is dm-crypt
> related, but it could be ac322de which was quite large.
>
> James or Tim,
>
> Can you try building ac322de? If that produces the problem, then there
> are only 3 more to try (unless this was actually a problem in 4.3 which
> was fixed in 4.3.y, but hopefully that isn't so).
>
> ccf21b6 is probably the next to test to rule out neil's big md patch,
> which Linus abreviated in the commit log so it must be quite long. OTOH,
> if dm-4.4-changes works, then I'm not sure what commit might produce the
> problem because the rest are not obviously relevant to the issue that are
> more recent.
So I decided to go ahead and bisect it today. Looks like the bad commit is
this one. The commit prior flushed the bcache writeback cache without
incident; this one does not and I guess caused this bcache regression.
(FWIW ac322de came up during bisection, and tested good.)
johnstonj@kernel-build:~/linux$ git bisect bad
dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7 is the first bad commit
commit dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7
Author: Mikulas Patocka <mpatocka@redhat.com>
Date: Wed Oct 21 16:34:20 2015 -0400
dm: eliminate unused "bioset" process for each bio-based DM device
Commit 54efd50bfd873e2dbf784e0b21a8027ba4299a3e ("block: make
generic_make_request handle arbitrarily sized bios") makes it possible
for block devices to process large bios. In doing so that commit
allocates a new queue->bio_split bioset for each block device, this
bioset is used for allocating bios when the driver needs to split large
bios.
Each bioset allocates a workqueue process, thus the above commit
increases the number of processes allocated per block device.
DM doesn't need the queue->bio_split bioset, thus we can deallocate it.
This reduces the number of allocated processes per bio-based DM device
from 3 to 2. Also remove the call to blk_queue_split(), it is not
needed because DM does its own splitting.
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
The patch for this commit is very brief; reproduced here:
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 9555843..64b50b7 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1763,8 +1763,6 @@ static void dm_make_request(struct request_queue *q, struct bio *bio)
map = dm_get_live_table(md, &srcu_idx);
- blk_queue_split(q, &bio, q->bio_split);
-
generic_start_io_acct(rw, bio_sectors(bio), &dm_disk(md)->part0);
/* if we're suspended, we have to queue this io for later */
@@ -2792,6 +2790,12 @@ int dm_setup_md_queue(struct mapped_device *md)
case DM_TYPE_BIO_BASED:
dm_init_old_md_queue(md);
blk_queue_make_request(md->queue, dm_make_request);
+ /*
+ * DM handles splitting bios as needed. Free the bio_split bioset
+ * since it won't be used (saves 1 process per bio-based DM device).
+ */
+ bioset_free(md->queue->bio_split);
+ md->queue->bio_split = NULL;
break;
}
Here is the bisect log:
johnstonj@kernel-build:~/linux$ git bisect log
git bisect start
# good: [6a13feb9c82803e2b815eca72fa7a9f5561d7861] Linux 4.3
git bisect good 6a13feb9c82803e2b815eca72fa7a9f5561d7861
# bad: [8005c49d9aea74d382f474ce11afbbc7d7130bec] Linux 4.4-rc1
git bisect bad 8005c49d9aea74d382f474ce11afbbc7d7130bec
# bad: [118c216e16c5ccb028cd03a0dcd56d17a07ff8d7] Merge tag 'staging-4.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect bad 118c216e16c5ccb028cd03a0dcd56d17a07ff8d7
# good: [e627078a0cbdc0c391efeb5a2c4eb287328fd633] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
git bisect good e627078a0cbdc0c391efeb5a2c4eb287328fd633
# good: [c17c6da659571a115c7b4983da6c6ac464317c34] staging: wilc1000: rename pfScanResult of struct scan_attr
git bisect good c17c6da659571a115c7b4983da6c6ac464317c34
# good: [7bdb7d554e0e433b92b63f3472523cc3067f8ab4] Staging: rtl8192u: ieee80211: corrected indent
git bisect good 7bdb7d554e0e433b92b63f3472523cc3067f8ab4
# good: [ac322de6bf5416cb145b58599297b8be73cd86ac] Merge tag 'md/4.4' of git://neil.brown.name/md
git bisect good ac322de6bf5416cb145b58599297b8be73cd86ac
# good: [a4d8e93c3182a54d8d21a4d1cec6538ae1be9e16] Merge tag 'usb-for-v4.4' of git://git.kernel.org/pub/scm/linux/kernel/git/balbi/usb into usb-next
git bisect good a4d8e93c3182a54d8d21a4d1cec6538ae1be9e16
# good: [4f56f3fdca43c9a18339b6e0c3b1aa2f57f6d0b0] serial: 8250: Tolerate clock variance for max baud rate
git bisect good 4f56f3fdca43c9a18339b6e0c3b1aa2f57f6d0b0
# good: [e052c6d15c61cc4caff2f06cbca72b183da9f15e] tty: Use unbound workqueue for all input workers
git bisect good e052c6d15c61cc4caff2f06cbca72b183da9f15e
# good: [b9ca0c948c921e960006aaf319a29c004917cdf6] uwb: neh: Use setup_timer
git bisect good b9ca0c948c921e960006aaf319a29c004917cdf6
# bad: [aad9ae4550755edc020b5c511a8b54f0104b2f47] dm switch: simplify conditional in alloc_region_table()
git bisect bad aad9ae4550755edc020b5c511a8b54f0104b2f47
# good: [a3d939ae7b5f82688a6d3450f95286eaea338328] dm: convert ffs to __ffs
git bisect good a3d939ae7b5f82688a6d3450f95286eaea338328
# bad: [00272c854ee17b804ce81ef706f611dac17f4f89] dm linear: remove redundant target name from error messages
git bisect bad 00272c854ee17b804ce81ef706f611dac17f4f89
# bad: [4c7da06f5a780bbf44ebd7547789e48536d0a823] dm persistent data: eliminate unnecessary return values
git bisect bad 4c7da06f5a780bbf44ebd7547789e48536d0a823
# bad: [dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7] dm: eliminate unused "bioset" process for each bio-based DM device
git bisect bad dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7
# first bad commit: [dbba42d8a9ebddcc1c1412e8457f79f3cb6ef6e7] dm: eliminate unused "bioset" process for each bio-based DM device
Commands used for testing:
# Make cache set
make-bcache --bucket 2M -C /dev/sdb
# Set up backing device crypto
cryptsetup luksFormat /dev/sdc
cryptsetup open --type luks /dev/sdc backCrypt
# Make backing device & enable writeback
make-bcache -B /dev/mapper/backCrypt
bcache-super-show /dev/sdb | grep cset.uuid | cut -f 3 > /sys/block/bcache0/bcache/attach
echo writeback > /sys/block/bcache0/bcache/cache_mode
# KILL SEQUENCE
cd /sys/block/bcache0/bcache
echo 0 > sequential_cutoff
# Verify that the cache is attached (i.e. does not say "no cache")
cat state
dd if=/dev/urandom of=/dev/bcache0 bs=1M count=250
cat dirty_data
cat state
# Next line causes severe disk thrashing and failure to flush writeback cache
# on bad commits.
echo 1 > detach
cat dirty_data
cat state
Hope this provides some insight into the problem...
James
^ permalink raw reply related
* Re: [PATCH] Need update superblock on time when deciding to do reshape
From: Xiao Ni @ 2016-05-22 2:14 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid, Jes Sorensen
In-Reply-To: <20160520175946.GA76216@kernel.org>
----- Original Message -----
> From: "Shaohua Li" <shli@kernel.org>
> To: "Xiao Ni" <xni@redhat.com>
> Cc: linux-raid@vger.kernel.org, "Jes Sorensen" <Jes.Sorensen@redhat.com>
> Sent: Saturday, May 21, 2016 1:59:46 AM
> Subject: Re: [PATCH] Need update superblock on time when deciding to do reshape
>
> On Tue, May 17, 2016 at 04:54:09PM +0800, Xiao Ni wrote:
> > Hi all
> >
> > If the disks are not enough to have spaces for relocating the data_offset,
> > it needs to run start_reshape and then run mdadm --grow --continue by
> > systemd. But mdadm --grow --continue fails because it checkes that
> > info->reshape_active is 0.
> >
> > The info->reshape_active is set to 1 when the superblock feature_map
> > have the flag MD_FEATURE_RESHAPE_ACTIVE. Superblock feature_map is set
> > MD_FEATURE_RESHAPE_ACTIVE as mddev->reshape_position != MaxSector.
> >
> > Function start_reshape calls raid5_start_reshape which changes
> > mddev->reshape_position to 0. Then in md_check_recovery it updates the
> > superblock to underlying devices. But there is a chance that the superblock
> > haven't written to underlying devices, the mdadm reads the superblock data.
> > So mdadm --grow --continue fails.
> >
> > The steps to reproduce this:
> > mdadm -CR /dev/md0 -l5 -n3 /dev/loop[0-2] --bitmap=internal
> > mdadm --wait /dev/md0
> > mdadm /dev/md0 -a /dev/loop3
> > mdadm --grow --raid-devices 4 /dev/md0
> > The loop device size is 500MB
> >
> > [root@storageqe-09 ~]# cat /proc/mdstat
> > Personalities : [raid6] [raid5] [raid4]
> > md0 : active raid5 loop3[4] loop2[3] loop1[1] loop0[0]
> > 1021952 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4]
> > [UUUU]
> > [>....................] reshape = 0.0% (1/510976) finish=0.0min
> > speed=255488K/sec
> > bitmap: 1/1 pages [4KB], 65536KB chunk
>
> what's the bad effect of the --continue failure? I think reshape will still
> continue. Doing a update super there is ok, but I'm wondering if it's the
> good
> way. Could mdadm wait for MD_FEATURE_RESHAPE_ACTIVE then let systemd run?
> Because sounds like we are working around systemd bug, kernel itself will
> write
> superblock anyway soon, so we probably working around in userspace.
There is no bad effect if --continue failure. It just miss one chance to
go on reshaping. Yes, as you said we can fix this in userspace. I tried
to start mdadm-grow-continue@.service 30 seconds later as mdadm-last-resort@.service
does. It can fix this too.
Sure it should be fixed this if mdadm try more times to check MD_FEATURE_RESHAPE_ACTIVE.
>
> > unused devices: <none>
> >
> > So if we update the superblock on time, mdadm can read the right superblock
> > data.
> >
> > Signed-off-by <xni@redhat.com>
> >
> > ---
> > drivers/md/md.c | 1 +
> > 1 file changed, 1 insertion(+)
> >
> > diff --git a/drivers/md/md.c b/drivers/md/md.c
> > index 14d3b37..7919606 100644
> > --- a/drivers/md/md.c
> > +++ b/drivers/md/md.c
> > @@ -4350,6 +4350,7 @@ action_store(struct mddev *mddev, const char *page,
> > size_t len)
> > else {
> > clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> > err = mddev->pers->start_reshape(mddev);
> > + md_update_sb(mddev, 1);
>
> write super even err != 0?
Ah, sorry for this. It should update superblock only err is 0.
At first I want to fix this in userspace, but I ask myself why shouldn't update
the superblock once start_reshape returns. There are some guys waiting for the
update. The function action_store should update the superblock in time to tell
the colleagues what happens now. Then I checked the places where md_update_sb
is called. I found it is indeed called in some places where the md device
changes. So I sent the patch here.
By the way, in size_store it update the superblock when err != 0. Is it right
to check err there?
Best Regards
Xiao
>
> Thanks,
> Shaohua
>
^ permalink raw reply
* Re: suggest disk numbers in a raidset?
From: John Stoffel @ 2016-05-21 18:29 UTC (permalink / raw)
To: d tbsky; +Cc: linux-raid
In-Reply-To: <CAC6SzH+MEJHmec7_9NF3u7zjW0C46LozJUKfo4Oy3faySCRpXw@mail.gmail.com>
>>>>> "d" == d tbsky <tbskyd@gmail.com> writes:
d> 2016-05-20 22:31 GMT+08:00 John Stoffel <john@stoffel.org>:
>> I wouldn't even think of using hardware RAID in this situation, and
>> I'd also not think about maximizing the size of the RAID6 volumes as
>> well, due to rebuild speed penalty. Another issue to think about is
>> chunk size as the number of members in a RAID array go up. Say you
>> have the default 64k block size (number pulled from thin air...), so
>> you need to have N * 64K worth of data before you can write a full
>> stripe of data. So as your writing data, you'll want to keep the
>> block size down.
d> thanks for sharing the thought. maybe I should lower the chunk size
d> for full stripe write.
Maybe... since you'll be doing large streaming writes, it might not be
a big problem in the long run. And esp since it's probably not high
performance either.
>> But back to goal. If you're writing large files, since I think NVR
>> refers to CCTV camera files, please correct me if wrong, you should
>> just stick with the defaults in terms of RAID defaults.
d> yes NVR refers CCTV files.
>> What I would do just just create a RAID6 array with 10 disks, so you
>> only have 8 x 4Tb of data, with two parity disks. Then create another
>> RAID6 with the remainng 10 disks. Then you would add them as PVs into
>> LVM, and then stripe acroos them. Something like this:
d> yes I will use lvm to combine the array if necessary. but 10 disks
d> with raid6 will use only 80% of disk capacity. I had use 16 disks
d> before and it seems ok.
Disk is cheap, but in your case it sounds like space/cost is the
driving factor.
>> # Since you want large space, make the extents use larger chunks here
>> # in the VG.
>> vgcreate -s 16 NVR /dev/md100 /dev/md101
d> thanks for the suggestion. I will study it.
>> I'd want redundant power supplies, some hot spare disks, a UPS, and a
>> rock solid hardware with plenty of memory. The other issue is that
>> unless you run a recent linux kernel, you might run into performance
>> problems with the RAID5/6 parity calculations being all done on a
>> single CPU core. Newer versions should have fixed this, but I don't
>> recall the exact version right now.
d> yes server hardware and environment is ready.
>> Also, think about backups. With this size of a system, backups are
>> going to be painful.... but maybe you don't care about backups of NVR
>> files past a certain time?
d> no backup for this indeed. if the data gone, just let time to re-collect it.
d> thanks again for your sharing!!
In that case, go for broke! But I'd still layer MD -> LVM ->
filesystem(s) just to give yourself flexibilty.
^ permalink raw reply
* Re: suggest disk numbers in a raidset?
From: d tbsky @ 2016-05-21 7:52 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <22335.8101.224476.822021@quad.stoffel.home>
2016-05-20 22:31 GMT+08:00 John Stoffel <john@stoffel.org>:
> I wouldn't even think of using hardware RAID in this situation, and
> I'd also not think about maximizing the size of the RAID6 volumes as
> well, due to rebuild speed penalty. Another issue to think about is
> chunk size as the number of members in a RAID array go up. Say you
> have the default 64k block size (number pulled from thin air...), so
> you need to have N * 64K worth of data before you can write a full
> stripe of data. So as your writing data, you'll want to keep the
> block size down.
thanks for sharing the thought. maybe I should lower the chunk size
for full stripe write.
> But back to goal. If you're writing large files, since I think NVR
> refers to CCTV camera files, please correct me if wrong, you should
> just stick with the defaults in terms of RAID defaults.
yes NVR refers CCTV files.
> What I would do just just create a RAID6 array with 10 disks, so you
> only have 8 x 4Tb of data, with two parity disks. Then create another
> RAID6 with the remainng 10 disks. Then you would add them as PVs into
> LVM, and then stripe acroos them. Something like this:
yes I will use lvm to combine the array if necessary.
but 10 disks with raid6 will use only 80% of disk capacity.
I had use 16 disks before and it seems ok.
> # Since you want large space, make the extents use larger chunks here
> # in the VG.
> vgcreate -s 16 NVR /dev/md100 /dev/md101
thanks for the suggestion. I will study it.
> I'd want redundant power supplies, some hot spare disks, a UPS, and a
> rock solid hardware with plenty of memory. The other issue is that
> unless you run a recent linux kernel, you might run into performance
> problems with the RAID5/6 parity calculations being all done on a
> single CPU core. Newer versions should have fixed this, but I don't
> recall the exact version right now.
yes server hardware and environment is ready.
> Also, think about backups. With this size of a system, backups are
> going to be painful.... but maybe you don't care about backups of NVR
> files past a certain time?
no backup for this indeed. if the data gone, just let time to re-collect it.
thanks again for your sharing!!
Regards,
tbskyd
^ permalink raw reply
* Mystery RCWs
From: Dallas Clement @ 2016-05-20 22:46 UTC (permalink / raw)
To: Linux-RAID
Hi All,
I've got a new little piece of NAS hardware I am working with and
trying to evaluate RAID 5, 6 performance on it. It has an ARM based
Annapurna Alpine SoC which apparently has a built-in XOR engine to
help out with RAID 5, 6. This little box has four spinning drives
which each have a transfer rate of about 150 MB/s.
I am only seeing a little over 200 MB/s sequential write throughput
for RAID 6 as measured with fio. This is a bit disappointing. I was
hoping that if writing stripe-aligned blocks I would get at least 250
MB/s.
I am measuring the peformance of writing 1M blocks to the array. My
RAID 6 array was constructed with a 512KB chunk size.
# fio --runtime=30 --ramp_time=5 --numjobs=4 --iodepth=32
--ioengine=libaio --direct=1 --filename=/dev/md10 --bs=1M --rw=write
-name poi
I am expecting with a 512KB chunk size that I will be writing stripe
aligned blocks.
I then used blktrace to see what is going on suspecting that there
might be excessive RMWs. I did not see any RMWs but I did see many
RCWs (about 20K).
9,10 3 66 0.701873200 22465 Q WS 347136 + 1024 [fio]
9,10 3 67 0.701925540 22465 U N [fio] 16
9,10 3 68 0.702259400 22465 Q WS 348160 + 1024 [fio]
9,10 3 69 0.702324340 22465 Q WS 349184 + 1024 [fio]
9,10 3 70 0.702378960 22465 U N [fio] 16
9,10 3 71 0.702627200 22465 Q WS 350208 + 1024 [fio]
9,10 3 72 0.702668180 22465 UT N [fio] 10
9,10 0 287 0.666149800 6504 C WS 300032 [0]
9,10 0 288 0.666154080 6504 C WS 299008 [0]
9,10 0 289 0.666491900 6504 C WS 301056 [0]
9,10 0 290 0.666495620 6504 C WS 302080 [0]
9,10 0 291 0.669055940 6504 C WS 305152 [0]
9,10 0 292 0.669059720 6504 C WS 306176 [0]
9,10 0 0 0.671655160 0 m N raid5 rcw 168960 1 1 0
9,10 0 0 0.671671240 0 m N raid5 rcw 169024 1 1 0
9,10 0 0 0.671679220 0 m N raid5 rcw 169088 1 1 0
9,10 0 0 0.671684620 0 m N raid5 rcw 169152 1 1 0
9,10 0 0 0.671697260 0 m N raid5 rcw 169216 1 1 0
9,10 0 0 0.671702560 0 m N raid5 rcw 169280 1 1 0
9,10 0 0 0.671708620 0 m N raid5 rcw 169344 1 1 0
9,10 0 0 0.671713600 0 m N raid5 rcw 169408 1 1 0
It seems that prior to the RCWs, this process 6504 starts doing some
writes. It looks like this is the md driver.
# ps -elf | grep 6504
1 S root 6504 2 0 80 0 - 0 md_thr May19 ?
00:00:36 [md10_raid6]
Can anyone explain what the md driver may be doing and why it's
triggering all these RCWs?
By the way this is running on a Annapurna modified 3.10 kernel:
3.10.20-031020-generic-sa #201311201536 SMP
Thanks,
Dallas
^ permalink raw reply
* Re: [PATCH] Need update superblock on time when deciding to do reshape
From: Shaohua Li @ 2016-05-20 17:59 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid, Jes.Sorensen
In-Reply-To: <1463475249-18658-1-git-send-email-xni@redhat.com>
On Tue, May 17, 2016 at 04:54:09PM +0800, Xiao Ni wrote:
> Hi all
>
> If the disks are not enough to have spaces for relocating the data_offset,
> it needs to run start_reshape and then run mdadm --grow --continue by
> systemd. But mdadm --grow --continue fails because it checkes that
> info->reshape_active is 0.
>
> The info->reshape_active is set to 1 when the superblock feature_map
> have the flag MD_FEATURE_RESHAPE_ACTIVE. Superblock feature_map is set
> MD_FEATURE_RESHAPE_ACTIVE as mddev->reshape_position != MaxSector.
>
> Function start_reshape calls raid5_start_reshape which changes
> mddev->reshape_position to 0. Then in md_check_recovery it updates the
> superblock to underlying devices. But there is a chance that the superblock
> haven't written to underlying devices, the mdadm reads the superblock data.
> So mdadm --grow --continue fails.
>
> The steps to reproduce this:
> mdadm -CR /dev/md0 -l5 -n3 /dev/loop[0-2] --bitmap=internal
> mdadm --wait /dev/md0
> mdadm /dev/md0 -a /dev/loop3
> mdadm --grow --raid-devices 4 /dev/md0
> The loop device size is 500MB
>
> [root@storageqe-09 ~]# cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 loop3[4] loop2[3] loop1[1] loop0[0]
> 1021952 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
> [>....................] reshape = 0.0% (1/510976) finish=0.0min speed=255488K/sec
> bitmap: 1/1 pages [4KB], 65536KB chunk
what's the bad effect of the --continue failure? I think reshape will still
continue. Doing a update super there is ok, but I'm wondering if it's the good
way. Could mdadm wait for MD_FEATURE_RESHAPE_ACTIVE then let systemd run?
Because sounds like we are working around systemd bug, kernel itself will write
superblock anyway soon, so we probably working around in userspace.
> unused devices: <none>
>
> So if we update the superblock on time, mdadm can read the right superblock data.
>
> Signed-off-by <xni@redhat.com>
>
> ---
> drivers/md/md.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 14d3b37..7919606 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -4350,6 +4350,7 @@ action_store(struct mddev *mddev, const char *page, size_t len)
> else {
> clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
> err = mddev->pers->start_reshape(mddev);
> + md_update_sb(mddev, 1);
write super even err != 0?
Thanks,
Shaohua
^ permalink raw reply
* Re: suggest disk numbers in a raidset?
From: John Stoffel @ 2016-05-20 14:31 UTC (permalink / raw)
To: d tbsky; +Cc: linux-raid
In-Reply-To: <CAC6SzHJ8Wi18opzDbzjtT+niM5QKOyOU3AAeo0npbDsV+sQOTQ@mail.gmail.com>
d> I need to create a raid6 array with about 20 disks, and it need to
d> grow up in the future, maybe add another 20 disks into the array.
d> I wonder how many disks can a software raid6 array handled? the LSI
d> hardware raid (2208/3108) can only put 32 disks in an array, and I
d> heard it may be just use 16 disks in a array is better in LSI.
d> the system will be using to put NVR recording files, so the disk
d> speed is not very important.
I wouldn't even think of using hardware RAID in this situation, and
I'd also not think about maximizing the size of the RAID6 volumes as
well, due to rebuild speed penalty. Another issue to think about is
chunk size as the number of members in a RAID array go up. Say you
have the default 64k block size (number pulled from thin air...), so
you need to have N * 64K worth of data before you can write a full
stripe of data. So as your writing data, you'll want to keep the
block size down.
But back to goal. If you're writing large files, since I think NVR
refers to CCTV camera files, please correct me if wrong, you should
just stick with the defaults in terms of RAID defaults.
What I would do just just create a RAID6 array with 10 disks, so you
only have 8 x 4Tb of data, with two parity disks. Then create another
RAID6 with the remainng 10 disks. Then you would add them as PVs into
LVM, and then stripe acroos them. Something like this:
mdadm --create /dev/md100 --level 6 -n 10 -x 0 --bitmap=internal /dev/sd[cdefghijkl]1
mdadm --create /dev/md101 --level 6 -n 10 -x 0 --bitmap=internal /dev/sd[mnopqrstuv]1
pvcreate /dev/md100
pvcreate /dev/md101
# Since you want large space, make the extents use larger chunks here
# in the VG.
vgcreate -s 16 NVR /dev/md100 /dev/md101
# Create a 30Tb volume...
lvcreate -L 30T --name vol1 NVR
# Make an XFS filesystem on the volume
mkfs -t xfs /dev/mapper/NVR_vol1
So the entire idea of the above is that you expand things by doing:
mdadm --create /dev/md102 --level 6 -n 10 -x 0 --bitmap=internal /dev/sda[cdefghijkl]1
mdadm --create /dev/md103 --level 6 -n 10 -x 0 --bitmap=internal /dev/sda[mnopqrstuv]1
pvcreate /dev/md102
pvcreate /dev/md103
vgextend NVR_data /dev/md102 /dev/md103
lvresize -L +30T --resizefs /dev/mapper/NVR_vol1
And now you've grown your volume without any impact! And you and
migrate LVs around and remove PVs (once empty) if you need to down the
line. Very flexible.
This is one of the big downsides of using ZFS in my mind, once you've
added in a physical device, you can never shrink the filesystem, only
grow it. Not that you mentioned using ZFS, but it's something to keep
in mind here.
But!!! I'd also think seriously about making more smaller volumes and
having your software spread stuff out across multiple filesystems.
XFS is good, but I'd be leery of such a huge filesystem on a system
like this.
I'd want redundant power supplies, some hot spare disks, a UPS, and a
rock solid hardware with plenty of memory. The other issue is that
unless you run a recent linux kernel, you might run into performance
problems with the RAID5/6 parity calculations being all done on a
single CPU core. Newer versions should have fixed this, but I don't
recall the exact version right now.
Also, think about backups. With this size of a system, backups are
going to be painful.... but maybe you don't care about backups of NVR
files past a certain time?
Good luck,
John
^ permalink raw reply
* Re: suggest disk numbers in a raidset?
From: d tbsky @ 2016-05-20 10:13 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20160520083830.GA3189@metamorpher.de>
2016-05-20 16:38 GMT+08:00 Andreas Klauer <Andreas.Klauer@metamorpher.de>:
> The question is whether you really want that many disks in a single array.
>
> After all, the more disks, the more likely you have more than one disk fail,
> due to various causes (not just the disk itself, also cable problems, etc.)
> Do you test your disks regularly? md raid checks, smart self tests, do you
> actually replace them when they start reallocating sectors, ...?
>
> Do you want everything to be gone when the raid fails or just half? ;)
>
> I'm not sure if I'd be comfortable with more than ~24 disks in a raid6.
>
>> in the future, maybe add another 20 disks into the array
>
> If you have 20 disks now you could create a 20 disk raid6 now;
> if you add 20 disks later you can choose... grow by 20? or just
> create another 20 disk raid set. I think I'd go for the latter.
>
> But it's a personal choice. You have to make it yourself.
> Just make sure you have good monitoring for your disks,
> if you don't detect disk errors early, raid redundancy might not save you.
>
> Regards
> Andreas Klauer
If I use LSI raid card, I would use 16 or 24 disks in a raid6 set,
since the limit is 32.
but I would like to know if there are other thoughts when using software raid..
thanks for the suggestion :)
Regards,
tbskyd
^ permalink raw reply
* Re: suggest disk numbers in a raidset?
From: Andreas Klauer @ 2016-05-20 8:38 UTC (permalink / raw)
To: d tbsky; +Cc: linux-raid
In-Reply-To: <CAC6SzHJ8Wi18opzDbzjtT+niM5QKOyOU3AAeo0npbDsV+sQOTQ@mail.gmail.com>
On Fri, May 20, 2016 at 01:13:43PM +0800, d tbsky wrote:
> I wonder how many disks can a software raid6 array handled?
mdadm claims 256
# mdadm --create /dev/md42 --level=6 -n 999 ...
mdadm: no more than 256 raid-devices supported for level 6
Actually trying to create a RAID with that many, using 256*4M loop devices on tmpfs
# mdadm --create /dev/md42 --level=6 -n 256 /dev/loop{0..255}
mdadm: Defaulting to version 1.2 metadata
Segmentation fault
[ 210.798267] md: bind<loop255>
[ 210.798898] md/raid:md42: not clean -- starting background reconstruction
[ 210.798909] divide error: 0000 [#1] SMP
[ 210.798925] CPU: 0 PID: 6397 Comm: mdadm Not tainted 4.5.5 #1
[ 210.798937] Hardware name: MSI MS-7817/B85M-E45 (MS-7817), BIOS V10.9 04/21/2015
[ 210.798953] task: ffff8800828dcec0 ti: ffff8803eced0000 task.ti: ffff8803eced0000
[ 210.798968] RIP: 0010:[<ffffffff814602d1>] [<ffffffff814602d1>] reciprocal_value+0x21/0x60
[ 210.798989] RSP: 0018:ffff8803eced3bc8 EFLAGS: 00010247
[ 210.799000] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000020
[ 210.799015] RDX: 0000000000000000 RSI: 000000000000001f RDI: 0000000000000000
[ 210.799029] RBP: 0000000000000080 R08: 00000000000154d8 R09: ffffffff81c708ad
[ 210.799043] R10: ffffea000dadc500 R11: 0000000000000002 R12: 0000000000001020
[ 210.799057] R13: 00000000024000c0 R14: 0000000000000100 R15: 0000000000000000
[ 210.799071] FS: 0000000001b3e880(0063) GS:ffff88041ea00000(0000) knlGS:0000000000000000
[ 210.799087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 210.799098] CR2: 0000561aa6bbe068 CR3: 000000036b775000 CR4: 00000000001406f0
[ 210.799112] Stack:
[ 210.799118] ffffffff8145aa0b ffff880082866000 0000000000000080 00000000024000c0
[ 210.799135] 0000000000000100 0000000000000100 ffffffff81e6b9a0 ffffffff816fff88
[ 210.799152] ffff880082866000 ffffe8ffffa01a90 ffff8803defde000 ffffffff8170184c
[ 210.799170] Call Trace:
[ 210.799179] [<ffffffff8145aa0b>] ? flex_array_alloc+0xbb/0xf0
[ 210.799193] [<ffffffff816fff88>] ? scribble_alloc+0x18/0x50
[ 210.799231] [<ffffffff8170184c>] ? alloc_scratch_buffer+0x7c/0x90
[ 210.799253] [<ffffffff817036d6>] ? setup_conf+0x3c6/0x800
[ 210.799267] [<ffffffff8117fd66>] ? printk+0x43/0x4b
[ 210.799288] [<ffffffff81704444>] ? raid5_run+0x854/0xad0
[ 210.799311] [<ffffffff8142d648>] ? __bioset_create+0x1c8/0x280
[ 210.799346] [<ffffffff81718eef>] ? md_run+0x27f/0x970
[ 210.799368] [<ffffffff81186c06>] ? free_hot_cold_page_list+0x26/0x40
[ 210.799381] [<ffffffff817195ea>] ? do_md_run+0xa/0xa0
[ 210.799402] [<ffffffff8171baae>] ? md_ioctl+0xe9e/0x1b60
[ 210.799427] [<ffffffff8119f5ae>] ? tlb_flush_mmu_free+0x2e/0x50
[ 210.799429] [<ffffffff811a0492>] ? tlb_finish_mmu+0x12/0x40
[ 210.799430] [<ffffffff811a669e>] ? unmap_region+0xbe/0xe0
[ 210.799441] [<ffffffff8143d7a7>] ? blkdev_ioctl+0x237/0x8c0
[ 210.799443] [<ffffffff811f0d74>] ? block_ioctl+0x34/0x40
[ 210.799445] [<ffffffff811d3568>] ? do_vfs_ioctl+0x88/0x590
[ 210.799446] [<ffffffff811a810d>] ? do_munmap+0x32d/0x450
[ 210.799447] [<ffffffff811d3aa6>] ? SyS_ioctl+0x36/0x70
[ 210.799460] [<ffffffff818b7e5b>] ? entry_SYSCALL_64_fastpath+0x16/0x6e
[ 210.799479] Code: ff ff ff 0f 1f 80 00 00 00 00 8d 47 ff be ff ff ff ff 89 ff 0f bd f0 8d 4e 01 b8 01 00 00 00 31 d2 48 d3 e0 48 29 f8 48 c1 e0 20 <48> f7 f7 85 c9 ba 01 00 00 00 0f 4f ca 83 c0 01 ba 00 00 00 00
[ 210.799480] RIP [<ffffffff814602d1>] reciprocal_value+0x21/0x60
[ 210.799481] RSP <ffff8803eced3bc8>
[ 210.805216] ---[ end trace c271c2a4daa3678c ]---
So that didn't work too well.
> the LSI hardware raid (2208/3108) can only put 32 disks in an array
I would be a little suprised if there was a 32 disk limit.
The question is whether you really want that many disks in a single array.
After all, the more disks, the more likely you have more than one disk fail,
due to various causes (not just the disk itself, also cable problems, etc.)
Do you test your disks regularly? md raid checks, smart self tests, do you
actually replace them when they start reallocating sectors, ...?
Do you want everything to be gone when the raid fails or just half? ;)
I'm not sure if I'd be comfortable with more than ~24 disks in a raid6.
> in the future, maybe add another 20 disks into the array
If you have 20 disks now you could create a 20 disk raid6 now;
if you add 20 disks later you can choose... grow by 20? or just
create another 20 disk raid set. I think I'd go for the latter.
But it's a personal choice. You have to make it yourself.
Just make sure you have good monitoring for your disks,
if you don't detect disk errors early, raid redundancy might not save you.
Regards
Andreas Klauer
^ permalink raw reply
* Write intent bitmap algorithm
From: Ankur Bose @ 2016-05-20 7:03 UTC (permalink / raw)
To: linux-raid
Hi All,
We are doing performance analysis of MD raid1 and wanted to know the flow/algorithm of the writes with write intent bitmap.
As per "man md" below lines gives the algo
"Before any write request is honored, md will make sure that the corresponding bit in the log is set. After a period of time with no writes to an area of the array, the corresponding bit will be cleared."
Here we wanted to know after what period of time and what is the logic to calculate this period of time which MD does ?
I see a function "void bitmap_daemon_work(struct mddev *mddev)" in bitmap.c file but trying to figure out the logic how it is used.
Thanks,
ankur.
^ permalink raw reply
* suggest disk numbers in a raidset?
From: d tbsky @ 2016-05-20 5:13 UTC (permalink / raw)
To: linux-raid
Hi:
I need to create a raid6 array with about 20 disks, and it need to
grow up in the future, maybe add another 20 disks into the array.
I wonder how many disks can a software raid6 array handled? the LSI
hardware raid (2208/3108) can only put 32 disks in an array, and I
heard it may be just use 16 disks in a array is better in LSI.
the system will be using to put NVR recording files, so the disk
speed is not very important.
thanks a lot for suggestion!
Regards,
tbskyd
^ permalink raw reply
* [GIT PULL] MD for 4.7-rc1
From: Shaohua Li @ 2016-05-19 18:34 UTC (permalink / raw)
To: torvalds; +Cc: linux-kernel, linux-raid, neilb
Hi Linus,
pleae pull MD update for 4.7. It includes several patches from Guoqing fixing
md-cluster bugs and several patches from Heinz fixing dm-raid bugs.
Thanks,
Shaohua
The following changes since commit 4810d9682971e8eee659f96e4f9d9154e3c6c0b4:
Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security (2016-05-04 11:14:00 -0700)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/shli/md.git tags/md/4.7-rc1
for you to fetch changes up to 1fa9a1ad0a9db3c745fe0c1bfa73fd87901fd7f3:
md-cluster: check the return value of process_recvd_msg (2016-05-09 09:24:04 -0700)
----------------------------------------------------------------
Guoqing Jiang (15):
md-cluster: change resync lock from asynchronous to synchronous
md-cluser: make resync_finish only called after pers->sync_request
md-cluster: wake up thread to continue recovery
md-cluster: unregister thread if err happened
md-cluster: fix locking when node joins cluster during message broadcast
md-cluster: change array_sectors and update size are not supported
md-cluster: wakeup thread if activated a spare disk
md-cluster: always setup in-memory bitmap
md-cluster: sync bitmap when node received RESYNCING msg
md-cluster/bitmap: fix wrong calcuation of offset
md-cluster/bitmap: fix wrong page num in bitmap_file_clear_bit and bitmap_file_set_bit
md-cluster/bitmap: unplug bitmap to sync dirty pages to disk
md: set MD_CHANGE_PENDING in a atomic region
md-cluster: gather resync infos and enable recv_thread after bitmap is ready
md-cluster: check the return value of process_recvd_msg
Heinz Mauelshagen (3):
md: md.c: fix oops in mddev_suspend for raid0
md: raid10: add prerequisite to run underneath dm-raid
md: raid5: add prerequisite to run underneath dm-raid
kbuild test robot (1):
md-cluster: fix ifnullfree.cocci warnings
Documentation/md-cluster.txt | 6 +++
drivers/md/bitmap.c | 88 +++++++++++++++++++++++++++++++++++-----
drivers/md/bitmap.h | 3 ++
drivers/md/md-cluster.c | 96 ++++++++++++++++++++++++++++++++++++--------
drivers/md/md-cluster.h | 1 +
drivers/md/md.c | 86 ++++++++++++++++++++++++---------------
drivers/md/raid1.c | 4 +-
drivers/md/raid10.c | 20 +++++----
drivers/md/raid5-cache.c | 4 +-
drivers/md/raid5.c | 10 +++--
include/linux/bitops.h | 16 ++++++++
11 files changed, 257 insertions(+), 77 deletions(-)
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox