* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Phil Turmel @ 2015-12-02 15:51 UTC (permalink / raw)
To: Robert Kierski, Dallas Clement; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <F7761B9B1D11B64BBB666019E9378117FDDC2A@CFWEX01.americas.cray.com>
On 12/02/2015 10:44 AM, Robert Kierski wrote:
> I've tried a variety of settings... ranging from 17 to 32768.
>
> Yes.. with stripe_cache_size set to 17, I see a C/T of rmw's. And my TP goes in the toilet -- even with the RAM disks, I get only about 30M/s.
Ok.
You mentioned you aren't using a filesystem. How are you testing?
Phil
ps. convention on kernel.org is to trim replies and bottom-post, or
interleave. Please do.
^ permalink raw reply
* two small raid5 cache updates
From: Christoph Hellwig @ 2015-12-02 16:06 UTC (permalink / raw)
To: shli, neilb; +Cc: linux-raid
The first is a trival cleanup, the second a similarly trivial reduction
in memory usage.
^ permalink raw reply
* [PATCH 1/2] raid5-cache: simplify r5l_move_io_unit_list
From: Christoph Hellwig @ 2015-12-02 16:06 UTC (permalink / raw)
To: shli, neilb; +Cc: linux-raid
In-Reply-To: <1449072399-15360-1-git-send-email-hch@lst.de>
It's only used for one kind of move, so make that explicit. Also clean
up the code a bit by using list_for_each_safe.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 32 +++++++++++++++-----------------
1 file changed, 15 insertions(+), 17 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index b887e04..3699c47 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -156,21 +156,6 @@ static void r5l_free_io_unit(struct r5l_log *log, struct r5l_io_unit *io)
kmem_cache_free(log->io_kc, io);
}
-static void r5l_move_io_unit_list(struct list_head *from, struct list_head *to,
- enum r5l_io_unit_state state)
-{
- struct r5l_io_unit *io;
-
- while (!list_empty(from)) {
- io = list_first_entry(from, struct r5l_io_unit, log_sibling);
- /* don't change list order */
- if (io->state >= state)
- list_move_tail(&io->log_sibling, to);
- else
- break;
- }
-}
-
static void __r5l_set_io_unit_state(struct r5l_io_unit *io,
enum r5l_io_unit_state state)
{
@@ -206,6 +191,20 @@ static void r5l_log_run_stripes(struct r5l_log *log)
}
}
+static void r5l_move_to_end_ios(struct r5l_log *log)
+{
+ struct r5l_io_unit *io, *next;
+
+ assert_spin_locked(&log->io_list_lock);
+
+ list_for_each_entry_safe(io, next, &log->running_ios, log_sibling) {
+ /* don't change list order */
+ if (io->state < IO_UNIT_IO_END)
+ break;
+ list_move_tail(&io->log_sibling, &log->io_end_ios);
+ }
+}
+
static void r5l_log_endio(struct bio *bio)
{
struct r5l_io_unit *io = bio->bi_private;
@@ -220,8 +219,7 @@ static void r5l_log_endio(struct bio *bio)
spin_lock_irqsave(&log->io_list_lock, flags);
__r5l_set_io_unit_state(io, IO_UNIT_IO_END);
if (log->need_cache_flush)
- r5l_move_io_unit_list(&log->running_ios, &log->io_end_ios,
- IO_UNIT_IO_END);
+ r5l_move_to_end_ios(log);
else
r5l_log_run_stripes(log);
spin_unlock_irqrestore(&log->io_list_lock, flags);
--
1.9.1
^ permalink raw reply related
* [PATCH 2/2] raid5-cache: free meta_page earlier
From: Christoph Hellwig @ 2015-12-02 16:06 UTC (permalink / raw)
To: shli, neilb; +Cc: linux-raid
In-Reply-To: <1449072399-15360-1-git-send-email-hch@lst.de>
Once the I/O completed we don't need the meta page anymore. As the iounits
can live on for a long time this reduces memory pressure a bit.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 3699c47..668e973 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -150,12 +150,6 @@ static bool r5l_has_free_space(struct r5l_log *log, sector_t size)
return log->device_size > used_size + size;
}
-static void r5l_free_io_unit(struct r5l_log *log, struct r5l_io_unit *io)
-{
- __free_page(io->meta_page);
- kmem_cache_free(log->io_kc, io);
-}
-
static void __r5l_set_io_unit_state(struct r5l_io_unit *io,
enum r5l_io_unit_state state)
{
@@ -215,6 +209,7 @@ static void r5l_log_endio(struct bio *bio)
md_error(log->rdev->mddev, log->rdev);
bio_put(bio);
+ __free_page(io->meta_page);
spin_lock_irqsave(&log->io_list_lock, flags);
__r5l_set_io_unit_state(io, IO_UNIT_IO_END);
@@ -552,7 +547,7 @@ static bool r5l_complete_finished_ios(struct r5l_log *log)
log->next_cp_seq = io->seq;
list_del(&io->log_sibling);
- r5l_free_io_unit(log, io);
+ kmem_cache_free(log->io_kc, io);
found = true;
}
--
1.9.1
^ permalink raw reply related
* [RFC] using mempools for raid5-cache
From: Christoph Hellwig @ 2015-12-02 16:10 UTC (permalink / raw)
To: shli, neilb; +Cc: linux-raid
Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
I've looked into replacing these with mempools and biosets, and for the
bio and the meta_page that's pretty trivial as they have short life times
and do make guaranteed progress. I'm massively struggling with the iounit
allocation, though. These can live on for a long time over log I/O, cache
flushing and last but not least RAID I/O, and every attempt at something
mempool-like results in reproducible deadlocks. I wonder if we need to
figure out some more efficient data structure to communicate the completion
status that doesn't rely on these fairly long living allocations from
the I/O path.
FYI, my last attempt to use the bio frontpad is below, but a mempool showed
pretty similar results:
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 2e3f22a..d2438be 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -33,12 +33,12 @@
*/
#define RECLAIM_MAX_FREE_SPACE (10 * 1024 * 1024 * 2) /* sector */
#define RECLAIM_MAX_FREE_SPACE_SHIFT (2)
-
+
/*
* We only need 2 bios per I/O unit to make progress, but ensure we
* have a few more available to not get too tight.
*/
-#define R5L_POOL_SIZE 1024
+#define R5L_POOL_SIZE 16384
struct r5l_log {
struct md_rdev *rdev;
@@ -75,7 +75,6 @@ struct r5l_log {
struct list_head finished_ios; /* io_units which settle down in log disk */
struct bio flush_bio;
- struct kmem_cache *io_kc;
struct bio_set *bs;
mempool_t *meta_pool;
@@ -120,6 +119,8 @@ struct r5l_io_unit {
int state;
bool need_split_bio;
+
+ struct bio bio;
};
/* r5l_io_unit state */
@@ -209,14 +210,13 @@ static void r5l_move_to_end_ios(struct r5l_log *log)
static void r5l_log_endio(struct bio *bio)
{
- struct r5l_io_unit *io = bio->bi_private;
+ struct r5l_io_unit *io = container_of(bio, struct r5l_io_unit, bio);
struct r5l_log *log = io->log;
unsigned long flags;
if (bio->bi_error)
md_error(log->rdev->mddev, log->rdev);
- bio_put(bio);
mempool_free(io->meta_page, log->meta_pool);
spin_lock_irqsave(&log->io_list_lock, flags);
@@ -284,11 +284,13 @@ static void r5_reserve_log_entry(struct r5l_log *log, struct r5l_io_unit *io)
static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
{
+ struct bio *bio;
struct r5l_io_unit *io;
struct r5l_meta_block *block;
- /* We can't handle memory allocate failure so far */
- io = kmem_cache_zalloc(log->io_kc, GFP_NOIO | __GFP_NOFAIL);
+ bio = r5l_bio_alloc(log);
+
+ io = container_of(bio, struct r5l_io_unit, bio);
io->log = log;
INIT_LIST_HEAD(&io->log_sibling);
INIT_LIST_HEAD(&io->stripe_list);
@@ -306,7 +308,7 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
io->meta_offset = sizeof(struct r5l_meta_block);
io->seq = log->seq++;
- io->current_bio = r5l_bio_alloc(log);
+ io->current_bio = bio;
io->current_bio->bi_end_io = r5l_log_endio;
io->current_bio->bi_private = io;
bio_add_page(io->current_bio, io->meta_page, PAGE_SIZE, 0);
@@ -556,7 +558,7 @@ static bool r5l_complete_finished_ios(struct r5l_log *log)
log->next_cp_seq = io->seq;
list_del(&io->log_sibling);
- kmem_cache_free(log->io_kc, io);
+ bio_put(&io->bio);
found = true;
}
@@ -1158,11 +1160,8 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
INIT_LIST_HEAD(&log->finished_ios);
bio_init(&log->flush_bio);
- log->io_kc = KMEM_CACHE(r5l_io_unit, 0);
- if (!log->io_kc)
- goto io_kc;
-
- log->bs = bioset_create(R5L_POOL_SIZE, 0);
+ log->bs = bioset_create(R5L_POOL_SIZE,
+ offsetof(struct r5l_io_unit, bio));
if (!log->bs)
goto io_bs;
@@ -1192,8 +1191,6 @@ reclaim_thread:
out_mempool:
bioset_free(log->bs);
io_bs:
- kmem_cache_destroy(log->io_kc);
-io_kc:
kfree(log);
return -EINVAL;
}
@@ -1203,6 +1200,5 @@ void r5l_exit_log(struct r5l_log *log)
md_unregister_thread(&log->reclaim_thread);
mempool_destroy(log->meta_pool);
bioset_free(log->bs);
- kmem_cache_destroy(log->io_kc);
kfree(log);
}
^ permalink raw reply related
* [PATCH 1/2] raid5-cache: use a bio_set
From: Christoph Hellwig @ 2015-12-02 16:10 UTC (permalink / raw)
To: shli, neilb; +Cc: linux-raid
In-Reply-To: <1449072638-15409-1-git-send-email-hch@lst.de>
This allows us to make guaranteed forward progress.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 668e973..ef59564 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -34,6 +34,12 @@
#define RECLAIM_MAX_FREE_SPACE (10 * 1024 * 1024 * 2) /* sector */
#define RECLAIM_MAX_FREE_SPACE_SHIFT (2)
+/*
+ * We only need 2 bios per I/O unit to make progress, but ensure we
+ * have a few more available to not get too tight.
+ */
+#define R5L_POOL_SIZE 1024
+
struct r5l_log {
struct md_rdev *rdev;
@@ -70,6 +76,7 @@ struct r5l_log {
struct bio flush_bio;
struct kmem_cache *io_kc;
+ struct bio_set *bs;
struct md_thread *reclaim_thread;
unsigned long reclaim_target; /* number of space that need to be
@@ -248,7 +255,7 @@ static void r5l_submit_current_io(struct r5l_log *log)
static struct bio *r5l_bio_alloc(struct r5l_log *log)
{
- struct bio *bio = bio_kmalloc(GFP_NOIO | __GFP_NOFAIL, BIO_MAX_PAGES);
+ struct bio *bio = bio_alloc_bioset(GFP_NOIO, BIO_MAX_PAGES, log->bs);
bio->bi_rw = WRITE;
bio->bi_bdev = log->rdev->bdev;
@@ -1153,6 +1160,10 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
if (!log->io_kc)
goto io_kc;
+ log->bs = bioset_create(R5L_POOL_SIZE, 0);
+ if (!log->bs)
+ goto io_bs;
+
log->reclaim_thread = md_register_thread(r5l_reclaim_thread,
log->rdev->mddev, "reclaim");
if (!log->reclaim_thread)
@@ -1170,6 +1181,8 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
error:
md_unregister_thread(&log->reclaim_thread);
reclaim_thread:
+ bioset_free(log->bs);
+io_bs:
kmem_cache_destroy(log->io_kc);
io_kc:
kfree(log);
@@ -1179,6 +1192,7 @@ io_kc:
void r5l_exit_log(struct r5l_log *log)
{
md_unregister_thread(&log->reclaim_thread);
+ bioset_free(log->bs);
kmem_cache_destroy(log->io_kc);
kfree(log);
}
--
1.9.1
^ permalink raw reply related
* [PATCH 2/2] raid5-cache: use a mempool for the metadata block
From: Christoph Hellwig @ 2015-12-02 16:10 UTC (permalink / raw)
To: shli, neilb; +Cc: linux-raid
In-Reply-To: <1449072638-15409-1-git-send-email-hch@lst.de>
We only have a limited number in flight, so use a page based mempool.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/md/raid5-cache.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index ef59564..2e3f22a 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -77,6 +77,7 @@ struct r5l_log {
struct kmem_cache *io_kc;
struct bio_set *bs;
+ mempool_t *meta_pool;
struct md_thread *reclaim_thread;
unsigned long reclaim_target; /* number of space that need to be
@@ -216,7 +217,7 @@ static void r5l_log_endio(struct bio *bio)
md_error(log->rdev->mddev, log->rdev);
bio_put(bio);
- __free_page(io->meta_page);
+ mempool_free(io->meta_page, log->meta_pool);
spin_lock_irqsave(&log->io_list_lock, flags);
__r5l_set_io_unit_state(io, IO_UNIT_IO_END);
@@ -293,8 +294,9 @@ static struct r5l_io_unit *r5l_new_meta(struct r5l_log *log)
INIT_LIST_HEAD(&io->stripe_list);
io->state = IO_UNIT_RUNNING;
- io->meta_page = alloc_page(GFP_NOIO | __GFP_NOFAIL | __GFP_ZERO);
+ io->meta_page = mempool_alloc(log->meta_pool, GFP_NOIO);
block = page_address(io->meta_page);
+ clear_page(block);
block->magic = cpu_to_le32(R5LOG_MAGIC);
block->version = R5LOG_VERSION;
block->seq = cpu_to_le64(log->seq);
@@ -1164,6 +1166,10 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
if (!log->bs)
goto io_bs;
+ log->meta_pool = mempool_create_page_pool(R5L_POOL_SIZE, 0);
+ if (!log->meta_pool)
+ goto out_mempool;
+
log->reclaim_thread = md_register_thread(r5l_reclaim_thread,
log->rdev->mddev, "reclaim");
if (!log->reclaim_thread)
@@ -1178,9 +1184,12 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev)
conf->log = log;
return 0;
+
error:
md_unregister_thread(&log->reclaim_thread);
reclaim_thread:
+ mempool_destroy(log->meta_pool);
+out_mempool:
bioset_free(log->bs);
io_bs:
kmem_cache_destroy(log->io_kc);
@@ -1192,6 +1201,7 @@ io_kc:
void r5l_exit_log(struct r5l_log *log)
{
md_unregister_thread(&log->reclaim_thread);
+ mempool_destroy(log->meta_pool);
bioset_free(log->bs);
kmem_cache_destroy(log->io_kc);
kfree(log);
--
1.9.1
^ permalink raw reply related
* RAID6 - "Invalid Argument" / "does not have a valid v1.2 superblock" ?!
From: Tobias Geiger @ 2015-12-02 17:04 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hello,
i need help assembling my raid6 again:
root@pc:~# mdadm -A /dev/md0 -v --force /dev/sdd2 /dev/sde2 /dev/sdg2
/dev/sdc2 /dev/sdb2 /dev/sdf2
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdd2 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sde2 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdg2 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdc2 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdb2 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdf2 is identified as a member of /dev/md0, slot 5.
mdadm: failed to add /dev/sde2 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdg2 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdc2 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdb2 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdf2 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdd2 to /dev/md0: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md0: Invalid argument
dmesg says to this:
[25717.423008] md: md0 stopped.
[25717.424958] md: sde2 does not have a valid v1.2 superblock, not importing!
[25717.424989] md: md_import_device returned -22
[25717.425266] md: sdg2 does not have a valid v1.2 superblock, not importing!
[25717.425963] md: md_import_device returned -22
[25717.426847] md: sdc2 does not have a valid v1.2 superblock, not importing!
[25717.426880] md: md_import_device returned -22
[25717.427908] md: sdb2 does not have a valid v1.2 superblock, not importing!
[25717.427952] md: md_import_device returned -22
[25717.428890] md: sdf2 does not have a valid v1.2 superblock, not importing!
[25717.428938] md: md_import_device returned -22
[25717.429668] md: sdd2 does not have a valid v1.2 superblock, not importing!
[25717.429716] md: md_import_device returned -22
[25717.430104] md: md0 stopped.
but i cant find any problems examining the devices:
/dev/sdb2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 078883e8:30c2ebe6:87576034:e85737b8
Name : pc:0 (local to host pc)
Creation Time : Fri Sep 5 01:06:21 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 7813699215 (3725.86 GiB 4000.61 GB)
Array Size : 15627124736 (14903.19 GiB 16002.18 GB)
Used Dev Size : 7813562368 (3725.80 GiB 4000.54 GB)
Data Offset : 198656 sectors
Super Offset : 8 sectors
Unused Space : before=198568 sectors, after=69263 sectors
State : clean
Device UUID : 02dab3ff:6e36ebf3:2012e2a0:e8b29f3f
Update Time : Wed Dec 2 10:13:53 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b75c2285 - correct
Events : 1903827
Layout : left-symmetric
Chunk Size : 4096K
Device Role : Active device 4
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 078883e8:30c2ebe6:87576034:e85737b8
Name : pc:0 (local to host pc)
Creation Time : Fri Sep 5 01:06:21 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 7813699215 (3725.86 GiB 4000.61 GB)
Array Size : 15627124736 (14903.19 GiB 16002.18 GB)
Used Dev Size : 7813562368 (3725.80 GiB 4000.54 GB)
Data Offset : 198656 sectors
Super Offset : 8 sectors
Unused Space : before=198568 sectors, after=69263 sectors
State : clean
Device UUID : c9aaa3e3:00589ba5:be8fe774:362d477a
Update Time : Wed Dec 2 10:13:53 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 5ba906b9 - correct
Events : 1903827
Layout : left-symmetric
Chunk Size : 4096K
Device Role : Active device 3
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 078883e8:30c2ebe6:87576034:e85737b8
Name : pc:0 (local to host pc)
Creation Time : Fri Sep 5 01:06:21 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 7813699215 (3725.86 GiB 4000.61 GB)
Array Size : 15627124736 (14903.19 GiB 16002.18 GB)
Used Dev Size : 7813562368 (3725.80 GiB 4000.54 GB)
Data Offset : 198656 sectors
Super Offset : 8 sectors
Unused Space : before=198568 sectors, after=69263 sectors
State : clean
Device UUID : 2b4e613e:1a7f062c:f8ba29a9:e5dcfaf8
Update Time : Wed Dec 2 10:13:53 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : efc7b1df - correct
Events : 1903827
Layout : left-symmetric
Chunk Size : 4096K
Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 078883e8:30c2ebe6:87576034:e85737b8
Name : pc:0 (local to host pc)
Creation Time : Fri Sep 5 01:06:21 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 7813699215 (3725.86 GiB 4000.61 GB)
Array Size : 15627124736 (14903.19 GiB 16002.18 GB)
Used Dev Size : 7813562368 (3725.80 GiB 4000.54 GB)
Data Offset : 198656 sectors
Super Offset : 8 sectors
Unused Space : before=198568 sectors, after=69263 sectors
State : clean
Device UUID : e80ba505:cc0b9904:9288c64a:5c289e62
Update Time : Wed Dec 2 10:13:53 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9ade0f9a - correct
Events : 1903827
Layout : left-symmetric
Chunk Size : 4096K
Device Role : Active device 1
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 078883e8:30c2ebe6:87576034:e85737b8
Name : pc:0 (local to host pc)
Creation Time : Fri Sep 5 01:06:21 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 7813699215 (3725.86 GiB 4000.61 GB)
Array Size : 15627124736 (14903.19 GiB 16002.18 GB)
Used Dev Size : 7813562368 (3725.80 GiB 4000.54 GB)
Data Offset : 198656 sectors
Super Offset : 8 sectors
Unused Space : before=198568 sectors, after=69263 sectors
State : clean
Device UUID : f769eefd:bd202fca:50966083:39a25647
Update Time : Wed Dec 2 10:13:53 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 76100a3c - correct
Events : 1903827
Layout : left-symmetric
Chunk Size : 4096K
Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 078883e8:30c2ebe6:87576034:e85737b8
Name : pc:0 (local to host pc)
Creation Time : Fri Sep 5 01:06:21 2014
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 7813699215 (3725.86 GiB 4000.61 GB)
Array Size : 15627124736 (14903.19 GiB 16002.18 GB)
Used Dev Size : 7813562368 (3725.80 GiB 4000.54 GB)
Data Offset : 198656 sectors
Super Offset : 8 sectors
Unused Space : before=198568 sectors, after=69263 sectors
State : clean
Device UUID : 7fad3e2f:85dd6849:127d7d47:58c166dd
Update Time : Wed Dec 2 10:13:53 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 80c71068 - correct
Events : 1903827
Layout : left-symmetric
Chunk Size : 4096K
Device Role : Active device 2
Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
except for the Checksum and the Device UUID the output is exactly the same for
each device...
i really hesitate to "recreate" the array with "--assume-clean", because of
the Unused-Space/ArraySize, which i wouldnt know how to reassemble in a
"mdadm --create ..." line ...
do you have any thoughts on that?
Thanks very much for your help!
Greetings
Tobias
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Dallas Clement @ 2015-12-02 19:50 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <565F136F.2090709@turmel.org>
On Wed, Dec 2, 2015 at 9:51 AM, Phil Turmel <philip@turmel.org> wrote:
> On 12/02/2015 10:44 AM, Robert Kierski wrote:
>> I've tried a variety of settings... ranging from 17 to 32768.
>>
>> Yes.. with stripe_cache_size set to 17, I see a C/T of rmw's. And my TP goes in the toilet -- even with the RAM disks, I get only about 30M/s.
>
> Ok.
>
> You mentioned you aren't using a filesystem. How are you testing?
>
> Phil
>
> ps. convention on kernel.org is to trim replies and bottom-post, or
> interleave. Please do.
Thank you all for your responses.
Keld,
> Did you test the performance of other raid types, such as RAID1 and the various layouts of RAID10 for the newer kernels?
I did try RAID 1 but not RAID 10. With RAID 1 I am seeing much higher
average and peak wMB/s and disk utilization than with RAID 5 and 6.
Though I need to run some more tests to compare the performance of
newer kernels with the 2.6.39.4 kernel. Will report on that a bit
later.
Roman,
> Do you use a write intent bitmap (internal?), what is your bitmap chunk size?
Yes, I do. After reading up on this, I see that it can negatively
affect write performance. The bitmap chunk size is 67108864.
> What is your stripe_cache_size set to?
strip_cache_size is 8192
Robert, like you I am observing that my CPU is mostly idle during RAID
5 or 6 write testing. Something else is throttling the traffic. Not
sure if there is some threshold crossing i.e. queue size, await time
etc that is causing this or if it is implementation problem.
I understand that the stripe cache grows dynamically in >= 4.1
kernels. Fwiw, adjusting the stripe cache made no difference in my
results.
Regards,
Dallas
^ permalink raw reply
* Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt
From: Alasdair G Kergon @ 2015-12-02 19:56 UTC (permalink / raw)
To: Baolin Wang
Cc: Mark Brown, Jens Axboe, keith.busch, Jan Kara, Arnd Bergmann,
Mike Snitzer, neilb, LKML, linux-raid, dm-devel, Garg, Dinesh, tj,
bart.vanassche, jmoyer, Alasdair G Kergon, Mikulas Patocka
In-Reply-To: <CAMz4kuLTuY5CpTfW4Gijesy1Pp9PxbWj076XeoV24Hj3EydifQ@mail.gmail.com>
On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote:
> These are the benchmarks for request based dm-crypt. Please check it.
Now please put request-based dm-crypt completely to one side and focus
just on the existing bio-based code. Why is it slower and what can be
adjusted to improve this?
People aren't going to take a request-based solution seriously until
you can explain in full detail *why* bio-based is slower AND why it's
impossible to improve its performance.
> For request based things, some sequential bios/requests can merged
> into one request to expand the IO size to be a big block handled by
> hardware engine at one time.
Bio-based also merges I/O so that does not provide justification.
Investigate in much more detail the actual merging and scheduling
involved in the cases you need to optimise. See if blktrace gives you
any clues, or add your own instrumentation. You could even look at some
of the patches we've had in the list archives for optimising bio-based
crypt in different situations.
Alasdair
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Dallas Clement @ 2015-12-03 0:12 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
In-Reply-To: <CAE9DZUQGeGFxWigXJr4+HoWCn7+TYj4QBECLce75_zugV14wZQ@mail.gmail.com>
On Wed, Dec 2, 2015 at 1:50 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Wed, Dec 2, 2015 at 9:51 AM, Phil Turmel <philip@turmel.org> wrote:
>> On 12/02/2015 10:44 AM, Robert Kierski wrote:
>>> I've tried a variety of settings... ranging from 17 to 32768.
>>>
>>> Yes.. with stripe_cache_size set to 17, I see a C/T of rmw's. And my TP goes in the toilet -- even with the RAM disks, I get only about 30M/s.
>>
>> Ok.
>>
>> You mentioned you aren't using a filesystem. How are you testing?
>>
>> Phil
>>
>> ps. convention on kernel.org is to trim replies and bottom-post, or
>> interleave. Please do.
>
> Thank you all for your responses.
>
> Keld,
>
>> Did you test the performance of other raid types, such as RAID1 and the various layouts of RAID10 for the newer kernels?
>
> I did try RAID 1 but not RAID 10. With RAID 1 I am seeing much higher
> average and peak wMB/s and disk utilization than with RAID 5 and 6.
> Though I need to run some more tests to compare the performance of
> newer kernels with the 2.6.39.4 kernel. Will report on that a bit
> later.
>
> Roman,
>
>> Do you use a write intent bitmap (internal?), what is your bitmap chunk size?
>
> Yes, I do. After reading up on this, I see that it can negatively
> affect write performance. The bitmap chunk size is 67108864.
>
>> What is your stripe_cache_size set to?
>
> strip_cache_size is 8192
>
> Robert, like you I am observing that my CPU is mostly idle during RAID
> 5 or 6 write testing. Something else is throttling the traffic. Not
> sure if there is some threshold crossing i.e. queue size, await time
> etc that is causing this or if it is implementation problem.
>
> I understand that the stripe cache grows dynamically in >= 4.1
> kernels. Fwiw, adjusting the stripe cache made no difference in my
> results.
>
> Regards,
>
> Dallas
Here is a summary of the performance differences I am seeing with the
3.10.69 kernel vs the 2.6.39.4 kernel (baseline):
RAID 0
bs = 512k - 3.5% slower
bs = 2048k - 1.5% slower
RAID 1
bs = 512k - 35% faster
bs = 2048k - 48% faster
RAID 5
bs = 512k - 22% slower
bs = 2048k - 28% slower
RAID 6
bs = 512k - 24% slower
bs = 2048k - 30% slower
Surprisingly RAID 1 is faster in the newer kernel, but RAID 5 & 6 much slower.
All measurements computed from bandwidth averages taken on 12 disk
array with XFS filesytem using fio with direct=1, sync=1,
invalidate=1.
Seems incredulous!?
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Phil Turmel @ 2015-12-03 2:18 UTC (permalink / raw)
To: Dallas Clement, linux-raid@vger.kernel.org
In-Reply-To: <CAE9DZUR=w9gbyqm0NzqrKk-G-mT3vd9Py54uEps7fpxC4LMBhg@mail.gmail.com>
On 12/02/2015 07:12 PM, Dallas Clement wrote:
> All measurements computed from bandwidth averages taken on 12 disk
> array with XFS filesytem using fio with direct=1, sync=1,
> invalidate=1.
Why do you need direct=1 and sync=1 ? Have you checked an strace from
the app you are trying to model that shows it uses these?
> Seems incredulous!?
Not with those options. Particularly sync=1. That causes an inode
stats update and a hardware queue flush after every write operation.
Support for that on various devices has changed over time.
I suspect if you do a bisect on the kernel to pinpoint the change(s)
that is doing this, you'll find a patch that closes a device-specific or
filesystem sync bug or something that enables deep queues for a device.
Modern software that needs file integrity guarantees make sparse use of
fdatasync and/or fsync and avoid sync entirely. You'll have a more
believable test if you use fsync_on_close=1 or end_fsync=1.
Phil
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Dallas Clement @ 2015-12-03 2:24 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <565FA66C.8060907@turmel.org>
On Wed, Dec 2, 2015 at 8:18 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/02/2015 07:12 PM, Dallas Clement wrote:
>> All measurements computed from bandwidth averages taken on 12 disk
>> array with XFS filesytem using fio with direct=1, sync=1,
>> invalidate=1.
>
> Why do you need direct=1 and sync=1 ? Have you checked an strace from
> the app you are trying to model that shows it uses these?
>
>> Seems incredulous!?
>
> Not with those options. Particularly sync=1. That causes an inode
> stats update and a hardware queue flush after every write operation.
> Support for that on various devices has changed over time.
>
> I suspect if you do a bisect on the kernel to pinpoint the change(s)
> that is doing this, you'll find a patch that closes a device-specific or
> filesystem sync bug or something that enables deep queues for a device.
>
> Modern software that needs file integrity guarantees make sparse use of
> fdatasync and/or fsync and avoid sync entirely. You'll have a more
> believable test if you use fsync_on_close=1 or end_fsync=1.
>
> Phil
Hi Phil. Hmm that makes sense that something may have changed wrt to
syncing. Basically what I am trying to do with my fio testing is
avoid any asynchronous or caching behavior.
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Dallas Clement @ 2015-12-03 2:33 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAE9DZUSRZ=bsga0G1UvfL3Dj-4W0ihXgOF=Ppqv=5-vz2+kAoQ@mail.gmail.com>
On Wed, Dec 2, 2015 at 8:24 PM, Dallas Clement
<dallas.a.clement@gmail.com> wrote:
> On Wed, Dec 2, 2015 at 8:18 PM, Phil Turmel <philip@turmel.org> wrote:
>> On 12/02/2015 07:12 PM, Dallas Clement wrote:
>>> All measurements computed from bandwidth averages taken on 12 disk
>>> array with XFS filesytem using fio with direct=1, sync=1,
>>> invalidate=1.
>>
>> Why do you need direct=1 and sync=1 ? Have you checked an strace from
>> the app you are trying to model that shows it uses these?
>>
>>> Seems incredulous!?
>>
>> Not with those options. Particularly sync=1. That causes an inode
>> stats update and a hardware queue flush after every write operation.
>> Support for that on various devices has changed over time.
>>
>> I suspect if you do a bisect on the kernel to pinpoint the change(s)
>> that is doing this, you'll find a patch that closes a device-specific or
>> filesystem sync bug or something that enables deep queues for a device.
>>
>> Modern software that needs file integrity guarantees make sparse use of
>> fdatasync and/or fsync and avoid sync entirely. You'll have a more
>> believable test if you use fsync_on_close=1 or end_fsync=1.
>>
>> Phil
>
> Hi Phil. Hmm that makes sense that something may have changed wrt to
> syncing. Basically what I am trying to do with my fio testing is
> avoid any asynchronous or caching behavior.
I'm not sure that the sync=1 has any effect in this case where I've
got direct=1 set (for non buffered I/O). I think the sync=1 flag only
matters for buffered I/O. I really shouldn't be setting that flag at
all.
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Phil Turmel @ 2015-12-03 2:34 UTC (permalink / raw)
To: Dallas Clement; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAE9DZUSRZ=bsga0G1UvfL3Dj-4W0ihXgOF=Ppqv=5-vz2+kAoQ@mail.gmail.com>
On 12/02/2015 09:24 PM, Dallas Clement wrote:
> On Wed, Dec 2, 2015 at 8:18 PM, Phil Turmel <philip@turmel.org> wrote:
>> I suspect if you do a bisect on the kernel to pinpoint the change(s)
>> that is doing this, you'll find a patch that closes a device-specific or
>> filesystem sync bug or something that enables deep queues for a device.
>>
>> Modern software that needs file integrity guarantees make sparse use of
>> fdatasync and/or fsync and avoid sync entirely. You'll have a more
>> believable test if you use fsync_on_close=1 or end_fsync=1.
>>
>> Phil
>
> Hi Phil. Hmm that makes sense that something may have changed wrt to
> syncing. Basically what I am trying to do with my fio testing is
> avoid any asynchronous or caching behavior.
I hope that if you really need this you are doing exhaustive testing on
failure modes -- I would be worried that these speed changes imply flaws
in the older kernels.
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Phil Turmel @ 2015-12-03 2:38 UTC (permalink / raw)
To: Dallas Clement; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAE9DZURYwqnMtbhRXVvdqoL5watEyiT8Ct9w1D_Fkr4kzyvPKw@mail.gmail.com>
On 12/02/2015 09:33 PM, Dallas Clement wrote:
> I'm not sure that the sync=1 has any effect in this case where I've
> got direct=1 set (for non buffered I/O). I think the sync=1 flag only
> matters for buffered I/O. I really shouldn't be setting that flag at
> all.
It's substantially different from direct=1. O_DIRECT just bypasses the
kernel's caches. O_SYNC flushes the file data and filesystem metadata,
and kills the device caches and queues.
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Dallas Clement @ 2015-12-03 2:51 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <565FAB29.9060405@turmel.org>
On Wed, Dec 2, 2015 at 8:38 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/02/2015 09:33 PM, Dallas Clement wrote:
>
>> I'm not sure that the sync=1 has any effect in this case where I've
>> got direct=1 set (for non buffered I/O). I think the sync=1 flag only
>> matters for buffered I/O. I really shouldn't be setting that flag at
>> all.
>
> It's substantially different from direct=1. O_DIRECT just bypasses the
> kernel's caches. O_SYNC flushes the file data and filesystem metadata,
> and kills the device caches and queues.
Isn't O_SYNC only applicable for buffered I/O or going through the
kernel caches? If I'm using O_DIRECT, seems like it should just
ignore this flag.
^ permalink raw reply
* Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-12-03 2:56 UTC (permalink / raw)
To: Baolin Wang, Mark Brown, Jens Axboe, keith.busch, Jan Kara,
Arnd Bergmann, Mike Snitzer, neilb, LKML, linux-raid, dm-devel,
Garg, Dinesh, tj, bart.vanassche, jmoyer, Alasdair G Kergon,
Mikulas Patocka
In-Reply-To: <20151202195657.GB11127@agk-dp.fab.redhat.com>
On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote:
> On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote:
>> These are the benchmarks for request based dm-crypt. Please check it.
>
> Now please put request-based dm-crypt completely to one side and focus
> just on the existing bio-based code. Why is it slower and what can be
> adjusted to improve this?
>
OK. I think I find something need to be point out.
1. From the IO block size test in the performance report, for the
request based, we can find it can not get the corresponding
performance if we just expand the IO size. Because In dm crypt, it
will map the data buffer of one request with scatterlists, and send
all scatterlists of one request to the encryption engine to encrypt or
decrypt. I found if the scatterlist list number is small and each
scatterlist length is bigger, it will improve the encryption speed,
that helps the engine palys best performance. But a big IO size does
not mean bigger scatterlists (maybe many scatterlists with small
length), that's why we can not get the corresponding performance if we
just expand the IO size I think.
2. Why bio based is slower?
If you understand 1, you can obviously understand the crypto engine
likes bigger scatterlists to improve the performance. But for bio
based, it only send one scatterlist (the scatterlist's length is
always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It
means if the bio size is 1M, the bio based will send 2048 times (evey
time the only one scatterlist length is 512 bytes) to crypto engine to
handle, which is more time-consuming and ineffective for the crypto
engine. But for request based, it can map the whole request with many
scatterlists (not just one scatterlist), and send all the scatterlists
to the crypto engine which can improve the performance, is it right?
Another optimization solution I think is we can expand the scatterlist
entry number for bio based.
> People aren't going to take a request-based solution seriously until
> you can explain in full detail *why* bio-based is slower AND why it's
> impossible to improve its performance.
>
>> For request based things, some sequential bios/requests can merged
>> into one request to expand the IO size to be a big block handled by
>> hardware engine at one time.
>
> Bio-based also merges I/O so that does not provide justification.
> Investigate in much more detail the actual merging and scheduling
> involved in the cases you need to optimise. See if blktrace gives you
> any clues, or add your own instrumentation. You could even look at some
> of the patches we've had in the list archives for optimising bio-based
> crypt in different situations.
>
> Alasdair
>
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Phil Turmel @ 2015-12-03 4:30 UTC (permalink / raw)
To: Dallas Clement; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAE9DZUQhwXYOA1mMOQX4UjPCBiRJuVKRf1woVmuW1RWO4WzL5Q@mail.gmail.com>
On 12/02/2015 09:51 PM, Dallas Clement wrote:
> On Wed, Dec 2, 2015 at 8:38 PM, Phil Turmel <philip@turmel.org> wrote:
>> On 12/02/2015 09:33 PM, Dallas Clement wrote:
>>
>>> I'm not sure that the sync=1 has any effect in this case where I've
>>> got direct=1 set (for non buffered I/O). I think the sync=1 flag only
>>> matters for buffered I/O. I really shouldn't be setting that flag at
>>> all.
>>
>> It's substantially different from direct=1. O_DIRECT just bypasses the
>> kernel's caches. O_SYNC flushes the file data and filesystem metadata,
>> and kills the device caches and queues.
>
> Isn't O_SYNC only applicable for buffered I/O or going through the
> kernel caches? If I'm using O_DIRECT, seems like it should just
> ignore this flag.
O_SYNC is orthogonal to whether the kernel caches are involved. It is
about ensuring that data *and* metadata are safely written all the way
to permanent media.
Phil
^ permalink raw reply
* Re: two small raid5 cache updates
From: Shaohua Li @ 2015-12-03 4:47 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: neilb, linux-raid
In-Reply-To: <1449072399-15360-1-git-send-email-hch@lst.de>
On Wed, Dec 02, 2015 at 05:06:37PM +0100, Christoph Hellwig wrote:
> The first is a trival cleanup, the second a similarly trivial reduction
> in memory usage.
Looks good, thanks!
Reviewed-by: Shaohua Li <shli@fb.com>
^ permalink raw reply
* Re: [RFC] using mempools for raid5-cache
From: Shaohua Li @ 2015-12-03 4:49 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: neilb, linux-raid
In-Reply-To: <1449072638-15409-1-git-send-email-hch@lst.de>
On Wed, Dec 02, 2015 at 05:10:36PM +0100, Christoph Hellwig wrote:
> Currently the raid5-cache code is heavily relying on GFP_NOFAIL allocations.
>
> I've looked into replacing these with mempools and biosets, and for the
> bio and the meta_page that's pretty trivial as they have short life times
> and do make guaranteed progress. I'm massively struggling with the iounit
> allocation, though. These can live on for a long time over log I/O, cache
> flushing and last but not least RAID I/O, and every attempt at something
> mempool-like results in reproducible deadlocks. I wonder if we need to
> figure out some more efficient data structure to communicate the completion
> status that doesn't rely on these fairly long living allocations from
> the I/O path.
>
> FYI, my last attempt to use the bio frontpad is below, but a mempool showed
> pretty similar results:
yep, the io unit and metadata are used to be allocated with a mempool,
but it's hard to calculate the pool size, so fall back to GFP_NOFAIL ...
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Dallas Clement @ 2015-12-03 4:49 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <565FC548.1060302@turmel.org>
On Wed, Dec 2, 2015 at 10:30 PM, Phil Turmel <philip@turmel.org> wrote:
> On 12/02/2015 09:51 PM, Dallas Clement wrote:
>> On Wed, Dec 2, 2015 at 8:38 PM, Phil Turmel <philip@turmel.org> wrote:
>>> On 12/02/2015 09:33 PM, Dallas Clement wrote:
>>>
>>>> I'm not sure that the sync=1 has any effect in this case where I've
>>>> got direct=1 set (for non buffered I/O). I think the sync=1 flag only
>>>> matters for buffered I/O. I really shouldn't be setting that flag at
>>>> all.
>>>
>>> It's substantially different from direct=1. O_DIRECT just bypasses the
>>> kernel's caches. O_SYNC flushes the file data and filesystem metadata,
>>> and kills the device caches and queues.
>>
>> Isn't O_SYNC only applicable for buffered I/O or going through the
>> kernel caches? If I'm using O_DIRECT, seems like it should just
>> ignore this flag.
>
> O_SYNC is orthogonal to whether the kernel caches are involved. It is
> about ensuring that data *and* metadata are safely written all the way
> to permanent media.
>
> Phil
Okay, that was my original intent, i.e. to avoid caching and buffering
as much as possible so that I could get a feel for true throughput
capability of the RAID device and the disks. Do you think it would be
better then to use sync=0 or fsync_on_close=1 for the sake of
evaluating RAID write performance?
^ permalink raw reply
* Re: [PATCH 1/2] raid5-cache: use a bio_set
From: Shaohua Li @ 2015-12-03 5:01 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: neilb, linux-raid
In-Reply-To: <1449072638-15409-2-git-send-email-hch@lst.de>
On Wed, Dec 02, 2015 at 05:10:37PM +0100, Christoph Hellwig wrote:
> This allows us to make guaranteed forward progress.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> drivers/md/raid5-cache.c | 16 +++++++++++++++-
> 1 file changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
> index 668e973..ef59564 100644
> --- a/drivers/md/raid5-cache.c
> +++ b/drivers/md/raid5-cache.c
> @@ -34,6 +34,12 @@
> #define RECLAIM_MAX_FREE_SPACE (10 * 1024 * 1024 * 2) /* sector */
> #define RECLAIM_MAX_FREE_SPACE_SHIFT (2)
>
> +/*
> + * We only need 2 bios per I/O unit to make progress, but ensure we
> + * have a few more available to not get too tight.
> + */
> +#define R5L_POOL_SIZE 1024
Looks reasonable, the recent arbitrary size bio makes things simpler. I
think we don't need to be that conservative. One metadata page can hold
at most
(PAGE_SIZE - sizeof(struct r5l_meta_block)) / sizeof(struct r5l_payload_data_parity)
pages.
Thanks,
Shaohua
^ permalink raw reply
* Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-12-03 10:36 UTC (permalink / raw)
To: Baolin Wang, Mark Brown, Jens Axboe, keith.busch, Jan Kara,
Arnd Bergmann, Mike Snitzer, neilb, LKML, linux-raid, dm-devel,
Garg, Dinesh, tj, bart.vanassche, jmoyer, Alasdair G Kergon,
Mikulas Patocka
In-Reply-To: <CAMz4kuL3qE-jE14F6ovAMRv6D-UFCdtiyUyi4ZDo+Lkr7_nA0Q@mail.gmail.com>
On 3 December 2015 at 10:56, Baolin Wang <baolin.wang@linaro.org> wrote:
> On 3 December 2015 at 03:56, Alasdair G Kergon <agk@redhat.com> wrote:
>> On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote:
>>> These are the benchmarks for request based dm-crypt. Please check it.
>>
>> Now please put request-based dm-crypt completely to one side and focus
>> just on the existing bio-based code. Why is it slower and what can be
>> adjusted to improve this?
>>
>
> OK. I think I find something need to be point out.
> 1. From the IO block size test in the performance report, for the
> request based, we can find it can not get the corresponding
> performance if we just expand the IO size. Because In dm crypt, it
> will map the data buffer of one request with scatterlists, and send
> all scatterlists of one request to the encryption engine to encrypt or
> decrypt. I found if the scatterlist list number is small and each
> scatterlist length is bigger, it will improve the encryption speed,
> that helps the engine palys best performance. But a big IO size does
> not mean bigger scatterlists (maybe many scatterlists with small
> length), that's why we can not get the corresponding performance if we
> just expand the IO size I think.
>
> 2. Why bio based is slower?
> If you understand 1, you can obviously understand the crypto engine
> likes bigger scatterlists to improve the performance. But for bio
> based, it only send one scatterlist (the scatterlist's length is
> always '1 << SECTOR_SHIFT' = 512) to the crypto engine at one time. It
> means if the bio size is 1M, the bio based will send 2048 times (evey
> time the only one scatterlist length is 512 bytes) to crypto engine to
> handle, which is more time-consuming and ineffective for the crypto
> engine. But for request based, it can map the whole request with many
> scatterlists (not just one scatterlist), and send all the scatterlists
> to the crypto engine which can improve the performance, is it right?
>
> Another optimization solution I think is we can expand the scatterlist
> entry number for bio based.
>
I did some testing about my assumption of expanding the scatterlist
entry number for bio based. I did some modification for the bio based
to support multiple scatterlists, then it will get the same
performance as the request based things.
1. bio based with expanding the scatterlist entry
time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct
1073741824 bytes (1.1 GB) copied, 94.5458 s, 11.4 MB/s
real 1m34.562s
user 0m0.030s
sys 0m3.850s
2. Sequential read 1G with requset based:
time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct
1073741824 bytes (1.1 GB) copied, 94.8922 s, 11.3 MB/s
real 1m34.908s
user 0m0.030s
sys 0m4.000s
From the data, we can find the bio based also can get the same
performance as the request based. So if someone still don't like the
request based things, I think we can optimize the bio based by
expanding the scatterlists number. Thanks.
>>
>> Alasdair
>>
>
>
>
> --
> Baolin.wang
> Best Regards
--
Baolin.wang
Best Regards
^ permalink raw reply
* Reshape stalls and fails when SELinux is enabled
From: Enrico Tagliavini @ 2015-12-03 11:04 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 3820 bytes --]
Hi everybody,
yesterday I tested a conversion of a 2 disk raid1 to a 4 disk raid5.
The reshape never completed, I rebooted the VM and the raid device was
unrecoverable. Backup-file was not useful unfortunately. Take a
comfortable seat, this is a bit long story, sorry :).
Environment:
- VM running on my laptop (Fedora 22), created with virt-manager,
qemu based. All disk images are qcow2 virtio devices.
- VM OS: centos 7 64 bit, fully updated as of today.
- mdadm version: 3.3.2-2.el7_1.1
- kernel version: 3.10.0-229.20.1.el7
- selinux policy targeted: 3.13.1-23.el7_1.21
How to reproduce:
* Attach four 1 GB disks to the VM, in the following example they are
vd[b,c,d,e].
* create a raid1 device with two disks: mdadm --create /dev/md0 -e
1.2 -n 2 -l raid1 -N raidtest /dev/vdb /dev/vdc
* [optional?]: I created a FS on it: mkfs.xfs -L test /dev/md0
* [optional?]: I created three files on it and noted the checksum, to
check for eventual corruptions
- mount /dev/md0 /mnt/raidtest/ && cd /mnt/raidtest/
- echo uno > first.txt ; echo due > second.txt ; touch third.txt
; shred -s 512M -f third.txt
- for file in *; do sha1sum $file >> sha1sum.txt; done
* check /proc/mdstat to make sure the raid1 sync operation is
complete (should really be fast but better safe than sorry)
* unmount /mnt/raidtest/
* add the two additional disks as spares:
- mdadm --manage /dev/md0 --add-spare /dev/vdd
- mdadm --manage /dev/md0 --add-spare /dev/vde
* grow and reshape: mdadm --grow /dev/md0 -n 4 --level=5
--backup-file /root/backup-md0 # (I also tried with
/var/local/backup-md0)
At this point an AVC will happen. Even if I'm in an interactive
session and, as such SELinux should not limit it normally, mdadm
process switches to mdadm_t type (maybe a forked process with its own
session group?) and is not allowed to write a file in the /root/ or
/var/ folders. This is ok, however mdadm keeps going instead of
aborting the reshape. It's running without a backup file, that's not
what the admin asked for since the --backup-file option is specified.
But even worst than this is that my reshape got stuck and never
completed. I waited a couple of hours but it remained at 0%. Something
was actually written to the backup file (which is weird given the AVC,
but it can be the original mdadm process not running under mdadm_t).
At this point I was kind of curious to test what would happen if a
distracted admin like me wont notice the problem and, days later,
would reboot the server due to security updates or anything else. The
result is an unrecoverable md array. I tried to assemble it back with
the backup file
mdadm --assemble /dev/md0 -u 6f53ec3e:d9868fef:12d3e243:8489561b
--backup-file /root/backup-md
But no way
mdadm: [sorry I copied wrong and the device name was lost] has an
active reshape - checking if critical section needs to be restored
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
I retried the entire procedure from scratch, but this time with before
mdadm --grow I set SELinux in permissive mode with setenforce 0.
Everything was butter smooth this time. Reshape was almost instant for
such a small array, data was checksumming correctly and my array was
level 5.
Now there might be a problem with the SELinux policy here, but
honestly I think mdadm should just abort, whatever the reason of the
problem was. There might be other scenarios not involving SELinux
causing the same problem.
It would also be nice to suggest the user, if SELinux is active, to
change the context of the backup file to something SELinux will permit
(mdadm_map_t, mdadm_var_run_t?).
Attached you can also find the AVC denial for my entire day of testing.
Thank you for your help.
Best regards.
Enrico Tagliavini
[-- Attachment #2: changeraid1_to_raid5.txt --]
[-- Type: text/plain, Size: 68 bytes --]
mdadm --grow /dev/md0 -n 4 --level=5 --backup-file=/root/backup-md0
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox