* [V2 PATCH 05/13] md-cluster: fix locking when node joins cluster during message broadcast
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
If a node joins the cluster while a message broadcast
is under way, a lock issue could happen as follows.
For a cluster which included two nodes, if node A is
calling __sendmsg before up-convert CR to EX on ack,
and node B released CR on ack. But if a new node C
joins the cluster and it doesn't receive the message
which A sent before, so it could hold CR on ack before
A up-convert CR to EX on ack.
So a node joining the cluster should get an EX lock on
the "token" first to ensure no broadcast is ongoing,
then release it after held CR on ack.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 76f88f7..30f1160 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -781,17 +781,24 @@ static int join(struct mddev *mddev, int nodes)
cinfo->token_lockres = lockres_init(mddev, "token", NULL, 0);
if (!cinfo->token_lockres)
goto err;
- cinfo->ack_lockres = lockres_init(mddev, "ack", ack_bast, 0);
- if (!cinfo->ack_lockres)
- goto err;
cinfo->no_new_dev_lockres = lockres_init(mddev, "no-new-dev", NULL, 0);
if (!cinfo->no_new_dev_lockres)
goto err;
+ ret = dlm_lock_sync(cinfo->token_lockres, DLM_LOCK_EX);
+ if (ret) {
+ ret = -EAGAIN;
+ pr_err("md-cluster: can't join cluster to avoid lock issue\n");
+ goto err;
+ }
+ cinfo->ack_lockres = lockres_init(mddev, "ack", ack_bast, 0);
+ if (!cinfo->ack_lockres)
+ goto err;
/* get sync CR lock on ACK. */
if (dlm_lock_sync(cinfo->ack_lockres, DLM_LOCK_CR))
pr_err("md-cluster: failed to get a sync CR lock on ACK!(%d)\n",
ret);
+ dlm_unlock_sync(cinfo->token_lockres);
/* get sync CR lock on no-new-dev. */
if (dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR))
pr_err("md-cluster: failed to get a sync CR lock on no-new-dev!(%d)\n", ret);
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 06/13] md-cluster: change array_sectors and update size are not supported
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
Currently, some features are not supported yet,
such as change array_sectors and update size, so
return EINVAL for them and listed it in document.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Documentation/md-cluster.txt | 6 ++++++
drivers/md/md.c | 8 ++++++++
2 files changed, 14 insertions(+)
diff --git a/Documentation/md-cluster.txt b/Documentation/md-cluster.txt
index c100c71..3888327 100644
--- a/Documentation/md-cluster.txt
+++ b/Documentation/md-cluster.txt
@@ -316,3 +316,9 @@ The algorithm is:
nodes are using the raid which is achieved by lock all bitmap
locks within the cluster, and also those locks are unlocked
accordingly.
+
+7. Unsupported features
+
+There are somethings which are not supported by cluster MD yet.
+
+- update size and change array_sectors.
diff --git a/drivers/md/md.c b/drivers/md/md.c
index dd83a50..8cc4bbc 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -4817,6 +4817,10 @@ array_size_store(struct mddev *mddev, const char *buf, size_t len)
if (err)
return err;
+ /* cluster raid doesn't support change array_sectors */
+ if (mddev_is_clustered(mddev))
+ return -EINVAL;
+
if (strncmp(buf, "default", 7) == 0) {
if (mddev->pers)
sectors = mddev->pers->size(mddev, 0, 0);
@@ -6438,6 +6442,10 @@ static int update_size(struct mddev *mddev, sector_t num_sectors)
int rv;
int fit = (num_sectors == 0);
+ /* cluster raid doesn't support update size */
+ if (mddev_is_clustered(mddev))
+ return -EINVAL;
+
if (mddev->pers->resize == NULL)
return -EINVAL;
/* The "num_sectors" is the number of sectors of each device that
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 07/13] md-cluster: wakeup thread if activated a spare disk
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
When a device is re-added, it will ultimately need
to be activated and that happens in md_check_recovery,
so we need to set MD_RECOVERY_NEEDED right after
remove_and_add_spares.
A specifical issue without the change is that when
one node perform fail/remove/readd on a disk, but
slave nodes could not add the disk back to array as
expected (added as missed instead of in sync). So
give slave nodes a chance to do resync.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 8cc4bbc..06f6e81 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -8694,6 +8694,11 @@ static void check_sb_changes(struct mddev *mddev, struct md_rdev *rdev)
ret = remove_and_add_spares(mddev, rdev2);
pr_info("Activated spare: %s\n",
bdevname(rdev2->bdev,b));
+ /* wakeup mddev->thread here, so array could
+ * perform resync with the new activated disk */
+ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
+ md_wakeup_thread(mddev->thread);
+
}
/* device faulty
* We just want to do the minimum to mark the disk
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 08/13] md-cluster: always setup in-memory bitmap
From: Guoqing Jiang @ 2016-05-02 15:50 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
The in-memory bitmap for raid is allocated on demand,
then for cluster scenario, it is possible that slave
node which received RESYNCING message doesn't have the
in-memory bitmap when master node is perform resyncing,
so we can't make bitmap is match up well among each
nodes.
So for cluster scenario, we need always preserve the
bitmap, and ensure the page will not be freed. And a
no_hijack flag is introduced to both bitmap_checkpage
and bitmap_get_counter, which makes cluster raid returns
fail once allocate failed.
And the next patch is relied on this change since it
keeps sync bitmap among each nodes during resyncing
stage.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 37 +++++++++++++++++++++++++++++++++++--
1 file changed, 35 insertions(+), 2 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index 3fe86b5..431da21 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -46,7 +46,7 @@ static inline char *bmname(struct bitmap *bitmap)
* allocated while we're using it
*/
static int bitmap_checkpage(struct bitmap_counts *bitmap,
- unsigned long page, int create)
+ unsigned long page, int create, int no_hijack)
__releases(bitmap->lock)
__acquires(bitmap->lock)
{
@@ -90,6 +90,9 @@ __acquires(bitmap->lock)
if (mappage == NULL) {
pr_debug("md/bitmap: map page allocation failed, hijacking\n");
+ /* We don't support hijack for cluster raid */
+ if (no_hijack)
+ return -ENOMEM;
/* failed - set the hijacked flag so that we can use the
* pointer as a counter */
if (!bitmap->bp[page].map)
@@ -1321,7 +1324,7 @@ __acquires(bitmap->lock)
sector_t csize;
int err;
- err = bitmap_checkpage(bitmap, page, create);
+ err = bitmap_checkpage(bitmap, page, create, 0);
if (bitmap->bp[page].hijacked ||
bitmap->bp[page].map == NULL)
@@ -2032,6 +2035,36 @@ int bitmap_resize(struct bitmap *bitmap, sector_t blocks,
chunks << chunkshift);
spin_lock_irq(&bitmap->counts.lock);
+ /* For cluster raid, need to pre-allocate bitmap */
+ if (mddev_is_clustered(bitmap->mddev)) {
+ unsigned long page;
+ for (page = 0; page < pages; page++) {
+ ret = bitmap_checkpage(&bitmap->counts, page, 1, 1);
+ if (ret) {
+ unsigned long k;
+
+ /* deallocate the page memory */
+ for (k = 0; k < page; k++) {
+ if (new_bp[k].map)
+ kfree(new_bp[k].map);
+ }
+
+ /* restore some fields from old_counts */
+ bitmap->counts.bp = old_counts.bp;
+ bitmap->counts.pages = old_counts.pages;
+ bitmap->counts.missing_pages = old_counts.pages;
+ bitmap->counts.chunkshift = old_counts.chunkshift;
+ bitmap->counts.chunks = old_counts.chunks;
+ bitmap->mddev->bitmap_info.chunksize = 1 << (old_counts.chunkshift +
+ BITMAP_BLOCK_SHIFT);
+ blocks = old_counts.chunks << old_counts.chunkshift;
+ pr_err("Could not pre-allocate in-memory bitmap for cluster raid\n");
+ break;
+ } else
+ bitmap->counts.bp[page].count += 1;
+ }
+ }
+
for (block = 0; block < blocks; ) {
bitmap_counter_t *bmc_old, *bmc_new;
int set;
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 09/13] md-cluster: sync bitmap when node received RESYNCING msg
From: Guoqing Jiang @ 2016-05-02 15:50 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462204216-2269-1-git-send-email-gqjiang@suse.com>
If the node received RESYNCING message which means
another node will perform resync with the area, then
we don't want to do it again in another node.
Let's set RESYNC_MASK and clear NEEDED_MASK for the
region from old-low to new-low which has finished
syncing, and the region from old-hi to new-hi is about
to syncing, bitmap_sync_with_cluste is introduced for
the purpose.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 21 +++++++++++++++++++++
drivers/md/bitmap.h | 3 +++
drivers/md/md-cluster.c | 27 +++++++++++++++++++++++++++
3 files changed, 51 insertions(+)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index 431da21..ac93d87 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -1597,6 +1597,27 @@ void bitmap_cond_end_sync(struct bitmap *bitmap, sector_t sector, bool force)
}
EXPORT_SYMBOL(bitmap_cond_end_sync);
+void bitmap_sync_with_cluster(struct mddev *mddev,
+ sector_t old_lo, sector_t old_hi,
+ sector_t new_lo, sector_t new_hi)
+{
+ struct bitmap *bitmap = mddev->bitmap;
+ sector_t sector, blocks = 0;
+
+ for (sector = old_lo; sector < new_lo; ) {
+ bitmap_end_sync(bitmap, sector, &blocks, 0);
+ sector += blocks;
+ }
+ WARN((blocks > new_lo) && old_lo, "alignment is not correct for lo\n");
+
+ for (sector = old_hi; sector < new_hi; ) {
+ bitmap_start_sync(bitmap, sector, &blocks, 0);
+ sector += blocks;
+ }
+ WARN((blocks > new_hi) && old_hi, "alignment is not correct for hi\n");
+}
+EXPORT_SYMBOL(bitmap_sync_with_cluster);
+
static void bitmap_set_memory_bits(struct bitmap *bitmap, sector_t offset, int needed)
{
/* For each chunk covered by any of these sectors, set the
diff --git a/drivers/md/bitmap.h b/drivers/md/bitmap.h
index 5e3fcd6..5b6dd63 100644
--- a/drivers/md/bitmap.h
+++ b/drivers/md/bitmap.h
@@ -258,6 +258,9 @@ int bitmap_start_sync(struct bitmap *bitmap, sector_t offset, sector_t *blocks,
void bitmap_end_sync(struct bitmap *bitmap, sector_t offset, sector_t *blocks, int aborted);
void bitmap_close_sync(struct bitmap *bitmap);
void bitmap_cond_end_sync(struct bitmap *bitmap, sector_t sector, bool force);
+void bitmap_sync_with_cluster(struct mddev *mddev,
+ sector_t old_lo, sector_t old_hi,
+ sector_t new_lo, sector_t new_hi);
void bitmap_unplug(struct bitmap *bitmap);
void bitmap_daemon_work(struct mddev *mddev);
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 30f1160..a55b5f4 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -85,6 +85,9 @@ struct md_cluster_info {
struct completion newdisk_completion;
wait_queue_head_t wait;
unsigned long state;
+ /* record the region in RESYNCING message */
+ sector_t sync_low;
+ sector_t sync_hi;
};
enum msg_type {
@@ -411,6 +414,30 @@ static void process_suspend_info(struct mddev *mddev,
md_wakeup_thread(mddev->thread);
return;
}
+
+ /*
+ * The bitmaps are not same for different nodes
+ * if RESYNCING is happening in one node, then
+ * the node which received the RESYNCING message
+ * probably will perform resync with the region
+ * [lo, hi] again, so we could reduce resync time
+ * a lot if we can ensure that the bitmaps among
+ * different nodes are match up well.
+ *
+ * sync_low/hi is used to record the region which
+ * arrived in the previous RESYNCING message,
+ *
+ * Call bitmap_sync_with_cluster to clear
+ * NEEDED_MASK and set RESYNC_MASK since
+ * resync thread is running in another node,
+ * so we don't need to do the resync again
+ * with the same section */
+ bitmap_sync_with_cluster(mddev, cinfo->sync_low,
+ cinfo->sync_hi,
+ lo, hi);
+ cinfo->sync_low = lo;
+ cinfo->sync_hi = hi;
+
s = kzalloc(sizeof(struct suspend_info), GFP_KERNEL);
if (!s)
return;
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 10/13] md-cluster/bitmap: fix wrong calcuation of offset
From: Guoqing Jiang @ 2016-05-02 15:50 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462204216-2269-1-git-send-email-gqjiang@suse.com>
The offset is wrong in bitmap_storage_alloc, we should
set it like below in bitmap_init_from_disk().
node_offset = bitmap->cluster_slot * (DIV_ROUND_UP(store->bytes, PAGE_SIZE));
Because 'offset' is only assigned to 'page->index' and
that is usually over-written by read_sb_page. So it does
not cause problem in general, but it still need to be fixed.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index ac93d87..cf93bb8 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -759,7 +759,7 @@ static int bitmap_storage_alloc(struct bitmap_storage *store,
bytes += sizeof(bitmap_super_t);
num_pages = DIV_ROUND_UP(bytes, PAGE_SIZE);
- offset = slot_number * (num_pages - 1);
+ offset = slot_number * num_pages;
store->filemap = kmalloc(sizeof(struct page *)
* num_pages, GFP_KERNEL);
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 11/13] md-cluster/bitmap: fix wrong page num in bitmap_file_clear_bit and bitmap_file_set_bit
From: Guoqing Jiang @ 2016-05-02 15:50 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462204216-2269-1-git-send-email-gqjiang@suse.com>
The pnum passed to set_page_attr and test_page_attr should from
0 to storage.file_pages - 1, but bitmap_file_set_bit and
bitmap_file_clear_bit call set_page_attr and test_page_attr with
page->index parameter while page->index has already added node_offset
before.
So we need to minus node_offset in both bitmap_file_clear_bit
and bitmap_file_set_bit.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index cf93bb8..de28c80 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -903,6 +903,11 @@ static void bitmap_file_set_bit(struct bitmap *bitmap, sector_t block)
struct page *page;
void *kaddr;
unsigned long chunk = block >> bitmap->counts.chunkshift;
+ struct bitmap_storage *store = &bitmap->storage;
+ unsigned long node_offset = 0;
+
+ if (mddev_is_clustered(bitmap->mddev))
+ node_offset = bitmap->cluster_slot * store->file_pages;
page = filemap_get_page(&bitmap->storage, chunk);
if (!page)
@@ -918,7 +923,7 @@ static void bitmap_file_set_bit(struct bitmap *bitmap, sector_t block)
kunmap_atomic(kaddr);
pr_debug("set file bit %lu page %lu\n", bit, page->index);
/* record page number so it gets flushed to disk when unplug occurs */
- set_page_attr(bitmap, page->index, BITMAP_PAGE_DIRTY);
+ set_page_attr(bitmap, page->index - node_offset, BITMAP_PAGE_DIRTY);
}
static void bitmap_file_clear_bit(struct bitmap *bitmap, sector_t block)
@@ -927,6 +932,11 @@ static void bitmap_file_clear_bit(struct bitmap *bitmap, sector_t block)
struct page *page;
void *paddr;
unsigned long chunk = block >> bitmap->counts.chunkshift;
+ struct bitmap_storage *store = &bitmap->storage;
+ unsigned long node_offset = 0;
+
+ if (mddev_is_clustered(bitmap->mddev))
+ node_offset = bitmap->cluster_slot * store->file_pages;
page = filemap_get_page(&bitmap->storage, chunk);
if (!page)
@@ -938,8 +948,8 @@ static void bitmap_file_clear_bit(struct bitmap *bitmap, sector_t block)
else
clear_bit_le(bit, paddr);
kunmap_atomic(paddr);
- if (!test_page_attr(bitmap, page->index, BITMAP_PAGE_NEEDWRITE)) {
- set_page_attr(bitmap, page->index, BITMAP_PAGE_PENDING);
+ if (!test_page_attr(bitmap, page->index - node_offset, BITMAP_PAGE_NEEDWRITE)) {
+ set_page_attr(bitmap, page->index - node_offset, BITMAP_PAGE_PENDING);
bitmap->allclean = 0;
}
}
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 12/13] md-cluster/bitmap: unplug bitmap to sync dirty pages to disk
From: Guoqing Jiang @ 2016-05-02 15:50 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462204216-2269-1-git-send-email-gqjiang@suse.com>
This patch is doing two distinct but related things.
1. It adds bitmap_unplug() for the main bitmap (mddev->bitmap). As bit
have been set, BITMAP_PAGE_DIRTY is set so bitmap_deamon_work() will
not write those pages out in its regular scans, only bitmap_unplug()
will. If there are no writes to the array, bitmap_unplug() won't be
called, so we need to call it explicitly here.
2. bitmap_write_all() is a bit of a confusing interface as it doesn't
actually write anything. The current code for writing "bitmap" works
but this change makes it a bit clearer.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index de28c80..4a05bac 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -1924,14 +1924,14 @@ int bitmap_copy_from_slot(struct mddev *mddev, int slot,
if (clear_bits) {
bitmap_update_sb(bitmap);
- /* Setting this for the ev_page should be enough.
- * And we do not require both write_all and PAGE_DIRT either
- */
+ /* BITMAP_PAGE_PENDING is set, but bitmap_unplug needs
+ * BITMAP_PAGE_DIRTY or _NEEDWRITE to write ... */
for (i = 0; i < bitmap->storage.file_pages; i++)
- set_page_attr(bitmap, i, BITMAP_PAGE_DIRTY);
- bitmap_write_all(bitmap);
+ if (test_page_attr(bitmap, i, BITMAP_PAGE_PENDING))
+ set_page_attr(bitmap, i, BITMAP_PAGE_NEEDWRITE);
bitmap_unplug(bitmap);
}
+ bitmap_unplug(mddev->bitmap);
*low = lo;
*high = hi;
err:
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 13/13] md-cluster: fix ifnullfree.cocci warnings
From: Guoqing Jiang @ 2016-05-02 15:50 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, kbuild test robot, Fengguang Wu
In-Reply-To: <1462204216-2269-1-git-send-email-gqjiang@suse.com>
From: kbuild test robot <lkp@intel.com>
drivers/md/bitmap.c:2049:6-11: WARNING: NULL check before freeing functions like kfree, debugfs_remove, debugfs_remove_recursive or usb_free_urb is not needed. Maybe consider reorganizing relevant code to avoid passing NULL values.
NULL check before some freeing functions is not needed.
Based on checkpatch warning
"kfree(NULL) is safe this check is probably not required"
and kfreeaddr.cocci by Julia Lawall.
Generated by: scripts/coccinelle/free/ifnullfree.cocci
Acked-by: Guoqing Jiang <gqjiang@suse.com>
Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
---
drivers/md/bitmap.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index 4a05bac..ad5a858 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -2076,8 +2076,7 @@ int bitmap_resize(struct bitmap *bitmap, sector_t blocks,
/* deallocate the page memory */
for (k = 0; k < page; k++) {
- if (new_bp[k].map)
- kfree(new_bp[k].map);
+ kfree(new_bp[k].map);
}
/* restore some fields from old_counts */
--
2.6.6
^ permalink raw reply related
* [GIT PULL] MD fix for 4.6-rc6
From: Shaohua Li @ 2016-05-02 17:32 UTC (permalink / raw)
To: torvalds; +Cc: linux-kernel, linux-raid, neilb
Hi Linus,
please pull MD fix for 4.6. This update includes several trival fixes. The only
important one is to fix MD bio merge, which has big performance impact.
Thanks,
Shaohua
The following changes since commit 5b5b7fd185e997ebc18f76b98b9f4ff148d3f5bb:
Merge branch 'parisc-4.6-3' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux (2016-04-09 14:10:20 -0700)
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/shli/md.git tags/md/4.6-rc6-fix
for you to fetch changes up to b8a0b8e94648179b92191e5cf4fd3c4379b31cc4:
raid5: delete unnecessary warnning (2016-04-29 14:18:03 -0700)
----------------------------------------------------------------
Dan Carpenter (1):
md/raid0: fix uninitialized variable bug
Michał Pecio (1):
md/raid0: remove empty line printk from dump_zones
Shaohua Li (2):
MD: make bio mergeable
raid5: delete unnecessary warnning
drivers/md/md.c | 2 ++
drivers/md/raid0.c | 2 +-
drivers/md/raid5.c | 2 --
3 files changed, 3 insertions(+), 3 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [V2 PATCH 00/13] The latest patches for md-cluster
From: Shaohua Li @ 2016-05-02 17:49 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: neilb, linux-raid
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
On Mon, May 02, 2016 at 11:33:07AM -0400, Guoqing Jiang wrote:
> Changes:
> 1. delete no_hijack parameter from bitmap_get_counter
> 2. add one patch from kbuild test robot to remove checkpatch warning
> 3. md-set-MD_CHANGE_PENDING-in-a-spinlocked-region.patch is removed
> from the patchset and it will be post later.
>
> For cluster raid1, we found some issues and some codes need to be
> improved during the past months, and all the patches are based on
> for-next branch of md tree.
>
> The patchset also available in github as follows:
>
> https://github.com/GuoqingJiang/linux/tree/md-for-next
Applied, thanks!
^ permalink raw reply
* RAID5 - reshape_position too early for auto-recovery - aborting
From: SharksArt @ 2016-05-03 18:17 UTC (permalink / raw)
To: linux-raid
Hi,
I've a little problem with my Xpenology NAS (based on Synology). After
remplacement a disk (500GB) for a bigger one (1TB), it start checking
parity and i let him doing it during the night. But the next day, my
NAS was unreachable. There was no intence activity (led non blinking)
so i decided to power off the NAS, wait a little and power on. Bad
luck, the only volume (about 4.5TB) was unusable and showing no
capacity, used space etc...
So, my NAS is build with Xpenology on an ESXi 5.5 with now 6x 1TB HDD
in RDM plus a virtual disk of 15MB for booting :
Cube> fdisk -l
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 311 2490240 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary
/dev/sda2 311 572 2097152 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary
/dev/sda3 588 121601 972036912 f Win95 Ext'd (LBA)
/dev/sda5 589 60801 483652864 fd Linux raid autodetect
/dev/sda6 60802 77825 136737232 fd Linux raid autodetect
/dev/sda7 77826 121601 351622672 fd Linux raid autodetect
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 311 2490240 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary
/dev/sdb2 311 572 2097152 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary
/dev/sdb3 588 121601 972036912 f Win95 Ext'd (LBA)
/dev/sdb5 589 60801 483652864 fd Linux raid autodetect
/dev/sdb6 60802 77825 136737232 fd Linux raid autodetect
/dev/sdb7 77826 121601 351622672 fd Linux raid autodetect
Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 311 2490240 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary
/dev/sdc2 311 572 2097152 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary
/dev/sdc3 588 121601 972036912 f Win95 Ext'd (LBA)
/dev/sdc5 589 60801 483652864 fd Linux raid autodetect
/dev/sdc6 60802 77825 136737232 fd Linux raid autodetect
/dev/sdc7 77826 121601 351622672 fd Linux raid autodetect
Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 311 2490240 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary
/dev/sdd2 311 572 2097152 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary
/dev/sdd3 588 121601 972036912 f Win95 Ext'd (LBA)
/dev/sdd5 589 60801 483652864 fd Linux raid autodetect
/dev/sdd6 60802 77825 136737232 fd Linux raid autodetect
/dev/sdd7 77826 121601 351622672 fd Linux raid autodetect
Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 311 2490240 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary
/dev/sde2 311 572 2097152 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary
/dev/sde3 588 121601 972036912 f Win95 Ext'd (LBA)
/dev/sde5 589 60801 483652864 fd Linux raid autodetect
/dev/sde6 60802 77825 136737232 fd Linux raid autodetect
/dev/sde7 77826 121601 351622672 fd Linux raid autodetect
Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdf1 1 311 2490240 fd Linux raid autodetect
Partition 1 does not end on cylinder boundary
/dev/sdf2 311 572 2097152 fd Linux raid autodetect
Partition 2 does not end on cylinder boundary
/dev/sdf3 588 121601 972036912 f Win95 Ext'd (LBA)
/dev/sdf5 589 60801 483652864 fd Linux raid autodetect
/dev/sdf6 60802 77825 136737232 fd Linux raid autodetect
/dev/sdf7 77826 121601 351622672 fd Linux raid autodetect
Disk /dev/sdaf: 16 MB, 16515072 bytes
4 heads, 32 sectors/track, 252 cylinders
Units = cylinders of 128 * 512 = 65536 bytes
Device Boot Start End Blocks Id System
/dev/sdaf1 * 1 252 16096+ e Win95 FAT16 (LBA)
// State after reboot :
Cube> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md4 : active raid5 sdf7[0] sda7[5] sdc7[4] sde7[3] sdd7[1]
1758107520 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
md2 : active raid5 sda5[9] sdd5[6] sdf5[5] sdc5[8] sde5[7]
2418258240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/5] [_UUUUU]
md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[4] sde2[3] sdf2[5]
2097088 blocks [12/6] [UUUUUU______]
md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[5] sdf1[4]
2490176 blocks [12/6] [UUUUUU______]
unused devices: <none>
// I've added sdb5 to md2 and sdb7 to md4 :
mdadm --manage /dev/md2 --add /dev/sdb5
mdadm: added /dev/sdb5
Cube> mdadm --manage /dev/md4 --add /dev/sdb7
mdadm: added /dev/sdb7
// Then after rebuild :
Cube> cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md4 : active raid5 sdb7[6] sdf7[0] sda7[5] sdc7[4] sde7[3] sdd7[1]
1758107520 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md2 : active raid5 sdb5[10] sda5[9] sdd5[6] sdf5[5] sdc5[8] sde5[7]
2418258240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md1 : active raid1 sda2[0] sdb2[1] sdc2[2] sdd2[4] sde2[3] sdf2[5]
2097088 blocks [12/6] [UUUUUU______]
md0 : active raid1 sda1[0] sdb1[1] sdc1[2] sdd1[3] sde1[5] sdf1[4]
2490176 blocks [12/6] [UUUUUU______]
unused devices: <none>
//
Cube> mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Thu Jul 10 21:13:14 2014
Raid Level : raid5
Array Size : 2418258240 (2306.23 GiB 2476.30 GB)
Used Dev Size : 483651648 (461.25 GiB 495.26 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Mon May 2 00:16:36 2016
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : cube:2
UUID : 103009d5:68bcfd24:c017fe46:a505b66b
Events : 36638
Number Major Minor RaidDevice State
10 8 21 0 active sync /dev/sdb5
9 8 5 1 active sync /dev/sda5
7 8 69 2 active sync /dev/sde5
8 8 37 3 active sync /dev/sdc5
5 8 85 4 active sync /dev/sdf5
6 8 53 5 active sync /dev/sdd5
//
Cube> mdadm -D /dev/md4
/dev/md4:
Version : 1.2
Creation Time : Sun Apr 5 11:43:39 2015
Raid Level : raid5
Array Size : 1758107520 (1676.66 GiB 1800.30 GB)
Used Dev Size : 351621504 (335.33 GiB 360.06 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Mon May 2 04:07:05 2016
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : cube:4
UUID : b0803ba8:eb54f644:84dc0b1f:b66dcc0f
Events : 6823
Number Major Minor RaidDevice State
0 8 87 0 active sync /dev/sdf7
1 8 55 1 active sync /dev/sdd7
3 8 71 2 active sync /dev/sde7
4 8 39 3 active sync /dev/sdc7
5 8 7 4 active sync /dev/sda7
6 8 23 5 active sync /dev/sdb7
// Good... but volume was not ok... because md3 has disappeared (RAID
informations one day before the crash) :
Cube> cat /etc/space/space_history_20160429_002313.xml
<?xml version="1.0" encoding="UTF-8"?>
<spaces>
<space path="/dev/vg1000/lv" reference="/volume1" >
<device>
<lvm path="/dev/vg1000"
uuid="5pSrIp-2W4j-LJUb-77OA-7g1h-Wdyz-pz83ku" designed_pv_counts="3"
status="normal" total_size="4476601630720" free_size="0"
pe_size="4194304">
<raids>
<raid path="/dev/md4" uuid="b0803ba8:eb54f644:84dc0b1f:b66dcc0f"
level="raid5" version="1.2">
<disks>
<disk status="normal" dev_path="/dev/sda7" model="Virtual SATA
Hard Drive " serial="00000000000000000001" partition_version="8"
slot="4">
</disk>
<disk status="normal" dev_path="/dev/sdb7" model="Virtual SATA
Hard Drive " serial="01000000000000000001" partition_version="8"
slot="5">
</disk>
<disk status="normal" dev_path="/dev/sdc7" model="Virtual SATA
Hard Drive " serial="02000000000000000001" partition_version="8"
slot="3">
</disk>
<disk status="normal" dev_path="/dev/sdd7" model="Virtual SATA
Hard Drive " serial="03000000000000000001" partition_version="7"
slot="1">
</disk>
<disk status="normal" dev_path="/dev/sde7" model="Virtual SATA
Hard Drive " serial="04000000000000000001" partition_version="8"
slot="2">
</disk>
<disk status="normal" dev_path="/dev/sdf7" model="Virtual SATA
Hard Drive " serial="05000000000000000001" partition_version="7"
slot="0">
</disk>
</disks>
</raid>
<raid path="/dev/md2" uuid="103009d5:68bcfd24:c017fe46:a505b66b"
level="raid5" version="1.2">
<disks>
<disk status="normal" dev_path="/dev/sda5" model="Virtual SATA
Hard Drive " serial="00000000000000000001" partition_version="8"
slot="1">
</disk>
<disk status="normal" dev_path="/dev/sdb5" model="Virtual SATA
Hard Drive " serial="01000000000000000001" partition_version="8"
slot="0">
</disk>
<disk status="normal" dev_path="/dev/sdc5" model="Virtual SATA
Hard Drive " serial="02000000000000000001" partition_version="8"
slot="3">
</disk>
<disk status="normal" dev_path="/dev/sdd5" model="Virtual SATA
Hard Drive " serial="03000000000000000001" partition_version="7"
slot="5">
</disk>
<disk status="normal" dev_path="/dev/sde5" model="Virtual SATA
Hard Drive " serial="04000000000000000001" partition_version="8"
slot="2">
</disk>
<disk status="normal" dev_path="/dev/sdf5" model="Virtual SATA
Hard Drive " serial="05000000000000000001" partition_version="7"
slot="4">
</disk>
</disks>
</raid>
<raid path="/dev/md3" uuid="bf5e3c59:d407890c:bb467cfa:f698458f"
level="raid5" version="1.2">
<disks>
<disk status="normal" dev_path="/dev/sda6" model="Virtual SATA
Hard Drive " serial="00000000000000000001" partition_version="8"
slot="4">
</disk>
<disk status="normal" dev_path="/dev/sdb6" model="Virtual SATA
Hard Drive " serial="01000000000000000001" partition_version="8"
slot="5">
</disk>
<disk status="normal" dev_path="/dev/sdc6" model="Virtual SATA
Hard Drive " serial="02000000000000000001" partition_version="8"
slot="1">
</disk>
<disk status="normal" dev_path="/dev/sdd6" model="Virtual SATA
Hard Drive " serial="03000000000000000001" partition_version="7"
slot="2">
</disk>
<disk status="normal" dev_path="/dev/sde6" model="Virtual SATA
Hard Drive " serial="04000000000000000001" partition_version="8"
slot="3">
</disk>
<disk status="normal" dev_path="/dev/sdf6" model="Virtual SATA
Hard Drive " serial="05000000000000000001" partition_version="7"
slot="0">
</disk>
</disks>
</raid>
</raids>
</lvm>
</device>
<reference>
<volumes>
<volume path="/volume1" dev_path="/dev/vg1000/lv">
</volume>
</volumes>
</reference>
</space>
</spaces>
// So i've scanned sd[abcdef]6 :
Cube> mdadm -E /dev/sda6
/dev/sda6:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : bf5e3c59:d407890c:bb467cfa:f698458f
Name : cube:3
Creation Time : Tue Jul 15 08:00:26 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 273472416 (130.40 GiB 140.02 GB)
Array Size : 1367361280 (652.01 GiB 700.09 GB)
Used Dev Size : 273472256 (130.40 GiB 140.02 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c7e8d7b9:26c766b7:71338730:ab9fbd18
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Fri Apr 29 00:23:11 2016
Checksum : 835fdf82 - correct
Events : 3383
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 4
Array State : AAAAAA ('A' == active, '.' == missing)
Cube> mdadm -E /dev/sdb6
/dev/sdb6:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : bf5e3c59:d407890c:bb467cfa:f698458f
Name : cube:3
Creation Time : Tue Jul 15 08:00:26 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 273472416 (130.40 GiB 140.02 GB)
Array Size : 1367361280 (652.01 GiB 700.09 GB)
Used Dev Size : 273472256 (130.40 GiB 140.02 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3ca84eae:4f12ab74:e534b2de:a22df4af
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Fri Apr 29 00:23:11 2016
Checksum : 7a7c798d - correct
Events : 3383
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing)
Cube> mdadm -E /dev/sdc6
/dev/sdc6:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : bf5e3c59:d407890c:bb467cfa:f698458f
Name : cube:3
Creation Time : Tue Jul 15 08:00:26 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 273472416 (130.40 GiB 140.02 GB)
Array Size : 1367361280 (652.01 GiB 700.09 GB)
Used Dev Size : 273472256 (130.40 GiB 140.02 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f50b2511:20431fd3:39385708:da820eb2
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Fri Apr 29 00:23:11 2016
Checksum : 678666a0 - correct
Events : 3383
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AAAAAA ('A' == active, '.' == missing)
Cube> mdadm -E /dev/sdd6
/dev/sdd6:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : bf5e3c59:d407890c:bb467cfa:f698458f
Name : cube:3
Creation Time : Tue Jul 15 08:00:26 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 273472416 (130.40 GiB 140.02 GB)
Array Size : 1367361280 (652.01 GiB 700.09 GB)
Used Dev Size : 273472256 (130.40 GiB 140.02 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 920dbf7c:fdea401f:99537d57:45d6b816
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Fri Apr 29 00:23:11 2016
Checksum : d3127ee1 - correct
Events : 3383
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 2
Array State : AAAAAA ('A' == active, '.' == missing)
Cube> mdadm -E /dev/sde6
/dev/sde6:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : bf5e3c59:d407890c:bb467cfa:f698458f
Name : cube:3
Creation Time : Tue Jul 15 08:00:26 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 273472416 (130.40 GiB 140.02 GB)
Array Size : 1367361280 (652.01 GiB 700.09 GB)
Used Dev Size : 273472256 (130.40 GiB 140.02 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 18c59328:dec7f04a:211a05a1:97d48710
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Fri Apr 29 00:23:11 2016
Checksum : ededd824 - correct
Events : 3383
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 3
Array State : AAAAAA ('A' == active, '.' == missing)
Cube> mdadm -E /dev/sdf6
/dev/sdf6:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : bf5e3c59:d407890c:bb467cfa:f698458f
Name : cube:3
Creation Time : Tue Jul 15 08:00:26 2014
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 273472416 (130.40 GiB 140.02 GB)
Array Size : 1367361280 (652.01 GiB 700.09 GB)
Used Dev Size : 273472256 (130.40 GiB 140.02 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1340d676:ebd4f963:c9f39dfb:40d7d9d4
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Fri Apr 29 00:23:11 2016
Checksum : 74243c7b - correct
Events : 3383
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing)
// All look ok (State clean, Array State AAAAAA and same events number 3383)
// So i've tried to assemble the array :
Cube> mdadm --assemble --verbose /dev/md3 /dev/sda6 /dev/sdb6
/dev/sdc6 /dev/sdd6 /dev/sde6 /dev/sdf6
mdadm: looking for devices for /dev/md3
mdadm: /dev/sda6 is identified as a member of /dev/md3, slot 4.
mdadm: /dev/sdb6 is identified as a member of /dev/md3, slot 5.
mdadm: /dev/sdc6 is identified as a member of /dev/md3, slot 1.
mdadm: /dev/sdd6 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/sde6 is identified as a member of /dev/md3, slot 3.
mdadm: /dev/sdf6 is identified as a member of /dev/md3, slot 0.
mdadm:/dev/md3 has an active reshape - checking if critical section
needs to be restored
mdadm: No backup metadata on device-5
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
Possibly you needed to specify the --backup-file
// As i don't have any backup-file i've tried with "--invalid-backup"
but this option doesn't exist on Synology so i stop the NAS and mount
all 6 disk into a fresh Ubuntu install (virtualised too) :
// First interesting thing, "md3" is now visible (but inactive) :
// Because of the system disk, the 6 disks are now sd[bcdefg] insted
of sd[abcdef]
tks@Tks:~$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md2 : active raid5 sdg5[5] sdf5[7] sdd5[8] sde5[6] sdc5[10] sdb5[9]
2418258240 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
md3 : inactive sdf6[4](S) sdg6[0](S) sde6[2](S) sdd6[5](S) sdc6[7](S) sdb6[6](S)
820417248 blocks super 1.2
md4 : active raid5 sdg7[0] sdf7[3] sde7[1] sdd7[4] sdc7[6] sdb7[5]
1758107520 blocks super 1.2 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
unused devices: <none>
tks@Tks:~$ sudo mdadm -D /dev/md3
/dev/md3:
Version : 1.2
Raid Level : raid0
Total Devices : 6
Persistence : Superblock is persistent
State : inactive
Delta Devices : 1, (-1->0)
New Level : raid5
New Layout : left-symmetric
New Chunksize : 64K
Name : cube:3
UUID : bf5e3c59:d407890c:bb467cfa:f698458f
Events : 3383
Number Major Minor RaidDevice
- 8 22 - /dev/sdb6
- 8 38 - /dev/sdc6
- 8 54 - /dev/sdd6
- 8 70 - /dev/sde6
- 8 86 - /dev/sdf6
- 8 102 - /dev/sdg6
// I've tried to assemble the disks :
tks@Tks:~$ sudo mdadm --assemble --verbose --invalid-backup --force
/dev/md3 /dev/sdb6 /dev/sdc6 /dev/sdd6 /dev/sde6 /dev/sdf6 /dev/sdg6
mdadm: looking for devices for /dev/md3
mdadm: /dev/sdb6 is identified as a member of /dev/md3, slot 4.
mdadm: /dev/sdc6 is identified as a member of /dev/md3, slot 5.
mdadm: /dev/sdd6 is identified as a member of /dev/md3, slot 1.
mdadm: /dev/sde6 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/sdf6 is identified as a member of /dev/md3, slot 3.
mdadm: /dev/sdg6 is identified as a member of /dev/md3, slot 0.
mdadm: :/dev/md3 has an active reshape - checking if critical section
needs to be restored
mdadm: No backup metadata on device-5
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/sdd6 to /dev/md3 as 1
mdadm: added /dev/sde6 to /dev/md3 as 2
mdadm: added /dev/sdf6 to /dev/md3 as 3
mdadm: added /dev/sdb6 to /dev/md3 as 4
mdadm: added /dev/sdc6 to /dev/md3 as 5
mdadm: added /dev/sdg6 to /dev/md3 as 0
mdadm: failed to RUN_ARRAY /dev/md3: Invalid argument
// and after lots of search, i found this error :
dmesg
md/raid:md3: reshape_position too early for auto-recovery - aborting.
md: pers->run() failed ...
// and i found some message of Phil Genera and Neil Brown about this
problem... but no solution for my case :(
I've tried to remove the last added disk to see but nothing more.
So, now, i need help because i'm a bit lost :-s i've seen that maybe
with "--create --assume-clean" and good options, it could works but
i'm not very confident... and i don't want to overwrite something.
I use mdadm v3.4 on Ubuntu 16.04x64.
The NAS :
Synology DSM 5.1-5022
Cube> uname -a
Linux Cube 3.2.40 #1 SMP Tue Mar 3 23:34:55 CST 2015 x86_64
GNU/Linux synology_bromolow_3615xs
mdadm 3.1.4
I hope that all information above will be useful for you to help me.
Thank you in advance for reading and if you have any ideas for restoring datas.
Laurent
^ permalink raw reply
* Re: Help needed: array inactive after grow attempt
From: Phil Turmel @ 2016-05-03 18:47 UTC (permalink / raw)
To: Andread Mayrhoff, linux-raid
In-Reply-To: <e7110adb04c7319afd4ae4ee856e4bec@spoilbase.eu>
On 04/28/2016 04:02 PM, Andread Mayrhoff wrote:
> Some more info:
I noticed this hasn't been answered in several days. Not sure if you
are still in trouble.
Anyways, since you are using metadata version 1.0, mdadm cannot use
starting position manipulations to avoid the need for a backup file.
Your mdadm -E reports all show the reshape position == 0, meaning it
didn't actually reshape anything.
You should try assembling with a --backup-file option and possibly also
the --invalid-backup option to see what happens. Include --verbose and
show the output here if it still doesn't work.
Phil
^ permalink raw reply
* Re: [PATCH 00/42] v7: separate operations from flags in the bio/request structs
From: Jeff Moyer @ 2016-05-03 20:44 UTC (permalink / raw)
To: mchristi-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA,
linux-block-u79uwXL29TY76Z2rM5mHXA, xfs-VZNHf3L845pBDgjK7y7TUQ,
ocfs2-devel-N0ozoZBvEnrZJqsBc5GL+g,
linux-scsi-u79uwXL29TY76Z2rM5mHXA,
konrad.wilk-QHcLZuEGTsvQT0dZR+AlfA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
philipp.reisner-63ez5xqkn6DQT0dZR+AlfA,
linux-f2fs-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA,
target-devel-u79uwXL29TY76Z2rM5mHXA,
linux-mtd-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r,
osd-dev-yNzVSZO3znNg9hUCZPvPmw,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA,
lars.ellenberg-63ez5xqkn6DQT0dZR+AlfA,
linux-ext4-u79uwXL29TY76Z2rM5mHXA,
linux-btrfs-u79uwXL29TY76Z2rM5mHXA,
drbd-dev-cunTk1MwBs8qoQakbn7OcQ
In-Reply-To: <1460747777-8479-1-git-send-email-mchristi-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
mchristi-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org writes:
> The following patches begin to cleanup the request->cmd_flags and
> bio->bi_rw mess. We currently use cmd_flags to specify the operation,
> attributes and state of the request. For bi_rw we use it for similar
> info and also the priority but then also have another bi_flags field
> for state. At some point, we abused them so much we just made cmd_flags
> 64 bits, so we could add more.
>
> The following patches seperate the operation (read, write discard,
> flush, etc) from cmd_flags/bi_rw.
>
> This patchset was made against linux-next from today April 15
> (git tag next-20160415).
>
> I put a git tree here:
> https://github.com/mikechristie/linux-kernel.git
> The patches are in the op branch.
Hi, Mike,
That git tree doesn't seem to exist. I did manage to apply your patch
set on top of next-20160415, though.
So... what testing did you do? ;-) I ran into the following problems:
- git clone fails
- yum segfaults
- many blktrace/blkparse issues, including incorrect cpu recorded in
traces, null task names, and blkparse outputting nothing for a trace
file several gigabytes in size.
After that, I decided to back out your patches and test the base
linux-next kernel. That kernel has none of those issues.
So, either I'm missing some dependencies, or I think we've got some
issues to iron out before this thing goes in. Before I dig any further,
am I missing something?
Cheers,
Jeff
^ permalink raw reply
* [PATCH 1/3] md: set MD_CHANGE_PENDING in a atomic region
From: Guoqing Jiang @ 2016-05-04 2:22 UTC (permalink / raw)
To: shli
Cc: neilb, linux-raid, Guoqing Jiang, Martin Kepplinger,
Andrew Morton, Denys Vlasenko, Sasha Levin, linux-kernel
Some code waits for a metadata update by:
1. flagging that it is needed (MD_CHANGE_DEVS or MD_CHANGE_CLEAN)
2. setting MD_CHANGE_PENDING and waking the management thread
3. waiting for MD_CHANGE_PENDING to be cleared
If the first two are done without locking, the code in md_update_sb()
which checks if it needs to repeat might test if an update is needed
before step 1, then clear MD_CHANGE_PENDING after step 2, resulting
in the wait returning early.
So make sure all places that set MD_CHANGE_PENDING are atomicial, and
bit_clear_unless (suggested by Neil) is introduced for the purpose.
Cc: Martin Kepplinger <martink@posteo.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: <linux-kernel@vger.kernel.org>
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md.c | 27 ++++++++++++++-------------
drivers/md/raid1.c | 4 ++--
drivers/md/raid10.c | 8 ++++----
drivers/md/raid5-cache.c | 4 ++--
drivers/md/raid5.c | 4 ++--
include/linux/bitops.h | 16 ++++++++++++++++
6 files changed, 40 insertions(+), 23 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 06f6e81..892d639 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2295,12 +2295,16 @@ repeat:
if (mddev_is_clustered(mddev)) {
if (test_and_clear_bit(MD_CHANGE_DEVS, &mddev->flags))
force_change = 1;
+ if (test_and_clear_bit(MD_CHANGE_CLEAN, &mddev->flags))
+ nospares = 1;
ret = md_cluster_ops->metadata_update_start(mddev);
/* Has someone else has updated the sb */
if (!does_sb_need_changing(mddev)) {
if (ret == 0)
md_cluster_ops->metadata_update_cancel(mddev);
- clear_bit(MD_CHANGE_PENDING, &mddev->flags);
+ bit_clear_unless(&mddev->flags, BIT(MD_CHANGE_PENDING),
+ BIT(MD_CHANGE_DEVS) |
+ BIT(MD_CHANGE_CLEAN));
return;
}
}
@@ -2434,15 +2438,11 @@ repeat:
if (mddev_is_clustered(mddev) && ret == 0)
md_cluster_ops->metadata_update_finish(mddev);
- spin_lock(&mddev->lock);
if (mddev->in_sync != sync_req ||
- test_bit(MD_CHANGE_DEVS, &mddev->flags)) {
+ !bit_clear_unless(&mddev->flags, BIT(MD_CHANGE_PENDING),
+ BIT(MD_CHANGE_DEVS) | BIT(MD_CHANGE_CLEAN)))
/* have to write it out again */
- spin_unlock(&mddev->lock);
goto repeat;
- }
- clear_bit(MD_CHANGE_PENDING, &mddev->flags);
- spin_unlock(&mddev->lock);
wake_up(&mddev->sb_wait);
if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery))
sysfs_notify(&mddev->kobj, NULL, "sync_completed");
@@ -8147,18 +8147,18 @@ void md_do_sync(struct md_thread *thread)
}
}
skip:
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
-
if (mddev_is_clustered(mddev) &&
ret == 0) {
/* set CHANGE_PENDING here since maybe another
* update is needed, so other nodes are informed */
- set_bit(MD_CHANGE_PENDING, &mddev->flags);
+ set_mask_bits(&mddev->flags, 0,
+ BIT(MD_CHANGE_PENDING) | BIT(MD_CHANGE_DEVS));
md_wakeup_thread(mddev->thread);
wait_event(mddev->sb_wait,
!test_bit(MD_CHANGE_PENDING, &mddev->flags));
md_cluster_ops->resync_finish(mddev);
- }
+ } else
+ set_bit(MD_CHANGE_DEVS, &mddev->flags);
spin_lock(&mddev->lock);
if (!test_bit(MD_RECOVERY_INTR, &mddev->recovery)) {
@@ -8550,6 +8550,7 @@ EXPORT_SYMBOL(md_finish_reshape);
int rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
int is_new)
{
+ struct mddev *mddev = rdev->mddev;
int rv;
if (is_new)
s += rdev->new_data_offset;
@@ -8559,8 +8560,8 @@ int rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
if (rv == 0) {
/* Make sure they get written out promptly */
sysfs_notify_dirent_safe(rdev->sysfs_state);
- set_bit(MD_CHANGE_CLEAN, &rdev->mddev->flags);
- set_bit(MD_CHANGE_PENDING, &rdev->mddev->flags);
+ set_mask_bits(&mddev->flags, 0,
+ BIT(MD_CHANGE_CLEAN) | BIT(MD_CHANGE_PENDING));
md_wakeup_thread(rdev->mddev->thread);
return 1;
} else
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index a7f2b9c..c7c8cde 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1474,8 +1474,8 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)
* if recovery is running, make sure it aborts.
*/
set_bit(MD_RECOVERY_INTR, &mddev->recovery);
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
- set_bit(MD_CHANGE_PENDING, &mddev->flags);
+ set_mask_bits(&mddev->flags, 0,
+ BIT(MD_CHANGE_DEVS) | BIT(MD_CHANGE_PENDING));
printk(KERN_ALERT
"md/raid1:%s: Disk failure on %s, disabling device.\n"
"md/raid1:%s: Operation continuing on %d devices.\n",
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index e3fd725..f21c4bb 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1102,8 +1102,8 @@ static void __make_request(struct mddev *mddev, struct bio *bio)
bio->bi_iter.bi_sector < conf->reshape_progress))) {
/* Need to update reshape_position in metadata */
mddev->reshape_position = conf->reshape_progress;
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
- set_bit(MD_CHANGE_PENDING, &mddev->flags);
+ set_mask_bits(&mddev->flags, 0,
+ BIT(MD_CHANGE_DEVS) | BIT(MD_CHANGE_PENDING));
md_wakeup_thread(mddev->thread);
wait_event(mddev->sb_wait,
!test_bit(MD_CHANGE_PENDING, &mddev->flags));
@@ -1591,8 +1591,8 @@ static void raid10_error(struct mddev *mddev, struct md_rdev *rdev)
set_bit(MD_RECOVERY_INTR, &mddev->recovery);
set_bit(Blocked, &rdev->flags);
set_bit(Faulty, &rdev->flags);
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
- set_bit(MD_CHANGE_PENDING, &mddev->flags);
+ set_mask_bits(&mddev->flags, 0,
+ BIT(MD_CHANGE_DEVS) | BIT(MD_CHANGE_PENDING));
spin_unlock_irqrestore(&conf->device_lock, flags);
printk(KERN_ALERT
"md/raid10:%s: Disk failure on %s, disabling device.\n"
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 9531f5f..ac51bc5 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -712,8 +712,8 @@ static void r5l_write_super_and_discard_space(struct r5l_log *log,
* in_teardown check workaround this issue.
*/
if (!log->in_teardown) {
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
- set_bit(MD_CHANGE_PENDING, &mddev->flags);
+ set_mask_bits(&mddev->flags, 0,
+ BIT(MD_CHANGE_DEVS) | BIT(MD_CHANGE_PENDING));
md_wakeup_thread(mddev->thread);
wait_event(mddev->sb_wait,
!test_bit(MD_CHANGE_PENDING, &mddev->flags) ||
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index e48c262..cd110ce 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -2514,8 +2514,8 @@ static void raid5_error(struct mddev *mddev, struct md_rdev *rdev)
set_bit(Blocked, &rdev->flags);
set_bit(Faulty, &rdev->flags);
- set_bit(MD_CHANGE_DEVS, &mddev->flags);
- set_bit(MD_CHANGE_PENDING, &mddev->flags);
+ set_mask_bits(&mddev->flags, 0,
+ BIT(MD_CHANGE_DEVS) | BIT(MD_CHANGE_PENDING));
printk(KERN_ALERT
"md/raid:%s: Disk failure on %s, disabling device.\n"
"md/raid:%s: Operation continuing on %d devices.\n",
diff --git a/include/linux/bitops.h b/include/linux/bitops.h
index defeaac..299e76b 100644
--- a/include/linux/bitops.h
+++ b/include/linux/bitops.h
@@ -227,6 +227,22 @@ static inline unsigned long __ffs64(u64 word)
})
#endif
+#ifndef bit_clear_unless
+#define bit_clear_unless(ptr, _clear, _test) \
+({ \
+ const typeof(*ptr) clear = (_clear), test = (_test); \
+ typeof(*ptr) old, new; \
+ \
+ do { \
+ old = ACCESS_ONCE(*ptr); \
+ new = old & ~clear; \
+ } while (!(old & test) && \
+ cmpxchg(ptr, old, new) != old); \
+ \
+ !(old & test); \
+})
+#endif
+
#ifndef find_last_bit
/**
* find_last_bit - find the last set bit in a memory region
--
2.6.6
^ permalink raw reply related
* [PATCH 2/3] md-cluster: gather resync infos and enable recv_thread after bitmap is ready
From: Guoqing Jiang @ 2016-05-04 2:22 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462328535-22566-1-git-send-email-gqjiang@suse.com>
The in-memory bitmap is not ready when node joins cluster,
so it doesn't make sense to make gather_all_resync_info()
called so earlier, we need to call it after the node's
bitmap is setup. Also, recv_thread could be wake up after
node joins cluster, but it could cause problem if node
receives RESYNCING message without persionality since
mddev->pers->quiesce is called in process_suspend_info.
This commit introduces a new cluster interface load_bitmaps
to fix above problems, load_bitmaps is called in bitmap_load
where bitmap and persionality are ready, and load_bitmaps
does the following tasks:
1. call gather_all_resync_info to load all the node's
bitmap info.
2. set MD_CLUSTER_ALREADY_IN_CLUSTER bit to recv_thread
could be wake up, and wake up recv_thread if there is
pending recv event.
Then ack_bast only wakes up recv_thread after IN_CLUSTER
bit is ready otherwise MD_CLUSTER_PENDING_RESYNC_EVENT is
set.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 3 +++
drivers/md/md-cluster.c | 27 ++++++++++++++++++++++-----
drivers/md/md-cluster.h | 1 +
3 files changed, 26 insertions(+), 5 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index ad5a858..d8129ec 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -1848,6 +1848,9 @@ int bitmap_load(struct mddev *mddev)
if (!bitmap)
goto out;
+ if (mddev_is_clustered(mddev))
+ md_cluster_ops->load_bitmaps(mddev, mddev->bitmap_info.nodes);
+
/* Clear out old bitmap info first: Either there is none, or we
* are resuming after someone else has possibly changed things,
* so we should forget old cached info.
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index a55b5f4..bee4085 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -61,6 +61,10 @@ struct resync_info {
* the lock.
*/
#define MD_CLUSTER_SEND_LOCKED_ALREADY 5
+/* We should receive message after node joined cluster and
+ * set up all the related infos such as bitmap and personality */
+#define MD_CLUSTER_ALREADY_IN_CLUSTER 6
+#define MD_CLUSTER_PENDING_RESYNC_EVENT 7
struct md_cluster_info {
@@ -376,8 +380,11 @@ static void ack_bast(void *arg, int mode)
struct dlm_lock_resource *res = arg;
struct md_cluster_info *cinfo = res->mddev->cluster_info;
- if (mode == DLM_LOCK_EX)
+ if (mode == DLM_LOCK_EX &&
+ test_bit(MD_CLUSTER_ALREADY_IN_CLUSTER, &cinfo->state))
md_wakeup_thread(cinfo->recv_thread);
+ else
+ set_bit(MD_CLUSTER_PENDING_RESYNC_EVENT, &cinfo->state);
}
static void __remove_suspend_info(struct md_cluster_info *cinfo, int slot)
@@ -846,10 +853,6 @@ static int join(struct mddev *mddev, int nodes)
if (!cinfo->resync_lockres)
goto err;
- ret = gather_all_resync_info(mddev, nodes);
- if (ret)
- goto err;
-
return 0;
err:
md_unregister_thread(&cinfo->recovery_thread);
@@ -867,6 +870,19 @@ err:
return ret;
}
+static void load_bitmaps(struct mddev *mddev, int total_slots)
+{
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+
+ /* load all the node's bitmap info for resync */
+ if (gather_all_resync_info(mddev, total_slots))
+ pr_err("md-cluster: failed to gather all resyn infos\n");
+ set_bit(MD_CLUSTER_ALREADY_IN_CLUSTER, &cinfo->state);
+ /* wake up recv thread in case something need to be handled */
+ if (test_and_clear_bit(MD_CLUSTER_PENDING_RESYNC_EVENT, &cinfo->state))
+ md_wakeup_thread(cinfo->recv_thread);
+}
+
static void resync_bitmap(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
@@ -1208,6 +1224,7 @@ static struct md_cluster_operations cluster_ops = {
.add_new_disk_cancel = add_new_disk_cancel,
.new_disk_ack = new_disk_ack,
.remove_disk = remove_disk,
+ .load_bitmaps = load_bitmaps,
.gather_bitmaps = gather_bitmaps,
.lock_all_bitmaps = lock_all_bitmaps,
.unlock_all_bitmaps = unlock_all_bitmaps,
diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
index 45ce6c9..e765499 100644
--- a/drivers/md/md-cluster.h
+++ b/drivers/md/md-cluster.h
@@ -23,6 +23,7 @@ struct md_cluster_operations {
void (*add_new_disk_cancel)(struct mddev *mddev);
int (*new_disk_ack)(struct mddev *mddev, bool ack);
int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
+ void (*load_bitmaps)(struct mddev *mddev, int total_slots);
int (*gather_bitmaps)(struct md_rdev *rdev);
int (*lock_all_bitmaps)(struct mddev *mddev);
void (*unlock_all_bitmaps)(struct mddev *mddev);
--
2.6.6
^ permalink raw reply related
* [PATCH 3/3] md-cluster: check the return value of process_recvd_msg
From: Guoqing Jiang @ 2016-05-04 2:22 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462328535-22566-1-git-send-email-gqjiang@suse.com>
We don't need to run the full path of recv_daemon
if process_recvd_msg doesn't return 0.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index bee4085..301207d 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -519,11 +519,13 @@ static void process_readd_disk(struct mddev *mddev, struct cluster_msg *msg)
__func__, __LINE__, le32_to_cpu(msg->raid_slot));
}
-static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
+static int process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
{
+ int ret = 0;
+
if (WARN(mddev->cluster_info->slot_number - 1 == le32_to_cpu(msg->slot),
"node %d received it's own msg\n", le32_to_cpu(msg->slot)))
- return;
+ return -1;
switch (le32_to_cpu(msg->type)) {
case METADATA_UPDATED:
process_metadata_update(mddev, msg);
@@ -546,9 +548,11 @@ static void process_recvd_msg(struct mddev *mddev, struct cluster_msg *msg)
__recover_slot(mddev, le32_to_cpu(msg->slot));
break;
default:
+ ret = -1;
pr_warn("%s:%d Received unknown message from %d\n",
__func__, __LINE__, msg->slot);
}
+ return ret;
}
/*
@@ -572,7 +576,9 @@ static void recv_daemon(struct md_thread *thread)
/* read lvb and wake up thread to process this message_lockres */
memcpy(&msg, message_lockres->lksb.sb_lvbptr, sizeof(struct cluster_msg));
- process_recvd_msg(thread->mddev, &msg);
+ ret = process_recvd_msg(thread->mddev, &msg);
+ if (ret)
+ goto out;
/*release CR on ack_lockres*/
ret = dlm_unlock_sync(ack_lockres);
@@ -586,6 +592,7 @@ static void recv_daemon(struct md_thread *thread)
ret = dlm_lock_sync(ack_lockres, DLM_LOCK_CR);
if (unlikely(ret != 0))
pr_info("lock CR on ack failed return %d\n", ret);
+out:
/*release CR on message_lockres*/
ret = dlm_unlock_sync(message_lockres);
if (unlikely(ret != 0))
--
2.6.6
^ permalink raw reply related
* Re: Write to the degraded raid5 will trigger the call trace dump when skip_copy is enabled
From: Joey Liao @ 2016-05-04 3:21 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid
In-Reply-To: <CAHvN=imVJgEtMmO6XtLCdQKCPiuVT2sCqpq_0qoMR-3aJ+ZzPw@mail.gmail.com>
Any suggestion for my question in the last e-mail?
Should the WARN_ON() be exist or not?
It is also related to skip_copy feature and I think it seems make
sense that maybe R5_UPTODATE should not be set in this case.
But I'm not really sure if my thinking is correct or not.
However, if it does not have to exist, would you mind delete it as
well or just modify the code?
2016-05-02 22:45 GMT+08:00 Joey Liao <joeyliao@qnap.com>:
> Hi Shaohua,
>
> Many thanks for your reply.
>
> How about the following WARN_ON() in handle_stripe_clean_event()???
>
> if (test_and_clear_bit(R5_SkipCopy, &dev->flags)) {
> WARN_ON(test_bit(R5_UPTODATE, &dev->flags));
> }
>
> Sometimes it will be triggered as well but I can't find the exactly
> way to reproduce it.
> I would like to know if it is suitable to remove it as well or not.
>
>
> 2016-04-30 5:17 GMT+08:00 Shaohua Li <shli@kernel.org>:
>> On Fri, Apr 29, 2016 at 06:54:10PM +0800, Joey Liao wrote:
>>> Hi all,
>>>
>>> For improving the I/O performance, we try to enable the raid 5
>>> skip_copy feature by default. It indeed has some benefit, however, it
>>> will always (not a random issue) dump the following call trace
>>> repeatedly when we try to write the raid 5 block device. It is related
>>> to the following codes in handle_stripe_clean_event() in raid5.c.
>>>
>>> WARN_ON(test_bit(R5_SkipCopy, &dev->flags));
>>> WARN_ON(dev->page != dev->orig_page);
>>>
>>> Is it a known bug for skip_copy feature?
>>> Does it do harm to the data integrity?
>>> If we would like to prevent this call trace, for your suggestion how
>>> should we do to modify the source code?
>>
>> Looks the two WARN_ON should be deleted. if the dev has R5_LOCKED, it's legit
>> the dev has SkipCopy set and page != orig_page. I'll delete the code. This will
>> not harm to data integrity.
>>
>> Thanks,
>> Shaohua
^ permalink raw reply
* Re: [PATCH 2/3] md-cluster: gather resync infos and enable recv_thread after bitmap is ready
From: Guoqing Jiang @ 2016-05-04 5:52 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid
In-Reply-To: <1462328535-22566-2-git-send-email-gqjiang@suse.com>
On 05/03/2016 10:22 PM, Guoqing Jiang wrote:
> The in-memory bitmap is not ready when node joins cluster,
> so it doesn't make sense to make gather_all_resync_info()
> called so earlier, we need to call it after the node's
> bitmap is setup. Also, recv_thread could be wake up after
> node joins cluster, but it could cause problem if node
> receives RESYNCING message without persionality since
> mddev->pers->quiesce is called in process_suspend_info.
>
> This commit introduces a new cluster interface load_bitmaps
> to fix above problems, load_bitmaps is called in bitmap_load
> where bitmap and persionality are ready, and load_bitmaps
> does the following tasks:
>
> 1. call gather_all_resync_info to load all the node's
> bitmap info.
> 2. set MD_CLUSTER_ALREADY_IN_CLUSTER bit to recv_thread
> could be wake up, and wake up recv_thread if there is
> pending recv event.
>
> Then ack_bast only wakes up recv_thread after IN_CLUSTER
> bit is ready otherwise MD_CLUSTER_PENDING_RESYNC_EVENT is
> set.
>
> Reviewed-by: NeilBrown <neilb@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> drivers/md/bitmap.c | 3 +++
> drivers/md/md-cluster.c | 27 ++++++++++++++++++++++-----
> drivers/md/md-cluster.h | 1 +
> 3 files changed, 26 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index ad5a858..d8129ec 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -1848,6 +1848,9 @@ int bitmap_load(struct mddev *mddev)
> if (!bitmap)
> goto out;
>
> + if (mddev_is_clustered(mddev))
> + md_cluster_ops->load_bitmaps(mddev, mddev->bitmap_info.nodes);
> +
> /* Clear out old bitmap info first: Either there is none, or we
> * are resuming after someone else has possibly changed things,
> * so we should forget old cached info.
> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> index a55b5f4..bee4085 100644
> --- a/drivers/md/md-cluster.c
> +++ b/drivers/md/md-cluster.c
> @@ -61,6 +61,10 @@ struct resync_info {
> * the lock.
> */
> #define MD_CLUSTER_SEND_LOCKED_ALREADY 5
> +/* We should receive message after node joined cluster and
> + * set up all the related infos such as bitmap and personality */
> +#define MD_CLUSTER_ALREADY_IN_CLUSTER 6
> +#define MD_CLUSTER_PENDING_RESYNC_EVENT 7
>
>
> struct md_cluster_info {
> @@ -376,8 +380,11 @@ static void ack_bast(void *arg, int mode)
> struct dlm_lock_resource *res = arg;
> struct md_cluster_info *cinfo = res->mddev->cluster_info;
>
> - if (mode == DLM_LOCK_EX)
> + if (mode == DLM_LOCK_EX &&
> + test_bit(MD_CLUSTER_ALREADY_IN_CLUSTER, &cinfo->state))
> md_wakeup_thread(cinfo->recv_thread);
> + else
> + set_bit(MD_CLUSTER_PENDING_RESYNC_EVENT, &cinfo->state);
> }
Hmm, the above should be:
- if (mode == DLM_LOCK_EX)
- md_wakeup_thread(cinfo->recv_thread);
+ if (mode == DLM_LOCK_EX) {
+ if (test_bit(MD_CLUSTER_ALREADY_IN_CLUSTER, &cinfo->state))
+ md_wakeup_thread(cinfo->recv_thread);
+ else
+ set_bit(MD_CLUSTER_PENDING_RESYNC_EVENT,
&cinfo->state);
+ }
I will update the patch.
Thanks,
Guoqing
>
> static void __remove_suspend_info(struct md_cluster_info *cinfo, int slot)
> @@ -846,10 +853,6 @@ static int join(struct mddev *mddev, int nodes)
> if (!cinfo->resync_lockres)
> goto err;
>
> - ret = gather_all_resync_info(mddev, nodes);
> - if (ret)
> - goto err;
> -
> return 0;
> err:
> md_unregister_thread(&cinfo->recovery_thread);
> @@ -867,6 +870,19 @@ err:
> return ret;
> }
>
> +static void load_bitmaps(struct mddev *mddev, int total_slots)
> +{
> + struct md_cluster_info *cinfo = mddev->cluster_info;
> +
> + /* load all the node's bitmap info for resync */
> + if (gather_all_resync_info(mddev, total_slots))
> + pr_err("md-cluster: failed to gather all resyn infos\n");
> + set_bit(MD_CLUSTER_ALREADY_IN_CLUSTER, &cinfo->state);
> + /* wake up recv thread in case something need to be handled */
> + if (test_and_clear_bit(MD_CLUSTER_PENDING_RESYNC_EVENT, &cinfo->state))
> + md_wakeup_thread(cinfo->recv_thread);
> +}
> +
> static void resync_bitmap(struct mddev *mddev)
> {
> struct md_cluster_info *cinfo = mddev->cluster_info;
> @@ -1208,6 +1224,7 @@ static struct md_cluster_operations cluster_ops = {
> .add_new_disk_cancel = add_new_disk_cancel,
> .new_disk_ack = new_disk_ack,
> .remove_disk = remove_disk,
> + .load_bitmaps = load_bitmaps,
> .gather_bitmaps = gather_bitmaps,
> .lock_all_bitmaps = lock_all_bitmaps,
> .unlock_all_bitmaps = unlock_all_bitmaps,
> diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
> index 45ce6c9..e765499 100644
> --- a/drivers/md/md-cluster.h
> +++ b/drivers/md/md-cluster.h
> @@ -23,6 +23,7 @@ struct md_cluster_operations {
> void (*add_new_disk_cancel)(struct mddev *mddev);
> int (*new_disk_ack)(struct mddev *mddev, bool ack);
> int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
> + void (*load_bitmaps)(struct mddev *mddev, int total_slots);
> int (*gather_bitmaps)(struct md_rdev *rdev);
> int (*lock_all_bitmaps)(struct mddev *mddev);
> void (*unlock_all_bitmaps)(struct mddev *mddev);
^ permalink raw reply
* [Update PATCH] md-cluster: gather resync infos and enable recv_thread after bitmap is ready
From: Guoqing Jiang @ 2016-05-04 6:17 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462328535-22566-2-git-send-email-gqjiang@suse.com>
The in-memory bitmap is not ready when node joins cluster,
so it doesn't make sense to make gather_all_resync_info()
called so earlier, we need to call it after the node's
bitmap is setup. Also, recv_thread could be wake up after
node joins cluster, but it could cause problem if node
receives RESYNCING message without persionality since
mddev->pers->quiesce is called in process_suspend_info.
This commit introduces a new cluster interface load_bitmaps
to fix above problems, load_bitmaps is called in bitmap_load
where bitmap and persionality are ready, and load_bitmaps
does the following tasks:
1. call gather_all_resync_info to load all the node's
bitmap info.
2. set MD_CLUSTER_ALREADY_IN_CLUSTER bit to recv_thread
could be wake up, and wake up recv_thread if there is
pending recv event.
Then ack_bast only wakes up recv_thread after IN_CLUSTER
bit is ready otherwise MD_CLUSTER_PENDING_RESYNC_EVENT is
set.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Changes:
1. s/MD_CLUSTER_PENDING_RESYNC_EVENT/MD_CLUSTER_PENDING_RECV_EVENT
2. adjust the logic in ack_bast
drivers/md/bitmap.c | 3 +++
drivers/md/md-cluster.c | 30 ++++++++++++++++++++++++------
drivers/md/md-cluster.h | 1 +
3 files changed, 28 insertions(+), 6 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index ad5a858..d8129ec 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -1848,6 +1848,9 @@ int bitmap_load(struct mddev *mddev)
if (!bitmap)
goto out;
+ if (mddev_is_clustered(mddev))
+ md_cluster_ops->load_bitmaps(mddev, mddev->bitmap_info.nodes);
+
/* Clear out old bitmap info first: Either there is none, or we
* are resuming after someone else has possibly changed things,
* so we should forget old cached info.
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index a55b5f4..bef6a47 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -61,6 +61,10 @@ struct resync_info {
* the lock.
*/
#define MD_CLUSTER_SEND_LOCKED_ALREADY 5
+/* We should receive message after node joined cluster and
+ * set up all the related infos such as bitmap and personality */
+#define MD_CLUSTER_ALREADY_IN_CLUSTER 6
+#define MD_CLUSTER_PENDING_RECV_EVENT 7
struct md_cluster_info {
@@ -376,8 +380,12 @@ static void ack_bast(void *arg, int mode)
struct dlm_lock_resource *res = arg;
struct md_cluster_info *cinfo = res->mddev->cluster_info;
- if (mode == DLM_LOCK_EX)
- md_wakeup_thread(cinfo->recv_thread);
+ if (mode == DLM_LOCK_EX) {
+ if (test_bit(MD_CLUSTER_ALREADY_IN_CLUSTER, &cinfo->state))
+ md_wakeup_thread(cinfo->recv_thread);
+ else
+ set_bit(MD_CLUSTER_PENDING_RECV_EVENT, &cinfo->state);
+ }
}
static void __remove_suspend_info(struct md_cluster_info *cinfo, int slot)
@@ -846,10 +854,6 @@ static int join(struct mddev *mddev, int nodes)
if (!cinfo->resync_lockres)
goto err;
- ret = gather_all_resync_info(mddev, nodes);
- if (ret)
- goto err;
-
return 0;
err:
md_unregister_thread(&cinfo->recovery_thread);
@@ -867,6 +871,19 @@ err:
return ret;
}
+static void load_bitmaps(struct mddev *mddev, int total_slots)
+{
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+
+ /* load all the node's bitmap info for resync */
+ if (gather_all_resync_info(mddev, total_slots))
+ pr_err("md-cluster: failed to gather all resyn infos\n");
+ set_bit(MD_CLUSTER_ALREADY_IN_CLUSTER, &cinfo->state);
+ /* wake up recv thread in case something need to be handled */
+ if (test_and_clear_bit(MD_CLUSTER_PENDING_RECV_EVENT, &cinfo->state))
+ md_wakeup_thread(cinfo->recv_thread);
+}
+
static void resync_bitmap(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
@@ -1208,6 +1225,7 @@ static struct md_cluster_operations cluster_ops = {
.add_new_disk_cancel = add_new_disk_cancel,
.new_disk_ack = new_disk_ack,
.remove_disk = remove_disk,
+ .load_bitmaps = load_bitmaps,
.gather_bitmaps = gather_bitmaps,
.lock_all_bitmaps = lock_all_bitmaps,
.unlock_all_bitmaps = unlock_all_bitmaps,
diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
index 45ce6c9..e765499 100644
--- a/drivers/md/md-cluster.h
+++ b/drivers/md/md-cluster.h
@@ -23,6 +23,7 @@ struct md_cluster_operations {
void (*add_new_disk_cancel)(struct mddev *mddev);
int (*new_disk_ack)(struct mddev *mddev, bool ack);
int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
+ void (*load_bitmaps)(struct mddev *mddev, int total_slots);
int (*gather_bitmaps)(struct md_rdev *rdev);
int (*lock_all_bitmaps)(struct mddev *mddev);
void (*unlock_all_bitmaps)(struct mddev *mddev);
--
2.6.6
^ permalink raw reply related
* [PATCH 0/2] Check node nums for cluster raid
From: Guoqing Jiang @ 2016-05-04 8:33 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid, Guoqing Jiang
For cluster raid, we do need at least two nodes for it,
the two patches add the checks before create and change
bitmap.
Thanks,
Guoqing
Guoqing Jiang (2):
Create: check the node nums when create clustered raid
super1: don't update node nums if it is not more than 1
Create.c | 7 ++++++-
super1.c | 5 +++++
2 files changed, 11 insertions(+), 1 deletion(-)
--
2.6.2
^ permalink raw reply
* [PATCH 1/2] Create: check the node nums when create clustered raid
From: Guoqing Jiang @ 2016-05-04 8:33 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1462350824-9872-1-git-send-email-gqjiang@suse.com>
It doesn't make sense to create a clustered raid
with only 1 node.
Reported-by: Zhilong Liu <zlliu@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Create.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/Create.c b/Create.c
index 1e4a6ee..717086b 100644
--- a/Create.c
+++ b/Create.c
@@ -114,8 +114,13 @@ int Create(struct supertype *st, char *mddev,
unsigned long long newsize;
int major_num = BITMAP_MAJOR_HI;
- if (s->bitmap_file && strcmp(s->bitmap_file, "clustered") == 0)
+ if (s->bitmap_file && strcmp(s->bitmap_file, "clustered") == 0) {
major_num = BITMAP_MAJOR_CLUSTERED;
+ if (c->nodes <= 1) {
+ pr_err("At least 2 nodes are needed for cluster-md\n");
+ return 1;
+ }
+ }
memset(&info, 0, sizeof(info));
if (s->level == UnSet && st && st->ss->default_geometry)
--
2.6.2
^ permalink raw reply related
* [PATCH 2/2] super1: don't update node nums if it is not more than 1
From: Guoqing Jiang @ 2016-05-04 8:33 UTC (permalink / raw)
To: Jes.Sorensen; +Cc: linux-raid, Guoqing Jiang
In-Reply-To: <1462350824-9872-1-git-send-email-gqjiang@suse.com>
We at least need two nodes for cluster raid so make the
check before update node nums.
Reported-by: Zhilong Liu <zlliu@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
super1.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/super1.c b/super1.c
index 8d5543f..972b470 100644
--- a/super1.c
+++ b/super1.c
@@ -2394,6 +2394,11 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
return -EINVAL;
}
+ if (bms->version == BITMAP_MAJOR_CLUSTERED && st->nodes <= 1) {
+ pr_err("Warning: cluster-md at least needs two nodes\n");
+ return -EINVAL;
+ }
+
/* Each node has an independent bitmap, it is necessary to calculate the
* space is enough or not, first get how many bytes for the total bitmap */
bm_space_per_node = calc_bitmap_size(bms, 4096);
--
2.6.2
^ permalink raw reply related
* Re: [PATCH 0/2] Check node nums for cluster raid
From: Jes Sorensen @ 2016-05-04 15:12 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-raid
In-Reply-To: <1462350824-9872-1-git-send-email-gqjiang@suse.com>
Guoqing Jiang <gqjiang@suse.com> writes:
> For cluster raid, we do need at least two nodes for it,
> the two patches add the checks before create and change
> bitmap.
>
> Thanks,
> Guoqing
>
> Guoqing Jiang (2):
> Create: check the node nums when create clustered raid
> super1: don't update node nums if it is not more than 1
>
> Create.c | 7 ++++++-
> super1.c | 5 +++++
> 2 files changed, 11 insertions(+), 1 deletion(-)
Hi Guoqing,
I am a little confused on this one - albeit I haven't looked at it in
detail. Why should it not be possible to start a cluster with one node?
In theory you should be able to do that, and then add nodes later?
Cheers,
Jes
^ permalink raw reply
* Re: [PATCH 0/2] Check node nums for cluster raid
From: Doug Ledford @ 2016-05-04 15:19 UTC (permalink / raw)
To: Jes Sorensen, Guoqing Jiang; +Cc: linux-raid
In-Reply-To: <wrfjinytx3fv.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1311 bytes --]
On 05/04/2016 11:12 AM, Jes Sorensen wrote:
> Guoqing Jiang <gqjiang@suse.com> writes:
>> For cluster raid, we do need at least two nodes for it,
>> the two patches add the checks before create and change
>> bitmap.
>>
>> Thanks,
>> Guoqing
>>
>> Guoqing Jiang (2):
>> Create: check the node nums when create clustered raid
>> super1: don't update node nums if it is not more than 1
>>
>> Create.c | 7 ++++++-
>> super1.c | 5 +++++
>> 2 files changed, 11 insertions(+), 1 deletion(-)
>
> Hi Guoqing,
>
> I am a little confused on this one - albeit I haven't looked at it in
> detail. Why should it not be possible to start a cluster with one node?
> In theory you should be able to do that, and then add nodes later?
Not typically. A single node of a cluster is likely the odd man out, so
starting it and allowing changes to the underlying device has a high
potential of creating split brain issues. For that reason, most cluster
setups require some minimum (usually 2) for a quorum before they will
start. Otherwise, given a three node cluster, you could end up with
three separate live filesystems and the need to merge changes between
them to bring the cluster back into sync.
--
Doug Ledford <dledford@redhat.com>
GPG KeyID: 0E572FDD
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox