* [V2 PATCH 05/13] md-cluster: fix locking when node joins cluster during message broadcast
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
If a node joins the cluster while a message broadcast
is under way, a lock issue could happen as follows.
For a cluster which included two nodes, if node A is
calling __sendmsg before up-convert CR to EX on ack,
and node B released CR on ack. But if a new node C
joins the cluster and it doesn't receive the message
which A sent before, so it could hold CR on ack before
A up-convert CR to EX on ack.
So a node joining the cluster should get an EX lock on
the "token" first to ensure no broadcast is ongoing,
then release it after held CR on ack.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 76f88f7..30f1160 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -781,17 +781,24 @@ static int join(struct mddev *mddev, int nodes)
cinfo->token_lockres = lockres_init(mddev, "token", NULL, 0);
if (!cinfo->token_lockres)
goto err;
- cinfo->ack_lockres = lockres_init(mddev, "ack", ack_bast, 0);
- if (!cinfo->ack_lockres)
- goto err;
cinfo->no_new_dev_lockres = lockres_init(mddev, "no-new-dev", NULL, 0);
if (!cinfo->no_new_dev_lockres)
goto err;
+ ret = dlm_lock_sync(cinfo->token_lockres, DLM_LOCK_EX);
+ if (ret) {
+ ret = -EAGAIN;
+ pr_err("md-cluster: can't join cluster to avoid lock issue\n");
+ goto err;
+ }
+ cinfo->ack_lockres = lockres_init(mddev, "ack", ack_bast, 0);
+ if (!cinfo->ack_lockres)
+ goto err;
/* get sync CR lock on ACK. */
if (dlm_lock_sync(cinfo->ack_lockres, DLM_LOCK_CR))
pr_err("md-cluster: failed to get a sync CR lock on ACK!(%d)\n",
ret);
+ dlm_unlock_sync(cinfo->token_lockres);
/* get sync CR lock on no-new-dev. */
if (dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR))
pr_err("md-cluster: failed to get a sync CR lock on no-new-dev!(%d)\n", ret);
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 04/13] md-cluster: unregister thread if err happened
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
The two threads need to be unregistered if a node
can't join cluster successfully.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 0d4ddf8..76f88f7 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -818,6 +818,8 @@ static int join(struct mddev *mddev, int nodes)
return 0;
err:
+ md_unregister_thread(&cinfo->recovery_thread);
+ md_unregister_thread(&cinfo->recv_thread);
lockres_free(cinfo->message_lockres);
lockres_free(cinfo->token_lockres);
lockres_free(cinfo->ack_lockres);
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 03/13] md-cluster: wake up thread to continue recovery
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
In recovery case, we need to set MD_RECOVERY_NEEDED
and wake up thread only if recover is not finished.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 12fbfec..0d4ddf8 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -284,11 +284,14 @@ static void recover_bitmaps(struct md_thread *thread)
goto dlm_unlock;
}
if (hi > 0) {
- /* TODO:Wait for current resync to get over */
- set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
if (lo < mddev->recovery_cp)
mddev->recovery_cp = lo;
- md_check_recovery(mddev);
+ /* wake up thread to continue resync in case resync
+ * is not finished */
+ if (mddev->recovery_cp != MaxSector) {
+ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
+ md_wakeup_thread(mddev->thread);
+ }
}
dlm_unlock:
dlm_unlock_sync(bm_lockres);
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 02/13] md-cluser: make resync_finish only called after pers->sync_request
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
It is not reasonable that cluster raid to release resync
lock before the last pers->sync_request has finished.
As the metadata will be changed when node performs resync,
we need to inform other nodes to update metadata, so the
MD_CHANGE_PENDING flag is set before finish resync.
Then metadata_update_finish is move ahead to ensure that
METADATA_UPDATED msg is sent before finish resync, and
metadata_update_start need to be run after "repeat:" label
accordingly.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md.c | 25 +++++++++++++------------
1 file changed, 13 insertions(+), 12 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 4fd7d77..dd83a50 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2291,6 +2291,7 @@ void md_update_sb(struct mddev *mddev, int force_change)
return;
}
+repeat:
if (mddev_is_clustered(mddev)) {
if (test_and_clear_bit(MD_CHANGE_DEVS, &mddev->flags))
force_change = 1;
@@ -2303,7 +2304,7 @@ void md_update_sb(struct mddev *mddev, int force_change)
return;
}
}
-repeat:
+
/* First make sure individual recovery_offsets are correct */
rdev_for_each(rdev, mddev) {
if (rdev->raid_disk >= 0 &&
@@ -2430,6 +2431,9 @@ repeat:
md_super_wait(mddev);
/* if there was a failure, MD_CHANGE_DEVS was set, and we re-write super */
+ if (mddev_is_clustered(mddev) && ret == 0)
+ md_cluster_ops->metadata_update_finish(mddev);
+
spin_lock(&mddev->lock);
if (mddev->in_sync != sync_req ||
test_bit(MD_CHANGE_DEVS, &mddev->flags)) {
@@ -2452,9 +2456,6 @@ repeat:
clear_bit(BlockedBadBlocks, &rdev->flags);
wake_up(&rdev->blocked_wait);
}
-
- if (mddev_is_clustered(mddev) && ret == 0)
- md_cluster_ops->metadata_update_finish(mddev);
}
EXPORT_SYMBOL(md_update_sb);
@@ -7785,7 +7786,6 @@ void md_do_sync(struct md_thread *thread)
struct md_rdev *rdev;
char *desc, *action = NULL;
struct blk_plug plug;
- bool cluster_resync_finished = false;
int ret;
/* just incase thread restarts... */
@@ -8103,11 +8103,6 @@ void md_do_sync(struct md_thread *thread)
mddev->curr_resync_completed = mddev->curr_resync;
sysfs_notify(&mddev->kobj, NULL, "sync_completed");
}
- /* tell personality and other nodes that we are finished */
- if (mddev_is_clustered(mddev)) {
- md_cluster_ops->resync_finish(mddev);
- cluster_resync_finished = true;
- }
mddev->pers->sync_request(mddev, max_sectors, &skipped);
if (!test_bit(MD_RECOVERY_CHECK, &mddev->recovery) &&
@@ -8147,9 +8142,15 @@ void md_do_sync(struct md_thread *thread)
set_bit(MD_CHANGE_DEVS, &mddev->flags);
if (mddev_is_clustered(mddev) &&
- test_bit(MD_RECOVERY_INTR, &mddev->recovery) &&
- !cluster_resync_finished)
+ ret == 0) {
+ /* set CHANGE_PENDING here since maybe another
+ * update is needed, so other nodes are informed */
+ set_bit(MD_CHANGE_PENDING, &mddev->flags);
+ md_wakeup_thread(mddev->thread);
+ wait_event(mddev->sb_wait,
+ !test_bit(MD_CHANGE_PENDING, &mddev->flags));
md_cluster_ops->resync_finish(mddev);
+ }
spin_lock(&mddev->lock);
if (!test_bit(MD_RECOVERY_INTR, &mddev->recovery)) {
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 01/13] md-cluster: change resync lock from asynchronous to synchronous
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
In-Reply-To: <1462203200-1375-1-git-send-email-gqjiang@suse.com>
If multiple nodes choose to attempt do resync at the same time
they need to be serialized so they don't duplicate effort. This
serialization is done by locking the 'resync' DLM lock.
Currently if a node cannot get the lock immediately it doesn't
request notification when the lock becomes available (i.e.
DLM_LKF_NOQUEUE is set), so it may not reliably find out when it
is safe to try again.
Rather than trying to arrange an async wake-up when the lock
becomes available, switch to using synchronous locking - this is
a lot easier to think about. As it is not permitted to block in
the 'raid1d' thread, move the locking to the resync thread. So
the rsync thread is forked immediately, but it blocks until the
resync lock is available. Once the lock is locked it checks again
if any resync action is needed.
A particular symptom of the current problem is that a node can
get stuck with "resync=pending" indefinitely.
Reviewed-by: NeilBrown <neilb@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 2 --
drivers/md/md.c | 23 ++++++++++++++---------
2 files changed, 14 insertions(+), 11 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index dd97d42..12fbfec 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -937,7 +937,6 @@ static void metadata_update_cancel(struct mddev *mddev)
static int resync_start(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
- cinfo->resync_lockres->flags |= DLM_LKF_NOQUEUE;
return dlm_lock_sync(cinfo->resync_lockres, DLM_LOCK_EX);
}
@@ -967,7 +966,6 @@ static int resync_info_update(struct mddev *mddev, sector_t lo, sector_t hi)
static int resync_finish(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
- cinfo->resync_lockres->flags &= ~DLM_LKF_NOQUEUE;
dlm_unlock_sync(cinfo->resync_lockres);
return resync_info_update(mddev, 0, 0);
}
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 14d3b37..4fd7d77 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -7786,6 +7786,7 @@ void md_do_sync(struct md_thread *thread)
char *desc, *action = NULL;
struct blk_plug plug;
bool cluster_resync_finished = false;
+ int ret;
/* just incase thread restarts... */
if (test_bit(MD_RECOVERY_DONE, &mddev->recovery))
@@ -7795,6 +7796,19 @@ void md_do_sync(struct md_thread *thread)
return;
}
+ if (mddev_is_clustered(mddev)) {
+ ret = md_cluster_ops->resync_start(mddev);
+ if (ret)
+ goto skip;
+
+ if (!(test_bit(MD_RECOVERY_SYNC, &mddev->recovery) ||
+ test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) ||
+ test_bit(MD_RECOVERY_RECOVER, &mddev->recovery))
+ && ((unsigned long long)mddev->curr_resync_completed
+ < (unsigned long long)mddev->resync_max_sectors))
+ goto skip;
+ }
+
if (test_bit(MD_RECOVERY_SYNC, &mddev->recovery)) {
if (test_bit(MD_RECOVERY_CHECK, &mddev->recovery)) {
desc = "data-check";
@@ -8226,18 +8240,9 @@ static void md_start_sync(struct work_struct *ws)
struct mddev *mddev = container_of(ws, struct mddev, del_work);
int ret = 0;
- if (mddev_is_clustered(mddev)) {
- ret = md_cluster_ops->resync_start(mddev);
- if (ret) {
- mddev->sync_thread = NULL;
- goto out;
- }
- }
-
mddev->sync_thread = md_register_thread(md_do_sync,
mddev,
"resync");
-out:
if (!mddev->sync_thread) {
if (!(mddev_is_clustered(mddev) && ret == -EAGAIN))
printk(KERN_ERR "%s: could not start resync"
--
2.6.6
^ permalink raw reply related
* [V2 PATCH 00/13] The latest patches for md-cluster
From: Guoqing Jiang @ 2016-05-02 15:33 UTC (permalink / raw)
To: shli; +Cc: neilb, linux-raid, Guoqing Jiang
Changes:
1. delete no_hijack parameter from bitmap_get_counter
2. add one patch from kbuild test robot to remove checkpatch warning
3. md-set-MD_CHANGE_PENDING-in-a-spinlocked-region.patch is removed
from the patchset and it will be post later.
For cluster raid1, we found some issues and some codes need to be
improved during the past months, and all the patches are based on
for-next branch of md tree.
The patchset also available in github as follows:
https://github.com/GuoqingJiang/linux/tree/md-for-next
Thanks,
Guoqing
Guoqing Jiang (12):
md-cluster: change resync lock from asynchronous to synchronous
md-cluser: make resync_finish only called after pers->sync_request
md-cluster: wake up thread to continue recovery
md-cluster: unregister thread if err happened
md-cluster: fix locking when node joins cluster during message
broadcast
md-cluster: change array_sectors and update size are not supported
md-cluster: wakeup thread if activated a spare disk
md-cluster: always setup in-memory bitmap
md-cluster: sync bitmap when node received RESYNCING msg
md-cluster/bitmap: fix wrong calcuation of offset
md-cluster/bitmap: fix wrong page num in bitmap_file_clear_bit and
bitmap_file_set_bit
md-cluster/bitmap: unplug bitmap to sync dirty pages to disk
kbuild test robot (1):
md-cluster: fix ifnullfree.cocci warnings
Documentation/md-cluster.txt | 6 ++++
drivers/md/bitmap.c | 85 ++++++++++++++++++++++++++++++++++++++------
drivers/md/bitmap.h | 3 ++
drivers/md/md-cluster.c | 53 ++++++++++++++++++++++-----
drivers/md/md.c | 61 ++++++++++++++++++++-----------
5 files changed, 168 insertions(+), 40 deletions(-)
--
2.6.6
^ permalink raw reply
* Re: Write to the degraded raid5 will trigger the call trace dump when skip_copy is enabled
From: Joey Liao @ 2016-05-02 14:45 UTC (permalink / raw)
To: Shaohua Li; +Cc: linux-raid
In-Reply-To: <20160429211719.GA104659@kernel.org>
Hi Shaohua,
Many thanks for your reply.
How about the following WARN_ON() in handle_stripe_clean_event()???
if (test_and_clear_bit(R5_SkipCopy, &dev->flags)) {
WARN_ON(test_bit(R5_UPTODATE, &dev->flags));
}
Sometimes it will be triggered as well but I can't find the exactly
way to reproduce it.
I would like to know if it is suitable to remove it as well or not.
2016-04-30 5:17 GMT+08:00 Shaohua Li <shli@kernel.org>:
> On Fri, Apr 29, 2016 at 06:54:10PM +0800, Joey Liao wrote:
>> Hi all,
>>
>> For improving the I/O performance, we try to enable the raid 5
>> skip_copy feature by default. It indeed has some benefit, however, it
>> will always (not a random issue) dump the following call trace
>> repeatedly when we try to write the raid 5 block device. It is related
>> to the following codes in handle_stripe_clean_event() in raid5.c.
>>
>> WARN_ON(test_bit(R5_SkipCopy, &dev->flags));
>> WARN_ON(dev->page != dev->orig_page);
>>
>> Is it a known bug for skip_copy feature?
>> Does it do harm to the data integrity?
>> If we would like to prevent this call trace, for your suggestion how
>> should we do to modify the source code?
>
> Looks the two WARN_ON should be deleted. if the dev has R5_LOCKED, it's legit
> the dev has SkipCopy set and page != orig_page. I'll delete the code. This will
> not harm to data integrity.
>
> Thanks,
> Shaohua
^ permalink raw reply
* Re: I was dump, I need help.
From: Robin Hill @ 2016-05-02 12:41 UTC (permalink / raw)
To: Patrice; +Cc: linux-raid
In-Reply-To: <5726128E.9090309@pboenig.de>
[-- Attachment #1: Type: text/plain, Size: 8377 bytes --]
On Sun May 01, 2016 at 04:28:30PM +0200, Patrice wrote:
> Hi Robin,
>
> thank you for your reply.
> Ok, I try not to panic but in my opinion that sounds bad. It seems to me
> like a mess.
> Why is there a RAID 1 und 6? I need a RAID 5.
>
It looks like you have a RAID1, a RAID6 and a RAID5. I'd guess that the
RAID1 and RAID6 store the OS for the NAS system, and the RAID5 is the
data.
> > are there any others which should be being
> > assembled into another array?
>
> There are no others. At least there should be only one partition on each
> HDD. I didn`t do partitioning.
>
>
> fdisk -l output:
> ------------------
>
> Disk /dev/sda: 4000.8 GB, 4000787030016 bytes
> 256 heads, 63 sectors/track, 484501 cylinders, total 7814037168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sda1 1 4294967295 2147483647+ ee GPT
> Partition 1 does not start on physical sector boundary.
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdb: 4000.8 GB, 4000787030016 bytes
> 256 heads, 63 sectors/track, 484501 cylinders, total 7814037168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 1 4294967295 2147483647+ ee GPT
> Partition 1 does not start on physical sector boundary.
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdc: 4000.8 GB, 4000787030016 bytes
> 256 heads, 63 sectors/track, 484501 cylinders, total 7814037168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdc1 1 4294967295 2147483647+ ee GPT
> Partition 1 does not start on physical sector boundary.
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util
> fdisk doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdd: 4000.8 GB, 4000787030016 bytes
> 256 heads, 63 sectors/track, 484501 cylinders, total 7814037168 sectors
> Units = sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 4096 bytes
> I/O size (minimum/optimal): 4096 bytes / 4096 bytes
> Disk identifier: 0x00000000
>
> Device Boot Start End Blocks Id System
> /dev/sdd1 1 4294967295 2147483647+ ee GPT
> Partition 1 does not start on physical sector boundary.
>
> -------------------------------------------------------------------------
>
Okay, so there's 4 4TB drives - they're using GPT partitions so fdisk
doesn't report anything useful here.
>
> mdadm -E /dev/sd* output:
> --------------------------
>
> /dev/sda3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 632ff5fd:65342524:9c9798d7:80e47e94
> Name : 119c1bce:data-0 (local to host 119c1bce)
> Creation Time : Sun Apr 3 06:27:49 2016
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7804333680 (3721.40 GiB 3995.82 GB)
> Array Size : 11706500352 (11164.19 GiB 11987.46 GB)
> Used Dev Size : 7804333568 (3721.40 GiB 3995.82 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=112 sectors
> State : clean
> Device UUID : 38adc372:3e0eba36:0f819758:950a0411
>
> Update Time : Sat Apr 30 23:03:32 2016
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : d7f5b303 - correct
> Events : 266
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 0
> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>
>
> /dev/sdb3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 632ff5fd:65342524:9c9798d7:80e47e94
> Name : 119c1bce:data-0 (local to host 119c1bce)
> Creation Time : Sun Apr 3 06:27:49 2016
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7804333680 (3721.40 GiB 3995.82 GB)
> Array Size : 11706500352 (11164.19 GiB 11987.46 GB)
> Used Dev Size : 7804333568 (3721.40 GiB 3995.82 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=112 sectors
> State : clean
> Device UUID : 655ee144:43c43771:0d8a6157:9b556584
>
> Update Time : Sat Apr 30 23:03:32 2016
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 56bc6e3b - correct
> Events : 266
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 1
> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>
>
> /dev/sdc3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 632ff5fd:65342524:9c9798d7:80e47e94
> Name : 119c1bce:data-0 (local to host 119c1bce)
> Creation Time : Sun Apr 3 06:27:49 2016
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7804333680 (3721.40 GiB 3995.82 GB)
> Array Size : 11706500352 (11164.19 GiB 11987.46 GB)
> Used Dev Size : 7804333568 (3721.40 GiB 3995.82 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=112 sectors
> State : clean
> Device UUID : d066d1aa:ffd1e432:e9ecdd9d:08540efa
>
> Update Time : Sat Apr 30 23:17:27 2016
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 3a7ce8f6 - correct
> Events : 273
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 2
> Array State : ..AA ('A' == active, '.' == missing, 'R' == replacing)
>
>
> /dev/sdd3:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 632ff5fd:65342524:9c9798d7:80e47e94
> Name : 119c1bce:data-0 (local to host 119c1bce)
> Creation Time : Sun Apr 3 06:27:49 2016
> Raid Level : raid5
> Raid Devices : 4
>
> Avail Dev Size : 7804333680 (3721.40 GiB 3995.82 GB)
> Array Size : 11706500352 (11164.19 GiB 11987.46 GB)
> Used Dev Size : 7804333568 (3721.40 GiB 3995.82 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> Unused Space : before=262056 sectors, after=112 sectors
> State : clean
> Device UUID : b8a43a56:2f833e72:7dd9f166:6f80b5a2
>
> Update Time : Sat Apr 30 23:17:27 2016
> Bad Block Log : 512 entries available at offset 72 sectors
> Checksum : 96faf109 - correct
> Events : 273
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 3
> Array State : ..AA ('A' == active, '.' == missing, 'R' == replacing)
>
I've removed the info for the first two partitions on each disk as those
arrays are assembling fine. The third partitions look to contain your
data array - the events for sda3 and sdb3 match at 266, and sdc3 and
sdd3 are on 273 (and show sda3 & sdb3 missing). A forced assembly should
work without any issues here - the array name would look to be
/dev/md/data-0, so:
mdadm -Af /dev/md/data-0 /dev/sd[abcd]3
That should assemble the array from 3 of the disks (probably sda3, sdc3
and sdd3) - you'll then need to add the other one back in and allow the
rebuild to complete. You should also do a check on the filesystem to
ensure there's no corruption.
Cheers,
Robin
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: I was dump, I need help.
From: Robin Hill @ 2016-05-01 11:30 UTC (permalink / raw)
To: Patrice; +Cc: linux-raid
In-Reply-To: <5725DCA9.9010901@pboenig.de>
On Sun May 01, 2016 at 12:38:33pm +0200, Patrice wrote:
> It has a Debian 7 installed and a RAID 5.
> Here some more Informations.
>
> Thank you and best regards.
> Patrice
>
>
>
>
> root@ReadyNAS:/proc# cat mdstat
> Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
> md1 : active raid6 sda2[0] sdc2[3] sdb2[2] sdd2[1]
> 1046528 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4]
> [UUUU]
>
> md0 : active raid1 sda1[4] sdd1[3] sdc1[2] sdb1[5]
> 4190208 blocks super 1.2 [4/4] [UUUU]
>
> unused devices: <none>
>
This shows two RAID devices, both up and running without any issues.
> /dev/sda1:
> /dev/sdb1:
> /dev/sdc1:
> /dev/sdd1:
The info provided here is just for the first partition on each disk.
There's definitely at least 2 partitions on each disk (as the second one
is used for md1 above) - are there any others which should be being
assembled into another array? "fdisk -l" will show the partitions for
all disks - if there's more than 2 on sda/b/c/d then we'll need to see
the "mdadm -E" report for each of the others.
Cheers,
Robin
--
___
( ' } | Robin Hill <robin@robinhill.me.uk> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |
^ permalink raw reply
* I was dump, I need help.
From: Patrice @ 2016-05-01 10:38 UTC (permalink / raw)
To: linux-raid
It has a Debian 7 installed and a RAID 5.
Here some more Informations.
Thank you and best regards.
Patrice
root@ReadyNAS:/proc# cat mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md1 : active raid6 sda2[0] sdc2[3] sdb2[2] sdd2[1]
1046528 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4]
[UUUU]
md0 : active raid1 sda1[4] sdd1[3] sdc1[2] sdb1[5]
4190208 blocks super 1.2 [4/4] [UUUU]
unused devices: <none>
-----------------------------------------------------------------------
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2495045b:2aa5bd3c:8cff9bb5:de4e769c
Name : 119c1bce:0 (local to host 119c1bce)
Creation Time : Sun Apr 3 06:27:48 2016
Raid Level : raid1
Raid Devices : 4
Avail Dev Size : 8380416 (4.00 GiB 4.29 GB)
Array Size : 4190208 (4.00 GiB 4.29 GB)
Data Offset : 8192 sectors
Super Offset : 8 sectors
Unused Space : before=8104 sectors, after=0 sectors
State : clean
Device UUID : 5a053636:aeb9cb79:8f7f8807:8263480b
Update Time : Sun May 1 11:29:51 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : cd6387b5 - correct
Events : 343
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
-----------------------------------------------------------------------
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2495045b:2aa5bd3c:8cff9bb5:de4e769c
Name : 119c1bce:0 (local to host 119c1bce)
Creation Time : Sun Apr 3 06:27:48 2016
Raid Level : raid1
Raid Devices : 4
Avail Dev Size : 8380416 (4.00 GiB 4.29 GB)
Array Size : 4190208 (4.00 GiB 4.29 GB)
Data Offset : 8192 sectors
Super Offset : 8 sectors
Unused Space : before=8104 sectors, after=0 sectors
State : clean
Device UUID : 58b53ace:3e80e778:41c7e259:a631d7f1
Update Time : Sun May 1 11:30:02 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9d6d1427 - correct
Events : 343
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
---------------------------------------------------------------------
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2495045b:2aa5bd3c:8cff9bb5:de4e769c
Name : 119c1bce:0 (local to host 119c1bce)
Creation Time : Sun Apr 3 06:27:48 2016
Raid Level : raid1
Raid Devices : 4
Avail Dev Size : 8380416 (4.00 GiB 4.29 GB)
Array Size : 4190208 (4.00 GiB 4.29 GB)
Data Offset : 8192 sectors
Super Offset : 8 sectors
Unused Space : before=8104 sectors, after=0 sectors
State : clean
Device UUID : c92580fb:f2a7cd6a:9090dcfd:20d59709
Update Time : Sun May 1 11:30:20 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 78531924 - correct
Events : 343
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
-----------------------------------------------------------------
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 2495045b:2aa5bd3c:8cff9bb5:de4e769c
Name : 119c1bce:0 (local to host 119c1bce)
Creation Time : Sun Apr 3 06:27:48 2016
Raid Level : raid1
Raid Devices : 4
Avail Dev Size : 8380416 (4.00 GiB 4.29 GB)
Array Size : 4190208 (4.00 GiB 4.29 GB)
Data Offset : 8192 sectors
Super Offset : 8 sectors
Unused Space : before=8104 sectors, after=0 sectors
State : active
Device UUID : 38deb03b:1d552d92:6532842d:34f5480d
Update Time : Sun May 1 11:30:26 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 133c40ae - correct
Events : 344
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply
* I was dump, I need help.
From: Patrice @ 2016-05-01 9:54 UTC (permalink / raw)
To: linux-raid
Hi,
I figured out that the Event counter is 343 on all 4 disks.
But can I just do
mdadm --assemble --force /dev/mdX <list of devices>
and it's all ok? I am worried about that it`s not and when I do that the
RAID will be distroyed. :-S
Best regards
Patrice
^ permalink raw reply
* I was dump, I need help.
From: Patrice @ 2016-05-01 9:39 UTC (permalink / raw)
To: linux-raid
Hi folks,
my name is Patrice and I was dump. I am new on this list and this topic.
Yesterday I pulled out all of the 4 harddisk of my Netgear NAS. *Info*
the NAS was running. Now my RAID is broken and on the webGUI it shows 0
data. I`ve read the two How-tTos about this topic but I am confused and
not sure what to do. That`s why I am asking you guys friendly if you
could help me, please.
Best regards
Patrice
^ permalink raw reply
* Re: Write to the degraded raid5 will trigger the call trace dump when skip_copy is enabled
From: Shaohua Li @ 2016-04-29 21:17 UTC (permalink / raw)
To: Joey Liao; +Cc: linux-raid
In-Reply-To: <CAHvN=inPP5ubYK1YkNW_B1b9ouhqBo8qMMfVcGSjddMYekDEng@mail.gmail.com>
On Fri, Apr 29, 2016 at 06:54:10PM +0800, Joey Liao wrote:
> Hi all,
>
> For improving the I/O performance, we try to enable the raid 5
> skip_copy feature by default. It indeed has some benefit, however, it
> will always (not a random issue) dump the following call trace
> repeatedly when we try to write the raid 5 block device. It is related
> to the following codes in handle_stripe_clean_event() in raid5.c.
>
> WARN_ON(test_bit(R5_SkipCopy, &dev->flags));
> WARN_ON(dev->page != dev->orig_page);
>
> Is it a known bug for skip_copy feature?
> Does it do harm to the data integrity?
> If we would like to prevent this call trace, for your suggestion how
> should we do to modify the source code?
Looks the two WARN_ON should be deleted. if the dev has R5_LOCKED, it's legit
the dev has SkipCopy set and page != orig_page. I'll delete the code. This will
not harm to data integrity.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH V3 08/13] md: set MD_CHANGE_PENDING in a spinlocked region
From: Shaohua Li @ 2016-04-29 18:21 UTC (permalink / raw)
To: NeilBrown; +Cc: Guoqing Jiang, linux-raid
In-Reply-To: <87eg9purw7.fsf@notabene.neil.brown.name>
On Fri, Apr 29, 2016 at 11:26:32AM +1000, NeilBrown wrote:
> On Thu, Apr 28 2016, Shaohua Li wrote:
>
> > On Wed, Apr 27, 2016 at 10:55:43PM -0400, Guoqing Jiang wrote:
> >>
> >>
> >> On 04/27/2016 11:27 AM, Shaohua Li wrote:
> >> >On Tue, Apr 26, 2016 at 09:56:26PM -0400, Guoqing Jiang wrote:
> >> >>Some code waits for a metadata update by:
> >> >>
> >> >>1. flagging that it is needed (MD_CHANGE_DEVS or MD_CHANGE_CLEAN)
> >> >>2. setting MD_CHANGE_PENDING and waking the management thread
> >> >>3. waiting for MD_CHANGE_PENDING to be cleared
> >> >>
> >> >>If the first two are done without locking, the code in md_update_sb()
> >> >>which checks if it needs to repeat might test if an update is needed
> >> >>before step 1, then clear MD_CHANGE_PENDING after step 2, resulting
> >> >>in the wait returning early.
> >> >>
> >> >>So make sure all places that set MD_CHANGE_PENDING are protected by
> >> >>mddev->lock.
> >> >>
> >> >>Reviewed-by: NeilBrown <neilb@suse.com>
> >> >>Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> >> >>---
> >> >>V3 changes:
> >> >>1. use spin_lock_irqsave/spin_unlock_irqrestore in error funcs and
> >> >> raid10's __make_request
> >> >shouldn't other places use spin_lock_irq/spin_unlock_irq? interrupt can occur
> >> >after you do spin_lock(), and if it's md_error, we deadlock.
> >>
> >> It could possible in theory if func was interrupted by md_error after it
> >> called spin_lock,
> >> but seems lots of place in md.c also use spin_lock/unlock for mddev->lock,
> >> take
> >> md_do_sync and md_update_sb as example, both of them used
> >> spin_lock(&mddev->lock)
> >> and spin_unlock(&mddev->lock) before.
> >>
> >> So I guess it will not cause trouble, otherwise, then we need to change all
> >> the usages of
> >> spin_lock/unlock(&mddev->lock), or introduce a new lock for this scenario. I
> >> am not sure
> >> which one is more acceptable.
> >
> > It doesn't cause trouble, because no interrupt/bh uses lock before. But now we
> > use it in softirq, that's the difference. Please enable lockdep, I think it
> > will complain. either we change all the locking to irq save or introducing a
> > new lock. either is ok.
>
> Thanks for catching this!
>
> As you say, the straight forward solution is change all the current
> spin_lock(&mddev->lock);
> to
> spin_lock_irq(&mddev->lock);
>
> and similar for unlock. Except where it can be called from interrupts
> we have to use spin_lock_irqsave().
>
> There is another option that occurs to me. Not sure if it is elegant or
> ugly, so I'm keen to see what you think.
>
> In the places where were set MD_CHANGE_PENDING and one other bit -
> either MD_CHANGE_DEVS or MD_CHANGE_CLEAN - we could use set_mask_bits
> to set them both atomically.
>
> set_mask_bits(&mddev->flags, BIT(MD_CHANGE_PENDING) | BIT(MD_CHANGE_DEVS));
>
> Then in md_update_sb, when deciding whether to loop back to "repeat:" we
> use a new "bit_clear_unless".
>
> #define bit_clear_unless(ptr, _clear, _test) \
> ({ \
> const typeof(*ptr) clear = (_clear), test = (_test); \
> typeof(*ptr) old, new; \
> \
> do { \
> old = ACCESS_ONCE(*ptr); \
> new = old & ~clear; \
> } while (!(old & test) && cmpxchg(ptr, old, new) != old);\
> \
> !(old & test); \
> })
>
> The code in md_update_sb() would be
>
> if (mddev->in_sync != sync_req ||
> !bit_clear_unless(&mddev->flags, BIT(MD_CHANGE_PENDING),
> BIT(MD_CHANGE_DEVS)|BIT(MD_CHANGE_CLEAN)))
> goto repeat;
>
>
> So if either DEV or CLEAN we set, PENDING would not be cleared and the
> code would goto repeat.
>
> What do you think?
Looks great, I like this idea. Thanks!
^ permalink raw reply
* Write to the degraded raid5 will trigger the call trace dump when skip_copy is enabled
From: Joey Liao @ 2016-04-29 10:54 UTC (permalink / raw)
To: linux-raid
Hi all,
For improving the I/O performance, we try to enable the raid 5
skip_copy feature by default. It indeed has some benefit, however, it
will always (not a random issue) dump the following call trace
repeatedly when we try to write the raid 5 block device. It is related
to the following codes in handle_stripe_clean_event() in raid5.c.
WARN_ON(test_bit(R5_SkipCopy, &dev->flags));
WARN_ON(dev->page != dev->orig_page);
Is it a known bug for skip_copy feature?
Does it do harm to the data integrity?
If we would like to prevent this call trace, for your suggestion how
should we do to modify the source code?
Thanks.
<4>[38795.430821] ------------[ cut here ]------------
<4>[38795.430822] WARNING: CPU: 1 PID: 28674 at
drivers/md/raid5.c:3447 handle_stripe_clean_event+0x379/0x3d0()
<4>[38795.430823] Modules linked in: ppp_deflate bsd_comp ppp_mppe
ppp_async ppp_generic crc_ccitt slhc xt_mark ipt_MASQUERADE
iptable_nat nf_nat_masquerade_ipv4 nf_nat_ipv4 nf_nat tun iscsi_tcp(O)
libiscsi_tcp(O) libiscsi(O) scsi_transport_iscsi(O) iscsi_target_mod
target_core_file target_core_iblock target_core_mod fbdisk(O) bonding
bridge stp uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core
snd_usb_caiaq snd_usb_audio snd_usbmidi_lib snd_seq_midi snd_rawmidi
fnotify(PO) udf isofs iTCO_wdt psnap llc tbs_keys(O) dm_c2f(O)
ufsd(PO) jnl(O) pl2303 usbserial intel_ips drbd(O) flashcache(O)
dm_thin_pool dm_bio_prison dm_persistent_data hal_netlink(O) coretemp
mlx4_en(O) mlx4_core(O) mlx_compat(O) ixgbe mdio igb e1000e(O) mpt3sas
mpt2sas scsi_transport_sas raid_class uas usb_storage xhci_pci
xhci_hcd usblp uhci_hcd ehci_pci ehci_hcd
<4>[38795.430839] CPU: 1 PID: 28674 Comm: md2_resync Tainted: P
W O 3.19.8 #1
<4>[38795.430840] Hardware name: INSYDE QV96/Type2 - Board Product
Name1, BIOS QV96IR23 10/21/2015
<4>[38795.430840] ffffffff81ab550a ffff88009510f9c8 ffffffff81362987
ffff88009510fa08
<4>[38795.430841] ffffffff8105447b ffff88009510fa68 ffff88009897b000
0000000000000000
<4>[38795.430843] ffff8807d742fe48 ffff8807d742fe40 ffff8807d742f800
ffff88009510fa18
<4>[38795.430844] Call Trace:
<4>[38795.430845] [<ffffffff81362987>] dump_stack+0x57/0x80
<4>[38795.430847] [<ffffffff8105447b>] warn_slowpath_common+0x8b/0xd0
<4>[38795.430848] [<ffffffff810544d5>] warn_slowpath_null+0x15/0x20
<4>[38795.430850] [<ffffffff8160a829>] handle_stripe_clean_event+0x379/0x3d0
<4>[38795.430851] [<ffffffff8160e2d2>] handle_stripe+0xe52/0x1b50
<4>[38795.430852] [<ffffffff816100af>] sync_request+0x5af/0xbf0
<4>[38795.430853] [<ffffffff810a0a4e>] ? try_to_del_timer_sync+0x4e/0x60
<4>[38795.430854] [<ffffffff810a0a9d>] ? del_timer_sync+0x3d/0x50
<4>[38795.430855] [<ffffffff818938a5>] ? schedule_timeout+0xf5/0x160
<4>[38795.430856] [<ffffffff810a0ab0>] ? del_timer_sync+0x50/0x50
<4>[38795.430858] [<ffffffff8161be39>] md_do_sync+0x9c9/0xf00
<4>[38795.430860] [<ffffffff810856c0>] ? woken_wake_function+0x10/0x10
<4>[38795.430861] [<ffffffff81890fe0>] ? __schedule+0x320/0x860
<4>[38795.430862] [<ffffffff81075d9d>] ? try_to_wake_up+0xed/0x290
<4>[38795.430864] [<ffffffff81619175>] md_thread+0x75/0x120
<4>[38795.430865] [<ffffffff81619100>] ? errors_store+0x70/0x70
<4>[38795.430866] [<ffffffff81619100>] ? errors_store+0x70/0x70
<4>[38795.430867] [<ffffffff8106d87e>] kthread+0xde/0xf0
<4>[38795.430869] [<ffffffff8106d7a0>] ? kthreadd+0x150/0x150
<4>[38795.430870] [<ffffffff818946c8>] ret_from_fork+0x58/0x90
<4>[38795.430871] [<ffffffff8106d7a0>] ? kthreadd+0x150/0x150
<4>[38795.430872] ---[ end trace 9d400a29547fb473 ]---
<4>[38795.430872] ------------[ cut here ]------------
<4>[38795.430874] WARNING: CPU: 1 PID: 28674 at
drivers/md/raid5.c:3448 handle_stripe_clean_event+0x1f2/0x3d0()
<4>[38795.430874] Modules linked in: ppp_deflate bsd_comp ppp_mppe
ppp_async ppp_generic crc_ccitt slhc xt_mark ipt_MASQUERADE
iptable_nat nf_nat_masquerade_ipv4 nf_nat_ipv4 nf_nat tun iscsi_tcp(O)
libiscsi_tcp(O) libiscsi(O) scsi_transport_iscsi(O) iscsi_target_mod
target_core_file target_core_iblock target_core_mod fbdisk(O) bonding
bridge stp uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core
snd_usb_caiaq snd_usb_audio snd_usbmidi_lib snd_seq_midi snd_rawmidi
fnotify(PO) udf isofs iTCO_wdt psnap llc tbs_keys(O) dm_c2f(O)
ufsd(PO) jnl(O) pl2303 usbserial intel_ips drbd(O) flashcache(O)
dm_thin_pool dm_bio_prison dm_persistent_data hal_netlink(O) coretemp
mlx4_en(O) mlx4_core(O) mlx_compat(O) ixgbe mdio igb e1000e(O) mpt3sas
mpt2sas scsi_transport_sas raid_class uas usb_storage xhci_pci
xhci_hcd usblp uhci_hcd ehci_pci ehci_hcd
<4>[38795.430891] CPU: 1 PID: 28674 Comm: md2_resync Tainted: P
W O 3.19.8 #1
<4>[38795.430891] Hardware name: INSYDE QV96/Type2 - Board Product
Name1, BIOS QV96IR23 10/21/2015
<4>[38795.430892] ffffffff81ab550a ffff88009510f9c8 ffffffff81362987
ffff88009510fa08
<4>[38795.430893] ffffffff8105447b ffff88009510fa68 ffff88009897b000
0000000000000000
<4>[38795.430894] ffff8807d742fe48 ffff8807d742fe40 ffff8807d742f800
ffff88009510fa18
<4>[38795.430895] Call Trace:
<4>[38795.430897] [<ffffffff81362987>] dump_stack+0x57/0x80
<4>[38795.430898] [<ffffffff8105447b>] warn_slowpath_common+0x8b/0xd0
<4>[38795.430899] [<ffffffff810544d5>] warn_slowpath_null+0x15/0x20
<4>[38795.430901] [<ffffffff8160a6a2>] handle_stripe_clean_event+0x1f2/0x3d0
<4>[38795.430902] [<ffffffff8160e2d2>] handle_stripe+0xe52/0x1b50
<4>[38795.430903] [<ffffffff816100af>] sync_request+0x5af/0xbf0
<4>[38795.430904] [<ffffffff810a0a4e>] ? try_to_del_timer_sync+0x4e/0x60
<4>[38795.430906] [<ffffffff810a0a9d>] ? del_timer_sync+0x3d/0x50
<4>[38795.430907] [<ffffffff818938a5>] ? schedule_timeout+0xf5/0x160
<4>[38795.430908] [<ffffffff810a0ab0>] ? del_timer_sync+0x50/0x50
<4>[38795.430909] [<ffffffff8161be39>] md_do_sync+0x9c9/0xf00
<4>[38795.430911] [<ffffffff810856c0>] ? woken_wake_function+0x10/0x10
<4>[38795.430912] [<ffffffff81890fe0>] ? __schedule+0x320/0x860
<4>[38795.430914] [<ffffffff81075d9d>] ? try_to_wake_up+0xed/0x290
<4>[38795.430915] [<ffffffff81619175>] md_thread+0x75/0x120
<4>[38795.430916] [<ffffffff81619100>] ? errors_store+0x70/0x70
<4>[38795.430918] [<ffffffff81619100>] ? errors_store+0x70/0x70
<4>[38795.430919] [<ffffffff8106d87e>] kthread+0xde/0xf0
<4>[38795.430920] [<ffffffff8106d7a0>] ? kthreadd+0x150/0x150
<4>[38795.430921] [<ffffffff818946c8>] ret_from_fork+0x58/0x90
<4>[38795.430922] [<ffffffff8106d7a0>] ? kthreadd+0x150/0x150
<4>[38795.430923] ---[ end trace 9d400a29547fb474 ]---
^ permalink raw reply
* Re: [PATCH] MD: make bio mergeable
From: Holger Kiehl @ 2016-04-29 9:23 UTC (permalink / raw)
To: Shaohua Li
Cc: linux-block, linux-kernel, linux-raid, qkrwngud825, Kernel-team,
Ming Lei, Jens Axboe, Neil Brown
In-Reply-To: <20160428211908.GA3116523@devbig084.prn1.facebook.com>
On Thu, 28 Apr 2016, Shaohua Li wrote:
> On Thu, Apr 28, 2016 at 08:00:22PM +0000, Holger Kiehl wrote:
> > Hello,
> >
> > On Mon, 25 Apr 2016, Shaohua Li wrote:
> >
> > > blk_queue_split marks bio unmergeable, which makes sense for normal bio.
> > > But if dispatching the bio to underlayer disk, the blk_queue_split
> > > checks are invalid, hence it's possible the bio becomes mergeable.
> > >
> > > In the reported bug, this bug causes trim against raid0 performance slash
> > > https://bugzilla.kernel.org/show_bug.cgi?id=117051
> > >
> > This patch makes a huge difference. On a system with two Samsung 850 Pro
> > in a MD Raid0 setup the time for fstrim went down from ~30min to 18sec!
> >
> > However, on another system with two Intel P3700 1.6TB NVMe PCIe SSD's
> > also setup as one big MD Raid0, the patch does not make any difference
> > at all. fstrim takes more then 4 hours!
>
> Does the raid0 cross two partitions or two SSD?
>
Two SSD's. Where it works, for the two Samsung 850 Pro SATA SSD it was
via partitions.
> can you post blktrace data in the bugzilloa, I'll track the bug there.
>
I did the blktrace on the two md raid0 devices /dev/nvme[01]n1 for 2 minutes
and attached them to the bug 117051 as a tar.bz2 file:
https://bugzilla.kernel.org/show_bug.cgi?id=117051
Please just ask if I have forgotten anything. And many thanks for looking
at this and all the good work!
Regards,
Holger
^ permalink raw reply
* Re: [PATCH V3 08/13] md: set MD_CHANGE_PENDING in a spinlocked region
From: NeilBrown @ 2016-04-29 1:26 UTC (permalink / raw)
To: Shaohua Li, Guoqing Jiang; +Cc: linux-raid
In-Reply-To: <20160428035802.GA90901@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 3524 bytes --]
On Thu, Apr 28 2016, Shaohua Li wrote:
> On Wed, Apr 27, 2016 at 10:55:43PM -0400, Guoqing Jiang wrote:
>>
>>
>> On 04/27/2016 11:27 AM, Shaohua Li wrote:
>> >On Tue, Apr 26, 2016 at 09:56:26PM -0400, Guoqing Jiang wrote:
>> >>Some code waits for a metadata update by:
>> >>
>> >>1. flagging that it is needed (MD_CHANGE_DEVS or MD_CHANGE_CLEAN)
>> >>2. setting MD_CHANGE_PENDING and waking the management thread
>> >>3. waiting for MD_CHANGE_PENDING to be cleared
>> >>
>> >>If the first two are done without locking, the code in md_update_sb()
>> >>which checks if it needs to repeat might test if an update is needed
>> >>before step 1, then clear MD_CHANGE_PENDING after step 2, resulting
>> >>in the wait returning early.
>> >>
>> >>So make sure all places that set MD_CHANGE_PENDING are protected by
>> >>mddev->lock.
>> >>
>> >>Reviewed-by: NeilBrown <neilb@suse.com>
>> >>Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
>> >>---
>> >>V3 changes:
>> >>1. use spin_lock_irqsave/spin_unlock_irqrestore in error funcs and
>> >> raid10's __make_request
>> >shouldn't other places use spin_lock_irq/spin_unlock_irq? interrupt can occur
>> >after you do spin_lock(), and if it's md_error, we deadlock.
>>
>> It could possible in theory if func was interrupted by md_error after it
>> called spin_lock,
>> but seems lots of place in md.c also use spin_lock/unlock for mddev->lock,
>> take
>> md_do_sync and md_update_sb as example, both of them used
>> spin_lock(&mddev->lock)
>> and spin_unlock(&mddev->lock) before.
>>
>> So I guess it will not cause trouble, otherwise, then we need to change all
>> the usages of
>> spin_lock/unlock(&mddev->lock), or introduce a new lock for this scenario. I
>> am not sure
>> which one is more acceptable.
>
> It doesn't cause trouble, because no interrupt/bh uses lock before. But now we
> use it in softirq, that's the difference. Please enable lockdep, I think it
> will complain. either we change all the locking to irq save or introducing a
> new lock. either is ok.
Thanks for catching this!
As you say, the straight forward solution is change all the current
spin_lock(&mddev->lock);
to
spin_lock_irq(&mddev->lock);
and similar for unlock. Except where it can be called from interrupts
we have to use spin_lock_irqsave().
There is another option that occurs to me. Not sure if it is elegant or
ugly, so I'm keen to see what you think.
In the places where were set MD_CHANGE_PENDING and one other bit -
either MD_CHANGE_DEVS or MD_CHANGE_CLEAN - we could use set_mask_bits
to set them both atomically.
set_mask_bits(&mddev->flags, BIT(MD_CHANGE_PENDING) | BIT(MD_CHANGE_DEVS));
Then in md_update_sb, when deciding whether to loop back to "repeat:" we
use a new "bit_clear_unless".
#define bit_clear_unless(ptr, _clear, _test) \
({ \
const typeof(*ptr) clear = (_clear), test = (_test); \
typeof(*ptr) old, new; \
\
do { \
old = ACCESS_ONCE(*ptr); \
new = old & ~clear; \
} while (!(old & test) && cmpxchg(ptr, old, new) != old);\
\
!(old & test); \
})
The code in md_update_sb() would be
if (mddev->in_sync != sync_req ||
!bit_clear_unless(&mddev->flags, BIT(MD_CHANGE_PENDING),
BIT(MD_CHANGE_DEVS)|BIT(MD_CHANGE_CLEAN)))
goto repeat;
So if either DEV or CLEAN we set, PENDING would not be cleared and the
code would goto repeat.
What do you think?
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Help needed: array inactive after grow attempt
From: Wols Lists @ 2016-04-28 23:30 UTC (permalink / raw)
To: Andread Mayrhoff, linux-raid
In-Reply-To: <c98c947ff2759669bcb5ccf87c3dd245@spoilbase.eu>
On 28/04/16 23:29, Andread Mayrhoff wrote:
> Hello Wol,
>
> that was already pretty helpful.
>
> I am able to recover the old 5-disk-status by recreating the array with
> --assume-clean (and a couple of parameters). So the data seems safe
> while we're speaking.
>
> However the reason for the problem (that still persists) is that the
> "grow" operation is, as you rightly say, stuck at 0%, and it doesn't
> progress at all.
>
> That was the reason for the crash in the first place, because when the
> machine was switched off a couple of hours after I started the
> reshaping, it was very likely still at 0%.
> Now, I'm going to catch up with the other reports on the mail list and
> try to determine if there's any way to bypass the problem.
Ahhhh ....
>
> Good night and thanks again...
>
That sounds very promising. This is a regular problem that crops up on
the list a lot, and iirc the fix is very simple. Get the latest mdadm :-)
You should find a lot of info on the mailing list :-)
Cheers,
Wol
>
> 2016-04-29 00:12, Wols Lists wrote:
>>> I might add
>>>
>>> ~/tmp # mdadm --stop /dev/md127
>>> mdadm: stopped /dev/md127
>>> ~/tmp # mdadm --assemble --scan
>>> mdadm: Failed to restore critical section for reshape, sorry.
>>> Possibly you needed to specify the --backup-file
>>>
>>> Thanks for helping me out of this...
>>
>> Have you got the latest mdadm? See the "stuck reshape operation" thread
>> for details.
>>
>> You could also try adding the --invalid-backup option to mdadm. It
>> shouldn't need a backup if you're adding more disk, though, so that
>> looks slightly odd to me.
>>
>> Beyond those tips I'll not stray - my raid-fu is not the best and while
>> I'm confident of this information I don't believe in encouraging others
>> to take risks. If those don't work I'll let a better expert than me
>> chime in.
>>
>> Cheers,
>> Wol
>
>
^ permalink raw reply
* Re: Help needed: array inactive after grow attempt
From: Adam Goryachev @ 2016-04-28 22:43 UTC (permalink / raw)
To: Andread Mayrhoff, linux-raid
In-Reply-To: <57228AD7.1050408@youngman.org.uk>
On 29/04/16 08:12, Wols Lists wrote:
> On 28/04/16 20:50, Andread Mayrhoff wrote:
>> I might add
>>
>> ~/tmp # mdadm --stop /dev/md127
>> mdadm: stopped /dev/md127
>> ~/tmp # mdadm --assemble --scan
>> mdadm: Failed to restore critical section for reshape, sorry.
>> Possibly you needed to specify the --backup-file
>>
>> Thanks for helping me out of this...
> Have you got the latest mdadm? See the "stuck reshape operation" thread
> for details.
>
> You could also try adding the --invalid-backup option to mdadm. It
> shouldn't need a backup if you're adding more disk, though, so that
> looks slightly odd to me.
>
> Beyond those tips I'll not stray - my raid-fu is not the best and while
> I'm confident of this information I don't believe in encouraging others
> to take risks. If those don't work I'll let a better expert than me
> chime in.
>
> Cheers,
> Wol
My suggestion (which isn't really a fix) is to take a quick look at disk
overlays, that will allow you to "play" with your disks without damaging
them, and retaining the ability to recover your data.
Once you do that, I think I've seen people use a combination of
--invalid-backup and --force to solve this kind of problem. The
interesting part would be to know if the reshape ever actually
started/made progress. Usually people are reporting that it sits on 0%
complete.
Hope that helps.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: Help needed: array inactive after grow attempt
From: Wols Lists @ 2016-04-28 22:12 UTC (permalink / raw)
To: Andread Mayrhoff, linux-raid
In-Reply-To: <7cfbdd22b8c20acdb95fababfa1a04a6@spoilbase.eu>
On 28/04/16 20:50, Andread Mayrhoff wrote:
> I might add
>
> ~/tmp # mdadm --stop /dev/md127
> mdadm: stopped /dev/md127
> ~/tmp # mdadm --assemble --scan
> mdadm: Failed to restore critical section for reshape, sorry.
> Possibly you needed to specify the --backup-file
>
> Thanks for helping me out of this...
Have you got the latest mdadm? See the "stuck reshape operation" thread
for details.
You could also try adding the --invalid-backup option to mdadm. It
shouldn't need a backup if you're adding more disk, though, so that
looks slightly odd to me.
Beyond those tips I'll not stray - my raid-fu is not the best and while
I'm confident of this information I don't believe in encouraging others
to take risks. If those don't work I'll let a better expert than me
chime in.
Cheers,
Wol
^ permalink raw reply
* Re: mdadm - stuck reshape operation
From: Peter Bates @ 2016-04-28 21:31 UTC (permalink / raw)
To: John Stoffel; +Cc: linux-raid
In-Reply-To: <22305.30340.565924.810276@quad.stoffel.home>
Thanks for the suggestions John.
I'm not running an SELinux setup. I did notice some of the Kernel
security settings enabled that I would never use so I've removed
those.
Dumping the drives to /dev/null didn't produce any errors.
During the whole process I haven't seen any disk level errors in dmesg
or syslog.
Here is a pastbin of an strace on the mdadm assemble command.
Probably showing my ignorance, but I can't strace the md6_raid kernel
thread can I?
http://pastebin.com/5q0K6w6r
Will upgrade mdadm over the weekend.and try increasing the dmesg log level to 7.
Peter Bates
peter.thebates@gmail.com
On 28 April 2016 at 12:33, John Stoffel <john@stoffel.org> wrote:
>>>>>> "Peter" == Peter Bates <peter.thebates@gmail.com> writes:
>
>
> Peter> I have a 3 disk RAID 5 array that I tried to add a 4th disk to.
>
>>> mdadm --add /dev/md6 /dev/sdb1
>>> mdadm --grow --raid-devices=4 /dev/md6
>
> Peter> This operation started successfully and proceeded until it hit 51.1%
>
>>> cat /proc/mdstat
> Peter> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
> Peter> [raid4] [multipath] [faulty]
> Peter> md6 : active raid5 sda1[0] sdb1[5] sdf1[3] sde1[4]
> Peter> 3906764800 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
> Peter> [==========>..........] reshape = 51.1% (998533632/1953382400)
> Peter> finish=9046506.1min speed=1K/sec
> Peter> bitmap: 0/15 pages [0KB], 65536KB chunk
>
> Peter> It has been sitting on the same 998533632 position for
> Peter> days. I've tried a few reboots, but it never progresses.
> Peter> Stopping the array, or trying to start the logical volume in it
> Peter> hangs. Altering the min / max speed parameters has no effect.
> Peter> When I reboot and resemble the array the speed indicated
> Peter> steadily drops to almost 0.
>
>>> mdadm --assemble /dev/md6 --verbose --uuid 90c2b5c3:3bbfa0d7:a5efaeed:726c43e2
>
> I looked back in my email archives, and I wonder if maybe you have
> SElinux enabled? If so, please turn it off and see if that helps.
>
> What happens when you use dd on each of the drives and dump the output
> to /dev/null?
>
> Are there any messages in the logs, or dmesg output after the stuff
> you showed? Can you maybe 'strace' the mdadm process, or even go grab
> the latest version using git from:
>
> git clone git://neil.brown.name/mdadm
>
> And see if compiling it yourself from the master might do the trick.
>
>
> Peter> I haven't tried anything more drastic than a reboot yet,
> Peter> Below is as much information as I can think to provide at this stage.
> Peter> Please let me know what else I can do.
> Peter> I'm happy to change kernels, kernel config or anything else require to
> Peter> get better info.
>
> Peter> Kernel: 4.4.3
> Peter> mdadm 3.4
>
>>> ps aux | grep md6
> Peter> root 5041 99.9 0.0 0 0 ? R 07:10 761:58 [md6_raid5]
> Peter> root 5042 0.0 0.0 0 0 ? D 07:10 0:00 [md6_reshape]
>
> Peter> This is consistent. 100% cpu on the raid component, but not the reshape
>
>>> mdadm --detail --verbose /dev/md6
> Peter> /dev/md6:
> Peter> Version : 1.2
> Peter> Creation Time : Fri Aug 29 21:13:52 2014
> Peter> Raid Level : raid5
> Peter> Array Size : 3906764800 (3725.78 GiB 4000.53 GB)
> Peter> Used Dev Size : 1953382400 (1862.89 GiB 2000.26 GB)
> Peter> Raid Devices : 4
> Peter> Total Devices : 4
> Peter> Persistence : Superblock is persistent
>
> Peter> Intent Bitmap : Internal
>
> Peter> Update Time : Wed Apr 27 07:10:07 2016
> Peter> State : clean, reshaping
> Peter> Active Devices : 4
> Peter> Working Devices : 4
> Peter> Failed Devices : 0
> Peter> Spare Devices : 0
>
> Peter> Layout : left-symmetric
> Peter> Chunk Size : 512K
>
> Peter> Reshape Status : 51% complete
> Peter> Delta Devices : 1, (3->4)
>
> Peter> Name : Alpheus:6 (local to host Alpheus)
> Peter> UUID : 90c2b5c3:3bbfa0d7:a5efaeed:726c43e2
> Peter> Events : 47975
>
> Peter> Number Major Minor RaidDevice State
> Peter> 0 8 1 0 active sync /dev/sda1
> Peter> 4 8 65 1 active sync /dev/sde1
> Peter> 3 8 81 2 active sync /dev/sdf1
> Peter> 5 8 17 3 active sync /dev/sdb1
>
>>> iostat
> Peter> Linux 4.4.3-gentoo (Alpheus) 04/27/2016 _x86_64_ (4 CPU)
>
> Peter> avg-cpu: %user %nice %system %iowait %steal %idle
> Peter> 1.84 0.00 24.50 0.09 0.00 73.57
>
> Peter> Looking at the individual disks I can see minor activity on the MD6
> Peter> members. This activity tends to match up with the overall rate
> Peter> reported by /proc/mdstat
>
> Peter> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
> Peter> sda 0.02 2.72 1.69 128570 79957
> Peter> sdb 0.01 0.03 1.69 1447 79889
> Peter> sdd 3.85 2.27 56.08 106928 2646042
> Peter> sde 0.02 2.73 1.69 128610 79961
> Peter> sdf 0.02 2.72 1.69 128128 79961
> Peter> sdc 4.08 5.44 56.08 256899 2646042
> Peter> md0 2.91 7.62 55.08 359714 2598725
> Peter> dm-0 0.00 0.03 0.00 1212 0
> Peter> dm-1 0.00 0.05 0.00 2151 9
> Peter> dm-2 2.65 6.52 3.42 307646 161296
> Peter> dm-3 0.19 1.03 51.66 48377 2437420
> Peter> md6 0.00 0.02 0.00 1036 0
>
>>> dmesg
> Peter> [ 1199.426995] md: bind<sde1>
> Peter> [ 1199.427779] md: bind<sdf1>
> Peter> [ 1199.428379] md: bind<sdb1>
> Peter> [ 1199.428592] md: bind<sda1>
> Peter> [ 1199.429260] md/raid:md6: reshape will continue
> Peter> [ 1199.429274] md/raid:md6: device sda1 operational as raid disk 0
> Peter> [ 1199.429275] md/raid:md6: device sdb1 operational as raid disk 3
> Peter> [ 1199.429276] md/raid:md6: device sdf1 operational as raid disk 2
> Peter> [ 1199.429277] md/raid:md6: device sde1 operational as raid disk 1
> Peter> [ 1199.429498] md/raid:md6: allocated 4338kB
> Peter> [ 1199.429807] md/raid:md6: raid level 5 active with 4 out of 4
> Peter> devices, algorithm 2
> Peter> [ 1199.429810] RAID conf printout:
> Peter> [ 1199.429811] --- level:5 rd:4 wd:4
> Peter> [ 1199.429812] disk 0, o:1, dev:sda1
> Peter> [ 1199.429814] disk 1, o:1, dev:sde1
> Peter> [ 1199.429816] disk 2, o:1, dev:sdf1
> Peter> [ 1199.429817] disk 3, o:1, dev:sdb1
> Peter> [ 1199.429993] created bitmap (15 pages) for device md6
> Peter> [ 1199.430297] md6: bitmap initialized from disk: read 1 pages, set 0
> Peter> of 29807 bits
> Peter> [ 1199.474604] md6: detected capacity change from 0 to 4000527155200
> Peter> [ 1199.474611] md: reshape of RAID array md6
> Peter> [ 1199.474613] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> Peter> [ 1199.474614] md: using maximum available idle IO bandwidth (but not
> Peter> more than 200000 KB/sec) for reshape.
> Peter> [ 1199.474617] md: using 128k window, over a total of 1953382400k.
>
>>> lsblk
> Peter> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> Peter> sda 8:0 0 1.8T 0 disk
> Peter> └─sda1 8:1 0 1.8T 0 part
> Peter> └─md6 9:6 0 3.7T 0 raid5
> Peter> sdb 8:16 0 1.8T 0 disk
> Peter> └─sdb1 8:17 0 1.8T 0 part
> Peter> └─md6 9:6 0 3.7T 0 raid5
> Peter> sdc 8:32 0 2.7T 0 disk
> Peter> ├─sdc1 8:33 0 16M 0 part
> Peter> └─sdc2 8:34 0 2.7T 0 part
> Peter> └─md0 9:0 0 2.7T 0 raid1
> Peter> ├─vg--mirror-swap 253:0 0 4G 0 lvm [SWAP]
> Peter> ├─vg--mirror-boot 253:1 0 256M 0 lvm /boot
> Peter> ├─vg--mirror-root 253:2 0 256G 0 lvm /
> Peter> └─vg--mirror-data--mirror 253:3 0 2.5T 0 lvm /data/mirror
> Peter> sdd 8:48 0 2.7T 0 disk
> Peter> ├─sdd1 8:49 0 16M 0 part
> Peter> └─sdd2 8:50 0 2.7T 0 part
> Peter> └─md0 9:0 0 2.7T 0 raid1
> Peter> ├─vg--mirror-swap 253:0 0 4G 0 lvm [SWAP]
> Peter> ├─vg--mirror-boot 253:1 0 256M 0 lvm /boot
> Peter> ├─vg--mirror-root 253:2 0 256G 0 lvm /
> Peter> └─vg--mirror-data--mirror 253:3 0 2.5T 0 lvm /data/mirror
> Peter> sde 8:64 0 1.8T 0 disk
> Peter> └─sde1 8:65 0 1.8T 0 part
> Peter> └─md6 9:6 0 3.7T 0 raid5
> Peter> sdf 8:80 0 1.8T 0 disk
> Peter> └─sdf1 8:81 0 1.8T 0 part
> Peter> └─md6 9:6 0 3.7T 0 raid5
>
> Peter> Thanks for any pointers
>
> Peter> Peter Bates
> Peter> peter.thebates@gmail.com
> Peter> --
> Peter> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> Peter> the body of a message to majordomo@vger.kernel.org
> Peter> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] MD: make bio mergeable
From: Shaohua Li @ 2016-04-28 21:19 UTC (permalink / raw)
To: Holger Kiehl
Cc: linux-block, linux-kernel, linux-raid, qkrwngud825, Kernel-team,
Ming Lei, Jens Axboe, Neil Brown
In-Reply-To: <alpine.LRH.2.20.1604281933200.31165@diagnostix.dwd.de>
On Thu, Apr 28, 2016 at 08:00:22PM +0000, Holger Kiehl wrote:
> Hello,
>
> On Mon, 25 Apr 2016, Shaohua Li wrote:
>
> > blk_queue_split marks bio unmergeable, which makes sense for normal bio.
> > But if dispatching the bio to underlayer disk, the blk_queue_split
> > checks are invalid, hence it's possible the bio becomes mergeable.
> >
> > In the reported bug, this bug causes trim against raid0 performance slash
> > https://bugzilla.kernel.org/show_bug.cgi?id=117051
> >
> This patch makes a huge difference. On a system with two Samsung 850 Pro
> in a MD Raid0 setup the time for fstrim went down from ~30min to 18sec!
>
> However, on another system with two Intel P3700 1.6TB NVMe PCIe SSD's
> also setup as one big MD Raid0, the patch does not make any difference
> at all. fstrim takes more then 4 hours!
Does the raid0 cross two partitions or two SSD?
can you post blktrace data in the bugzilloa, I'll track the bug there.
Thanks,
Shaohua
^ permalink raw reply
* Re: Help needed: array inactive after grow attempt
From: Andread Mayrhoff @ 2016-04-28 20:02 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <7cfbdd22b8c20acdb95fababfa1a04a6@spoilbase.eu>
Some more info:
/dev/sdb1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : b120f4d9:d1ba5648:e1359c5d:7a36372e
Name : n54l:raid5.11111111.eu
Creation Time : Sat Aug 31 10:09:28 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 5859448552 (2794.00 GiB 3000.04 GB)
Array Size : 14648584960 (13969.98 GiB 15000.15 GB)
Used Dev Size : 5859433984 (2794.00 GiB 3000.03 GB)
Super Offset : 5859448816 sectors
Unused Space : before=0 sectors, after=14816 sectors
State : clean
Device UUID : 1f97ccdb:5d734f5a:69ec1e62:4b767bd1
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Thu Apr 28 19:51:47 2016
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 9cee067c - correct
Events : 60868
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 0
Array State : AAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : b120f4d9:d1ba5648:e1359c5d:7a36372e
Name : n54l:raid5.11111111.eu
Creation Time : Sat Aug 31 10:09:28 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 5859448552 (2794.00 GiB 3000.04 GB)
Array Size : 14648584960 (13969.98 GiB 15000.15 GB)
Used Dev Size : 5859433984 (2794.00 GiB 3000.03 GB)
Super Offset : 5859448816 sectors
Unused Space : before=0 sectors, after=14816 sectors
State : clean
Device UUID : c0d6ef32:c96f2287:c28a747e:3dd7ac5e
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Thu Apr 28 19:51:47 2016
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : ca6b41e2 - correct
Events : 60868
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 1
Array State : AAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : b120f4d9:d1ba5648:e1359c5d:7a36372e
Name : n54l:raid5.11111111.eu
Creation Time : Sat Aug 31 10:09:28 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 5859448552 (2794.00 GiB 3000.04 GB)
Array Size : 14648584960 (13969.98 GiB 15000.15 GB)
Used Dev Size : 5859433984 (2794.00 GiB 3000.03 GB)
Super Offset : 5859448816 sectors
Unused Space : before=0 sectors, after=14816 sectors
State : clean
Device UUID : 935f90f7:4b0dc9d7:f66aa9d6:20d11f62
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Thu Apr 28 19:51:47 2016
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 3b5a4242 - correct
Events : 60868
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 2
Array State : AAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : b120f4d9:d1ba5648:e1359c5d:7a36372e
Name : n54l:raid5.11111111.eu
Creation Time : Sat Aug 31 10:09:28 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 5859448552 (2794.00 GiB 3000.04 GB)
Array Size : 14648584960 (13969.98 GiB 15000.15 GB)
Used Dev Size : 5859433984 (2794.00 GiB 3000.03 GB)
Super Offset : 5859448816 sectors
Unused Space : before=0 sectors, after=14816 sectors
State : clean
Device UUID : 8a681fff:9c9fea74:5fa541e0:d3903bfe
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Thu Apr 28 19:51:47 2016
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 85bed7a3 - correct
Events : 60868
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 3
Array State : AAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : b120f4d9:d1ba5648:e1359c5d:7a36372e
Name : n54l:raid5.11111111.eu
Creation Time : Sat Aug 31 10:09:28 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 5859448552 (2794.00 GiB 3000.04 GB)
Array Size : 14648584960 (13969.98 GiB 15000.15 GB)
Used Dev Size : 5859433984 (2794.00 GiB 3000.03 GB)
Super Offset : 5859448816 sectors
Unused Space : before=0 sectors, after=14816 sectors
State : clean
Device UUID : 7ab63bf4:aaf2b5fb:7e4ced0f:5942004e
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Thu Apr 28 19:51:47 2016
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 8116d149 - correct
Events : 60868
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 4
Array State : AAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : b120f4d9:d1ba5648:e1359c5d:7a36372e
Name : n54l:raid5.11111111.eu
Creation Time : Sat Aug 31 10:09:28 2013
Raid Level : raid5
Raid Devices : 6
Avail Dev Size : 5859448552 (2794.00 GiB 3000.04 GB)
Array Size : 14648584960 (13969.98 GiB 15000.15 GB)
Used Dev Size : 5859433984 (2794.00 GiB 3000.03 GB)
Super Offset : 5859448816 sectors
Unused Space : before=0 sectors, after=14816 sectors
State : clean
Device UUID : 49b2e0e8:dff3e0c7:15184780:0ff9f26d
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (5->6)
Update Time : Thu Apr 28 19:51:47 2016
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : d233509b - correct
Events : 60868
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 5
Array State : AAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
> I might add
>
> ~/tmp # mdadm --stop /dev/md127
> mdadm: stopped /dev/md127
> ~/tmp # mdadm --assemble --scan
> mdadm: Failed to restore critical section for reshape, sorry.
> Possibly you needed to specify the --backup-file
>
> Thanks for helping me out of this...
>
>> Hello,
>>
>> I believe I need some help from an mdadm expert.
>>
>> I tried to add a disk to an existing RAID5-array (/dev/md127) that
>> consisted of 5 disks: /dev/sd[bcdef]1
>> The system is a 4.5.2-2-kernel, x86_64-architecture, mdadm 3.3.1.
>> I attempted to add partition /dev/sdg1 to that array, in an attempt to
>> create a 6-disk-RAID5-array.
>> To achieve that goal, I typed "mdadm --add /dev/md127 /dev/sdg1".
>> As I found that working, I attempted to grow the array by typing
>> "mdadm --grow /dev/md127 --raid-devices=6".
>>
>> I left my machine, and when I returned, I found it had been switched
>> off by my... ahem, anyway, it had been switched off.
>>
>> I powered it on again, "cat /proc/mdstat" returned
>>
>>>>
>> Personalities : [raid6] [raid5] [raid4]
>> md127 : inactive sdg1[9](S) sdf1[8](S) sdc1[5](S) sdb1[6](S)
>> sde1[4](S) sdd1[7](S)
>> 17578345656 blocks super 1.0
>>
>> unused devices: <none>
>> <<
>>
>> mdadm --detail /dev/md127 returns
>>
>>>>
>> /dev/md127:
>> Version : 1.0
>> Raid Level : raid0
>> Total Devices : 6
>> Persistence : Superblock is persistent
>>
>> State : inactive
>>
>> Delta Devices : 1, (-1->0)
>> New Level : raid5
>> New Layout : left-symmetric
>> New Chunksize : 128K
>>
>> Name : n54l:raid5.11111111.eu
>> UUID : b120f4d9:d1ba5648:e1359c5d:7a36372e
>> Events : 60868
>>
>> Number Major Minor RaidDevice
>>
>> - 8 17 - /dev/sdb1
>> - 8 33 - /dev/sdc1
>> - 8 49 - /dev/sdd1
>> - 8 65 - /dev/sde1
>> - 8 81 - /dev/sdf1
>> - 8 97 - /dev/sdg1
>> <<
>>
>> Now, before I do anything extreme like unplugging that disk again or
>> forcing a recreate, could an expert suggest the next logical step I
>> should do. I know this sounds pretty defensive, but rather than
>> playing "Raid-hero with an erased set of disks" I'd go along with
>> "Raid-idiot that needed experts' advice which is why he still's got
>> all his data".
>>
>> Thanks for your advice!
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid"
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] MD: make bio mergeable
From: Holger Kiehl @ 2016-04-28 20:00 UTC (permalink / raw)
To: Shaohua Li
Cc: linux-block, linux-kernel, linux-raid, qkrwngud825, Kernel-team,
Ming Lei, Jens Axboe, Neil Brown
In-Reply-To: <384a0e0c7d6f2700aadbcbdef003cece88fa7dd7.1461626533.git.shli@fb.com>
Hello,
On Mon, 25 Apr 2016, Shaohua Li wrote:
> blk_queue_split marks bio unmergeable, which makes sense for normal bio.
> But if dispatching the bio to underlayer disk, the blk_queue_split
> checks are invalid, hence it's possible the bio becomes mergeable.
>
> In the reported bug, this bug causes trim against raid0 performance slash
> https://bugzilla.kernel.org/show_bug.cgi?id=117051
>
This patch makes a huge difference. On a system with two Samsung 850 Pro
in a MD Raid0 setup the time for fstrim went down from ~30min to 18sec!
However, on another system with two Intel P3700 1.6TB NVMe PCIe SSD's
also setup as one big MD Raid0, the patch does not make any difference
at all. fstrim takes more then 4 hours!
Any idea what could be wrong?
Regards,
Holger
> Reported-by: Park Ju Hyung <qkrwngud825@gmail.com>
> Fixes: 6ac45aeb6bca(block: avoid to merge splitted bio)
> Cc: stable@vger.kernel.org (v4.3+)
> Cc: Ming Lei <ming.lei@canonical.com>
> Cc: Jens Axboe <axboe@fb.com>
> Cc: Neil Brown <neilb@suse.de>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/md.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 194580f..14d3b37 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -284,6 +284,8 @@ static blk_qc_t md_make_request(struct request_queue *q, struct bio *bio)
> * go away inside make_request
> */
> sectors = bio_sectors(bio);
> + /* bio could be mergeable after passing to underlayer */
> + bio->bi_rw &= ~REQ_NOMERGE;
> mddev->pers->make_request(mddev, bio);
>
> cpu = part_stat_lock();
> --
> 2.8.0.rc2
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: Help needed: array inactive after grow attempt
From: Andread Mayrhoff @ 2016-04-28 19:50 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1d5a8c36a6802a9e7e32eb056165f200@spoilbase.eu>
I might add
~/tmp # mdadm --stop /dev/md127
mdadm: stopped /dev/md127
~/tmp # mdadm --assemble --scan
mdadm: Failed to restore critical section for reshape, sorry.
Possibly you needed to specify the --backup-file
Thanks for helping me out of this...
Am 2016-04-28 21:27, schrieb Andread Mayrhoff:
> Hello,
>
> I believe I need some help from an mdadm expert.
>
> I tried to add a disk to an existing RAID5-array (/dev/md127) that
> consisted of 5 disks: /dev/sd[bcdef]1
> The system is a 4.5.2-2-kernel, x86_64-architecture, mdadm 3.3.1.
> I attempted to add partition /dev/sdg1 to that array, in an attempt to
> create a 6-disk-RAID5-array.
> To achieve that goal, I typed "mdadm --add /dev/md127 /dev/sdg1".
> As I found that working, I attempted to grow the array by typing
> "mdadm --grow /dev/md127 --raid-devices=6".
>
> I left my machine, and when I returned, I found it had been switched
> off by my... ahem, anyway, it had been switched off.
>
> I powered it on again, "cat /proc/mdstat" returned
>
>>>
> Personalities : [raid6] [raid5] [raid4]
> md127 : inactive sdg1[9](S) sdf1[8](S) sdc1[5](S) sdb1[6](S)
> sde1[4](S) sdd1[7](S)
> 17578345656 blocks super 1.0
>
> unused devices: <none>
> <<
>
> mdadm --detail /dev/md127 returns
>
>>>
> /dev/md127:
> Version : 1.0
> Raid Level : raid0
> Total Devices : 6
> Persistence : Superblock is persistent
>
> State : inactive
>
> Delta Devices : 1, (-1->0)
> New Level : raid5
> New Layout : left-symmetric
> New Chunksize : 128K
>
> Name : n54l:raid5.11111111.eu
> UUID : b120f4d9:d1ba5648:e1359c5d:7a36372e
> Events : 60868
>
> Number Major Minor RaidDevice
>
> - 8 17 - /dev/sdb1
> - 8 33 - /dev/sdc1
> - 8 49 - /dev/sdd1
> - 8 65 - /dev/sde1
> - 8 81 - /dev/sdf1
> - 8 97 - /dev/sdg1
> <<
>
> Now, before I do anything extreme like unplugging that disk again or
> forcing a recreate, could an expert suggest the next logical step I
> should do. I know this sounds pretty defensive, but rather than
> playing "Raid-hero with an erased set of disks" I'd go along with
> "Raid-idiot that needed experts' advice which is why he still's got
> all his data".
>
> Thanks for your advice!
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox