* Re: The subarray is loaded container by load_container
From: Artur Paszkiewicz @ 2016-08-09 8:05 UTC (permalink / raw)
To: Jes Sorensen
Cc: Xiao Ni, linux-raid, tomasz.majchrzak, aleksey.obitotskiy,
pawel.baldysiak
In-Reply-To: <wrfjshuk1irn.fsf@redhat.com>
On 08/04/2016 08:38 PM, Jes Sorensen wrote:
> Do you have a version of this patch you would like me to apply to the
> official mdadm tree?
Hi Jes,
Yes I do, I'll send the patch in a moment.
Artur
^ permalink raw reply
* Re: No '--add-journal' helper page in "man mdadm" and "mdadm --manage --help"
From: Song Liu @ 2016-08-08 16:19 UTC (permalink / raw)
To: Yi Zhang, linux-raid@vger.kernel.org; +Cc: Shaohua Li, Jes.Sorensen@redhat.com
In-Reply-To: <522591825.11267578.1470309562558.JavaMail.zimbra@redhat.com>
Hi Yi,
Thanks for catching the miss. I will soon send patch for the man page entry.
Song
>> On 8/4/16, 4:19 AM, "Yi Zhang" <yizhan@redhat.com> wrote:
Hello shaohua/Song
Since write-journal has been integrated in mdadm/kernel, I found still no --add-journal usage on related pages, do you plan to add it?
Best Regards,
Yi Zhang
^ permalink raw reply
* [PATCH 2/2] MD: make sure raid5-cache superblock write with reconfig_mutex hold
From: shli @ 2016-08-08 15:37 UTC (permalink / raw)
To: linux-raid; +Cc: yizhan, gqjiang, neilb, Shaohua Li
In-Reply-To: <2a17e29eea7636568f6327f4de5fbe90b861fb04.1470670241.git.shli@fb.com>
From: Shaohua Li <shli@fb.com>
The raid5-cache .quiesce assumes it's called with reconfig_mutex hold. It's
true at most time but there are some exceptions. Since raid5-cache .quiesce
could write superblock, it must be called with reconfig_mutex hold. We already
have facility in .quiesce to indicate if it's called with the mutex hold. This
patch makes raid5-cache .quiesce use the facility.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/raid5-cache.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 551158c..e4070d2 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -97,6 +97,7 @@ struct r5l_log {
bool need_cache_flush;
bool in_teardown;
+ bool reconfig_mutex_locked;
};
/*
@@ -712,7 +713,7 @@ static void r5l_write_super_and_discard_space(struct r5l_log *log,
* md_check_recovery() fails, so the PENDING never get cleared. The
* in_teardown check workaround this issue.
*/
- if (!log->in_teardown) {
+ if (!log->in_teardown || !log->reconfig_mutex_locked) {
set_mask_bits(&mddev->flags, 0,
BIT(MD_CHANGE_DEVS) | BIT(MD_CHANGE_PENDING));
md_wakeup_thread(mddev->thread);
@@ -826,6 +827,7 @@ void r5l_quiesce(struct r5l_log *log, int state, bool reconfig_mutex_locked)
struct mddev *mddev;
if (!log || state == 2)
return;
+ log->reconfig_mutex_locked = reconfig_mutex_locked;
if (state == 0) {
log->in_teardown = 0;
/*
@@ -850,6 +852,7 @@ void r5l_quiesce(struct r5l_log *log, int state, bool reconfig_mutex_locked)
md_unregister_thread(&log->reclaim_thread);
r5l_do_reclaim(log);
}
+ log->reconfig_mutex_locked = false;
}
bool r5l_log_disk_error(struct r5conf *conf)
--
2.7.4
^ permalink raw reply related
* [PATCH 1/2] MD: add extra parameter for .quiesce
From: shli @ 2016-08-08 15:37 UTC (permalink / raw)
To: linux-raid; +Cc: yizhan, gqjiang, neilb, Shaohua Li
From: Shaohua Li <shli@fb.com>
Add extra parameter to indicate if .quiesce is called with reconfig_mutex hold.
The raid5-cache .quiesce will use the parameter to avoid a deadlock.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/bitmap.c | 12 ++++++------
drivers/md/linear.c | 3 ++-
drivers/md/md-cluster.c | 8 ++++----
drivers/md/md.c | 36 ++++++++++++++++++------------------
drivers/md/md.h | 2 +-
drivers/md/raid0.c | 3 ++-
drivers/md/raid1.c | 3 ++-
drivers/md/raid10.c | 3 ++-
drivers/md/raid5-cache.c | 2 +-
drivers/md/raid5.c | 5 +++--
drivers/md/raid5.h | 3 ++-
11 files changed, 43 insertions(+), 37 deletions(-)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index 13041ee..94cb895 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -2044,7 +2044,7 @@ int bitmap_resize(struct bitmap *bitmap, sector_t blocks,
}
if (!init)
- bitmap->mddev->pers->quiesce(bitmap->mddev, 1);
+ bitmap->mddev->pers->quiesce(bitmap->mddev, 1, true);
store.file = bitmap->storage.file;
bitmap->storage.file = NULL;
@@ -2158,7 +2158,7 @@ int bitmap_resize(struct bitmap *bitmap, sector_t blocks,
if (!init) {
bitmap_unplug(bitmap);
- bitmap->mddev->pers->quiesce(bitmap->mddev, 0);
+ bitmap->mddev->pers->quiesce(bitmap->mddev, 0, true);
}
ret = 0;
err:
@@ -2207,9 +2207,9 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
goto out;
}
if (mddev->pers) {
- mddev->pers->quiesce(mddev, 1);
+ mddev->pers->quiesce(mddev, 1, true);
bitmap_destroy(mddev);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 0, true);
}
mddev->bitmap_info.offset = 0;
if (mddev->bitmap_info.file) {
@@ -2246,7 +2246,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
mddev->bitmap_info.offset = offset;
if (mddev->pers) {
struct bitmap *bitmap;
- mddev->pers->quiesce(mddev, 1);
+ mddev->pers->quiesce(mddev, 1, true);
bitmap = bitmap_create(mddev, -1);
if (IS_ERR(bitmap))
rv = PTR_ERR(bitmap);
@@ -2256,7 +2256,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
if (rv)
mddev->bitmap_info.offset = 0;
}
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 0, true);
if (rv) {
bitmap_destroy(mddev);
goto out;
diff --git a/drivers/md/linear.c b/drivers/md/linear.c
index 70ff888..25cb10f 100644
--- a/drivers/md/linear.c
+++ b/drivers/md/linear.c
@@ -279,7 +279,8 @@ static void linear_status (struct seq_file *seq, struct mddev *mddev)
seq_printf(seq, " %dk rounding", mddev->chunk_sectors / 2);
}
-static void linear_quiesce(struct mddev *mddev, int state)
+static void linear_quiesce(struct mddev *mddev, int state,
+ bool reconfig_mutex_locked)
{
}
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 41573f1..d9a71c5 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -406,7 +406,7 @@ static void remove_suspend_info(struct mddev *mddev, int slot)
spin_lock_irq(&cinfo->suspend_lock);
__remove_suspend_info(cinfo, slot);
spin_unlock_irq(&cinfo->suspend_lock);
- mddev->pers->quiesce(mddev, 2);
+ mddev->pers->quiesce(mddev, 2, false);
}
@@ -452,14 +452,14 @@ static void process_suspend_info(struct mddev *mddev,
s->slot = slot;
s->lo = lo;
s->hi = hi;
- mddev->pers->quiesce(mddev, 1);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 1, false);
+ mddev->pers->quiesce(mddev, 0, false);
spin_lock_irq(&cinfo->suspend_lock);
/* Remove existing entry (if exists) before adding */
__remove_suspend_info(cinfo, slot);
list_add(&s->list, &cinfo->suspend_list);
spin_unlock_irq(&cinfo->suspend_lock);
- mddev->pers->quiesce(mddev, 2);
+ mddev->pers->quiesce(mddev, 2, false);
}
static void process_add_new_disk(struct mddev *mddev, struct cluster_msg *cmsg)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 2c3ab6f..ac151b9 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -312,7 +312,7 @@ void mddev_suspend(struct mddev *mddev)
return;
synchronize_rcu();
wait_event(mddev->sb_wait, atomic_read(&mddev->active_io) == 0);
- mddev->pers->quiesce(mddev, 1);
+ mddev->pers->quiesce(mddev, 1, true);
del_timer_sync(&mddev->safemode_timer);
}
@@ -323,7 +323,7 @@ void mddev_resume(struct mddev *mddev)
if (--mddev->suspended)
return;
wake_up(&mddev->sb_wait);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 0, true);
set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
md_wakeup_thread(mddev->thread);
@@ -4664,11 +4664,11 @@ suspend_lo_store(struct mddev *mddev, const char *buf, size_t len)
mddev->suspend_lo = new;
if (new >= old)
/* Shrinking suspended region */
- mddev->pers->quiesce(mddev, 2);
+ mddev->pers->quiesce(mddev, 2, true);
else {
/* Expanding suspended region - need to wait */
- mddev->pers->quiesce(mddev, 1);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 1, true);
+ mddev->pers->quiesce(mddev, 0, true);
}
err = 0;
unlock:
@@ -4707,11 +4707,11 @@ suspend_hi_store(struct mddev *mddev, const char *buf, size_t len)
mddev->suspend_hi = new;
if (new <= old)
/* Shrinking suspended region */
- mddev->pers->quiesce(mddev, 2);
+ mddev->pers->quiesce(mddev, 2, true);
else {
/* Expanding suspended region - need to wait */
- mddev->pers->quiesce(mddev, 1);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 1, true);
+ mddev->pers->quiesce(mddev, 0, true);
}
err = 0;
unlock:
@@ -5509,8 +5509,8 @@ static void mddev_detach(struct mddev *mddev)
atomic_read(&bitmap->behind_writes) == 0);
}
if (mddev->pers && mddev->pers->quiesce) {
- mddev->pers->quiesce(mddev, 1);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 1, true);
+ mddev->pers->quiesce(mddev, 0, true);
}
md_unregister_thread(&mddev->thread);
if (mddev->queue)
@@ -6321,7 +6321,7 @@ static int set_bitmap_file(struct mddev *mddev, int fd)
return -ENOENT; /* cannot remove what isn't there */
err = 0;
if (mddev->pers) {
- mddev->pers->quiesce(mddev, 1);
+ mddev->pers->quiesce(mddev, 1, true);
if (fd >= 0) {
struct bitmap *bitmap;
@@ -6336,7 +6336,7 @@ static int set_bitmap_file(struct mddev *mddev, int fd)
bitmap_destroy(mddev);
fd = -1; /* make sure to put the file */
}
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 0, true);
}
if (fd < 0) {
struct file *f = mddev->bitmap_info.file;
@@ -6617,7 +6617,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
mddev->bitmap_info.default_offset;
mddev->bitmap_info.space =
mddev->bitmap_info.default_space;
- mddev->pers->quiesce(mddev, 1);
+ mddev->pers->quiesce(mddev, 1, true);
bitmap = bitmap_create(mddev, -1);
if (!IS_ERR(bitmap)) {
mddev->bitmap = bitmap;
@@ -6626,7 +6626,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
rv = PTR_ERR(bitmap);
if (rv)
bitmap_destroy(mddev);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 0, true);
} else {
/* remove the bitmap */
if (!mddev->bitmap) {
@@ -6650,9 +6650,9 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
mddev->bitmap_info.nodes = 0;
md_cluster_ops->leave(mddev);
}
- mddev->pers->quiesce(mddev, 1);
+ mddev->pers->quiesce(mddev, 1, true);
bitmap_destroy(mddev);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 0, true);
mddev->bitmap_info.offset = 0;
}
}
@@ -7945,8 +7945,8 @@ void md_do_sync(struct md_thread *thread)
* region.
*/
if (mddev->bitmap) {
- mddev->pers->quiesce(mddev, 1);
- mddev->pers->quiesce(mddev, 0);
+ mddev->pers->quiesce(mddev, 1, false);
+ mddev->pers->quiesce(mddev, 0, false);
}
}
diff --git a/drivers/md/md.h b/drivers/md/md.h
index 20c6675..5035ffd 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -500,7 +500,7 @@ struct md_personality
* 1 - no new requests allowed
* others - reserved
*/
- void (*quiesce) (struct mddev *mddev, int state);
+ void (*quiesce) (struct mddev *mddev, int state, bool reconfig_mutex_locked);
/* takeover is used to transition an array from one
* personality to another. The new personality must be able
* to handle the data in the current layout.
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index c3d4390..faa56c0 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -658,7 +658,8 @@ static void *raid0_takeover(struct mddev *mddev)
return ERR_PTR(-EINVAL);
}
-static void raid0_quiesce(struct mddev *mddev, int state)
+static void raid0_quiesce(struct mddev *mddev, int state,
+ bool reconfig_mutex_locked)
{
}
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 46168ef..98cc597 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -3135,7 +3135,8 @@ static int raid1_reshape(struct mddev *mddev)
return 0;
}
-static void raid1_quiesce(struct mddev *mddev, int state)
+static void raid1_quiesce(struct mddev *mddev, int state,
+ bool reconfig_mutex_locked)
{
struct r1conf *conf = mddev->private;
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 1a632a8..6d1006d 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -3773,7 +3773,8 @@ static void raid10_free(struct mddev *mddev, void *priv)
kfree(conf);
}
-static void raid10_quiesce(struct mddev *mddev, int state)
+static void raid10_quiesce(struct mddev *mddev, int state,
+ bool reconfig_mutex_locked)
{
struct r10conf *conf = mddev->private;
diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 5504ce2..551158c 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -821,7 +821,7 @@ static void r5l_wake_reclaim(struct r5l_log *log, sector_t space)
md_wakeup_thread(log->reclaim_thread);
}
-void r5l_quiesce(struct r5l_log *log, int state)
+void r5l_quiesce(struct r5l_log *log, int state, bool reconfig_mutex_locked)
{
struct mddev *mddev;
if (!log || state == 2)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 4f8f524..7949dbf 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -7623,7 +7623,8 @@ static void raid5_finish_reshape(struct mddev *mddev)
}
}
-static void raid5_quiesce(struct mddev *mddev, int state)
+static void raid5_quiesce(struct mddev *mddev, int state,
+ bool reconfig_mutex_locked)
{
struct r5conf *conf = mddev->private;
@@ -7657,7 +7658,7 @@ static void raid5_quiesce(struct mddev *mddev, int state)
unlock_all_device_hash_locks_irq(conf);
break;
}
- r5l_quiesce(conf->log, state);
+ r5l_quiesce(conf->log, state, reconfig_mutex_locked);
}
static void *raid45_takeover_raid0(struct mddev *mddev, int level)
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 517d4b6..ef8eb2e 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -633,6 +633,7 @@ extern void r5l_write_stripe_run(struct r5l_log *log);
extern void r5l_flush_stripe_to_raid(struct r5l_log *log);
extern void r5l_stripe_write_finished(struct stripe_head *sh);
extern int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio);
-extern void r5l_quiesce(struct r5l_log *log, int state);
+extern void r5l_quiesce(struct r5l_log *log, int state,
+ bool reconfig_mutex_locked);
extern bool r5l_log_disk_error(struct r5conf *conf);
#endif
--
2.7.4
^ permalink raw reply related
* [PATCH -next] dm flakey: fix error return code in flakey_ctr()
From: Wei Yongjun @ 2016-08-08 14:09 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Shaohua Li, Vivek Goyal
Cc: Wei Yongjun, dm-devel, linux-raid
Fix to return error code -EINVAL instead of 0(r is overwrote to
0 by dm_read_arg() call), as done elsewhere in this function.
Fixes: e80d1c805a3b ("dm: do not override error code returned
from dm_get_device()")
Signed-off-by: Wei Yongjun <weiyj.lk@gmail.com>
---
drivers/md/dm-flakey.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/md/dm-flakey.c b/drivers/md/dm-flakey.c
index 19db13e..f0cb0fe 100644
--- a/drivers/md/dm-flakey.c
+++ b/drivers/md/dm-flakey.c
@@ -200,11 +200,13 @@ static int flakey_ctr(struct dm_target *ti, unsigned int argc, char **argv)
if (!(fc->up_interval + fc->down_interval)) {
ti->error = "Total (up + down) interval is zero";
+ r = -EINVAL;
goto bad;
}
if (fc->up_interval + fc->down_interval < fc->up_interval) {
ti->error = "Interval overflow";
+ r = -EINVAL;
goto bad;
}
^ permalink raw reply related
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Mikael Abrahamsson @ 2016-08-08 11:40 UTC (permalink / raw)
To: Wols Lists; +Cc: linux-raid
In-Reply-To: <57A7869D.2090203@youngman.org.uk>
On Sun, 7 Aug 2016, Wols Lists wrote:
> Note that I think this dmesg stuff overlaps with the last lot, namely
> the start of this is the tail end of the array starting successfully
> last time.
It helps if you supply output of /proc/mdstat before each operation, and
also adding verbose output to mdadm command.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Wols Lists @ 2016-08-07 19:06 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1608072002400.3593@uplift.swm.pp.se>
On 07/08/16 19:05, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
>
>> I'm guessing it should be "mdadm --grow --raid-devices=3
>> --backup-file=whatever", but I think I've already tried that, and that's
>> what got me into this mess :-)
>
> Try without backup file (it shouldn't be needed when growing under these
> circumstances). If it doesn't start rebuilding after issuing --grow, try
> --continue when the array is actually up and running (don't stop it when
> it's "frozen" and says rebuilding but doesn't actually show any
> rebuilding going on). That has worked for others.
>
> Use latest git version of mdadm.
>
> Please post output from all commands and dmesg if it still fails.
>
Guess what ...
Note that I think this dmesg stuff overlaps with the last lot, namely
the start of this is the tail end of the array starting successfully
last time.
I'm using ./mdadm, so that's the version I downloaded from Neil's
repository yesterday ... I'll investigate the source when I get the
chance (wives!!! :-)
So is this virtualbox and fuse playing up? Dunno ... (this vm is
mono-processor - shouldn't make any difference ...)
Cheers,
Wol
mdadm: /dev/md127 has been started with 2 drives and 2 spares.
kanga:/home/anthony/mdadm # mount /dev/md127 /mnt/testarray/
kanga:/home/anthony/mdadm # ./madm /dev/md127 --grow --raid-devices=3
bash: ./madm: No such file or directory
kanga:/home/anthony/mdadm # ./mdadm /dev/md127 --grow --raid-devices=3
mdadm: Failed to initiate reshape!
kanga:/home/anthony/mdadm # ./mdadm /dev/md127 --grow --continue
kanga:/home/anthony/mdadm #
>
[21390.529578] md/raid:md127: device sdb operational as raid disk 0
[21390.529618] md/raid:md127: device sdc operational as raid disk 1
[21390.531258] md/raid:md127: allocated 2250kB
[21390.534250] md/raid:md127: raid level 5 active with 2 out of 2
devices, algorithm 2
[21390.534259] RAID conf printout:
[21390.534263] --- level:5 rd:2 wd:2
[21390.534269] disk 0, o:1, dev:sdb
[21390.534273] disk 1, o:1, dev:sdc
[21390.534419] md127: detected capacity change from 0 to 8581545984
[21390.539650] RAID conf printout:
[21390.539682] --- level:5 rd:2 wd:2
[21390.539690] disk 0, o:1, dev:sdb
[21390.539694] disk 1, o:1, dev:sdc
[21390.539698] RAID conf printout:
[21390.539701] --- level:5 rd:2 wd:2
[21390.539704] disk 0, o:1, dev:sdb
[21390.539707] disk 1, o:1, dev:sdc
[21435.972237] EXT4-fs (md127): mounted filesystem with ordered data
mode. Opts: (null)
[22157.683863] device-mapper: uevent: version 1.0.3
[22157.686018] device-mapper: ioctl: 4.31.0-ioctl (2015-3-12)
initialised: dm-devel@redhat.com
[22165.688223] fuse init (API version 7.23)
[22243.094448] 06:10:19.795789 control Error: Getting host message
failed with VERR_INTERRUPTED
[22244.094733] 06:10:20.796092 control Guest control service stopped
[22244.094789] 06:10:20.796207 control Guest control worker returned
with rc=VERR_INTERRUPTED
[22244.095871] 06:10:20.797229 main Session 0 is about to close ...
[22244.095922] 06:10:20.797343 main Stopping all guest processes ...
[22244.095956] 06:10:20.797379 main Closing all guest files ...
[22244.099090] 06:10:20.800461 main Ended.
[22244.641855] VBoxService 5.0.24_SUSE r108355 (verbosity: 0)
linux.amd64 (openSUSE Buildservice) release log
00:00:00.000257 main Log opened 2016-08-07T18:07:43.777555000Z
[22244.641971] 00:00:00.000481 main OS Product: Linux
[22244.642016] 00:00:00.000526 main OS Release: 4.1.15-8-default
[22244.642056] 00:00:00.000564 main OS Version: #1 SMP PREEMPT Wed
Jan 20 16:41:00 UTC 2016 (0e3b3ab)
[22244.642206] 00:00:00.000604 main Executable: /usr/sbin/VBoxService
00:00:00.000605 main Process ID: 12443
00:00:00.000606 main Package type: LINUX_64BITS_GENERIC (OSE)
[22244.643245] 00:00:00.001736 main 5.0.24_SUSE r108355 started.
Verbose level = 0
[22567.452608] SFW2-INext-DROP-DEFLT IN=eth0 OUT=
MAC=08:00:27:0b:0f:57:52:54:00:12:35:02:08:00 SRC=10.0.2.2 DST=10.0.2.15
LEN=576 TOS=0x10 PREC=0x00 TTL=64 ID=2252 PROTO=UDP SPT=67 DPT=68 LEN=556
[25196.727211] RAID conf printout:
[25196.727217] --- level:5 rd:2 wd:2
[25196.727238] disk 0, o:1, dev:sdb
[25196.727240] disk 1, o:1, dev:sdc
[25196.727246] RAID conf printout:
[25196.727248] --- level:5 rd:2 wd:2
[25196.727250] disk 0, o:1, dev:sdb
[25196.727251] disk 1, o:1, dev:sdc
anthony@kanga:/mnt/anthony>
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Mikael Abrahamsson @ 2016-08-07 18:05 UTC (permalink / raw)
To: Wols Lists; +Cc: linux-raid
In-Reply-To: <57A776AE.3050308@youngman.org.uk>
On Sun, 7 Aug 2016, Wols Lists wrote:
> I'm guessing it should be "mdadm --grow --raid-devices=3
> --backup-file=whatever", but I think I've already tried that, and that's
> what got me into this mess :-)
Try without backup file (it shouldn't be needed when growing under these
circumstances). If it doesn't start rebuilding after issuing --grow, try
--continue when the array is actually up and running (don't stop it when
it's "frozen" and says rebuilding but doesn't actually show any rebuilding
going on). That has worked for others.
Use latest git version of mdadm.
Please post output from all commands and dmesg if it still fails.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Wols Lists @ 2016-08-07 17:58 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1608071923410.3593@uplift.swm.pp.se>
On 07/08/16 18:24, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
>
>> [17431.752149] md/raid:md127: reshape_position too early for
>> auto-recovery - aborting.
>
> Does the hint in the below URL help?
>
> http://www.spinics.net/lists/raid/msg52319.html
>
> (to use "--update=revert-reshape")
>
Okay - it's worked - as in I've got a working array back - a two-drive
raid 5 array with two spares.
So hwo do I now get it to rebuild successfully to give me a working full
raid 5?
I'm guessing it should be "mdadm --grow --raid-devices=3
--backup-file=whatever", but I think I've already tried that, and that's
what got me into this mess :-)
I'm trying to get access to the linux-raid wiki, so once I've got this
working, and I've got the access, all this stuff is going to be written
up and posted to the wiki :-)
Cheers,
Wol
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Wols Lists @ 2016-08-07 17:50 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1608071923410.3593@uplift.swm.pp.se>
On 07/08/16 18:24, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
>
>> [17431.752149] md/raid:md127: reshape_position too early for
>> auto-recovery - aborting.
>
> Does the hint in the below URL help?
>
> http://www.spinics.net/lists/raid/msg52319.html
>
> (to use "--update=revert-reshape")
>
I have tried it - I'm just looking at the url now to have another go ...
Cheers,
Wol
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Mikael Abrahamsson @ 2016-08-07 17:24 UTC (permalink / raw)
To: Wols Lists; +Cc: linux-raid
In-Reply-To: <57A766DC.4000207@youngman.org.uk>
On Sun, 7 Aug 2016, Wols Lists wrote:
> [17431.752149] md/raid:md127: reshape_position too early for
> auto-recovery - aborting.
Does the hint in the below URL help?
http://www.spinics.net/lists/raid/msg52319.html
(to use "--update=revert-reshape")
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Wols Lists @ 2016-08-07 16:50 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1608071630050.3593@uplift.swm.pp.se>
On 07/08/16 15:31, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
>
> Please --stop it, then --assemble with verbose output and include output
> from that command plus anything in dmesg at the same time. What happens
> when you then try to --run it?
>
Output below ...
kanga:/home/anthony/mdadm # ./mdadm --stop /dev/md127
mdadm: stopped /dev/md127
kanga:/home/anthony/mdadm # mdadm -v --assemble /dev/md127 /dev/sdb
/dev/sdc /dev/sdd /dev/sde
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdd is identified as a member of /dev/md127, slot -1.
mdadm: /dev/sde is identified as a member of /dev/md127, slot 2.
mdadm: added /dev/sdc to /dev/md127 as 1
mdadm: added /dev/sde to /dev/md127 as 2
mdadm: added /dev/sdd to /dev/md127 as -1
mdadm: added /dev/sdb to /dev/md127 as 0
mdadm: failed to RUN_ARRAY /dev/md127: Invalid argument
kanga:/home/anthony/mdadm #
[17372.305472] md: md127 stopped.
[17372.305483] md: unbind<sdc>
[17372.312124] md: export_rdev(sdc)
[17372.312198] md: unbind<sde>
[17372.317792] md: export_rdev(sde)
[17372.317868] md: unbind<sdb>
[17372.320344] md: export_rdev(sdb)
[17431.701768] md: md127 stopped.
[17431.710474] md: bind<sdc>
[17431.711203] md: bind<sde>
[17431.712843] md: bind<sdd>
[17431.713735] md: bind<sdb>
[17431.752149] md/raid:md127: reshape_position too early for
auto-recovery - aborting.
[17431.752155] md: pers->run() failed ...
[17431.752289] md: md127 stopped.
[17431.752297] md: unbind<sdb>
[17431.760544] md: export_rdev(sdb)
[17431.760602] md: unbind<sdd>
[17431.772124] md: export_rdev(sdd)
[17431.772144] md: unbind<sde>
[17431.780125] md: export_rdev(sde)
[17431.780144] md: unbind<sdc>
[17431.792163] md: export_rdev(sdc)
[17431.909807] md: bind<sde>
[17431.942187] md: bind<sdc>
[17431.962809] md: bind<sdd>
[17431.989609] md: bind<sdb>
[17432.026378] md/raid:md127: reshape_position too early for
auto-recovery - aborting.
[17432.026385] md: pers->run() failed ...
anthony@kanga:/mnt/anthony>
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Mikael Abrahamsson @ 2016-08-07 14:31 UTC (permalink / raw)
To: Wols Lists; +Cc: linux-raid
In-Reply-To: <57A724ED.5070808@youngman.org.uk>
On Sun, 7 Aug 2016, Wols Lists wrote:
Please --stop it, then --assemble with verbose output and include output
from that command plus anything in dmesg at the same time. What happens
when you then try to --run it?
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Wols Lists @ 2016-08-07 12:09 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <57A719CC.6080301@youngman.org.uk>
On 07/08/16 12:21, Wols Lists wrote:
> On 07/08/16 10:54, Mikael Abrahamsson wrote:
>> > On Sun, 7 Aug 2016, Wols Lists wrote:
>> >
>>> >> I also can't run the array :-(
>> >
>> > What does "cat /proc/mdstat" say?
> The vm is shut down right now, but it said inactive, four drives,
> [UUU_], iirc. I'll post it when I get it back up. Didn't mention raid
> type, so that could be messed up somewhere.
>
> Two drives are mirrored, it should be rebuilding on to third, and a
> fourth is spare (ready to rebuild into raid6)
/dev/sdd seems to have disappeared - maybe I removed it ... I've been
mucking about a fair bit ...
Note that "mdadm" is the version that came with the system, "./mdadm" is
the latest version downloaded via git.
And being SuSE, with systemd, "--continue" is supposed to fire off a
background mdadm, I believe. ps tells me that isn't happening...
kanga:/home/anthony # cd mdadm/
kanga:/home/anthony/mdadm # ./mdadm -D /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Fri Aug 5 18:16:24 2016
Raid Level : raid5
Used Dev Size : 8380416 (7.99 GiB 8.58 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Sun Aug 7 01:20:23 2016
State : active, Not Started
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Delta Devices : 1, (2->3)
Name : kanga:testarray (local to host kanga)
UUID : cf52ebc0:886a35cd:688274b4:3f16096c
Events : 157
Number Major Minor RaidDevice State
4 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
3 8 64 2 active sync /dev/sde
kanga:/home/anthony/mdadm # mdadm /dev/md127 --grow --continue
kanga:/home/anthony/mdadm # ps -fea | grep mdadm
root 1953 1855 0 13:02 pts/0 00:00:00 grep --color=auto mdadm
kanga:/home/anthony/mdadm # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : inactive sdc[1] sde[3] sdb[4]
25141248 blocks super 1.2
unused devices: <none>
kanga:/home/anthony/mdadm # ./mdadm /dev/md127 --grow --continue
kanga:/home/anthony/mdadm # ps -fea | grep mdadm
root 1967 1855 0 13:03 pts/0 00:00:00 grep --color=auto mdadm
kanga:/home/anthony/mdadm #
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Wols Lists @ 2016-08-07 11:21 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1608071154110.3593@uplift.swm.pp.se>
On 07/08/16 10:54, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
>
>> I also can't run the array :-(
>
> What does "cat /proc/mdstat" say?
The vm is shut down right now, but it said inactive, four drives,
[UUU_], iirc. I'll post it when I get it back up. Didn't mention raid
type, so that could be messed up somewhere.
Two drives are mirrored, it should be rebuilding on to third, and a
fourth is spare (ready to rebuild into raid6)
>
> Have you tried "--continue"?
>
The last paragraph before the one you quoted said "yes". Actually,
multiple times.
Cheers,
Wol
^ permalink raw reply
* Re: Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Mikael Abrahamsson @ 2016-08-07 9:54 UTC (permalink / raw)
To: Wols Lists; +Cc: linux-raid
In-Reply-To: <57A69012.1010406@youngman.org.uk>
On Sun, 7 Aug 2016, Wols Lists wrote:
> I also can't run the array :-(
What does "cat /proc/mdstat" say?
Have you tried "--continue"?
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Raid5 reshape stuck at 0% - SuSE leap 42.1
From: Wols Lists @ 2016-08-07 1:34 UTC (permalink / raw)
To: linux-raid
Yes I know this is a common occurrence, but it's bitten my test system.
I've now got an array converting from raid1 to raid5, that will build
but won't run, and won't grow.
I suspect it's a similar problem to the one that bit Fedora - in that
"mdadm --grow --continue" comes back with no error messages, but I can
find no evidence that a background mdadm is actually running. As
mentioned earlier, AppArmor is disabled, so I don't know why it won't
work. Any ideas?
I also can't run the array :-(
Note that it's a test vm specifically for playing with raid, so no data
is at stake, but equally, these things shouldn't fail and land you with
a non-working array (and yes, I have downloaded the latest mdadm -
unfortunately I have been mix-n-matching the distro and bleeding-edge
versions).
Cheers,
Wol
/dev/md127:
Version : 1.2
Creation Time : Fri Aug 5 18:16:24 2016
Raid Level : raid5
Used Dev Size : 8380416 (7.99 GiB 8.58 GB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Sun Aug 7 01:20:23 2016
State : active, Not Started
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Delta Devices : 1, (2->3)
Name : kanga:testarray (local to host kanga)
UUID : cf52ebc0:886a35cd:688274b4:3f16096c
Events : 157
Number Major Minor RaidDevice State
4 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
3 8 64 2 active sync /dev/sde
2 8 48 - spare /dev/sdd
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x44
Array UUID : cf52ebc0:886a35cd:688274b4:3f16096c
Name : kanga:testarray (local to host kanga)
Creation Time : Fri Aug 5 18:16:24 2016
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 16760832 (7.99 GiB 8.58 GB)
Array Size : 16760832 (15.98 GiB 17.16 GB)
Data Offset : 16384 sectors
New Offset : 16128 sectors
Super Offset : 8 sectors
State : clean
Device UUID : e9e6426d:d33b9c14:b8bce7e8:780f848e
Reshape pos'n : 0
Delta Devices : 1 (2->3)
Update Time : Sun Aug 7 01:20:23 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 41eb0fd7 - correct
Events : 157
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 0
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x44
Array UUID : cf52ebc0:886a35cd:688274b4:3f16096c
Name : kanga:testarray (local to host kanga)
Creation Time : Fri Aug 5 18:16:24 2016
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 16760832 (7.99 GiB 8.58 GB)
Array Size : 16760832 (15.98 GiB 17.16 GB)
Data Offset : 16384 sectors
New Offset : 16128 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9566866f:cc4cca0d:d8b54faf:f44e2e97
Reshape pos'n : 0
Delta Devices : 1 (2->3)
Update Time : Sun Aug 7 01:20:23 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c6ed915 - correct
Events : 157
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 1
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x44
Array UUID : cf52ebc0:886a35cd:688274b4:3f16096c
Name : kanga:testarray (local to host kanga)
Creation Time : Fri Aug 5 18:16:24 2016
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 16760832 (7.99 GiB 8.58 GB)
Array Size : 16760832 (15.98 GiB 17.16 GB)
Data Offset : 16384 sectors
New Offset : 16128 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fed79c82:8510d13a:496617f5:c1381778
Reshape pos'n : 0
Delta Devices : 1 (2->3)
Update Time : Sun Aug 7 01:20:23 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 733ca876 - correct
Events : 157
Layout : left-symmetric
Chunk Size : 64K
Device Role : spare
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : cf52ebc0:886a35cd:688274b4:3f16096c
Name : kanga:testarray (local to host kanga)
Creation Time : Fri Aug 5 18:16:24 2016
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 16760832 (7.99 GiB 8.58 GB)
Array Size : 16760832 (15.98 GiB 17.16 GB)
Data Offset : 16384 sectors
Super Offset : 8 sectors
Unused Space : before=16296 sectors, after=0 sectors
State : clean
Device UUID : 51271324:1b29e46a:521658dd:5747e61e
Reshape pos'n : 0
Delta Devices : 1 (2->3)
Update Time : Sun Aug 7 01:20:23 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d3d5cfbd - correct
Events : 157
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 2
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply
* Re: Unable to convert raid1 to raid5
From: Wols Lists @ 2016-08-07 1:00 UTC (permalink / raw)
To: Glenn Enright; +Cc: mdraid
In-Reply-To: <CAN7jt2060xqsQB+nAnWO4LmqdUySkU_AWpq=UdFte=Ht0zuoPw@mail.gmail.com>
On 07/08/16 01:32, Glenn Enright wrote:
> On 7/08/2016 12:01 pm, "Wols Lists" <antlists@youngman.org.uk
> <mailto:antlists@youngman.org.uk>> wrote:
>>
>> On 05/08/16 21:16, Wols Lists wrote:
>> > In my testing of xosview, I've been mucking about with a vm and raid.
>> > xosview is looking quite promising (I've got a few comments about it,
>> > but never mind).
>> >
>> > BUT. In mucking about with raid 1, I increased my raid devices to three.
>> > I now just can NOT convert the array to raid 5! I've been mucking around
>> > with all sorts of things trying to get it to work, but finally two error
>> > messages make things clear.
>> >
>> Following up to myself - suddenly thought "I know what's wrong". So I
>> stopped the array, and of course couldn't access it, it was no longer
>> there. So I assembled but didn't run it, and it worked fine.
>>
>> Simples, once you realise what's wrong - you can ADD devices to a
>> running array, but you can't REMOVE them.
>>
>> Cheers,
>> Wol
>>
>
> You can remove em if you mark em as failed first. Eg
>
> Mdadm /dev/mdx --fail /dev/sdc1 --remove /dev/sdc1
>
> Best, Glenn
>
Except - if you read my original post - I was trying to TOTALLY remove
the device!
mdadm --grow -raid-devices=2
THAT was the problem - I had a 3-device mirror, and you can't convert
that to raid5! Even if you've --fail --remove'd the third device!
In other words, "--grow --raid-devices=more" will work on a running
device, "--grow --raid-devices=less" will only work on an array that is
built but not running.
I now have the problem that my "--grow --level=5" has fallen foul of the
"reshape stuck at zero" problem, and I can now neither run the array,
nor get the reshape working ... :-(
At least it's a test vm, specifically for mucking about with raid, so if
I trash it all and start again it's no loss, but it's worrying if a
reshape threatens to lose all your data for you! Especially as I intend
to do it live at some point!
Cheers,
Wol
^ permalink raw reply
* Re: Unable to convert raid1 to raid5
From: Wols Lists @ 2016-08-06 23:59 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <57A4F436.9080402@youngman.org.uk>
On 05/08/16 21:16, Wols Lists wrote:
> In my testing of xosview, I've been mucking about with a vm and raid.
> xosview is looking quite promising (I've got a few comments about it,
> but never mind).
>
> BUT. In mucking about with raid 1, I increased my raid devices to three.
> I now just can NOT convert the array to raid 5! I've been mucking around
> with all sorts of things trying to get it to work, but finally two error
> messages make things clear.
>
Following up to myself - suddenly thought "I know what's wrong". So I
stopped the array, and of course couldn't access it, it was no longer
there. So I assembled but didn't run it, and it worked fine.
Simples, once you realise what's wrong - you can ADD devices to a
running array, but you can't REMOVE them.
Cheers,
Wol
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: Wols Lists @ 2016-08-06 18:04 UTC (permalink / raw)
To: bobzer, Peter Grandi; +Cc: Linux RAID
In-Reply-To: <CADzS=aqLeMibSf-=U1+-_pcoGDu043exZ9ytUR6XLUwZ=VDOWw@mail.gmail.com>
On 06/08/16 00:32, bobzer wrote:
> thank you very much for all your help
>
> I manage to borrow 2 3TB drive
> So i was thinking of dd copie my image to one, use the other to
> reconstruct my raid
> so at this point i got my first raid5 clean
> then i prepare my raid6 degraded with the the 3 4TB and do a rsync
>
> another solution could be to move the image disk to one 3TB
> create the raid6 degraded with the 3 4TB
> move the image in the raid 6
> replace the 2 2TB with the 2 3TB (with a DD copy ?)
> reuse the 2 2TB to have a clean raid 6
> rsync from the degraded raid5 to the clean raid6
>
> I prefere the first solution because the raid 5 is clean and so it
> should be safer but i worrying about the stress of rebuild the raid
Okay. So rebuilding a raid could tip it over the edge.
But you now have four spare drives. So the obvious solution is
create a new raid6 with 2 x 4TB and 2 x 3TB drives.
Copy from the old raid to the new one - the reads won't stress the old
raid anywhere near as much as a rebuild.
Replace one of the 3TB disks with the 4TB that's currently in the old raid.
Raid0 the 2 x 2TB disks from the old raid, and replace the second 3TB disk.
At no point are you rebuilding a degraded raid, and when your data is at
its greatest risk (no backup, raid reshaping) you have a raid6 so you
could survive a disk failure. Oh - and always use mdadm --replace so the
raid always has spare disks ...
If you want to do it in place, ddrescue the two 2TB drives on to the two
3TB drives, so again you can raid0 the 2 x 2TB together, but I think
that's still taking a risk.
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH][RESEND] md: Prevent IO hold during accessing to faulty raid5 array
From: Shaohua Li @ 2016-08-06 5:01 UTC (permalink / raw)
To: Alexey Obitotskiy; +Cc: linux-raid
In-Reply-To: <1470211376-25201-1-git-send-email-aleksey.obitotskiy@intel.com>
On Wed, Aug 03, 2016 at 10:02:56AM +0200, Alexey Obitotskiy wrote:
> After array enters in faulty state (e.g. number of failed drives
> becomes more then accepted for raid5 level) it sets error flags
> (one of this flags is MD_CHANGE_PENDING). For internal metadata
> arrays MD_CHANGE_PENDING cleared into md_update_sb, but not for
> external metadata arrays. MD_CHANGE_PENDING flag set prevents to
> finish all new or non-finished IOs to array and hold them in
> pending state. In some cases this can leads to deadlock situation.
>
> For example, we have faulty array (2 of 4 drives failed) and
> udev handle array state changes and blkid started (or other
> userspace application that used array to read/write) but unable
> to finish reads due to IO hold. At the same time we unable to get
> exclusive access to array (to stop array in our case) because
> another external application still use this array.
>
> Fix makes possible to return IO with errors immediately.
> So external application can finish working with array and
> give exclusive access to other applications to perform
> required management actions with array.
>
> Signed-off-by: Alexey Obitotskiy <aleksey.obitotskiy@intel.com>
> ---
> drivers/md/raid5.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 6c1149d..99471b6 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -4692,7 +4692,9 @@ finish:
> }
>
> if (!bio_list_empty(&s.return_bi)) {
> - if (test_bit(MD_CHANGE_PENDING, &conf->mddev->flags)) {
> + if (test_bit(MD_CHANGE_PENDING, &conf->mddev->flags) &&
> + (s.failed <= conf->max_degraded ||
> + conf->mddev->external == 0)) {
> spin_lock_irq(&conf->device_lock);
> bio_list_merge(&conf->return_bi, &s.return_bi);
> spin_unlock_irq(&conf->device_lock);
So the external metadata array will have the potential race Neil's patch
(c3cce6cda162eb) tried to fix. But we probably can't do too much for it.
Applied this one.
Thanks,
Shaohua
^ permalink raw reply
* Re: INFO: task systemd-udevd:794 blocked for more than 600 seconds observed after mdadm -G operation
From: Shaohua Li @ 2016-08-06 4:45 UTC (permalink / raw)
To: yizhan; +Cc: Guoqing Jiang, linux-raid, shli, NeilBrown, songliubraving
In-Reply-To: <40d28573-38a6-25ad-7956-95bd0c74b996@redhat.com>
On Fri, Aug 05, 2016 at 03:05:40PM +0800, yizhan wrote:
>
>
> On 08/05/2016 02:39 PM, Guoqing Jiang wrote:
> >
> >
> >On 08/04/2016 12:01 PM, Yi Zhang wrote:
> >
> >>[ 1201.734199] INFO: task systemd-udevd:794 blocked for more than 600
> >>seconds.
> >>[ 1201.734204] Not tainted 4.7.0 #5
> >>[ 1201.734204] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>disables this message.
> >>[ 1201.734206] systemd-udevd D ffff8800310a77b0 0 794 1
> >>0x00000000
> >>[ 1201.734209] ffff8800310a77b0 0000000000000001 ffff880031de0000
> >>ffff8807cd3c5688
> >>[ 1201.734211] ffff8800310a8000 0000000000000000 0000000000000002
> >>ffff8807c3e4c100
> >>[ 1201.734213] 0000000000000000 ffff8800310a77c8 ffffffff816e4935
> >>ffff8807cd3c5400
> >>[ 1201.734214] Call Trace:
> >>[ 1201.734220] [<ffffffff816e4935>] schedule+0x35/0x80
> >>[ 1201.734226] [<ffffffffa09a0ba9>] raid5_make_request+0x8b9/0xce0
> >>[raid456]
> >>[ 1201.734229] [<ffffffff810d0240>] ? prepare_to_wait_event+0xf0/0xf0
> >>[ 1201.734233] [<ffffffff8156758e>] md_make_request+0xee/0x230
> >>[ 1201.734236] [<ffffffff81320803>] generic_make_request+0x103/0x1d0
> >>[ 1201.734237] [<ffffffff81320947>] submit_bio+0x77/0x150
> >>[ 1201.734240] [<ffffffff813183bb>] ? bio_alloc_bioset+0x1ab/0x2d0
> >>[ 1201.734242] [<ffffffff81254ecd>] ? alloc_page_buffers+0x5d/0xb0
> >>[ 1201.734244] [<ffffffff81256a1f>] submit_bh_wbc+0x12f/0x160
> >>[ 1201.734246] [<ffffffff81256d58>] block_read_full_page+0x208/0x380
> >>[ 1201.734248] [<ffffffff81259000>] ? I_BDEV+0x20/0x20
> >>[ 1201.734250] [<ffffffff812597e8>] blkdev_readpage+0x18/0x20
> >>[ 1201.734253] [<ffffffff81197718>] do_read_cache_page+0x138/0x300
> >>[ 1201.734255] [<ffffffff812597d0>] ? blkdev_writepages+0x40/0x40
> >>[ 1201.734257] [<ffffffff811978f9>] read_cache_page+0x19/0x20
> >>[ 1201.734259] [<ffffffff81333b50>] read_dev_sector+0x80/0xb0
> >>[ 1201.734261] [<ffffffff81337f14>] read_lba+0x104/0x1c0
> >>[ 1201.734263] [<ffffffff8133852a>] find_valid_gpt+0xfa/0x710
> >>[ 1201.734266] [<ffffffff811d7d00>] ?
> >>vmap_page_range_noflush+0x190/0x380
> >>[ 1201.734267] [<ffffffff81338b40>] ? find_valid_gpt+0x710/0x710
> >>[ 1201.734269] [<ffffffff81338bc9>] efi_partition+0x89/0x440
> >>[ 1201.734271] [<ffffffff81359209>] ? snprintf+0x49/0x70
> >>[ 1201.734273] [<ffffffff81338b40>] ? find_valid_gpt+0x710/0x710
> >>[ 1201.734275] [<ffffffff81335b31>] check_partition+0x101/0x1f0
> >>[ 1201.734277] [<ffffffff81334462>] rescan_partitions+0xc2/0x2c0
> >>[ 1201.734279] [<ffffffff8132f294>] __blkdev_reread_part+0x64/0x70
> >>[ 1201.734281] [<ffffffff8132f2c3>] blkdev_reread_part+0x23/0x40
> >>[ 1201.734282] [<ffffffff8132fd3a>] blkdev_ioctl+0x46a/0x8f0
> >>[ 1201.734284] [<ffffffff81259aa1>] block_ioctl+0x41/0x50
> >>[ 1201.734286] [<ffffffff812323b6>] do_vfs_ioctl+0xa6/0x5c0
> >>[ 1201.734288] [<ffffffff81232949>] SyS_ioctl+0x79/0x90
> >>[ 1201.734290] [<ffffffff81274d67>] ? SyS_flock+0x117/0x1a0
> >>[ 1201.734292] [<ffffffff816e86b2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
> >>[ 1201.734305] INFO: task md0_reshape:4089 blocked for more than 600
> >>seconds.
> >>[ 1201.734306] Not tainted 4.7.0 #5
> >>[ 1201.734307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>disables this message.
> >>[ 1201.734308] md0_reshape D ffff88080db77ac0 0 4089 2
> >>0x00000080
> >>[ 1201.734310] ffff88080db77ac0 0000000000000000 ffff8807d6724380
> >>ffff8807cd3c5670
> >>[ 1201.734311] ffff88080db78000 ffff88080db77b20 ffff8807cd3c5670
> >>0000000000000c20
> >>[ 1201.734313] ffff8807cd3c5418 ffff88080db77ad8 ffffffff816e4935
> >>ffff8807cd3c5400
> >>[ 1201.734314] Call Trace:
> >>[ 1201.734316] [<ffffffff816e4935>] schedule+0x35/0x80
> >>[ 1201.734319] [<ffffffffa099fe8b>] raid5_get_active_stripe+0x20b/0x670
> >>[raid456]
> >>[ 1201.734321] [<ffffffff810d0240>] ? prepare_to_wait_event+0xf0/0xf0
> >>[ 1201.734323] [<ffffffffa09a4733>] reshape_request+0x4d3/0x8a0
> >>[raid456]
> >>[ 1201.734326] [<ffffffffa09a4e21>] raid5_sync_request+0x321/0x3a0
> >>[raid456]
> >>[ 1201.734327] [<ffffffff810cfc54>] ? __wake_up+0x44/0x50
> >>[ 1201.734329] [<ffffffff8156ce97>] md_do_sync+0x9e7/0xf60
> >>[ 1201.734332] [<ffffffff810c200d>] ? update_curr+0xed/0x180
> >>[ 1201.734333] [<ffffffff810c025e>] ? account_entity_dequeue+0xae/0xd0
> >>[ 1201.734336] [<ffffffff81098d43>] ? kernel_sigaction+0x43/0xe0
> >>[ 1201.734339] [<ffffffff81566dc6>] md_thread+0x136/0x150
> >>[ 1201.734341] [<ffffffff81566c90>] ? find_pers+0x70/0x70
> >>[ 1201.734343] [<ffffffff810ab678>] kthread+0xd8/0xf0
> >>[ 1201.734344] [<ffffffff816e88bf>] ret_from_fork+0x1f/0x40
> >>[ 1201.734346] [<ffffffff810ab5a0>] ? kthread_park+0x60/0x60
> >>
> >>
> >>Reproduce steps:
> >>
> >>#mdadm -CR /dev/md0 -l5 -n4 /dev/sd[b-e]1 --write-journal /dev/sdf1
> >>#mdadm --wait /dev/md0
> >>#mdadm /dev/md0 -f /dev/sdf1
> >>#mdadm /dev/md0 -r /dev/sdf1
> >>#mdadm /dev/md0 -a /dev/sdf1
> >>#mdadm -D /dev/md0
> >>/dev/md0:
> >> Version : 1.2
> >> Creation Time : Thu Aug 4 20:37:41 2016
> >> Raid Level : raid5
> >> Array Size : 31432704 (29.98 GiB 32.19 GB)
> >> Used Dev Size : 10477568 (9.99 GiB 10.73 GB)
> >> Raid Devices : 4
> >> Total Devices : 5
> >> Persistence : Superblock is persistent
> >>
> >> Update Time : Thu Aug 4 20:40:16 2016
> >> State : clean
> >> Active Devices : 4
> >>Working Devices : 5
> >> Failed Devices : 0
> >> Spare Devices : 1
> >>
> >> Layout : left-symmetric
> >> Chunk Size : 512K
> >>
> >> Name : dhcp-12-125.nay.redhat.com:0 (local to host
> >>dhcp-12-125.nay.redhat.com)
> >> UUID : 09607542:a8dccc81:f5d2f80c:c586923f
> >> Events : 22
> >>
> >> Number Major Minor RaidDevice State
> >> 0 8 17 0 active sync /dev/sdb1
> >> 1 8 33 1 active sync /dev/sdc1
> >> 2 8 49 2 active sync /dev/sdd1
> >> 5 8 65 3 active sync /dev/sde1
> >>
> >> 4 8 81 - spare /dev/sdf1
> >>
> >>#mdadm -G -n5 /dev/md0
> >
> >Also with 4.7+, I can't find the same issue, but md0_reshape is in 'D'
> >state from my side
> >(actually I can't find reshape is in-progress by 'cat /proc/mdstat').
> >
> Hi guoqing
> Thanks for you detailed log.
> The trace cannot be print soon, I think you need wait for more time, the
> time from below is more than 15m.
> Aug 4 20:39:37 dhcp-12-125 kernel: md/raid:md0: Disk failure on sdf1,
> disabling device.#012md/raid:md0: Operation continuing on 4 devices.
> Aug 4 20:39:56 dhcp-12-125 kernel: md: unbind<sdf1>
> Aug 4 20:39:56 dhcp-12-125 kernel: md: export_rdev(sdf1)
> Aug 4 20:40:16 dhcp-12-125 kernel: md: bind<sdf1>
> Aug 4 20:40:47 dhcp-12-125 kernel: md: reshape of RAID array md0
> Aug 4 20:40:47 dhcp-12-125 kernel: md: minimum _guaranteed_ speed: 1000
> KB/sec/disk.
> Aug 4 20:40:47 dhcp-12-125 kernel: md: using maximum available idle IO
> bandwidth (but not more than 200000 KB/sec) for reshape.
> Aug 4 20:40:47 dhcp-12-125 kernel: md: using 128k window, over a total of
> 10477568k
> Aug 4 20:55:56 dhcp-12-125 kernel: INFO: task systemd-udevd:794 blocked for
> more than 600 seconds.
Adding Song who knows a lot in this side.
If an array is created with journal and journal disk fails, we will make the
array read-only. So we should disable reshape too. Likely we have a bug here.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH 1/3] MD: hold mddev lock for .quiesce in md_do_sync
From: Shaohua Li @ 2016-08-06 4:14 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Shaohua Li
In-Reply-To: <87y44dnrz2.fsf@notabene.neil.brown.name>
On Thu, Aug 04, 2016 at 01:16:49PM +1000, Neil Brown wrote:
> On Wed, Aug 03 2016, NeilBrown wrote:
>
> > [ Unknown signature status ]
> > On Sun, Jul 31 2016, shli@kernel.org wrote:
> >
> >> From: Shaohua Li <shli@fb.com>
> >>
> >> .quiesce is called with mddev lock hold at most places. There are few
> >> exceptions. Calling .quesce without the lock hold could create races. For
> >> example, the .quesce of raid1 can't be recursively. The purpose of the patches
> >> is to fix a race in raid5-cache. The raid5-cache .quesce will write md
> >> superblock and should be called with mddev lock hold.
> >>
> >> Cc: NeilBrown <neilb@suse.com>
> >> Signed-off-by: Shaohua Li <shli@fb.com>
> >
> > Acked-by: NeilBrown <neilb@suse.com>
> >
> > This should be safe but I'm not sure I really like it.
> > The raid1 quiesce could be changed so that it can be called recursively.
> > The raid5-cache situation would be harder to get right and maybe this is
> > the best solution... It's just that 'quiesce' should be a fairly
> > light-weight operation, just waiting for pending requests to flush. It
> > shouldn't really *need* a lock.
>
> Actually, the more I think about this, the less I like it.
>
> I would much rather make .quiesce lighter weight so that no locking was
> needed.
>
> For r5l_quiesce, that probable means removed the "r5l_do_reclaim()".
> Stopping and restarting the reclaim thread seems reasonable, but calling
> r5l_do_reclaim() should not be needed. It should be done periodically
> by the thread, and at 'stop' time, but otherwise isn't needed.
> You would need to hold some mutex while calling md_register_thread, but
> that could be probably be log->io_mutex, or maybe even some other new
> mutex
We will have the same deadlock issue with just stopping/restarting the reclaim
thread. As stopping the thread will wait for the thread, which probably is
doing r5l_do_reclaim and writting the superblock. Since we are writting the
superblock, we must hold the reconfig_mutex.
Letting raid5_quiesce call r5l_do_reclaim gives us a clean log. Just
stop/restart the reclaim thread can't guarantee this, as it's possible some
space aren't reclaimed yet. A clean log will simplify a lot of things, for
example we change the layout of the array. The log doesn't need to remember
which part is for the old layout and which part is the new layout.
I think we can add a new parameter for .quiesce to indicate if reconfig_mutex
is hold. raid5_quiesce can check the parameter and hold reconfig_mutex if
necessary.
Thanks,
Shaohua
^ permalink raw reply
* Re: [PATCH 4/8] md-cluster: introduce dlm_lock_sync_interruptible to fix tasks hang
From: Shaohua Li @ 2016-08-06 3:59 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: linux-raid
In-Reply-To: <57A15977.205@suse.com>
On Wed, Aug 03, 2016 at 10:39:51AM +0800, Guoqing Jiang wrote:
>
>
> On 08/03/2016 06:36 AM, Shaohua Li wrote:
> >On Mon, Aug 01, 2016 at 11:24:34PM -0400, Guoqing Jiang wrote:
> >>
> >>On 08/01/2016 06:20 PM, Shaohua Li wrote:
> >>>On Thu, Jul 28, 2016 at 02:16:48AM -0400, Guoqing Jiang wrote:
> >>>>When some node leaves cluster, then it's bitmap need to be
> >>>>synced by another node, so "md*_recover" thread is triggered
> >>>>for the purpose. However, with below steps. we can find tasks
> >>>>hang happened either in B or C.
> >>>>
> >>>>1. Node A create a resyncing cluster raid1, assemble it in
> >>>> other two nodes (B and C).
> >>>>2. stop array in B and C.
> >>>>3. stop array in A.
> >>>>
> >>>>linux44:~ # ps aux|grep md|grep D
> >>>>root 5938 0.0 0.1 19852 1964 pts/0 D+ 14:52 0:00 mdadm -S md0
> >>>>root 5939 0.0 0.0 0 0 ? D 14:52 0:00 [md0_recover]
> >>>>
> >>>>linux44:~ # cat /proc/5939/stack
> >>>>[<ffffffffa04cf321>] dlm_lock_sync+0x71/0x90 [md_cluster]
> >>>>[<ffffffffa04d0705>] recover_bitmaps+0x125/0x220 [md_cluster]
> >>>>[<ffffffffa052105d>] md_thread+0x16d/0x180 [md_mod]
> >>>>[<ffffffff8107ad94>] kthread+0xb4/0xc0
> >>>>[<ffffffff8152a518>] ret_from_fork+0x58/0x90
> >>>>
> >>>>linux44:~ # cat /proc/5938/stack
> >>>>[<ffffffff8107afde>] kthread_stop+0x6e/0x120
> >>>>[<ffffffffa0519da0>] md_unregister_thread+0x40/0x80 [md_mod]
> >>>>[<ffffffffa04cfd20>] leave+0x70/0x120 [md_cluster]
> >>>>[<ffffffffa0525e24>] md_cluster_stop+0x14/0x30 [md_mod]
> >>>>[<ffffffffa05269ab>] bitmap_free+0x14b/0x150 [md_mod]
> >>>>[<ffffffffa0523f3b>] do_md_stop+0x35b/0x5a0 [md_mod]
> >>>>[<ffffffffa0524e83>] md_ioctl+0x873/0x1590 [md_mod]
> >>>>[<ffffffff81288464>] blkdev_ioctl+0x214/0x7d0
> >>>>[<ffffffff811dd3dd>] block_ioctl+0x3d/0x40
> >>>>[<ffffffff811b92d4>] do_vfs_ioctl+0x2d4/0x4b0
> >>>>[<ffffffff811b9538>] SyS_ioctl+0x88/0xa0
> >>>>[<ffffffff8152a5c9>] system_call_fastpath+0x16/0x1b
> >>>>
> >>>>The problem is caused by recover_bitmaps can't reliably abort
> >>>>when the thread is unregistered. So dlm_lock_sync_interruptible
> >>>>is introduced to detect the thread's situation to fix the problem.
> >>>>
> >>>>Reviewed-by: NeilBrown <neilb@suse.com>
> >>>>Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> >>>>---
> >>>> drivers/md/md-cluster.c | 38 +++++++++++++++++++++++++++++++++++++-
> >>>> 1 file changed, 37 insertions(+), 1 deletion(-)
> >>>>
> >>>>diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
> >>>>index ea2699e..f3d584e 100644
> >>>>--- a/drivers/md/md-cluster.c
> >>>>+++ b/drivers/md/md-cluster.c
> >>>>@@ -10,6 +10,8 @@
> >>>> #include <linux/module.h>
> >>>>+#include <linux/completion.h>
> >>>>+#include <linux/kthread.h>
> >>>> #include <linux/dlm.h>
> >>>> #include <linux/sched.h>
> >>>> #include <linux/raid/md_p.h>
> >>>>@@ -141,6 +143,40 @@ static int dlm_unlock_sync(struct dlm_lock_resource *res)
> >>>> return dlm_lock_sync(res, DLM_LOCK_NL);
> >>>> }
> >>>>+/* An variation of dlm_lock_sync, which make lock request could
> >>>>+ * be interrupted */
> >>>>+static int dlm_lock_sync_interruptible(struct dlm_lock_resource *res, int mode,
> >>>>+ struct mddev *mddev)
> >>>>+{
> >>>>+ int ret = 0;
> >>>>+
> >>>>+ ret = dlm_lock(res->ls, mode, &res->lksb,
> >>>>+ res->flags, res->name, strlen(res->name),
> >>>>+ 0, sync_ast, res, res->bast);
> >>>>+ if (ret)
> >>>>+ return ret;
> >>>>+
> >>>>+ wait_event(res->completion.wait,
> >>>>+ res->completion.done || kthread_should_stop());
> >>>can you convert it to a waitq? Directly using the .wait/.done of completion is
> >>>really intrusive.
> >>Maybe, but we still need completion for dlm_lock_resource otherwise there
> >>are different types of dlm_lock_resource, we also need to keep align with
> >>sync_ast as dlm_lock_sync did.
> >Yes, we need a waitq and variable like completion.done to indicate the event is
> >done, and convert the completion API to waitq API in other places like
> >sync_ast. The point is not using the opaque data structure of 'struct
> >completion'. Diving into implementation details of a unrelated data structure
> >(completion here) is really intrusive.
>
> I don't want to argue about the intrusive, but we have to create something
> which could be achieved by the existed thing.
>
> OTOH, convert completion to waitqueue should be done in a new patch,
> and it is a big change though achieveable, we need to do full test for it.
I can't see why this is a big change. Simply adding a waitq and a variable,
replace wait_for_completion with wait_event checking the condition and replace
complete with setting the variable and wake_up. That's pretty straightforward
and no risk. I'd rather see the first patch converts completion to waitq and
this patch addes the kthread_should_stop() check.
Thanks,
Shaohua
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: bobzer @ 2016-08-05 23:32 UTC (permalink / raw)
To: Peter Grandi, Wols Lists; +Cc: Linux RAID
In-Reply-To: <22424.53061.186858.220134@tree.ty.sabi.co.uk>
thank you very much for all your help
I manage to borrow 2 3TB drive
So i was thinking of dd copie my image to one, use the other to
reconstruct my raid
so at this point i got my first raid5 clean
then i prepare my raid6 degraded with the the 3 4TB and do a rsync
another solution could be to move the image disk to one 3TB
create the raid6 degraded with the 3 4TB
move the image in the raid 6
replace the 2 2TB with the 2 3TB (with a DD copy ?)
reuse the 2 2TB to have a clean raid 6
rsync from the degraded raid5 to the clean raid6
I prefere the first solution because the raid 5 is clean and so it
should be safer but i worrying about the stress of rebuild the raid
i like the second solution because i don't need to rebuild the raid 5
and so i don't take the risk that my 2T disk died
I mean, they should be ok but i prefere to just have to read on them
than to write for the rebuild (last time it's the stress of the
rebuild which destroy my raid...)
the interest is that there if one 2TB died it more likely to happend
when is in the raid 6 not when i read from them to copy in the 3T, and
when my data are in 3tb they should be safe
there is plenty more solution available to me now that i got these 2
drives except that i have to give then back as soon as possible
anyway i still don't know which scenario is the best
but thanks for your help
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox