* Re: [PATCH 7/8] md: skip resync for raid array with journal
From: Neil Brown @ 2015-09-30 4:24 UTC (permalink / raw)
To: Shaohua Li, linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <7c5622ab3335f31a27f807d8fee27f8bb60675ae.1441221530.git.shli@fb.com>
[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]
Shaohua Li <shli@fb.com> writes:
> If a raid array has journal, the journal can guarantee the consistency,
> we can skip resync after a unclean shutdown. The exception is raid
> creation or user initiated resync, which we still do a raid resync.
>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/md.c | 4 ++++
> drivers/md/md.h | 1 +
> 2 files changed, 5 insertions(+)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index b3f9eed..95824fb 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -1669,6 +1669,8 @@ static int super_1_validate(struct mddev *mddev, struct md_rdev *rdev)
> }
> set_bit(Journal, &rdev->flags);
> rdev->journal_tail = le64_to_cpu(sb->journal_tail);
> + if (mddev->recovery_cp == MaxSector)
> + set_bit(MD_JOURNAL_CLEAN, &mddev->flags);
> break;
> default:
> rdev->saved_raid_disk = role;
> @@ -1711,6 +1713,8 @@ static void super_1_sync(struct mddev *mddev, struct md_rdev *rdev)
> sb->events = cpu_to_le64(mddev->events);
> if (mddev->in_sync)
> sb->resync_offset = cpu_to_le64(mddev->recovery_cp);
> + else if (test_bit(MD_JOURNAL_CLEAN, &mddev->flags))
> + sb->resync_offset = cpu_to_le64(MaxSector);
> else
> sb->resync_offset = cpu_to_le64(0);
>
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index 226f4ba..0288a0b 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -236,6 +236,7 @@ struct mddev {
> #define MD_STILL_CLOSED 4 /* If set, then array has not been opened since
> * md_ioctl checked on it.
> */
> +#define MD_JOURNAL_CLEAN 5 /* A raid with journal is already clean */
>
> int suspended;
> atomic_t active_io;
> --
> 1.8.1
This looks right as far as it goes, but I don't think it goes far
enough.
The particular scenario that bothers me is if the array is started
without the journal being present.
I cannot see anything to prevent that - is there?
In that case we need to assume that the array is not in-sync,
and we need to clear MD_FEATURE_JOURNAL so if it gets stopped and then
assembled again with the stale journal doesn't get used.
One unfortunate side effect of that is that you couldn't stop the array
cleanly (leaving the journal effectively empty) and then restart with no
journal and no resync. Is that a problem I wonder?
I'm not sure what the best solution is here, but we need a clear
understanding of what happens if you try to assemble an array without
the journal where previously it had one, and I don't think the current
code gets it right.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 8/8] raid5-cache: add trim support for log
From: Neil Brown @ 2015-09-30 4:14 UTC (permalink / raw)
To: Shaohua Li, linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <4ac2b019b16111478c46ea6decb672d5a4de7de3.1441221530.git.shli@fb.com>
[-- Attachment #1: Type: text/plain, Size: 2320 bytes --]
Shaohua Li <shli@fb.com> writes:
> Since superblock is updated infrequently, we do a simple trim of log
> disk (a synchronous trim)
>
> Signed-off-by: Shaohua Li <shli@fb.com>
> ---
> drivers/md/raid5-cache.c | 30 +++++++++++++++++++++++++++++-
> 1 file changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
> index 27fb513..410b85b 100644
> --- a/drivers/md/raid5-cache.c
> +++ b/drivers/md/raid5-cache.c
> @@ -639,6 +639,34 @@ static void r5l_kick_io_unit(struct r5l_log *log)
> }
>
> static void r5l_write_super(struct r5l_log *log, sector_t cp);
> +static void r5l_write_super_and_discard_space(struct r5l_log *log,
> + sector_t end)
> +{
> + struct block_device *bdev = log->rdev->bdev;
> +
> + r5l_write_super(log, end);
> +
> + if (!blk_queue_discard(bdev_get_queue(bdev)))
> + return;
> +
> + /* discard destroy old data in log, so force a super update */
> + md_update_sb(log->rdev->mddev, 1);
I don't think this can go here.
md_update_sb() is currently always called with ->reconfig_mutex held,
and I think that should stay.
Instead you could:
set_bit(MD_CHANGE_PENDING, &mddev->flags);
md_wakeup_thread(mddev->thread);
wait_event(mddev->sb_wait,
!test_bit(MD_CHANGE_PENDING, &mddev->flags));
a bit like md_write_start().
Thanks,
NeilBrown
> +
> + if (log->last_checkpoint < end) {
> + blkdev_issue_discard(bdev,
> + log->last_checkpoint + log->rdev->data_offset,
> + end - log->last_checkpoint, GFP_NOIO, 0);
> + } else {
> + blkdev_issue_discard(bdev,
> + log->last_checkpoint + log->rdev->data_offset,
> + log->device_size - log->last_checkpoint,
> + GFP_NOIO, 0);
> + blkdev_issue_discard(bdev, log->rdev->data_offset, end,
> + GFP_NOIO, 0);
> + }
> +}
> +
> +
> static void r5l_do_reclaim(struct r5l_log *log)
> {
> struct r5l_io_unit *io, *last;
> @@ -694,7 +722,7 @@ static void r5l_do_reclaim(struct r5l_log *log)
> * here, because the log area might be reused soon and we don't want to
> * confuse recovery
> * */
> - r5l_write_super(log, last->log_start);
> + r5l_write_super_and_discard_space(log, last->log_start);
>
> mutex_lock(&log->io_mutex);
> log->last_checkpoint = last->log_start;
> --
> 1.8.1
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 2/3] md-cluster: Improve md_reload_sb to be less error prone
From: kbuild test robot @ 2015-09-30 1:56 UTC (permalink / raw)
Cc: kbuild-all, linux-raid, Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-3-git-send-email-rgoldwyn@suse.com>
[-- Attachment #1: Type: text/plain, Size: 1749 bytes --]
Hi Goldwyn,
[auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore]
config: openrisc-allmodconfig (attached as .config)
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
git checkout 825961765139461e82fb00395588061f26b21693
# save the attached .config to linux build tree
make.cross ARCH=openrisc
All warnings (new ones prefixed by >>):
drivers/md/md.c: In function 'check_sb_changes':
>> drivers/md/md.c:8975:3: warning: format '%lu' expects type 'long unsigned int', but argument 4 has type 'sector_t'
vim +8975 drivers/md/md.c
8959 }
8960 /* device faulty
8961 * We just want to do the minimum to mark the disk
8962 * as faulty. The recovery is performed by the
8963 * one who initiated the error.
8964 */
8965 if ((role == 0xfffe) || (role == 0xfffd)) {
8966 __md_error(mddev, rdev2, false);
8967 clear_bit(Blocked, &rdev2->flags);
8968 remove_and_add_spares(mddev, rdev2);
8969 }
8970 }
8971 }
8972
8973 /* recovery_cp changed */
8974 if (le64_to_cpu(sb->resync_offset) != mddev->recovery_cp) {
> 8975 pr_info("%s:%d recovery_cp changed from %lu to %lu\n", __func__,
8976 __LINE__, mddev->recovery_cp,
8977 (unsigned long) le64_to_cpu(sb->resync_offset));
8978 mddev->recovery_cp = le64_to_cpu(sb->resync_offset);
8979 }
8980
8981 /* Finally set the event to be up to date */
8982 mddev->events = le64_to_cpu(sb->events);
8983 }
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 35004 bytes --]
^ permalink raw reply
* [RFC PATCH] md-cluster: does_sb_need_changing() can be static
From: kbuild test robot @ 2015-09-29 16:30 UTC (permalink / raw)
Cc: kbuild-all, linux-raid, Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-4-git-send-email-rgoldwyn@suse.com>
Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
---
md.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 09d1384..ae90e3c 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2196,7 +2196,7 @@ static void sync_sbs(struct mddev *mddev, int nospares)
}
}
-bool does_sb_need_changing(struct mddev *mddev)
+static bool does_sb_need_changing(struct mddev *mddev)
{
struct md_rdev *rdev;
struct mdp_superblock_1 *sb;
^ permalink raw reply related
* Re: [PATCH 3/3] md-cluster: Perform a lazy update
From: kbuild test robot @ 2015-09-29 16:30 UTC (permalink / raw)
Cc: kbuild-all, linux-raid, Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-4-git-send-email-rgoldwyn@suse.com>
Hi Goldwyn,
[auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore]
reproduce:
# apt-get install sparse
make ARCH=x86_64 allmodconfig
make C=1 CF=-D__CHECK_ENDIAN__
sparse warnings: (new ones prefixed by >>)
drivers/md/md.c:1480:34: sparse: cast to restricted __le64
drivers/md/md.c:1760:40: sparse: incorrect type in assignment (different base types)
drivers/md/md.c:1760:40: expected unsigned long long [unsigned] [long] [long long] [usertype] <noident>
drivers/md/md.c:1760:40: got restricted __le64 [usertype] <noident>
drivers/md/md.c:1835:26: sparse: incorrect type in assignment (different base types)
drivers/md/md.c:1835:26: expected restricted __le64 [usertype] super_offset
drivers/md/md.c:1835:26: got unsigned long [unsigned] [usertype] sb_start
drivers/md/md.c:2230:43: sparse: cast to restricted __le64
>> drivers/md/md.c:2230:43: sparse: cast from restricted __le32
>> drivers/md/md.c:2199:6: sparse: symbol 'does_sb_need_changing' was not declared. Should it be static?
drivers/md/md.c:7061:6: sparse: symbol '__md_error' was not declared. Should it be static?
Please review and possibly fold the followup patch.
vim +2230 drivers/md/md.c
2193 sync_super(mddev, rdev);
2194 rdev->sb_loaded = 1;
2195 }
2196 }
2197 }
2198
> 2199 bool does_sb_need_changing(struct mddev *mddev)
2200 {
2201 struct md_rdev *rdev;
2202 struct mdp_superblock_1 *sb;
2203 int role;
2204
2205 /* Find a good rdev */
2206 rdev_for_each_rcu(rdev, mddev)
2207 if ((rdev->raid_disk >= 0) && !test_bit(Faulty, &rdev->flags))
2208 break;
2209
2210 /* No good device found. */
2211 if (!rdev)
2212 return false;
2213
2214 sb = page_address(rdev->sb_page);
2215 /* Check if a device has become faulty or a spare become active */
2216 rdev_for_each(rdev, mddev) {
2217 role = le16_to_cpu(sb->dev_roles[rdev->desc_nr]);
2218 /* Device activated? */
2219 if (role == 0xffff && rdev->raid_disk >=0 && !test_bit(Faulty, &rdev->flags))
2220 return true;
2221 /* Device turned faulty? */
2222 if (test_bit(Faulty, &rdev->flags) && (role < 0xfffd))
2223 return true;
2224 }
2225
2226 /* Check if any mddev parameters have changed */
2227 if ((mddev->dev_sectors != le64_to_cpu(sb->size)) ||
2228 (mddev->reshape_position != le64_to_cpu(sb->reshape_position)) ||
2229 (mddev->recovery_cp != le64_to_cpu(sb->resync_offset)) ||
> 2230 (mddev->layout != le64_to_cpu(sb->layout)) ||
2231 (mddev->raid_disks != le32_to_cpu(sb->raid_disks)) ||
2232 (mddev->chunk_sectors != le32_to_cpu(sb->chunksize))
2233 )
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
^ permalink raw reply
* [RFC PATCH] md-cluster: __md_error() can be static
From: kbuild test robot @ 2015-09-29 16:22 UTC (permalink / raw)
Cc: kbuild-all, linux-raid, Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-3-git-send-email-rgoldwyn@suse.com>
Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
---
md.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 2db697b..294f9e4 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -7028,7 +7028,7 @@ void md_unregister_thread(struct md_thread **threadp)
}
EXPORT_SYMBOL(md_unregister_thread);
-void __md_error(struct mddev *mddev, struct md_rdev *rdev, bool writeout)
+static void __md_error(struct mddev *mddev, struct md_rdev *rdev, bool writeout)
{
if (!rdev || test_bit(Faulty, &rdev->flags))
return;
^ permalink raw reply related
* Re: [PATCH 2/3] md-cluster: Improve md_reload_sb to be less error prone
From: kbuild test robot @ 2015-09-29 16:22 UTC (permalink / raw)
Cc: kbuild-all, linux-raid, Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-3-git-send-email-rgoldwyn@suse.com>
Hi Goldwyn,
[auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore]
reproduce:
# apt-get install sparse
make ARCH=x86_64 allmodconfig
make C=1 CF=-D__CHECK_ENDIAN__
sparse warnings: (new ones prefixed by >>)
drivers/md/md-cluster.c:213:15: sparse: restricted __le64 degrades to integer
>> drivers/md/md-cluster.c:424:29: sparse: cast to restricted __le32
drivers/md/md-cluster.c:564:20: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:564:20: expected int [signed] slot
drivers/md/md-cluster.c:564:20: got restricted __le32 [usertype] <noident>
drivers/md/md-cluster.c:823:19: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:823:19: expected int [signed] [addressable] type
drivers/md/md-cluster.c:823:19: got restricted __le32 [usertype] <noident>
>> drivers/md/md-cluster.c:829:40: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:829:40: expected int [signed] [addressable] [assigned] raid_slot
drivers/md/md-cluster.c:829:40: got restricted __le32 [usertype] <noident>
drivers/md/md-cluster.c:858:19: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:858:19: expected int [signed] type
drivers/md/md-cluster.c:858:19: got restricted __le32 [usertype] <noident>
drivers/md/md-cluster.c:859:19: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:859:19: expected int [signed] slot
drivers/md/md-cluster.c:859:19: got restricted __le32 [usertype] <noident>
drivers/md/md-cluster.c:860:18: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:860:18: expected unsigned long [unsigned] [usertype] low
drivers/md/md-cluster.c:860:18: got restricted __le64 [usertype] <noident>
drivers/md/md-cluster.c:861:19: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:861:19: expected unsigned long [unsigned] [usertype] high
drivers/md/md-cluster.c:861:19: got restricted __le64 [usertype] <noident>
drivers/md/md-cluster.c:880:27: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:880:27: expected int [signed] type
drivers/md/md-cluster.c:880:27: got restricted __le32 [usertype] <noident>
drivers/md/md-cluster.c:881:27: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:881:27: expected int [signed] slot
drivers/md/md-cluster.c:881:27: got restricted __le32 [usertype] <noident>
drivers/md/md-cluster.c:919:19: sparse: incorrect type in assignment (different base types)
drivers/md/md-cluster.c:919:19: expected int [signed] [addressable] type
drivers/md/md-cluster.c:919:19: got restricted __le32 [usertype] <noident>
--
drivers/md/md.c:1480:34: sparse: cast to restricted __le64
drivers/md/md.c:1760:40: sparse: incorrect type in assignment (different base types)
drivers/md/md.c:1760:40: expected unsigned long long [unsigned] [long] [long long] [usertype] <noident>
drivers/md/md.c:1760:40: got restricted __le64 [usertype] <noident>
drivers/md/md.c:1835:26: sparse: incorrect type in assignment (different base types)
drivers/md/md.c:1835:26: expected restricted __le64 [usertype] super_offset
drivers/md/md.c:1835:26: got unsigned long [unsigned] [usertype] sb_start
>> drivers/md/md.c:7031:6: sparse: symbol '__md_error' was not declared. Should it be static?
Please review and possibly fold the followup patch.
vim +424 drivers/md/md-cluster.c
408 len = snprintf(disk_uuid, 64, "DEVICE_UUID=");
409 sprintf(disk_uuid + len, "%pU", cmsg->uuid);
410 snprintf(raid_slot, 16, "RAID_DISK=%d", cmsg->raid_slot);
411 pr_info("%s:%d Sending kobject change with %s and %s\n", __func__, __LINE__, disk_uuid, raid_slot);
412 init_completion(&cinfo->newdisk_completion);
413 set_bit(MD_CLUSTER_WAITING_FOR_NEWDISK, &cinfo->state);
414 kobject_uevent_env(&disk_to_dev(mddev->gendisk)->kobj, KOBJ_CHANGE, envp);
415 wait_for_completion_timeout(&cinfo->newdisk_completion,
416 NEW_DEV_TIMEOUT);
417 clear_bit(MD_CLUSTER_WAITING_FOR_NEWDISK, &cinfo->state);
418 }
419
420
421 static void process_metadata_update(struct mddev *mddev, struct cluster_msg *msg)
422 {
423 struct md_cluster_info *cinfo = mddev->cluster_info;
> 424 md_reload_sb(mddev, le32_to_cpu(msg->raid_slot));
425 dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
426 }
427
428 static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
429 {
430 struct md_rdev *rdev = md_find_rdev_nr_rcu(mddev, msg->raid_slot);
431
432 if (rdev)
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
^ permalink raw reply
* Re: [PATCH 2/3] md-cluster: Improve md_reload_sb to be less error prone
From: kbuild test robot @ 2015-09-29 16:02 UTC (permalink / raw)
Cc: kbuild-all, linux-raid, Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-3-git-send-email-rgoldwyn@suse.com>
[-- Attachment #1: Type: text/plain, Size: 2435 bytes --]
Hi Goldwyn,
[auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore]
config: i386-defconfig (attached as .config)
reproduce:
git checkout 825961765139461e82fb00395588061f26b21693
# save the attached .config to linux build tree
make ARCH=i386
All warnings (new ones prefixed by >>):
In file included from include/linux/printk.h:6:0,
from include/linux/kernel.h:13,
from include/linux/sched.h:17,
from include/linux/kthread.h:5,
from drivers/md/md.c:35:
drivers/md/md.c: In function 'check_sb_changes':
>> include/linux/kern_levels.h:4:18: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'sector_t {aka long long unsigned int}' [-Wformat=]
#define KERN_SOH "\001" /* ASCII Start Of Header */
^
include/linux/kern_levels.h:13:19: note: in expansion of macro 'KERN_SOH'
#define KERN_INFO KERN_SOH "6" /* informational */
^
include/linux/printk.h:259:9: note: in expansion of macro 'KERN_INFO'
printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
^
>> drivers/md/md.c:8975:3: note: in expansion of macro 'pr_info'
pr_info("%s:%d recovery_cp changed from %lu to %lu\n", __func__,
^
vim +/pr_info +8975 drivers/md/md.c
8959 }
8960 /* device faulty
8961 * We just want to do the minimum to mark the disk
8962 * as faulty. The recovery is performed by the
8963 * one who initiated the error.
8964 */
8965 if ((role == 0xfffe) || (role == 0xfffd)) {
8966 __md_error(mddev, rdev2, false);
8967 clear_bit(Blocked, &rdev2->flags);
8968 remove_and_add_spares(mddev, rdev2);
8969 }
8970 }
8971 }
8972
8973 /* recovery_cp changed */
8974 if (le64_to_cpu(sb->resync_offset) != mddev->recovery_cp) {
> 8975 pr_info("%s:%d recovery_cp changed from %lu to %lu\n", __func__,
8976 __LINE__, mddev->recovery_cp,
8977 (unsigned long) le64_to_cpu(sb->resync_offset));
8978 mddev->recovery_cp = le64_to_cpu(sb->resync_offset);
8979 }
8980
8981 /* Finally set the event to be up to date */
8982 mddev->events = le64_to_cpu(sb->events);
8983 }
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 23761 bytes --]
^ permalink raw reply
* Re: [PATCH 2/3] md-cluster: Improve md_reload_sb to be less error prone
From: kbuild test robot @ 2015-09-29 15:49 UTC (permalink / raw)
Cc: kbuild-all, linux-raid, Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-3-git-send-email-rgoldwyn@suse.com>
[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]
Hi Goldwyn,
[auto build test results on v4.3-rc3 -- if it's inappropriate base, please ignore]
config: parisc-c3000_defconfig (attached as .config)
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
git checkout 825961765139461e82fb00395588061f26b21693
# save the attached .config to linux build tree
make.cross ARCH=parisc
All warnings (new ones prefixed by >>):
drivers/md/md.c: In function 'check_sb_changes':
>> drivers/md/md.c:8975:3: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'sector_t' [-Wformat=]
pr_info("%s:%d recovery_cp changed from %lu to %lu\n", __func__,
^
vim +8975 drivers/md/md.c
8959 }
8960 /* device faulty
8961 * We just want to do the minimum to mark the disk
8962 * as faulty. The recovery is performed by the
8963 * one who initiated the error.
8964 */
8965 if ((role == 0xfffe) || (role == 0xfffd)) {
8966 __md_error(mddev, rdev2, false);
8967 clear_bit(Blocked, &rdev2->flags);
8968 remove_and_add_spares(mddev, rdev2);
8969 }
8970 }
8971 }
8972
8973 /* recovery_cp changed */
8974 if (le64_to_cpu(sb->resync_offset) != mddev->recovery_cp) {
> 8975 pr_info("%s:%d recovery_cp changed from %lu to %lu\n", __func__,
8976 __LINE__, mddev->recovery_cp,
8977 (unsigned long) le64_to_cpu(sb->resync_offset));
8978 mddev->recovery_cp = le64_to_cpu(sb->resync_offset);
8979 }
8980
8981 /* Finally set the event to be up to date */
8982 mddev->events = le64_to_cpu(sb->events);
8983 }
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 13572 bytes --]
^ permalink raw reply
* [PATCH 3/3] md-cluster: Perform a lazy update
From: Goldwyn Rodrigues @ 2015-09-29 15:21 UTC (permalink / raw)
To: linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-1-git-send-email-rgoldwyn@suse.com>
In a clustered environment, a change such as marking a device faulty,
can be recorded by any of the nodes. This is communicated to all the
nodes and re-recording such a change is unnecessary, and quite often
pretty disruptive.
With this patch, just before the update, we detect for the changes
and if the changes are already in superblock, we abort the update
after clearing all the flags
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md.c | 115 ++++++++++++++++++++++++++++++++------------------------
1 file changed, 66 insertions(+), 49 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index fe3ce06..e732d73 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2199,6 +2199,46 @@ static void sync_sbs(struct mddev *mddev, int nospares)
}
}
+bool does_sb_need_changing(struct mddev *mddev)
+{
+ struct md_rdev *rdev;
+ struct mdp_superblock_1 *sb;
+ int role;
+
+ /* Find a good rdev */
+ rdev_for_each_rcu(rdev, mddev)
+ if ((rdev->raid_disk >= 0) && !test_bit(Faulty, &rdev->flags))
+ break;
+
+ /* No good device found. */
+ if (!rdev)
+ return false;
+
+ sb = page_address(rdev->sb_page);
+ /* Check if a device has become faulty or a spare become active */
+ rdev_for_each(rdev, mddev) {
+ role = le16_to_cpu(sb->dev_roles[rdev->desc_nr]);
+ /* Device activated? */
+ if (role == 0xffff && rdev->raid_disk >=0 && !test_bit(Faulty, &rdev->flags))
+ return true;
+ /* Device turned faulty? */
+ if (test_bit(Faulty, &rdev->flags) && (role < 0xfffd))
+ return true;
+ }
+
+ /* Check if any mddev parameters have changed */
+ if ((mddev->dev_sectors != le64_to_cpu(sb->size)) ||
+ (mddev->reshape_position != le64_to_cpu(sb->reshape_position)) ||
+ (mddev->recovery_cp != le64_to_cpu(sb->resync_offset)) ||
+ (mddev->layout != le64_to_cpu(sb->layout)) ||
+ (mddev->raid_disks != le32_to_cpu(sb->raid_disks)) ||
+ (mddev->chunk_sectors != le32_to_cpu(sb->chunksize))
+ )
+ return true;
+
+ return false;
+}
+
void md_update_sb(struct mddev *mddev, int force_change)
{
struct md_rdev *rdev;
@@ -2211,6 +2251,27 @@ void md_update_sb(struct mddev *mddev, int force_change)
set_bit(MD_CHANGE_DEVS, &mddev->flags);
return;
}
+
+ if (mddev_is_clustered(mddev)) {
+ int change_devs;
+recheck:
+ change_devs = test_and_clear_bit(MD_CHANGE_DEVS, &mddev->flags);
+ md_cluster_ops->metadata_update_start(mddev);
+ /* Has someone else has updated the sb */
+ if (!does_sb_need_changing(mddev)) {
+ md_cluster_ops->metadata_update_cancel(mddev);
+ /* Check if someone set the bits while we
+ * were waiting on lock
+ */
+ if (test_bit(MD_CHANGE_DEVS, &mddev->flags))
+ goto recheck;
+ clear_bit(MD_CHANGE_PENDING, &mddev->flags);
+ return;
+ }
+
+ if (change_devs)
+ set_bit(MD_CHANGE_DEVS, &mddev->flags);
+ }
repeat:
/* First make sure individual recovery_offsets are correct */
rdev_for_each(rdev, mddev) {
@@ -2359,6 +2420,9 @@ repeat:
clear_bit(BlockedBadBlocks, &rdev->flags);
wake_up(&rdev->blocked_wait);
}
+
+ if (mddev_is_clustered(mddev))
+ md_cluster_ops->metadata_update_finish(mddev);
}
EXPORT_SYMBOL(md_update_sb);
@@ -2496,13 +2560,9 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
if (mddev_is_clustered(mddev))
md_cluster_ops->remove_disk(mddev, rdev);
md_kick_rdev_from_array(rdev);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
if (mddev->pers)
md_update_sb(mddev, 1);
md_new_event(mddev);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
err = 0;
}
} else if (cmd_match(buf, "writemostly")) {
@@ -4063,12 +4123,8 @@ size_store(struct mddev *mddev, const char *buf, size_t len)
if (err)
return err;
if (mddev->pers) {
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
err = update_size(mddev, sectors);
md_update_sb(mddev, 1);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
} else {
if (mddev->dev_sectors == 0 ||
mddev->dev_sectors > sectors)
@@ -5306,8 +5362,6 @@ static void md_clean(struct mddev *mddev)
static void __md_stop_writes(struct mddev *mddev)
{
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
set_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
flush_workqueue(md_misc_wq);
if (mddev->sync_thread) {
@@ -5326,8 +5380,6 @@ static void __md_stop_writes(struct mddev *mddev)
mddev->in_sync = 1;
md_update_sb(mddev, 1);
}
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
}
void md_stop_writes(struct mddev *mddev)
@@ -5995,9 +6047,6 @@ static int hot_remove_disk(struct mddev *mddev, dev_t dev)
if (!rdev)
return -ENXIO;
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
-
clear_bit(Blocked, &rdev->flags);
remove_and_add_spares(mddev, rdev);
@@ -6011,13 +6060,8 @@ static int hot_remove_disk(struct mddev *mddev, dev_t dev)
md_update_sb(mddev, 1);
md_new_event(mddev);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
-
return 0;
busy:
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_cancel(mddev);
printk(KERN_WARNING "md: cannot remove active disk %s from %s ...\n",
bdevname(rdev->bdev,b), mdname(mddev));
return -EBUSY;
@@ -6068,14 +6112,12 @@ static int hot_add_disk(struct mddev *mddev, dev_t dev)
goto abort_export;
}
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
clear_bit(In_sync, &rdev->flags);
rdev->desc_nr = -1;
rdev->saved_raid_disk = -1;
err = bind_rdev_to_array(rdev, mddev);
if (err)
- goto abort_clustered;
+ goto abort_export;
/*
* The rest should better be atomic, we can have disk failures
@@ -6085,9 +6127,6 @@ static int hot_add_disk(struct mddev *mddev, dev_t dev)
rdev->raid_disk = -1;
md_update_sb(mddev, 1);
-
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
/*
* Kick recovery, maybe this spare has to be added to the
* array immediately.
@@ -6097,9 +6136,6 @@ static int hot_add_disk(struct mddev *mddev, dev_t dev)
md_new_event(mddev);
return 0;
-abort_clustered:
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_cancel(mddev);
abort_export:
export_rdev(rdev);
return err;
@@ -6417,8 +6453,6 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
return rv;
}
}
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
if (info->size >= 0 && mddev->dev_sectors / 2 != info->size)
rv = update_size(mddev, (sector_t)info->size * 2);
@@ -6476,12 +6510,8 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
}
}
md_update_sb(mddev, 1);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
return rv;
err:
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_cancel(mddev);
return rv;
}
@@ -7603,11 +7633,7 @@ int md_allow_write(struct mddev *mddev)
mddev->safemode == 0)
mddev->safemode = 1;
spin_unlock(&mddev->lock);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
md_update_sb(mddev, 0);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
sysfs_notify_dirent_safe(mddev->sysfs_state);
} else
spin_unlock(&mddev->lock);
@@ -8186,13 +8212,8 @@ void md_check_recovery(struct mddev *mddev)
sysfs_notify_dirent_safe(mddev->sysfs_state);
}
- if (mddev->flags & MD_UPDATE_SB_FLAGS) {
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
+ if (mddev->flags & MD_UPDATE_SB_FLAGS)
md_update_sb(mddev, 0);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
- }
if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) &&
!test_bit(MD_RECOVERY_DONE, &mddev->recovery)) {
@@ -8290,8 +8311,6 @@ void md_reap_sync_thread(struct mddev *mddev)
set_bit(MD_CHANGE_DEVS, &mddev->flags);
}
}
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_start(mddev);
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
mddev->pers->finish_reshape)
mddev->pers->finish_reshape(mddev);
@@ -8304,8 +8323,6 @@ void md_reap_sync_thread(struct mddev *mddev)
rdev->saved_raid_disk = -1;
md_update_sb(mddev, 1);
- if (mddev_is_clustered(mddev))
- md_cluster_ops->metadata_update_finish(mddev);
clear_bit(MD_RECOVERY_RUNNING, &mddev->recovery);
clear_bit(MD_RECOVERY_DONE, &mddev->recovery);
clear_bit(MD_RECOVERY_SYNC, &mddev->recovery);
--
1.8.5.6
^ permalink raw reply related
* [PATCH 2/3] md-cluster: Improve md_reload_sb to be less error prone
From: Goldwyn Rodrigues @ 2015-09-29 15:21 UTC (permalink / raw)
To: linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-1-git-send-email-rgoldwyn@suse.com>
md_reload_sb is too simplistic and it explicitly needs to determine
the changes made by the writing node. However, there are multiple areas
where a simple reload could fail.
Instead, read the superblock of one of the "good" rdevs and update
the necessary information:
- read the superblock into a newly allocated page, by temporarily
swapping out rdev->sb_page and calling ->load_super.
- if that fails return
- if it succeeds, call check_sb_changes
1. iterates over list of active devices and checks the matching
dev_roles[] value.
If that is 'faulty', the device must be marked as faulty
- call md_error to mark the device as faulty. Make sure
not to set CHANGE_DEVS and wakeup mddev->thread or else
it would initiate a resync process, which is the responsibility
of the "primary" node.
- clear the Blocked bit
- Call remove_and_add_spares() to hot remove the device.
If the device is 'spare':
- call remove_and_add_spares() to get the number of spares
added in this operation.
- Reduce mddev->degraded to mark the array as not degraded.
2. reset recovery_cp
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md-cluster.c | 27 ++++++-----
drivers/md/md.c | 119 ++++++++++++++++++++++++++++++++++++++++--------
drivers/md/md.h | 2 +-
3 files changed, 117 insertions(+), 31 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index b94a2e6..4645a93 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -424,8 +424,7 @@ static void process_add_new_disk(struct mddev *mddev, struct cluster_msg *cmsg)
static void process_metadata_update(struct mddev *mddev, struct cluster_msg *msg)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
-
- md_reload_sb(mddev);
+ md_reload_sb(mddev, le32_to_cpu(msg->raid_slot));
dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
}
@@ -811,11 +810,23 @@ static int metadata_update_finish(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
struct cluster_msg cmsg;
- int ret;
+ struct md_rdev *rdev;
+ int ret = 0;
memset(&cmsg, 0, sizeof(cmsg));
cmsg.type = cpu_to_le32(METADATA_UPDATED);
- ret = __sendmsg(cinfo, &cmsg);
+ cmsg.raid_slot = -1;
+ /* Pick up a good active device number to send.
+ */
+ rdev_for_each(rdev, mddev)
+ if (rdev->raid_disk > -1 && !test_bit(Faulty, &rdev->flags)) {
+ cmsg.raid_slot = cpu_to_le32(rdev->desc_nr);
+ break;
+ }
+ if (cmsg.raid_slot >= 0)
+ ret = __sendmsg(cinfo, &cmsg);
+ else
+ pr_warn("md-cluster: No good device id found to send\n");
unlock_comm(cinfo);
return ret;
}
@@ -899,15 +910,9 @@ static int add_new_disk_start(struct mddev *mddev, struct md_rdev *rdev)
static int add_new_disk_finish(struct mddev *mddev)
{
- struct cluster_msg cmsg;
- struct md_cluster_info *cinfo = mddev->cluster_info;
- int ret;
/* Write sb and inform others */
md_update_sb(mddev, 1);
- cmsg.type = METADATA_UPDATED;
- ret = __sendmsg(cinfo, &cmsg);
- unlock_comm(cinfo);
- return ret;
+ return metadata_update_finish(mddev);
}
static int new_disk_ack(struct mddev *mddev, bool ack)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 883f675..fe3ce06 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -7035,7 +7035,7 @@ void md_unregister_thread(struct md_thread **threadp)
}
EXPORT_SYMBOL(md_unregister_thread);
-void md_error(struct mddev *mddev, struct md_rdev *rdev)
+void __md_error(struct mddev *mddev, struct md_rdev *rdev, bool writeout)
{
if (!rdev || test_bit(Faulty, &rdev->flags))
return;
@@ -7046,15 +7046,24 @@ void md_error(struct mddev *mddev, struct md_rdev *rdev)
if (mddev->degraded)
set_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
sysfs_notify_dirent_safe(rdev->sysfs_state);
- set_bit(MD_RECOVERY_INTR, &mddev->recovery);
- set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
- md_wakeup_thread(mddev->thread);
+ if (writeout) {
+ set_bit(MD_RECOVERY_INTR, &mddev->recovery);
+ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
+ md_wakeup_thread(mddev->thread);
+ }
if (mddev->event_work.func)
queue_work(md_misc_wq, &mddev->event_work);
md_new_event_inintr(mddev);
}
EXPORT_SYMBOL(md_error);
+void md_error(struct mddev *mddev, struct md_rdev *rdev)
+{
+ __md_error(mddev, rdev, true);
+
+}
+
+
/* seq_file implementation /proc/mdstat */
static void status_unused(struct seq_file *seq)
@@ -7989,7 +7998,7 @@ void md_do_sync(struct md_thread *thread)
EXPORT_SYMBOL_GPL(md_do_sync);
static int remove_and_add_spares(struct mddev *mddev,
- struct md_rdev *this)
+ struct md_rdev *this)
{
struct md_rdev *rdev;
int spares = 0;
@@ -8919,25 +8928,97 @@ err_wq:
return ret;
}
-void md_reload_sb(struct mddev *mddev)
+static void check_sb_changes(struct mddev *mddev, struct md_rdev *rdev)
{
- struct md_rdev *rdev, *tmp;
+ struct mdp_superblock_1 *sb = page_address(rdev->sb_page);
+ struct md_rdev *rdev2;
+ int role, ret;
+ char b[BDEVNAME_SIZE];
- rdev_for_each_safe(rdev, tmp, mddev) {
- rdev->sb_loaded = 0;
- ClearPageUptodate(rdev->sb_page);
+ /* Check for change of roles in the active devices */
+ rdev_for_each(rdev2, mddev) {
+ if (test_bit(Faulty, &rdev2->flags))
+ continue;
+
+ /* Check if the roles changed */
+ role = le16_to_cpu(sb->dev_roles[rdev2->desc_nr]);
+ if (role != rdev2->raid_disk) {
+ /* got activated */
+ if (rdev2->raid_disk == -1 && role != 0xffff) {
+ ret = remove_and_add_spares(mddev, rdev2);
+ /* This should ideally be done at the end of
+ * a resync operation.
+ */
+ if (ret) {
+ set_bit(In_sync, &rdev2->flags);
+ mddev->degraded -= ret;
+ }
+ pr_info("Activated spare: %s\n",
+ bdevname(rdev2->bdev,b));
+ continue;
+ }
+ /* device faulty
+ * We just want to do the minimum to mark the disk
+ * as faulty. The recovery is performed by the
+ * one who initiated the error.
+ */
+ if ((role == 0xfffe) || (role == 0xfffd)) {
+ __md_error(mddev, rdev2, false);
+ clear_bit(Blocked, &rdev2->flags);
+ remove_and_add_spares(mddev, rdev2);
+ }
+ }
}
- mddev->raid_disks = 0;
- analyze_sbs(mddev);
- rdev_for_each_safe(rdev, tmp, mddev) {
- struct mdp_superblock_1 *sb = page_address(rdev->sb_page);
- /* since we don't write to faulty devices, we figure out if the
- * disk is faulty by comparing events
- */
- if (mddev->events > sb->events)
- set_bit(Faulty, &rdev->flags);
+
+ /* recovery_cp changed */
+ if (le64_to_cpu(sb->resync_offset) != mddev->recovery_cp) {
+ pr_info("%s:%d recovery_cp changed from %lu to %lu\n", __func__,
+ __LINE__, mddev->recovery_cp,
+ (unsigned long) le64_to_cpu(sb->resync_offset));
+ mddev->recovery_cp = le64_to_cpu(sb->resync_offset);
+ }
+
+ /* Finally set the event to be up to date */
+ mddev->events = le64_to_cpu(sb->events);
+}
+
+void md_reload_sb(struct mddev *mddev, int nr)
+{
+ struct md_rdev *rdev;
+ struct page *swapout;
+ int err;
+
+ /* Find the rdev */
+ rdev_for_each_rcu(rdev, mddev) {
+ if (rdev->desc_nr == nr)
+ break;
+ }
+
+ if (!rdev || rdev->desc_nr != nr) {
+ pr_warn("%s: %d Could not find rdev with nr %d\n", __func__, __LINE__, nr);
+ return;
+ }
+ /* Store the sb page of the rdev in the swapout temporary
+ * variable in case we err in the future
+ */
+ swapout = rdev->sb_page;
+ rdev->sb_page = NULL;
+ alloc_disk_sb(rdev);
+ ClearPageUptodate(rdev->sb_page);
+ rdev->sb_loaded = 0;
+ err = super_types[mddev->major_version].load_super(rdev, NULL, mddev->minor_version);
+
+ if (err < 0) {
+ pr_warn("%s: %d Could not reload rdev(%d) err: %d. Restoring old values\n",
+ __func__, __LINE__, nr, err);
+ put_page(rdev->sb_page);
+ rdev->sb_page = swapout;
+ rdev->sb_loaded = 1;
+ return;
}
+ check_sb_changes(mddev, rdev);
+ put_page(swapout);
}
EXPORT_SYMBOL(md_reload_sb);
diff --git a/drivers/md/md.h b/drivers/md/md.h
index ab33957..2ea0035 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -658,7 +658,7 @@ extern struct bio *bio_alloc_mddev(gfp_t gfp_mask, int nr_iovecs,
struct mddev *mddev);
extern void md_unplug(struct blk_plug_cb *cb, bool from_schedule);
-extern void md_reload_sb(struct mddev *mddev);
+extern void md_reload_sb(struct mddev *mddev, int raid_disk);
extern void md_update_sb(struct mddev *mddev, int force);
extern void md_kick_rdev_from_array(struct md_rdev * rdev);
struct md_rdev *md_find_rdev_nr_rcu(struct mddev *mddev, int nr);
--
1.8.5.6
^ permalink raw reply related
* [PATCH 1/3] remove_and_add_spares() to activate specific rdev
From: Goldwyn Rodrigues @ 2015-09-29 15:21 UTC (permalink / raw)
To: linux-raid; +Cc: Goldwyn Rodrigues
In-Reply-To: <1443540089-5398-1-git-send-email-rgoldwyn@suse.com>
remove_and_add_spares() checks for all devices to activate spare.
Change it to activate a specific device passed as spare.
So, remove_and_add_spares() can be used to activate spares in
slot_store() as well.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
drivers/md/md.c | 16 ++++++----------
1 file changed, 6 insertions(+), 10 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 9798a99..883f675 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2691,15 +2691,9 @@ slot_store(struct md_rdev *rdev, const char *buf, size_t len)
rdev->saved_raid_disk = -1;
clear_bit(In_sync, &rdev->flags);
clear_bit(Bitmap_sync, &rdev->flags);
- err = rdev->mddev->pers->
- hot_add_disk(rdev->mddev, rdev);
- if (err) {
- rdev->raid_disk = -1;
- return err;
- } else
- sysfs_notify_dirent_safe(rdev->sysfs_state);
- if (sysfs_link_rdev(rdev->mddev, rdev))
- /* failure here is OK */;
+ remove_and_add_spares(rdev->mddev, rdev);
+ if (rdev->raid_disk == -1)
+ return -EBUSY;
/* don't wakeup anyone, leave that to userspace. */
} else {
if (slot >= rdev->mddev->raid_disks &&
@@ -8018,10 +8012,12 @@ static int remove_and_add_spares(struct mddev *mddev,
if (removed && mddev->kobj.sd)
sysfs_notify(&mddev->kobj, NULL, "degraded");
- if (this)
+ if (this && removed)
goto no_add;
rdev_for_each(rdev, mddev) {
+ if (this && this != rdev)
+ continue;
if (rdev->raid_disk >= 0 &&
!test_bit(In_sync, &rdev->flags) &&
!test_bit(Faulty, &rdev->flags))
--
1.8.5.6
^ permalink raw reply related
* [PATCH 0/3] md-cluster: A better way for METADATA_UPDATED processing
From: Goldwyn Rodrigues @ 2015-09-29 15:21 UTC (permalink / raw)
To: linux-raid
The processing of METADATA_UPDATED message is too simple and prone to
errors. Besides, it would not update the internal data structures as
required.
This set of patches reads the superblock from one of the device of the MD
and checks for changes in the in-memory data structures. If there is a change,
it performs the necessary actions to keep the internal data structures
as it would be in the primary node.
An example is if a devices turns faulty. The algorithm is:
1. The primary node marks the device as faulty and updates the superblock
2. The primary node sends METADATA_UPDATED with an advisory device number to the rest of the nodes.
3. The secondary node on receiving the METADATA_UPDATED message
3.1 Reads the superblock
3.2 Detects a device has failed by comparing with memory structure
3.3 Calls the necessary functions to record the failure and get the device out of the active array.
3.4 Acknowledges the message.
^ permalink raw reply
* Re: raid5-cache I/O path improvements V2
From: Christoph Hellwig @ 2015-09-28 14:01 UTC (permalink / raw)
To: Shaohua Li
Cc: Neil Brown, Christoph Hellwig, linux-raid@vger.kernel.org,
Kernel Team, dan.j.williams@intel.com
In-Reply-To: <20150915215458.GA1943628@devbig084.prn1.facebook.com>
So the summary is that for now you want me to resend with a patch
to opt into using FUA?
Also do you have a git tree as a baseline somewhere? I've been collecting
the patches you've sent, but with increasing time I fear I might have
missed something.
^ permalink raw reply
* Start reshape failed
From: Xiao Ni @ 2015-09-28 8:52 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <152634201.28921946.1443426236655.JavaMail.zimbra@redhat.com>
Hi Neil
When I add one disk to a 3disks raid5 and try to run --grow to let raid devices to 4.
It's failed to start reshape. I used 4.3.0-rc2 and the latest mdadm git tree.
[root@storageqe-09 md]# mdadm --grow /dev/md0 --raid-devices=4
mdadm: Failed to initiate reshape!
[root@storageqe-09 md]# uname -r
4.3.0-rc2
[root@storageqe-09 md]# mdadm --version
mdadm - v3.3.4-24-g86a406c - 28th September 2015
After some analysis, I found some hints.
A: when run --grow, it want to write reshape to sync_action. Before that, it set SET_ARRAY_INFO
in impose_reshape first. When set SET_ARRAY_INFO, it get mutex lock mddev->reconfig_mutex, and
will call mddev_resume. In mddev_resume it set MD_RECOVERY_NEEDED.
B: At the same time raid5d run. And it call md_check_recovery. But it can't get the lock
mddev->reconfig_mutex. So it misses the chance to clear MD_RECOVERY_NEEDED.
After A, it write reshape to sync_action. It calls action_store. It will return EBUSY because
the MD_RECOVERY_NEEDED is already set.
} else if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) ||
test_bit(MD_RECOVERY_NEEDED, &mddev->recovery))
return -EBUSY;
It's a little complex. I add md_check_recovery in action_store and the problem can be fixed.
But I think maybe it's not a right way to fix this.
Could you give some suggestions?
Best Regards
Xiao
^ permalink raw reply
* Re: Guidance on reshape stuck at 0% after --grow
From: Guillaume Paumier @ 2015-09-27 23:53 UTC (permalink / raw)
To: Wols Lists; +Cc: Mikael Abrahamsson, linux-raid
In-Reply-To: <CAKWBTuFTANRTWP692hf9Et_8GPvx=uLvYowKRCLdhQPMqhCKKg@mail.gmail.com>
Hello,
After a discussion on IRC, I rebooted the machine to boot using
SystemRescueCD, in order to see if a newer version would work better.
It didn't, but after rebooting into the regular OS of the machine, the
array was assembled. It's still not running, though:
mdadm -D gives the following output:
___________________________________________
# mdadm -D /dev/md0
---------------------------------------------------------------------------
/dev/md0:
Version : 1.0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Used Dev Size : -1
Raid Devices : 8
Total Devices : 9
Persistence : Superblock is persistent
Update Time : Tue Sep 22 13:25:43 2015
State : active, Not Started
Active Devices : 9
Working Devices : 9
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 128K
Delta Devices : -1, (9->8)
Name : (redacted):0
UUID : eea59047:120a0365:353da182:6787e030
Events : 35500
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
2 8 97 2 active sync /dev/sdg1
3 8 113 3 active sync /dev/sdh1
4 8 129 4 active sync /dev/sdi1
5 8 145 5 active sync /dev/sdj1
7 8 81 6 active sync /dev/sdf1
6 8 65 7 active sync /dev/sde1
8 8 17 8 active sync /dev/sdb1
___________________________________________
Note that "Raid Devices : 8", "Total Devices : 9" but /dev/sdb1 is
still listed as RaidDevice #8 (i.e. the 9th device) and in "active
sync". I'm not sure how this works.
dmesg still indicates that the reshape can't proceed:
___________________________________________
# dmesg | grep md
---------------------------------------------------------------------------
[ 4.366341] md: bind<sdg1>
[ 4.368685] md: bind<sdd1>
[ 4.370618] md: bind<sdf1>
[ 4.387630] md: bind<sdh1>
[ 4.392266] md: bind<sdc1>
[ 4.397084] md: bind<sdb1>
[ 4.411668] md: bind<sdj1>
[ 4.453465] md: bind<sde1>
[ 4.489803] md: bind<sdi1>
[ 39.129640] md: raid6 personality registered for level 6
[ 39.129643] md: raid5 personality registered for level 5
[ 39.129644] md: raid4 personality registered for level 4
[ 39.133609] md/raid:md0: reshape_position too early for
auto-recovery - aborting.
[ 39.133613] md: pers->run() failed ...
___________________________________________
At this point, I'm wondering if it would help to remove sdb1 from the
array. Does that sound like a good idea? If so, should I attempt to
"fail" it and format it? Or try to shrink to 8 devices instead of 9?
For reference:
___________________________________________
mdadm --examine /dev/sd[b-j]1
-------------------------------------------------------------------------------
mdadm --examine /dev/sd[b-j]1
/dev/sdb1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033128 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : bde6e5ab:5f4a56e9:10ad82fc:301d477b
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Bad Block Log : 512 entries available at offset -8 sectors
Checksum : 283907e0 - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 8
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : e1b689b5:b4a2c5a7:56057b69:a9101af0
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : d20f7c1e - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 0
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 1d8e74d3:9abd37f8:f2cf0ab8:02fdcfd6
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : 75b22537 - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 1
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : ddf17d3d:ea944bfb:6886cc91:3366f55f
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : 45b67ff2 - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 7
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 38675f59:ea412b1f:67d6ed9a:a33fc5dd
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : c68cbbd - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 6
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : b24758e6:042412c5:9b5a3c06:f167aedf
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : ac803ace - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 2
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 00e47d82:b49c3905:3ed961fe:40a5f259
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : fb370bbe - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 3
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : a7e34040:fa12382f:c2ef3d85:9c95b1d0
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : e093888c - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 4
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj1:
Magic : a92b4efc
Version : 1.0
Feature Map : 0x5
Array UUID : eea59047:120a0365:353da182:6787e030
Name : (redacted):0
Creation Time : Thu Aug 1 12:23:07 2013
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 7814033136 (3726.02 GiB 4000.78 GB)
Array Size : 23442098688 (22356.13 GiB 24004.71 GB)
Used Dev Size : 7814032896 (3726.02 GiB 4000.78 GB)
Super Offset : 7814033392 sectors
Unused Space : before=0 sectors, after=480 sectors
State : clean
Device UUID : 9d89c55d:9f4a2181:6b87922f:0681d580
Internal Bitmap : -16 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (9->8)
Update Time : Tue Sep 22 13:25:43 2015
Checksum : aa79f183 - correct
Events : 35500
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 5
Array State : AAAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
___________________________________________
--
Guillaume Paumier
^ permalink raw reply
* Re: Guidance on reshape stuck at 0% after --grow
From: Guillaume Paumier @ 2015-09-27 0:59 UTC (permalink / raw)
To: Wols Lists; +Cc: Mikael Abrahamsson, linux-raid
In-Reply-To: <5602EB5E.6010703@youngman.org.uk>
Hello,
On 23 September 2015 at 11:11, Wols Lists <antlists@youngman.org.uk> wrote:
>
> Read up on loopback devices. Sorry, never done it, can't help
> personally, but it sticks a ram-disk layer between mdadm and the actual
> disk.
>
> So the disks themselves are read-only, and you can try things out to see
> what works. If it fails, you throw the loopback layer away, if it works
> you can run it on the disk itself.
Thank you; that seems similar to what is described at
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file
, and was also suggested in
http://www.spinics.net/lists/raid/msg49669.html
It seems I'm running into another issue, described at the bottom of
https://raid.wiki.kernel.org/index.php/RAID_Recovery#Restore_array_by_recreating_.28after_multiple_device_failure.29
:
"If upon running the above with the --size parameter you get, as one
of the authors of this page did, an error such as: "mdadm: /dev/sdb1
is smaller than given size. xxxK < yyyK + metadata", you may have
stumbled upon a problem where the array was initially created with an
earlier version of mdadm that reserved less device space. The solution
seems to be to find an earlier version of mdadm to run with the
creation command above"
This seems consistent with the errors I'm getting, in the form:
mdadm: /dev/sdc1 is smaller than given size. 3906885632K < 3907016448K
+ metadata
I found the list of previous versions of mdadm at
https://www.kernel.org/pub/linux/utils/raid/mdadm/ , but I'm not sure
which one to pick, and how easy it is to use those earlier versions.
According to my logs, I originally created the array in late July
2013. I'm not afraid of compiling it from source, but any additional
information would be helpful.
--
Guillaume Paumier
^ permalink raw reply
* Re: Unable to assemble RAID6 after Ubuntu>Arch switch
From: Mathias Burén @ 2015-09-26 18:01 UTC (permalink / raw)
To: Phil Turmel; +Cc: Alexander Afonyashin, Linux-RAID
In-Reply-To: <56068280.2090009@turmel.org>
On 26 September 2015 at 12:33, Phil Turmel <philip@turmel.org> wrote:
> On 09/26/2015 08:16 AM, Mathias Burén wrote:
>> Hi,
>>
>> Would it be possible to somehow manually assemble array under Arch?
>> Since the Arch kernel doesn't see /dev/sda1 and /dev/sdb1 (fdisk sees
>> them but they do not appear in /dev) I can't assemble arrays.
>> partprobe doesn't help. Any ideas why the partitions wouldn't populate in /dev ?
>
> Is there any chance you've managed to get a GPT label onto those disks?
> And your arch kernel has omitted GPT support?
>
> Phil
>
Good idea Phil, but it's enabled by default:
$ zgrep CONFIG_EFI_PARTITION /proc/config.gz
CONFIG_EFI_PARTITION=y
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Unable to assemble RAID6 after Ubuntu>Arch switch
From: Mathias Burén @ 2015-09-26 12:16 UTC (permalink / raw)
To: Alexander Afonyashin; +Cc: Linux-RAID
In-Reply-To: <CAETWcftkkJr_MjP5ReBsOCW5ioCu81KQcdwr_1-=qUD25e1ssw@mail.gmail.com>
Hi,
Would it be possible to somehow manually assemble array under Arch?
Since the Arch kernel doesn't see /dev/sda1 and /dev/sdb1 (fdisk sees
them but they do not appear in /dev) I can't assemble arrays.
partprobe doesn't help. Any ideas why the partitions wouldn't populate in /dev ?
Thanks
Mathias
On 25 September 2015 at 09:47, Alexander Afonyashin
<a.afonyashin@madnet-team.ru> wrote:
> Hi,
>
> I can suppose that 2 superblocks exist on disk /dev/sda (since you
> used disks at start and switch to partitions later). One is recorded
> for raid built with /dev/sdX devices and 2nd - for raid with /dev/sdX1
> partitions. The only idea so far is to assemble raid on Ubuntu, remove
> /dev/sda1 from raid (leave it in degraded state), then zero superblock
> both for /dev/sda and /dev/sda1, and, finally, add /dev/sda1 back as
> 'new' drive, wait for raid rebuild, perform these steps with other
> drives. Not sure if it's optimal, though.
>
> Regards,
> Alexander
>
> On Thu, Sep 24, 2015 at 9:17 PM, Mathias Burén <mathias.buren@gmail.com> wrote:
>> Yeah it's odd. I believe that ages ago I had an array on the full
>> disks (not partitions). I've since switched to RAID based on
>> partitions (this array that works in Ubuntu but not Arch). Maybe that
>> is where the confusion comes from.
>>
>> I just realized I provided the examine data for the old unused array
>> within Arch, and you're right, sda1 and sdb1 don't exist under Arch:
>>
>> $ sudo partprobe --summary
>> /dev/sda: msdos partitions 1
>> /dev/sdb: msdos partitions 1
>> /dev/sdc: msdos partitions 1 2
>> /dev/sdd: msdos partitions 1 2
>> /dev/sde: msdos partitions 1 2
>> /dev/sdf: msdos partitions 1 2 3
>>
>> $ sudo fdisk -l /dev/sda
>> Disk /dev/sda: 1.8 TiB, 1998998994944 bytes, 3904294912 sectors
>> Units: sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 512 bytes
>> I/O size (minimum/optimal): 512 bytes / 512 bytes
>> Disklabel type: dos
>> Disk identifier: 0x056cc1c0
>>
>> Device Boot Start End Sectors Size Id Type
>> /dev/sda1 2048 3904294911 3904292864 1.8T fd Linux raid autodetect
>>
>> $ sudo fdisk -l /dev/sdb
>> Disk /dev/sdb: 1.8 TiB, 1998998994944 bytes, 3904294912 sectors
>> Units: sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 512 bytes
>> I/O size (minimum/optimal): 512 bytes / 512 bytes
>> Disklabel type: dos
>> Disk identifier: 0x9fb14a5c
>>
>> Device Boot Start End Sectors Size Id Type
>> /dev/sdb1 2048 3904294911 3904292864 1.8T fd Linux raid autodetect
>>
>> $ ls -la /dev/sda*
>> brw-rw---- 1 root disk 8, 0 Sep 24 20:14 /dev/sda
>> $ ls -la /dev/sdb*
>> brw-rw---- 1 root disk 8, 16 Sep 24 20:14 /dev/sdb
>>
>>
>> So for some reason the kernel does not see the partitions even though
>> partprobe and fdisk show them.
>>
>> Regards,
>> Mathias
>>
>> On 24 September 2015 at 10:17, Alexander Afonyashin
>> <a.afonyashin@madnet-team.ru> wrote:
>>> Hi Matias,
>>>
>>> I wonder why Ubuntu detects raid6 parts as /dev/sdX1 (and counts only
>>> 5 disks) while Arch detects /dev/sdX (and counts 6 disks)? Try to
>>> assemble raid6 on Arch manually again by issuing:
>>>
>>> mdadm -S /dev/md0
>>> mdadm -A /dev/md0 /dev/sdb /dev/sda /dev/sde /dev/sdd /dev/sdc -
>>> exactly as shown in mdadm's output on Ubuntu
>>>
>>> P.S. Still confused why Arch doesn't recognize /dev/sda1 etc.
>>>
>>> Regards,
>>> Alexander
>>>
>>> On Wed, Sep 23, 2015 at 9:35 PM, Mathias Burén <mathias.buren@gmail.com> wrote:
>>>> Hi Alexander,
>>>>
>>>> I didn't try to grow the array or anything, just swapped out the OS.
>>>> When I boot back into Ubuntu it assembles fine.
>>>>
>>>> This is from within Ubuntu (I did a mdadm --assemble --scan using
>>>> mdadm - v3.3.2-7-g21dc471 - 03th November 2014)
>>>>
>>>> mdadm -D on RAID6
>>>> /dev/md0:
>>>> Version : 1.2
>>>> Creation Time : Thu Nov 20 23:52:58 2014
>>>> Raid Level : raid6
>>>> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
>>>> Used Dev Size : 1952015360 (1861.59 GiB 1998.86 GB)
>>>> Raid Devices : 5
>>>> Total Devices : 5
>>>> Persistence : Superblock is persistent
>>>>
>>>> Intent Bitmap : Internal
>>>>
>>>> Update Time : Sun Sep 20 19:47:50 2015
>>>> State : clean
>>>> Active Devices : 5
>>>> Working Devices : 5
>>>> Failed Devices : 0
>>>> Spare Devices : 0
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 512K
>>>>
>>>> Name : ion:0 (local to host ion)
>>>> UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
>>>> Events : 59736
>>>>
>>>> Number Major Minor RaidDevice State
>>>> 0 8 17 0 active sync /dev/sdb1
>>>> 1 8 33 1 active sync /dev/sdc1
>>>> 2 8 49 2 active sync /dev/sdd1
>>>> 3 8 65 3 active sync /dev/sde1
>>>> 4 8 1 4 active sync /dev/sda1
>>>>
>>>>
>>>> mdadm -D on RAID0
>>>> /dev/md1:
>>>> Version : 1.2
>>>> Creation Time : Thu Nov 20 23:53:48 2014
>>>> Raid Level : raid0
>>>> Array Size : 4098048 (3.91 GiB 4.20 GB)
>>>> Raid Devices : 3
>>>> Total Devices : 3
>>>> Persistence : Superblock is persistent
>>>>
>>>> Update Time : Thu Nov 20 23:53:48 2014
>>>> State : clean
>>>> Active Devices : 3
>>>> Working Devices : 3
>>>> Failed Devices : 0
>>>> Spare Devices : 0
>>>>
>>>> Chunk Size : 512K
>>>>
>>>> Name : ion:1 (local to host ion)
>>>> UUID : a20b70d4:7ee17e3f:abab74f8:dadb8cd8
>>>> Events : 0
>>>>
>>>> Number Major Minor RaidDevice State
>>>> 0 8 34 0 active sync /dev/sdc2
>>>> 1 8 50 1 active sync /dev/sdd2
>>>> 2 8 66 2 active sync /dev/sde2
>>>>
>>>>
>>>>
>>>> lsdrv from within Ubuntu
>>>> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
>>>> Logic MegaRAID SAS 1068
>>>> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
>>>> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>> │ └sda1 1.82t [8:1] MD raid6 (4/5) (w/ sdb1,sdc1,sdd1,sde1) in_sync
>>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>>> │ └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>>> │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>>> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
>>>> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>> └sdb1 1.82t [8:17] MD raid6 (0/5) (w/ sda1,sdc1,sdd1,sde1) in_sync
>>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>>> └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>>> ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>>> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
>>>> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>>>> ├scsi 1:0:0:0 ATA Corsair CSSD-F60 {10326505580009990027}
>>>> │└sdf 55.90g [8:80] Partitioned (dos)
>>>> │ ├sdf1 243.00m [8:81] ext2 {b45c13c8-4246-43ee-ac2c-f9bb5de099f8}
>>>> │ │└Mounted as /dev/sdf1 @ /boot
>>>> │ ├sdf2 1.00k [8:82] Partitioned (dos)
>>>> │ └sdf5 55.66g [8:85] PV LVM2_member 55.66g used, 0 free
>>>> {zZ5Dgy-E7v8-Y1Mi-EuqG-uXUb-DseW-s7goxG}
>>>> │ └VG ion 55.66g 0 free {XSaCyP-phLN-m4IH-2cJ7-1XXw-b3uh-qWBl8N}
>>>> │ ├dm-0 52.41g [252:0] LV root ext4 {63342e0d-1b4f-4475-9835-d3a2a6610e8f}
>>>> │ │└Mounted as /dev/mapper/ion-root @ /
>>>> │ └dm-1 3.25g [252:1] LV swap_1 Empty/Unknown
>>>> │ └dm-2 3.25g [252:2] swap {0c81625d-ee9e-4a7a-9613-410c1cf53e72}
>>>> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
>>>> │└sdc 1.82t [8:32] MD raid6 (6) inactive 'ion:md0'
>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>> │ ├sdc1 1.82t [8:33] MD raid6 (1/5) (w/ sda1,sdb1,sdd1,sde1) in_sync
>>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>>> │ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>>> │ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>>> │ └sdc2 1.30g [8:34] MD raid0 (0/3) (w/ sdd2,sde2) in_sync 'ion:1'
>>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>>> │ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>>>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>>>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>>>> ├scsi 3:x:x:x [Empty]
>>>> ├scsi 4:x:x:x [Empty]
>>>> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
>>>> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>> │ ├sdd1 1.82t [8:49] MD raid6 (2/5) (w/ sda1,sdb1,sdc1,sde1) in_sync
>>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>>> │ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>>> │ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>>> │ └sdd2 1.30g [8:50] MD raid0 (1/3) (w/ sdc2,sde2) in_sync 'ion:1'
>>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>>> │ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>>>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>>>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>>>> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
>>>> └sde 2.73t [8:64] Partitioned (dos)
>>>> ├sde1 1.82t [8:65] MD raid6 (3/5) (w/ sda1,sdb1,sdc1,sdd1) in_sync
>>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>>> │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>>> │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>>> ├sde2 1.30g [8:66] MD raid0 (2/3) (w/ sdc2,sdd2) in_sync 'ion:1'
>>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>>> │└md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>>>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>>>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>>>> └sde3 184.98g [8:67] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>>>> └Mounted as /dev/sde3 @ /media/198GB
>>>> Other Block Devices
>>>> ├loop0 0.00k [7:0] Empty/Unknown
>>>> ├loop1 0.00k [7:1] Empty/Unknown
>>>> ├loop2 0.00k [7:2] Empty/Unknown
>>>> ├loop3 0.00k [7:3] Empty/Unknown
>>>> ├loop4 0.00k [7:4] Empty/Unknown
>>>> ├loop5 0.00k [7:5] Empty/Unknown
>>>> ├loop6 0.00k [7:6] Empty/Unknown
>>>> ├loop7 0.00k [7:7] Empty/Unknown
>>>> ├md127 0.00k [9:127] MD vnone () clear, None (None) None {None}
>>>> │ Empty/Unknown
>>>> ├ram0 64.00m [1:0] Empty/Unknown
>>>> ├ram1 64.00m [1:1] Empty/Unknown
>>>> ├ram2 64.00m [1:2] Empty/Unknown
>>>> ├ram3 64.00m [1:3] Empty/Unknown
>>>> ├ram4 64.00m [1:4] Empty/Unknown
>>>> ├ram5 64.00m [1:5] Empty/Unknown
>>>> ├ram6 64.00m [1:6] Empty/Unknown
>>>> ├ram7 64.00m [1:7] Empty/Unknown
>>>> ├ram8 64.00m [1:8] Empty/Unknown
>>>> ├ram9 64.00m [1:9] Empty/Unknown
>>>> ├ram10 64.00m [1:10] Empty/Unknown
>>>> ├ram11 64.00m [1:11] Empty/Unknown
>>>> ├ram12 64.00m [1:12] Empty/Unknown
>>>> ├ram13 64.00m [1:13] Empty/Unknown
>>>> ├ram14 64.00m [1:14] Empty/Unknown
>>>> └ram15 64.00m [1:15] Empty/Unknown
>>>>
>>>>
>>>> /proc/mdstat on Ubuntu
>>>> Personalities : [raid6] [raid5] [raid4] [raid0]
>>>> md0 : active raid6 sdb1[0] sda1[4] sde1[3] sdd1[2] sdc1[1]
>>>> 5856046080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
>>>> bitmap: 0/15 pages [0KB], 65536KB chunk
>>>>
>>>> md1 : active raid0 sdc2[0] sde2[2] sdd2[1]
>>>> 4098048 blocks super 1.2 512k chunks
>>>>
>>>> Regards,
>>>> Mathias
>>>>
>>>> On 23 September 2015 at 07:24, Alexander Afonyashin
>>>> <a.afonyashin@madnet-team.ru> wrote:
>>>>> Hi,
>>>>>
>>>>> I wonder why metainfo from /dev/sdf1 thinks that there're only 5
>>>>> devices in raid6. Did you try to grow your array recently?
>>>>> Show the output of mdadm -D /dev/mdX when array is assembled on Ubuntu.
>>>>> Can you assemble the array with 4 disks: /dev/sd[a-d]?
>>>>>
>>>>> P.S. According to mdadm output metainfo on /dev/sdf1 claims that it's
>>>>> from another md-array (check Array UUID field). You may need to zero
>>>>> metadata on /dev/sdf1 and re-add in to existing raid6 array.
>>>>>
>>>>> Regards,
>>>>> Alexander
>>>>>
>>>>> On Wed, Sep 23, 2015 at 12:30 AM, Mathias Burén <mathias.buren@gmail.com> wrote:
>>>>>> Hi (please reply-all)
>>>>>>
>>>>>> I've a RAID6 array (sda sdb sdd sde sdf1) that I can't assemble under
>>>>>> Arch, but it worked fine under Ubuntu. mdadm - v3.3.4 - 3rd August
>>>>>> 2015, kernel 4.1.6
>>>>>>
>>>>>> Here is the mdadm --examine for each drive:
>>>>>>
>>>>>>
>>>>>>
>>>>>> [root@ion ~]# mdadm --examine /dev/sda
>>>>>> /dev/sda:
>>>>>> Magic : a92b4efc
>>>>>> Version : 1.2
>>>>>> Feature Map : 0x0
>>>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>>>> Name : ion:md0 (local to host ion)
>>>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>>>> Raid Level : raid6
>>>>>> Raid Devices : 6
>>>>>>
>>>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>>>> Data Offset : 262144 sectors
>>>>>> Super Offset : 8 sectors
>>>>>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>>>>>> State : clean
>>>>>> Device UUID : a09fc60d:5c4a27a5:4b89bc33:29b01582
>>>>>>
>>>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>>>> Checksum : 528563ee - correct
>>>>>> Events : 97557
>>>>>>
>>>>>> Layout : left-symmetric
>>>>>> Chunk Size : 512K
>>>>>>
>>>>>> Device Role : Active device 3
>>>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>>
>>>>>> [root@ion ~]# mdadm --examine /dev/sdb
>>>>>> /dev/sdb:
>>>>>> Magic : a92b4efc
>>>>>> Version : 1.2
>>>>>> Feature Map : 0x0
>>>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>>>> Name : ion:md0 (local to host ion)
>>>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>>>> Raid Level : raid6
>>>>>> Raid Devices : 6
>>>>>>
>>>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>>>> Data Offset : 262144 sectors
>>>>>> Super Offset : 8 sectors
>>>>>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>>>>>> State : clean
>>>>>> Device UUID : 93568b01:632395bf:7d0082a5:db9b6ff9
>>>>>>
>>>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>>>> Checksum : 49d756ca - correct
>>>>>> Events : 97557
>>>>>>
>>>>>> Layout : left-symmetric
>>>>>> Chunk Size : 512K
>>>>>>
>>>>>> Device Role : Active device 0
>>>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>>
>>>>>>
>>>>>> [root@ion ~]# mdadm --examine /dev/sdd
>>>>>> /dev/sdd:
>>>>>> Magic : a92b4efc
>>>>>> Version : 1.2
>>>>>> Feature Map : 0x0
>>>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>>>> Name : ion:md0 (local to host ion)
>>>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>>>> Raid Level : raid6
>>>>>> Raid Devices : 6
>>>>>>
>>>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>>>> Data Offset : 262144 sectors
>>>>>> Super Offset : 8 sectors
>>>>>> Unused Space : before=262064 sectors, after=1200 sectors
>>>>>> State : clean
>>>>>> Device UUID : 78df2586:cb5649aa:e0b6d211:d92dc224
>>>>>>
>>>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>>>> Checksum : 50c95b7c - correct
>>>>>> Events : 97557
>>>>>>
>>>>>> Layout : left-symmetric
>>>>>> Chunk Size : 512K
>>>>>>
>>>>>> Device Role : Active device 5
>>>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>>
>>>>>> [root@ion ~]# mdadm --examine /dev/sde
>>>>>> /dev/sde:
>>>>>> Magic : a92b4efc
>>>>>> Version : 1.2
>>>>>> Feature Map : 0x0
>>>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>>>> Name : ion:md0 (local to host ion)
>>>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>>>> Raid Level : raid6
>>>>>> Raid Devices : 6
>>>>>>
>>>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>>>> Data Offset : 262144 sectors
>>>>>> Super Offset : 8 sectors
>>>>>> Unused Space : before=262064 sectors, after=1200 sectors
>>>>>> State : clean
>>>>>> Device UUID : 41712f8c:255b0f3e:0e345f7b:e1504e42
>>>>>>
>>>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>>>> Checksum : 71b191d6 - correct
>>>>>> Events : 97557
>>>>>>
>>>>>> Layout : left-symmetric
>>>>>> Chunk Size : 512K
>>>>>>
>>>>>> Device Role : Active device 1
>>>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>>
>>>>>> [root@ion ~]# mdadm --examine /dev/sdf1
>>>>>> /dev/sdf1:
>>>>>> Magic : a92b4efc
>>>>>> Version : 1.2
>>>>>> Feature Map : 0x1
>>>>>> Array UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
>>>>>> Name : ion:0 (local to host ion)
>>>>>> Creation Time : Thu Nov 20 23:52:58 2014
>>>>>> Raid Level : raid6
>>>>>> Raid Devices : 5
>>>>>>
>>>>>> Avail Dev Size : 3904030720 (1861.59 GiB 1998.86 GB)
>>>>>> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
>>>>>> Data Offset : 262144 sectors
>>>>>> Super Offset : 8 sectors
>>>>>> Unused Space : before=262056 sectors, after=0 sectors
>>>>>> State : clean
>>>>>> Device UUID : eaa0dcba:d04a7c16:6c256916:67a491ae
>>>>>>
>>>>>> Internal Bitmap : 8 sectors from superblock
>>>>>> Update Time : Sun Sep 20 19:47:50 2015
>>>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>>>> Checksum : b1f6f725 - correct
>>>>>> Events : 59736
>>>>>>
>>>>>> Layout : left-symmetric
>>>>>> Chunk Size : 512K
>>>>>>
>>>>>> Device Role : Active device 3
>>>>>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here is ldrv:
>>>>>>
>>>>>> [root@ion ~]# python2 lsdrv
>>>>>> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
>>>>>> Logic MegaRAID SAS 1068
>>>>>> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
>>>>>> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
>>>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>>>> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
>>>>>> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
>>>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>>>> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
>>>>>> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>>>>>> ├scsi 1:0:0:0 ATA INTEL SSDSA2BW16 {BTPR2062006T160DGN}
>>>>>> │└sdc 149.05g [8:32] Partitioned (dos)
>>>>>> │ ├sdc1 256.00m [8:33] ext4 'boot' {f7727d21-646d-42f9-844b-50591b7f8358}
>>>>>> │ │└Mounted as /dev/sdc1 @ /boot
>>>>>> │ └sdc2 140.00g [8:34] PV LVM2_member 40.00g used, 100.00g free
>>>>>> {KGQ5TJ-mVvL-1Ccd-CKae-1QvS-6AFn-8jFQAw}
>>>>>> │ └VG ArchVG 140.00g 100.00g free {F2vaKY-XO4m-UEih-f9Fh-sHgX-cRsN-SLi3Oo}
>>>>>> │ ├dm-1 32.00g [254:1] LV root ext4 'root'
>>>>>> {82fa2951-25c8-4e36-bf36-9f4f7747ac46}
>>>>>> │ │└Mounted as /dev/mapper/ArchVG-root @ /
>>>>>> │ └dm-0 8.00g [254:0] LV swap swap {0e3f4596-fa50-4fed-875f-f8427084d9b5}
>>>>>> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
>>>>>> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
>>>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>>>> ├scsi 3:x:x:x [Empty]
>>>>>> ├scsi 4:x:x:x [Empty]
>>>>>> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
>>>>>> │└sde 1.82t [8:64] MD raid6 (6) inactive 'ion:md0'
>>>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>>>> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
>>>>>> └sdf 2.73t [8:80] Partitioned (dos)
>>>>>> ├sdf1 1.82t [8:81] MD raid6 (5) inactive 'ion:0'
>>>>>> {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>>>>> ├sdf2 1.30g [8:82] MD raid0 (3) inactive 'ion:1'
>>>>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>>>>> └sdf3 184.98g [8:83] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>>>>>>
>>>>>>
>>>>>> If I try:
>>>>>>
>>>>>> [root@ion ~]# mdadm --assemble --scan
>>>>>> mdadm: /dev/md/0 assembled from 1 drive - not enough to start the array.
>>>>>> mdadm: /dev/md/1 assembled from 1 drive - not enough to start the array.
>>>>>>
>>>>>> From dmesg:
>>>>>>
>>>>>> [ 1227.671344] sda: sda1
>>>>>> [ 1227.680392] md: sda does not have a valid v1.2 superblock, not importing!
>>>>>> [ 1227.680414] md: md_import_device returned -22
>>>>>> [ 1227.680462] md: md0 stopped.
>>>>>> [ 1227.707969] sdb: sdb1
>>>>>> [ 1227.718598] md: sdb does not have a valid v1.2 superblock, not importing!
>>>>>> [ 1227.718611] md: md_import_device returned -22
>>>>>> [ 1227.718631] md: md0 stopped.
>>>>>> [ 1286.334542] md: md0 stopped.
>>>>>> [ 1286.338250] md: bind<sdf1>
>>>>>> [ 1286.338390] md: md0 stopped.
>>>>>> [ 1286.338400] md: unbind<sdf1>
>>>>>> [ 1286.348350] md: export_rdev(sdf1)
>>>>>> [ 1286.372268] md: bind<sdf1>
>>>>>> [ 1286.373390] md: md1 stopped.
>>>>>> [ 1286.373936] md: bind<sdf2>
>>>>>> [ 1286.373977] md: md1 stopped.
>>>>>> [ 1286.373983] md: unbind<sdf2>
>>>>>> [ 1286.388061] md: export_rdev(sdf2)
>>>>>> [ 1286.405140] md: bind<sdf2>
>>>>>>
>>>>>> [root@ion ~]# cat /proc/mdstat
>>>>>> Personalities :
>>>>>> md1 : inactive sdf2[2](S)
>>>>>> 1366104 blocks super 1.2
>>>>>>
>>>>>> md0 : inactive sdf1[3](S)
>>>>>> 1952015360 blocks super 1.2
>>>>>>
>>>>>> unused devices: <none>
>>>>>>
>>>>>>
>>>>>> If I try manually (I'm not sure of the order though):
>>>>>>
>>>>>> [root@ion ~]# mdadm --assemble --readonly --verbose /dev/md0 /dev/sda
>>>>>> /dev/sdb /dev/sdd /dev/sde
>>>>>> mdadm: looking for devices for /dev/md0
>>>>>> mdadm: /dev/sda is identified as a member of /dev/md0, slot 3.
>>>>>> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
>>>>>> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 5.
>>>>>> mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
>>>>>> mdadm: added /dev/sde to /dev/md0 as 1
>>>>>> mdadm: no uptodate device for slot 2 of /dev/md0
>>>>>> mdadm: failed to add /dev/sda to /dev/md0: Invalid argument
>>>>>> mdadm: no uptodate device for slot 4 of /dev/md0
>>>>>> mdadm: added /dev/sdd to /dev/md0 as 5
>>>>>> mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument
>>>>>> mdadm: /dev/md0 assembled from 2 drives - need 5 to start (use --run to insist).
>>>>>>
>>>>>>
>>>>>> Any idea where I should start?
>>>>>>
>>>>>> Thanks
>>>>>> Mathias
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Unable to assemble RAID6 after Ubuntu>Arch switch
From: Phil Turmel @ 2015-09-26 11:33 UTC (permalink / raw)
To: Mathias Burén, Alexander Afonyashin; +Cc: Linux-RAID
In-Reply-To: <CADNH=7HEpag54Z7+aey+skWiThL93H2sgPpxVS77pqp4R_1NAQ@mail.gmail.com>
On 09/26/2015 08:16 AM, Mathias Burén wrote:
> Hi,
>
> Would it be possible to somehow manually assemble array under Arch?
> Since the Arch kernel doesn't see /dev/sda1 and /dev/sdb1 (fdisk sees
> them but they do not appear in /dev) I can't assemble arrays.
> partprobe doesn't help. Any ideas why the partitions wouldn't populate in /dev ?
Is there any chance you've managed to get a GPT label onto those disks?
And your arch kernel has omitted GPT support?
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Unable to assemble RAID6 after Ubuntu>Arch switch
From: Alexander Afonyashin @ 2015-09-25 8:47 UTC (permalink / raw)
To: Mathias Burén; +Cc: Linux-RAID
In-Reply-To: <CADNH=7Eyy-K3ivHAt_oYtTWdkeSDA7itZ+1Y+-EHdz8Y9_U0ww@mail.gmail.com>
Hi,
I can suppose that 2 superblocks exist on disk /dev/sda (since you
used disks at start and switch to partitions later). One is recorded
for raid built with /dev/sdX devices and 2nd - for raid with /dev/sdX1
partitions. The only idea so far is to assemble raid on Ubuntu, remove
/dev/sda1 from raid (leave it in degraded state), then zero superblock
both for /dev/sda and /dev/sda1, and, finally, add /dev/sda1 back as
'new' drive, wait for raid rebuild, perform these steps with other
drives. Not sure if it's optimal, though.
Regards,
Alexander
On Thu, Sep 24, 2015 at 9:17 PM, Mathias Burén <mathias.buren@gmail.com> wrote:
> Yeah it's odd. I believe that ages ago I had an array on the full
> disks (not partitions). I've since switched to RAID based on
> partitions (this array that works in Ubuntu but not Arch). Maybe that
> is where the confusion comes from.
>
> I just realized I provided the examine data for the old unused array
> within Arch, and you're right, sda1 and sdb1 don't exist under Arch:
>
> $ sudo partprobe --summary
> /dev/sda: msdos partitions 1
> /dev/sdb: msdos partitions 1
> /dev/sdc: msdos partitions 1 2
> /dev/sdd: msdos partitions 1 2
> /dev/sde: msdos partitions 1 2
> /dev/sdf: msdos partitions 1 2 3
>
> $ sudo fdisk -l /dev/sda
> Disk /dev/sda: 1.8 TiB, 1998998994944 bytes, 3904294912 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x056cc1c0
>
> Device Boot Start End Sectors Size Id Type
> /dev/sda1 2048 3904294911 3904292864 1.8T fd Linux raid autodetect
>
> $ sudo fdisk -l /dev/sdb
> Disk /dev/sdb: 1.8 TiB, 1998998994944 bytes, 3904294912 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x9fb14a5c
>
> Device Boot Start End Sectors Size Id Type
> /dev/sdb1 2048 3904294911 3904292864 1.8T fd Linux raid autodetect
>
> $ ls -la /dev/sda*
> brw-rw---- 1 root disk 8, 0 Sep 24 20:14 /dev/sda
> $ ls -la /dev/sdb*
> brw-rw---- 1 root disk 8, 16 Sep 24 20:14 /dev/sdb
>
>
> So for some reason the kernel does not see the partitions even though
> partprobe and fdisk show them.
>
> Regards,
> Mathias
>
> On 24 September 2015 at 10:17, Alexander Afonyashin
> <a.afonyashin@madnet-team.ru> wrote:
>> Hi Matias,
>>
>> I wonder why Ubuntu detects raid6 parts as /dev/sdX1 (and counts only
>> 5 disks) while Arch detects /dev/sdX (and counts 6 disks)? Try to
>> assemble raid6 on Arch manually again by issuing:
>>
>> mdadm -S /dev/md0
>> mdadm -A /dev/md0 /dev/sdb /dev/sda /dev/sde /dev/sdd /dev/sdc -
>> exactly as shown in mdadm's output on Ubuntu
>>
>> P.S. Still confused why Arch doesn't recognize /dev/sda1 etc.
>>
>> Regards,
>> Alexander
>>
>> On Wed, Sep 23, 2015 at 9:35 PM, Mathias Burén <mathias.buren@gmail.com> wrote:
>>> Hi Alexander,
>>>
>>> I didn't try to grow the array or anything, just swapped out the OS.
>>> When I boot back into Ubuntu it assembles fine.
>>>
>>> This is from within Ubuntu (I did a mdadm --assemble --scan using
>>> mdadm - v3.3.2-7-g21dc471 - 03th November 2014)
>>>
>>> mdadm -D on RAID6
>>> /dev/md0:
>>> Version : 1.2
>>> Creation Time : Thu Nov 20 23:52:58 2014
>>> Raid Level : raid6
>>> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
>>> Used Dev Size : 1952015360 (1861.59 GiB 1998.86 GB)
>>> Raid Devices : 5
>>> Total Devices : 5
>>> Persistence : Superblock is persistent
>>>
>>> Intent Bitmap : Internal
>>>
>>> Update Time : Sun Sep 20 19:47:50 2015
>>> State : clean
>>> Active Devices : 5
>>> Working Devices : 5
>>> Failed Devices : 0
>>> Spare Devices : 0
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 512K
>>>
>>> Name : ion:0 (local to host ion)
>>> UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
>>> Events : 59736
>>>
>>> Number Major Minor RaidDevice State
>>> 0 8 17 0 active sync /dev/sdb1
>>> 1 8 33 1 active sync /dev/sdc1
>>> 2 8 49 2 active sync /dev/sdd1
>>> 3 8 65 3 active sync /dev/sde1
>>> 4 8 1 4 active sync /dev/sda1
>>>
>>>
>>> mdadm -D on RAID0
>>> /dev/md1:
>>> Version : 1.2
>>> Creation Time : Thu Nov 20 23:53:48 2014
>>> Raid Level : raid0
>>> Array Size : 4098048 (3.91 GiB 4.20 GB)
>>> Raid Devices : 3
>>> Total Devices : 3
>>> Persistence : Superblock is persistent
>>>
>>> Update Time : Thu Nov 20 23:53:48 2014
>>> State : clean
>>> Active Devices : 3
>>> Working Devices : 3
>>> Failed Devices : 0
>>> Spare Devices : 0
>>>
>>> Chunk Size : 512K
>>>
>>> Name : ion:1 (local to host ion)
>>> UUID : a20b70d4:7ee17e3f:abab74f8:dadb8cd8
>>> Events : 0
>>>
>>> Number Major Minor RaidDevice State
>>> 0 8 34 0 active sync /dev/sdc2
>>> 1 8 50 1 active sync /dev/sdd2
>>> 2 8 66 2 active sync /dev/sde2
>>>
>>>
>>>
>>> lsdrv from within Ubuntu
>>> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
>>> Logic MegaRAID SAS 1068
>>> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
>>> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>> │ └sda1 1.82t [8:1] MD raid6 (4/5) (w/ sdb1,sdc1,sdd1,sde1) in_sync
>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>> │ └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>> │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
>>> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>> └sdb1 1.82t [8:17] MD raid6 (0/5) (w/ sda1,sdc1,sdd1,sde1) in_sync
>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>> └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>> ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
>>> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>>> ├scsi 1:0:0:0 ATA Corsair CSSD-F60 {10326505580009990027}
>>> │└sdf 55.90g [8:80] Partitioned (dos)
>>> │ ├sdf1 243.00m [8:81] ext2 {b45c13c8-4246-43ee-ac2c-f9bb5de099f8}
>>> │ │└Mounted as /dev/sdf1 @ /boot
>>> │ ├sdf2 1.00k [8:82] Partitioned (dos)
>>> │ └sdf5 55.66g [8:85] PV LVM2_member 55.66g used, 0 free
>>> {zZ5Dgy-E7v8-Y1Mi-EuqG-uXUb-DseW-s7goxG}
>>> │ └VG ion 55.66g 0 free {XSaCyP-phLN-m4IH-2cJ7-1XXw-b3uh-qWBl8N}
>>> │ ├dm-0 52.41g [252:0] LV root ext4 {63342e0d-1b4f-4475-9835-d3a2a6610e8f}
>>> │ │└Mounted as /dev/mapper/ion-root @ /
>>> │ └dm-1 3.25g [252:1] LV swap_1 Empty/Unknown
>>> │ └dm-2 3.25g [252:2] swap {0c81625d-ee9e-4a7a-9613-410c1cf53e72}
>>> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
>>> │└sdc 1.82t [8:32] MD raid6 (6) inactive 'ion:md0'
>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>> │ ├sdc1 1.82t [8:33] MD raid6 (1/5) (w/ sda1,sdb1,sdd1,sde1) in_sync
>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>> │ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>> │ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>> │ └sdc2 1.30g [8:34] MD raid0 (0/3) (w/ sdd2,sde2) in_sync 'ion:1'
>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>> │ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>>> ├scsi 3:x:x:x [Empty]
>>> ├scsi 4:x:x:x [Empty]
>>> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
>>> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>> │ ├sdd1 1.82t [8:49] MD raid6 (2/5) (w/ sda1,sdb1,sdc1,sde1) in_sync
>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>> │ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>> │ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>> │ └sdd2 1.30g [8:50] MD raid0 (1/3) (w/ sdc2,sde2) in_sync 'ion:1'
>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>> │ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>>> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
>>> └sde 2.73t [8:64] Partitioned (dos)
>>> ├sde1 1.82t [8:65] MD raid6 (3/5) (w/ sda1,sdb1,sdc1,sdd1) in_sync
>>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>> │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>>> │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>>> ├sde2 1.30g [8:66] MD raid0 (2/3) (w/ sdc2,sdd2) in_sync 'ion:1'
>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>> │└md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>>> └sde3 184.98g [8:67] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>>> └Mounted as /dev/sde3 @ /media/198GB
>>> Other Block Devices
>>> ├loop0 0.00k [7:0] Empty/Unknown
>>> ├loop1 0.00k [7:1] Empty/Unknown
>>> ├loop2 0.00k [7:2] Empty/Unknown
>>> ├loop3 0.00k [7:3] Empty/Unknown
>>> ├loop4 0.00k [7:4] Empty/Unknown
>>> ├loop5 0.00k [7:5] Empty/Unknown
>>> ├loop6 0.00k [7:6] Empty/Unknown
>>> ├loop7 0.00k [7:7] Empty/Unknown
>>> ├md127 0.00k [9:127] MD vnone () clear, None (None) None {None}
>>> │ Empty/Unknown
>>> ├ram0 64.00m [1:0] Empty/Unknown
>>> ├ram1 64.00m [1:1] Empty/Unknown
>>> ├ram2 64.00m [1:2] Empty/Unknown
>>> ├ram3 64.00m [1:3] Empty/Unknown
>>> ├ram4 64.00m [1:4] Empty/Unknown
>>> ├ram5 64.00m [1:5] Empty/Unknown
>>> ├ram6 64.00m [1:6] Empty/Unknown
>>> ├ram7 64.00m [1:7] Empty/Unknown
>>> ├ram8 64.00m [1:8] Empty/Unknown
>>> ├ram9 64.00m [1:9] Empty/Unknown
>>> ├ram10 64.00m [1:10] Empty/Unknown
>>> ├ram11 64.00m [1:11] Empty/Unknown
>>> ├ram12 64.00m [1:12] Empty/Unknown
>>> ├ram13 64.00m [1:13] Empty/Unknown
>>> ├ram14 64.00m [1:14] Empty/Unknown
>>> └ram15 64.00m [1:15] Empty/Unknown
>>>
>>>
>>> /proc/mdstat on Ubuntu
>>> Personalities : [raid6] [raid5] [raid4] [raid0]
>>> md0 : active raid6 sdb1[0] sda1[4] sde1[3] sdd1[2] sdc1[1]
>>> 5856046080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
>>> bitmap: 0/15 pages [0KB], 65536KB chunk
>>>
>>> md1 : active raid0 sdc2[0] sde2[2] sdd2[1]
>>> 4098048 blocks super 1.2 512k chunks
>>>
>>> Regards,
>>> Mathias
>>>
>>> On 23 September 2015 at 07:24, Alexander Afonyashin
>>> <a.afonyashin@madnet-team.ru> wrote:
>>>> Hi,
>>>>
>>>> I wonder why metainfo from /dev/sdf1 thinks that there're only 5
>>>> devices in raid6. Did you try to grow your array recently?
>>>> Show the output of mdadm -D /dev/mdX when array is assembled on Ubuntu.
>>>> Can you assemble the array with 4 disks: /dev/sd[a-d]?
>>>>
>>>> P.S. According to mdadm output metainfo on /dev/sdf1 claims that it's
>>>> from another md-array (check Array UUID field). You may need to zero
>>>> metadata on /dev/sdf1 and re-add in to existing raid6 array.
>>>>
>>>> Regards,
>>>> Alexander
>>>>
>>>> On Wed, Sep 23, 2015 at 12:30 AM, Mathias Burén <mathias.buren@gmail.com> wrote:
>>>>> Hi (please reply-all)
>>>>>
>>>>> I've a RAID6 array (sda sdb sdd sde sdf1) that I can't assemble under
>>>>> Arch, but it worked fine under Ubuntu. mdadm - v3.3.4 - 3rd August
>>>>> 2015, kernel 4.1.6
>>>>>
>>>>> Here is the mdadm --examine for each drive:
>>>>>
>>>>>
>>>>>
>>>>> [root@ion ~]# mdadm --examine /dev/sda
>>>>> /dev/sda:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x0
>>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>>> Name : ion:md0 (local to host ion)
>>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>>> Raid Level : raid6
>>>>> Raid Devices : 6
>>>>>
>>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>>>>> State : clean
>>>>> Device UUID : a09fc60d:5c4a27a5:4b89bc33:29b01582
>>>>>
>>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>>> Checksum : 528563ee - correct
>>>>> Events : 97557
>>>>>
>>>>> Layout : left-symmetric
>>>>> Chunk Size : 512K
>>>>>
>>>>> Device Role : Active device 3
>>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>
>>>>> [root@ion ~]# mdadm --examine /dev/sdb
>>>>> /dev/sdb:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x0
>>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>>> Name : ion:md0 (local to host ion)
>>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>>> Raid Level : raid6
>>>>> Raid Devices : 6
>>>>>
>>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>>>>> State : clean
>>>>> Device UUID : 93568b01:632395bf:7d0082a5:db9b6ff9
>>>>>
>>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>>> Checksum : 49d756ca - correct
>>>>> Events : 97557
>>>>>
>>>>> Layout : left-symmetric
>>>>> Chunk Size : 512K
>>>>>
>>>>> Device Role : Active device 0
>>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>
>>>>>
>>>>> [root@ion ~]# mdadm --examine /dev/sdd
>>>>> /dev/sdd:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x0
>>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>>> Name : ion:md0 (local to host ion)
>>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>>> Raid Level : raid6
>>>>> Raid Devices : 6
>>>>>
>>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> Unused Space : before=262064 sectors, after=1200 sectors
>>>>> State : clean
>>>>> Device UUID : 78df2586:cb5649aa:e0b6d211:d92dc224
>>>>>
>>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>>> Checksum : 50c95b7c - correct
>>>>> Events : 97557
>>>>>
>>>>> Layout : left-symmetric
>>>>> Chunk Size : 512K
>>>>>
>>>>> Device Role : Active device 5
>>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>
>>>>> [root@ion ~]# mdadm --examine /dev/sde
>>>>> /dev/sde:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x0
>>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>>> Name : ion:md0 (local to host ion)
>>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>>> Raid Level : raid6
>>>>> Raid Devices : 6
>>>>>
>>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> Unused Space : before=262064 sectors, after=1200 sectors
>>>>> State : clean
>>>>> Device UUID : 41712f8c:255b0f3e:0e345f7b:e1504e42
>>>>>
>>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>>> Checksum : 71b191d6 - correct
>>>>> Events : 97557
>>>>>
>>>>> Layout : left-symmetric
>>>>> Chunk Size : 512K
>>>>>
>>>>> Device Role : Active device 1
>>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>
>>>>> [root@ion ~]# mdadm --examine /dev/sdf1
>>>>> /dev/sdf1:
>>>>> Magic : a92b4efc
>>>>> Version : 1.2
>>>>> Feature Map : 0x1
>>>>> Array UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
>>>>> Name : ion:0 (local to host ion)
>>>>> Creation Time : Thu Nov 20 23:52:58 2014
>>>>> Raid Level : raid6
>>>>> Raid Devices : 5
>>>>>
>>>>> Avail Dev Size : 3904030720 (1861.59 GiB 1998.86 GB)
>>>>> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
>>>>> Data Offset : 262144 sectors
>>>>> Super Offset : 8 sectors
>>>>> Unused Space : before=262056 sectors, after=0 sectors
>>>>> State : clean
>>>>> Device UUID : eaa0dcba:d04a7c16:6c256916:67a491ae
>>>>>
>>>>> Internal Bitmap : 8 sectors from superblock
>>>>> Update Time : Sun Sep 20 19:47:50 2015
>>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>>> Checksum : b1f6f725 - correct
>>>>> Events : 59736
>>>>>
>>>>> Layout : left-symmetric
>>>>> Chunk Size : 512K
>>>>>
>>>>> Device Role : Active device 3
>>>>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Here is ldrv:
>>>>>
>>>>> [root@ion ~]# python2 lsdrv
>>>>> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
>>>>> Logic MegaRAID SAS 1068
>>>>> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
>>>>> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
>>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>>> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
>>>>> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
>>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>>> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
>>>>> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>>>>> ├scsi 1:0:0:0 ATA INTEL SSDSA2BW16 {BTPR2062006T160DGN}
>>>>> │└sdc 149.05g [8:32] Partitioned (dos)
>>>>> │ ├sdc1 256.00m [8:33] ext4 'boot' {f7727d21-646d-42f9-844b-50591b7f8358}
>>>>> │ │└Mounted as /dev/sdc1 @ /boot
>>>>> │ └sdc2 140.00g [8:34] PV LVM2_member 40.00g used, 100.00g free
>>>>> {KGQ5TJ-mVvL-1Ccd-CKae-1QvS-6AFn-8jFQAw}
>>>>> │ └VG ArchVG 140.00g 100.00g free {F2vaKY-XO4m-UEih-f9Fh-sHgX-cRsN-SLi3Oo}
>>>>> │ ├dm-1 32.00g [254:1] LV root ext4 'root'
>>>>> {82fa2951-25c8-4e36-bf36-9f4f7747ac46}
>>>>> │ │└Mounted as /dev/mapper/ArchVG-root @ /
>>>>> │ └dm-0 8.00g [254:0] LV swap swap {0e3f4596-fa50-4fed-875f-f8427084d9b5}
>>>>> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
>>>>> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
>>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>>> ├scsi 3:x:x:x [Empty]
>>>>> ├scsi 4:x:x:x [Empty]
>>>>> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
>>>>> │└sde 1.82t [8:64] MD raid6 (6) inactive 'ion:md0'
>>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>>> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
>>>>> └sdf 2.73t [8:80] Partitioned (dos)
>>>>> ├sdf1 1.82t [8:81] MD raid6 (5) inactive 'ion:0'
>>>>> {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>>>> ├sdf2 1.30g [8:82] MD raid0 (3) inactive 'ion:1'
>>>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>>>> └sdf3 184.98g [8:83] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>>>>>
>>>>>
>>>>> If I try:
>>>>>
>>>>> [root@ion ~]# mdadm --assemble --scan
>>>>> mdadm: /dev/md/0 assembled from 1 drive - not enough to start the array.
>>>>> mdadm: /dev/md/1 assembled from 1 drive - not enough to start the array.
>>>>>
>>>>> From dmesg:
>>>>>
>>>>> [ 1227.671344] sda: sda1
>>>>> [ 1227.680392] md: sda does not have a valid v1.2 superblock, not importing!
>>>>> [ 1227.680414] md: md_import_device returned -22
>>>>> [ 1227.680462] md: md0 stopped.
>>>>> [ 1227.707969] sdb: sdb1
>>>>> [ 1227.718598] md: sdb does not have a valid v1.2 superblock, not importing!
>>>>> [ 1227.718611] md: md_import_device returned -22
>>>>> [ 1227.718631] md: md0 stopped.
>>>>> [ 1286.334542] md: md0 stopped.
>>>>> [ 1286.338250] md: bind<sdf1>
>>>>> [ 1286.338390] md: md0 stopped.
>>>>> [ 1286.338400] md: unbind<sdf1>
>>>>> [ 1286.348350] md: export_rdev(sdf1)
>>>>> [ 1286.372268] md: bind<sdf1>
>>>>> [ 1286.373390] md: md1 stopped.
>>>>> [ 1286.373936] md: bind<sdf2>
>>>>> [ 1286.373977] md: md1 stopped.
>>>>> [ 1286.373983] md: unbind<sdf2>
>>>>> [ 1286.388061] md: export_rdev(sdf2)
>>>>> [ 1286.405140] md: bind<sdf2>
>>>>>
>>>>> [root@ion ~]# cat /proc/mdstat
>>>>> Personalities :
>>>>> md1 : inactive sdf2[2](S)
>>>>> 1366104 blocks super 1.2
>>>>>
>>>>> md0 : inactive sdf1[3](S)
>>>>> 1952015360 blocks super 1.2
>>>>>
>>>>> unused devices: <none>
>>>>>
>>>>>
>>>>> If I try manually (I'm not sure of the order though):
>>>>>
>>>>> [root@ion ~]# mdadm --assemble --readonly --verbose /dev/md0 /dev/sda
>>>>> /dev/sdb /dev/sdd /dev/sde
>>>>> mdadm: looking for devices for /dev/md0
>>>>> mdadm: /dev/sda is identified as a member of /dev/md0, slot 3.
>>>>> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
>>>>> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 5.
>>>>> mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
>>>>> mdadm: added /dev/sde to /dev/md0 as 1
>>>>> mdadm: no uptodate device for slot 2 of /dev/md0
>>>>> mdadm: failed to add /dev/sda to /dev/md0: Invalid argument
>>>>> mdadm: no uptodate device for slot 4 of /dev/md0
>>>>> mdadm: added /dev/sdd to /dev/md0 as 5
>>>>> mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument
>>>>> mdadm: /dev/md0 assembled from 2 drives - need 5 to start (use --run to insist).
>>>>>
>>>>>
>>>>> Any idea where I should start?
>>>>>
>>>>> Thanks
>>>>> Mathias
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Bad raid0 bio too large problem
From: Neil Brown @ 2015-09-25 4:23 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <wrfjwpvgvupf.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 3060 bytes --]
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Neil Brown <neilb@suse.de> writes:
>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>
>>> Neil Brown <neilb@suse.de> writes:
>>>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>>>
>>>>> Hi Neil,
>>>>>
>>>>> I think we have some bad side effects with this patch:
>>>>>
>>>>> commit 199dc6ed5179251fa6158a461499c24bdd99c836
>>>>> Author: NeilBrown <neilb@suse.com>
>>>>> Date: Mon Aug 3 13:11:47 2015 +1000
>>>>>
>>>>> md/raid0: update queue parameter in a safer location.
>>>>>
>>>>> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
>>>>> of queue parameters (e.g. max number of sectors per bio) is
>>>>> done in the wrong place.
>>>>> It should be part of ->run, but it is actually part of ->takeover.
>>>>> This means it happens before level_store() calls:
>>>>>
>>>>> blk_set_stacking_limits(&mddev->queue->limits);
>>>>>
>>>>> Running the '03r0assem' test suite fills my kernel log with output like
>>>>> below. Yi Zhang also had issues where writes failed too.
>>>>>
>>>>> robably something we need to resolve for 4.2-final or revert the
>>>>> offending patch.
>>>>>
>>>>> Cheers,
>>>>> Jes
>>>>>
>>>>> md: bind<loop0>
>>>>> md: bind<loop1>
>>>>> md: bind<loop2>
>>>>> md/raid0:md2: md_size is 116736 sectors.
>>>>> md: RAID0 configuration for md2 - 1 zone
>>>>> md: zone0=[loop0/loop1/loop2]
>>>>> zone-offset= 0KB, device-offset= 0KB, size= 58368KB
>>>>>
>>>>> md2: detected capacity change from 0 to 59768832
>>>>> bio too big device loop0 (296 > 255)
>>>>> bio too big device loop0 (272 > 255)
>>>>
>>>> 1/ Why do you blame that particular patch?
>>>>
>>>> 2/ Where is that error message coming from? I cannot find "bio too big"
>>>> in the kernel (except in a comment).
>>>> Commit: 54efd50bfd87 ("block: make generic_make_request handle
>>>> arbitrarily sized bios")
>>>> removed the only instance of the error message that I know of.
>>>>
>>>> Which kernel exactly are you testing?
>>>
>>> I blame it because of bisect - I revert that patch and the issue goes
>>> away.
>>>
>>> I checked out 199dc6ed5179251fa6158a461499c24bdd99c836 in Linus' tree,
>>> see the bio too large. I revert it and it goes away.
>>
>> Well that's pretty convincing - thanks.
>> And as you say - it is tagged for -stable so really needs to be fixed.
>>
>> Stares at the code again. And again.
>>
>> Ahhh. that patch moved the
>> blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
>> to after
>> disk_stack_limits(...);
>>
>> That is wrong.
>>
>> Could you confirm that this fixes your test?
>
> I refuse to do that! .... since Xiao beat me to it! Thanks!
>
> I was half way bisecting my way through it last night. For some reason
> the problem was reproducible in 4.2 if I applied the offending patch,
> but not in 4.3-rc2.
>
> Any chance you'll push these to your git tree in the near future?
Pushed.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Unable to assemble RAID6 after Ubuntu>Arch switch
From: Mathias Burén @ 2015-09-24 18:17 UTC (permalink / raw)
To: Alexander Afonyashin; +Cc: Linux-RAID
In-Reply-To: <CAETWcfvF1o8mNVw1t-=7WNZZTz774eTnj1tG6UckrXbDvWxdXg@mail.gmail.com>
Yeah it's odd. I believe that ages ago I had an array on the full
disks (not partitions). I've since switched to RAID based on
partitions (this array that works in Ubuntu but not Arch). Maybe that
is where the confusion comes from.
I just realized I provided the examine data for the old unused array
within Arch, and you're right, sda1 and sdb1 don't exist under Arch:
$ sudo partprobe --summary
/dev/sda: msdos partitions 1
/dev/sdb: msdos partitions 1
/dev/sdc: msdos partitions 1 2
/dev/sdd: msdos partitions 1 2
/dev/sde: msdos partitions 1 2
/dev/sdf: msdos partitions 1 2 3
$ sudo fdisk -l /dev/sda
Disk /dev/sda: 1.8 TiB, 1998998994944 bytes, 3904294912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x056cc1c0
Device Boot Start End Sectors Size Id Type
/dev/sda1 2048 3904294911 3904292864 1.8T fd Linux raid autodetect
$ sudo fdisk -l /dev/sdb
Disk /dev/sdb: 1.8 TiB, 1998998994944 bytes, 3904294912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x9fb14a5c
Device Boot Start End Sectors Size Id Type
/dev/sdb1 2048 3904294911 3904292864 1.8T fd Linux raid autodetect
$ ls -la /dev/sda*
brw-rw---- 1 root disk 8, 0 Sep 24 20:14 /dev/sda
$ ls -la /dev/sdb*
brw-rw---- 1 root disk 8, 16 Sep 24 20:14 /dev/sdb
So for some reason the kernel does not see the partitions even though
partprobe and fdisk show them.
Regards,
Mathias
On 24 September 2015 at 10:17, Alexander Afonyashin
<a.afonyashin@madnet-team.ru> wrote:
> Hi Matias,
>
> I wonder why Ubuntu detects raid6 parts as /dev/sdX1 (and counts only
> 5 disks) while Arch detects /dev/sdX (and counts 6 disks)? Try to
> assemble raid6 on Arch manually again by issuing:
>
> mdadm -S /dev/md0
> mdadm -A /dev/md0 /dev/sdb /dev/sda /dev/sde /dev/sdd /dev/sdc -
> exactly as shown in mdadm's output on Ubuntu
>
> P.S. Still confused why Arch doesn't recognize /dev/sda1 etc.
>
> Regards,
> Alexander
>
> On Wed, Sep 23, 2015 at 9:35 PM, Mathias Burén <mathias.buren@gmail.com> wrote:
>> Hi Alexander,
>>
>> I didn't try to grow the array or anything, just swapped out the OS.
>> When I boot back into Ubuntu it assembles fine.
>>
>> This is from within Ubuntu (I did a mdadm --assemble --scan using
>> mdadm - v3.3.2-7-g21dc471 - 03th November 2014)
>>
>> mdadm -D on RAID6
>> /dev/md0:
>> Version : 1.2
>> Creation Time : Thu Nov 20 23:52:58 2014
>> Raid Level : raid6
>> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
>> Used Dev Size : 1952015360 (1861.59 GiB 1998.86 GB)
>> Raid Devices : 5
>> Total Devices : 5
>> Persistence : Superblock is persistent
>>
>> Intent Bitmap : Internal
>>
>> Update Time : Sun Sep 20 19:47:50 2015
>> State : clean
>> Active Devices : 5
>> Working Devices : 5
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Name : ion:0 (local to host ion)
>> UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
>> Events : 59736
>>
>> Number Major Minor RaidDevice State
>> 0 8 17 0 active sync /dev/sdb1
>> 1 8 33 1 active sync /dev/sdc1
>> 2 8 49 2 active sync /dev/sdd1
>> 3 8 65 3 active sync /dev/sde1
>> 4 8 1 4 active sync /dev/sda1
>>
>>
>> mdadm -D on RAID0
>> /dev/md1:
>> Version : 1.2
>> Creation Time : Thu Nov 20 23:53:48 2014
>> Raid Level : raid0
>> Array Size : 4098048 (3.91 GiB 4.20 GB)
>> Raid Devices : 3
>> Total Devices : 3
>> Persistence : Superblock is persistent
>>
>> Update Time : Thu Nov 20 23:53:48 2014
>> State : clean
>> Active Devices : 3
>> Working Devices : 3
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Chunk Size : 512K
>>
>> Name : ion:1 (local to host ion)
>> UUID : a20b70d4:7ee17e3f:abab74f8:dadb8cd8
>> Events : 0
>>
>> Number Major Minor RaidDevice State
>> 0 8 34 0 active sync /dev/sdc2
>> 1 8 50 1 active sync /dev/sdd2
>> 2 8 66 2 active sync /dev/sde2
>>
>>
>>
>> lsdrv from within Ubuntu
>> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
>> Logic MegaRAID SAS 1068
>> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
>> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
>> {0ad2603e-e432-83ee-0218-077398e716ef}
>> │ └sda1 1.82t [8:1] MD raid6 (4/5) (w/ sdb1,sdc1,sdd1,sde1) in_sync
>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>> │ └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>> │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
>> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
>> {0ad2603e-e432-83ee-0218-077398e716ef}
>> └sdb1 1.82t [8:17] MD raid6 (0/5) (w/ sda1,sdc1,sdd1,sde1) in_sync
>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>> └md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>> ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
>> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>> ├scsi 1:0:0:0 ATA Corsair CSSD-F60 {10326505580009990027}
>> │└sdf 55.90g [8:80] Partitioned (dos)
>> │ ├sdf1 243.00m [8:81] ext2 {b45c13c8-4246-43ee-ac2c-f9bb5de099f8}
>> │ │└Mounted as /dev/sdf1 @ /boot
>> │ ├sdf2 1.00k [8:82] Partitioned (dos)
>> │ └sdf5 55.66g [8:85] PV LVM2_member 55.66g used, 0 free
>> {zZ5Dgy-E7v8-Y1Mi-EuqG-uXUb-DseW-s7goxG}
>> │ └VG ion 55.66g 0 free {XSaCyP-phLN-m4IH-2cJ7-1XXw-b3uh-qWBl8N}
>> │ ├dm-0 52.41g [252:0] LV root ext4 {63342e0d-1b4f-4475-9835-d3a2a6610e8f}
>> │ │└Mounted as /dev/mapper/ion-root @ /
>> │ └dm-1 3.25g [252:1] LV swap_1 Empty/Unknown
>> │ └dm-2 3.25g [252:2] swap {0c81625d-ee9e-4a7a-9613-410c1cf53e72}
>> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
>> │└sdc 1.82t [8:32] MD raid6 (6) inactive 'ion:md0'
>> {0ad2603e-e432-83ee-0218-077398e716ef}
>> │ ├sdc1 1.82t [8:33] MD raid6 (1/5) (w/ sda1,sdb1,sdd1,sde1) in_sync
>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>> │ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>> │ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>> │ └sdc2 1.30g [8:34] MD raid0 (0/3) (w/ sdd2,sde2) in_sync 'ion:1'
>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>> │ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>> ├scsi 3:x:x:x [Empty]
>> ├scsi 4:x:x:x [Empty]
>> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
>> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
>> {0ad2603e-e432-83ee-0218-077398e716ef}
>> │ ├sdd1 1.82t [8:49] MD raid6 (2/5) (w/ sda1,sdb1,sdc1,sde1) in_sync
>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>> │ │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>> │ │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>> │ └sdd2 1.30g [8:50] MD raid0 (1/3) (w/ sdc2,sde2) in_sync 'ion:1'
>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>> │ └md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
>> └sde 2.73t [8:64] Partitioned (dos)
>> ├sde1 1.82t [8:65] MD raid6 (3/5) (w/ sda1,sdb1,sdc1,sdd1) in_sync
>> 'ion:0' {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>> │└md0 5.45t [9:0] MD v1.2 raid6 (5) clean, 512k Chunk
>> {4cae433f:a40afcf5:f9aba91d:d8217b69}
>> │ ext4 '6TB_RAID6' {9e3c1fbe-8228-4b38-9047-66a5e2429e5f}
>> ├sde2 1.30g [8:66] MD raid0 (2/3) (w/ sdc2,sdd2) in_sync 'ion:1'
>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>> │└md1 3.91g [9:1] MD v1.2 raid0 (3) clean, 512k Chunk, None (None)
>> None {a20b70d4:7ee17e3f:abab74f8:dadb8cd8}
>> │ ext4 '4GB_RAID0' {c8327dd0-9d3f-4457-a73a-c9c7d5a0ee3f}
>> └sde3 184.98g [8:67] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>> └Mounted as /dev/sde3 @ /media/198GB
>> Other Block Devices
>> ├loop0 0.00k [7:0] Empty/Unknown
>> ├loop1 0.00k [7:1] Empty/Unknown
>> ├loop2 0.00k [7:2] Empty/Unknown
>> ├loop3 0.00k [7:3] Empty/Unknown
>> ├loop4 0.00k [7:4] Empty/Unknown
>> ├loop5 0.00k [7:5] Empty/Unknown
>> ├loop6 0.00k [7:6] Empty/Unknown
>> ├loop7 0.00k [7:7] Empty/Unknown
>> ├md127 0.00k [9:127] MD vnone () clear, None (None) None {None}
>> │ Empty/Unknown
>> ├ram0 64.00m [1:0] Empty/Unknown
>> ├ram1 64.00m [1:1] Empty/Unknown
>> ├ram2 64.00m [1:2] Empty/Unknown
>> ├ram3 64.00m [1:3] Empty/Unknown
>> ├ram4 64.00m [1:4] Empty/Unknown
>> ├ram5 64.00m [1:5] Empty/Unknown
>> ├ram6 64.00m [1:6] Empty/Unknown
>> ├ram7 64.00m [1:7] Empty/Unknown
>> ├ram8 64.00m [1:8] Empty/Unknown
>> ├ram9 64.00m [1:9] Empty/Unknown
>> ├ram10 64.00m [1:10] Empty/Unknown
>> ├ram11 64.00m [1:11] Empty/Unknown
>> ├ram12 64.00m [1:12] Empty/Unknown
>> ├ram13 64.00m [1:13] Empty/Unknown
>> ├ram14 64.00m [1:14] Empty/Unknown
>> └ram15 64.00m [1:15] Empty/Unknown
>>
>>
>> /proc/mdstat on Ubuntu
>> Personalities : [raid6] [raid5] [raid4] [raid0]
>> md0 : active raid6 sdb1[0] sda1[4] sde1[3] sdd1[2] sdc1[1]
>> 5856046080 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
>> bitmap: 0/15 pages [0KB], 65536KB chunk
>>
>> md1 : active raid0 sdc2[0] sde2[2] sdd2[1]
>> 4098048 blocks super 1.2 512k chunks
>>
>> Regards,
>> Mathias
>>
>> On 23 September 2015 at 07:24, Alexander Afonyashin
>> <a.afonyashin@madnet-team.ru> wrote:
>>> Hi,
>>>
>>> I wonder why metainfo from /dev/sdf1 thinks that there're only 5
>>> devices in raid6. Did you try to grow your array recently?
>>> Show the output of mdadm -D /dev/mdX when array is assembled on Ubuntu.
>>> Can you assemble the array with 4 disks: /dev/sd[a-d]?
>>>
>>> P.S. According to mdadm output metainfo on /dev/sdf1 claims that it's
>>> from another md-array (check Array UUID field). You may need to zero
>>> metadata on /dev/sdf1 and re-add in to existing raid6 array.
>>>
>>> Regards,
>>> Alexander
>>>
>>> On Wed, Sep 23, 2015 at 12:30 AM, Mathias Burén <mathias.buren@gmail.com> wrote:
>>>> Hi (please reply-all)
>>>>
>>>> I've a RAID6 array (sda sdb sdd sde sdf1) that I can't assemble under
>>>> Arch, but it worked fine under Ubuntu. mdadm - v3.3.4 - 3rd August
>>>> 2015, kernel 4.1.6
>>>>
>>>> Here is the mdadm --examine for each drive:
>>>>
>>>>
>>>>
>>>> [root@ion ~]# mdadm --examine /dev/sda
>>>> /dev/sda:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>> Name : ion:md0 (local to host ion)
>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>> Raid Level : raid6
>>>> Raid Devices : 6
>>>>
>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>> Data Offset : 262144 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>>>> State : clean
>>>> Device UUID : a09fc60d:5c4a27a5:4b89bc33:29b01582
>>>>
>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>> Checksum : 528563ee - correct
>>>> Events : 97557
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 512K
>>>>
>>>> Device Role : Active device 3
>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>>> [root@ion ~]# mdadm --examine /dev/sdb
>>>> /dev/sdb:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>> Name : ion:md0 (local to host ion)
>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>> Raid Level : raid6
>>>> Raid Devices : 6
>>>>
>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>> Data Offset : 262144 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=262064 sectors, after=18446744073706818560 sectors
>>>> State : clean
>>>> Device UUID : 93568b01:632395bf:7d0082a5:db9b6ff9
>>>>
>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>> Checksum : 49d756ca - correct
>>>> Events : 97557
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 512K
>>>>
>>>> Device Role : Active device 0
>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>>>
>>>> [root@ion ~]# mdadm --examine /dev/sdd
>>>> /dev/sdd:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>> Name : ion:md0 (local to host ion)
>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>> Raid Level : raid6
>>>> Raid Devices : 6
>>>>
>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>> Data Offset : 262144 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=262064 sectors, after=1200 sectors
>>>> State : clean
>>>> Device UUID : 78df2586:cb5649aa:e0b6d211:d92dc224
>>>>
>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>> Checksum : 50c95b7c - correct
>>>> Events : 97557
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 512K
>>>>
>>>> Device Role : Active device 5
>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>>> [root@ion ~]# mdadm --examine /dev/sde
>>>> /dev/sde:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x0
>>>> Array UUID : 0ad2603e:e43283ee:02180773:98e716ef
>>>> Name : ion:md0 (local to host ion)
>>>> Creation Time : Tue Feb 5 17:33:27 2013
>>>> Raid Level : raid6
>>>> Raid Devices : 6
>>>>
>>>> Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
>>>> Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
>>>> Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
>>>> Data Offset : 262144 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=262064 sectors, after=1200 sectors
>>>> State : clean
>>>> Device UUID : 41712f8c:255b0f3e:0e345f7b:e1504e42
>>>>
>>>> Update Time : Tue Nov 4 21:43:49 2014
>>>> Checksum : 71b191d6 - correct
>>>> Events : 97557
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 512K
>>>>
>>>> Device Role : Active device 1
>>>> Array State : AA.AAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>>> [root@ion ~]# mdadm --examine /dev/sdf1
>>>> /dev/sdf1:
>>>> Magic : a92b4efc
>>>> Version : 1.2
>>>> Feature Map : 0x1
>>>> Array UUID : 4cae433f:a40afcf5:f9aba91d:d8217b69
>>>> Name : ion:0 (local to host ion)
>>>> Creation Time : Thu Nov 20 23:52:58 2014
>>>> Raid Level : raid6
>>>> Raid Devices : 5
>>>>
>>>> Avail Dev Size : 3904030720 (1861.59 GiB 1998.86 GB)
>>>> Array Size : 5856046080 (5584.76 GiB 5996.59 GB)
>>>> Data Offset : 262144 sectors
>>>> Super Offset : 8 sectors
>>>> Unused Space : before=262056 sectors, after=0 sectors
>>>> State : clean
>>>> Device UUID : eaa0dcba:d04a7c16:6c256916:67a491ae
>>>>
>>>> Internal Bitmap : 8 sectors from superblock
>>>> Update Time : Sun Sep 20 19:47:50 2015
>>>> Bad Block Log : 512 entries available at offset 72 sectors
>>>> Checksum : b1f6f725 - correct
>>>> Events : 59736
>>>>
>>>> Layout : left-symmetric
>>>> Chunk Size : 512K
>>>>
>>>> Device Role : Active device 3
>>>> Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Here is ldrv:
>>>>
>>>> [root@ion ~]# python2 lsdrv
>>>> PCI [megaraid_sas] 02:0e.0 RAID bus controller: LSI Logic / Symbios
>>>> Logic MegaRAID SAS 1068
>>>> ├scsi 0:2:0:0 LSI MegaRAID 84016E {00ede206d4a731011c50ff5e02b00506}
>>>> │└sda 1.82t [8:0] MD raid6 (6) inactive 'ion:md0'
>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>> └scsi 0:2:1:0 LSI MegaRAID 84016E {0036936cc4270000ff50ff5e02b00506}
>>>> └sdb 1.82t [8:16] MD raid6 (6) inactive 'ion:md0'
>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>> PCI [ahci] 00:1f.2 SATA controller: Intel Corporation 8 Series/C220
>>>> Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>>>> ├scsi 1:0:0:0 ATA INTEL SSDSA2BW16 {BTPR2062006T160DGN}
>>>> │└sdc 149.05g [8:32] Partitioned (dos)
>>>> │ ├sdc1 256.00m [8:33] ext4 'boot' {f7727d21-646d-42f9-844b-50591b7f8358}
>>>> │ │└Mounted as /dev/sdc1 @ /boot
>>>> │ └sdc2 140.00g [8:34] PV LVM2_member 40.00g used, 100.00g free
>>>> {KGQ5TJ-mVvL-1Ccd-CKae-1QvS-6AFn-8jFQAw}
>>>> │ └VG ArchVG 140.00g 100.00g free {F2vaKY-XO4m-UEih-f9Fh-sHgX-cRsN-SLi3Oo}
>>>> │ ├dm-1 32.00g [254:1] LV root ext4 'root'
>>>> {82fa2951-25c8-4e36-bf36-9f4f7747ac46}
>>>> │ │└Mounted as /dev/mapper/ArchVG-root @ /
>>>> │ └dm-0 8.00g [254:0] LV swap swap {0e3f4596-fa50-4fed-875f-f8427084d9b5}
>>>> ├scsi 2:0:0:0 ATA SAMSUNG HD204UI {S2H7JR0B501861}
>>>> │└sdd 1.82t [8:48] MD raid6 (6) inactive 'ion:md0'
>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>> ├scsi 3:x:x:x [Empty]
>>>> ├scsi 4:x:x:x [Empty]
>>>> ├scsi 5:0:0:0 ATA WDC WD20EARS-00J {WD-WCAWZ2036074}
>>>> │└sde 1.82t [8:64] MD raid6 (6) inactive 'ion:md0'
>>>> {0ad2603e-e432-83ee-0218-077398e716ef}
>>>> └scsi 6:0:0:0 ATA ST3000DM001-1CH1 {W1F2PZGH}
>>>> └sdf 2.73t [8:80] Partitioned (dos)
>>>> ├sdf1 1.82t [8:81] MD raid6 (5) inactive 'ion:0'
>>>> {4cae433f-a40a-fcf5-f9ab-a91dd8217b69}
>>>> ├sdf2 1.30g [8:82] MD raid0 (3) inactive 'ion:1'
>>>> {a20b70d4-7ee1-7e3f-abab-74f8dadb8cd8}
>>>> └sdf3 184.98g [8:83] ext4 '198GB' {e4fc427a-4ec1-48dc-8d9b-bfcffe06d42f}
>>>>
>>>>
>>>> If I try:
>>>>
>>>> [root@ion ~]# mdadm --assemble --scan
>>>> mdadm: /dev/md/0 assembled from 1 drive - not enough to start the array.
>>>> mdadm: /dev/md/1 assembled from 1 drive - not enough to start the array.
>>>>
>>>> From dmesg:
>>>>
>>>> [ 1227.671344] sda: sda1
>>>> [ 1227.680392] md: sda does not have a valid v1.2 superblock, not importing!
>>>> [ 1227.680414] md: md_import_device returned -22
>>>> [ 1227.680462] md: md0 stopped.
>>>> [ 1227.707969] sdb: sdb1
>>>> [ 1227.718598] md: sdb does not have a valid v1.2 superblock, not importing!
>>>> [ 1227.718611] md: md_import_device returned -22
>>>> [ 1227.718631] md: md0 stopped.
>>>> [ 1286.334542] md: md0 stopped.
>>>> [ 1286.338250] md: bind<sdf1>
>>>> [ 1286.338390] md: md0 stopped.
>>>> [ 1286.338400] md: unbind<sdf1>
>>>> [ 1286.348350] md: export_rdev(sdf1)
>>>> [ 1286.372268] md: bind<sdf1>
>>>> [ 1286.373390] md: md1 stopped.
>>>> [ 1286.373936] md: bind<sdf2>
>>>> [ 1286.373977] md: md1 stopped.
>>>> [ 1286.373983] md: unbind<sdf2>
>>>> [ 1286.388061] md: export_rdev(sdf2)
>>>> [ 1286.405140] md: bind<sdf2>
>>>>
>>>> [root@ion ~]# cat /proc/mdstat
>>>> Personalities :
>>>> md1 : inactive sdf2[2](S)
>>>> 1366104 blocks super 1.2
>>>>
>>>> md0 : inactive sdf1[3](S)
>>>> 1952015360 blocks super 1.2
>>>>
>>>> unused devices: <none>
>>>>
>>>>
>>>> If I try manually (I'm not sure of the order though):
>>>>
>>>> [root@ion ~]# mdadm --assemble --readonly --verbose /dev/md0 /dev/sda
>>>> /dev/sdb /dev/sdd /dev/sde
>>>> mdadm: looking for devices for /dev/md0
>>>> mdadm: /dev/sda is identified as a member of /dev/md0, slot 3.
>>>> mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
>>>> mdadm: /dev/sdd is identified as a member of /dev/md0, slot 5.
>>>> mdadm: /dev/sde is identified as a member of /dev/md0, slot 1.
>>>> mdadm: added /dev/sde to /dev/md0 as 1
>>>> mdadm: no uptodate device for slot 2 of /dev/md0
>>>> mdadm: failed to add /dev/sda to /dev/md0: Invalid argument
>>>> mdadm: no uptodate device for slot 4 of /dev/md0
>>>> mdadm: added /dev/sdd to /dev/md0 as 5
>>>> mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument
>>>> mdadm: /dev/md0 assembled from 2 drives - need 5 to start (use --run to insist).
>>>>
>>>>
>>>> Any idea where I should start?
>>>>
>>>> Thanks
>>>> Mathias
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 1/2] md: clear CHANGE_PENDING in readonly array
From: Shaohua Li @ 2015-09-24 16:47 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid, Kernel-team, songliubraving, hch, dan.j.williams
In-Reply-To: <87fv24wjiu.fsf@notabene.neil.brown.name>
On Thu, Sep 24, 2015 at 02:03:53PM +1000, Neil Brown wrote:
> Shaohua Li <shli@fb.com> writes:
>
> > On Wed, Sep 23, 2015 at 04:05:33PM +1000, Neil Brown wrote:
> >> Shaohua Li <shli@fb.com> writes:
> >>
> >> > If faulty disks of an array are more than allowed degraded number, the
> >> > array enters error handling. It will be marked as read-only with
> >> > MD_CHANGE_PENDING/RECOVERY_NEEDED set. But currently recovery doesn't
> >> > clear CHANGE_PENDING bit for read-only array. If MD_CHANGE_PENDING is
> >> > set for a raid5 array, all returned IO will be hold on a list till the
> >> > bit is clear. But recovery nevery clears this bit, the IO is always in
> >> > pending state and nevery finish. This has bad effects like upper layer
> >> > can't get an IO error and the array can't be stopped.
> >> >
> >> > Signed-off-by: Shaohua Li <shli@fb.com>
> >> > ---
> >> > drivers/md/md.c | 1 +
> >> > 1 file changed, 1 insertion(+)
> >> >
> >> > diff --git a/drivers/md/md.c b/drivers/md/md.c
> >> > index 95824fb..c596b73 100644
> >> > --- a/drivers/md/md.c
> >> > +++ b/drivers/md/md.c
> >> > @@ -8209,6 +8209,7 @@ void md_check_recovery(struct mddev *mddev)
> >> > md_reap_sync_thread(mddev);
> >> > clear_bit(MD_RECOVERY_RECOVER, &mddev->recovery);
> >> > clear_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> >> > + clear_bit(MD_CHANGE_PENDING, &mddev->flags);
> >> > goto unlock;
> >> > }
> >> >
> >> > --
> >> > 1.8.1
> >>
> >> Hi,
> >> I can see that clearing MD_CHANGE_PENDING there is probably correct -
> >> bug introduced by
> >> Commit: c3cce6cda162 ("md/raid5: ensure device failure recorded before write request returns.")
> >>
> >> However I don't understand your reasoning. You say that the array is
> >> marked as read-only, but I don't see how that would happen. What
> >> causes the array to be marked "read-only"?
> >
> > It's set read-only by mdadm. I didn't look carefully, but looks there is
> > disk failure event, mdadm is invoked automatically by some background
> > daemon. It's a ubuntu distribution.
>
> Thanks.
> This raises a couple of questions.
>
> 1/ What should md_set_readonly do if it finds that MD_CHANGE_PENDING is
> set?
> Maybe it should wait for md_check_recovery to get run which should
> clear the bit, after probably writing out the metadata.
>
> 2/ Why didn't md_check_recovery already do that before mdadm had a
> chance to set the array read-only?
> I guess that is just a timing thing. md_check_recovery could be
> delayed, and mdadm could get called by udev rather quickly.
>
> I think I'll get md_set_readonly to
> wait_event(mddev->sb_wait,
> !test_bit(MD_CHANGE_PENDING, &mddev->flags));
>
> because I think that is the right thing to do. But if the array is
> already read-only that won't help, so I'll still need you patch.
>
> Would you be able to test that the following patch (without your patch)
> also fixes the symptom?
Yes, the wait_event patch fixes the issue (without my patch).
Thanks,
Shaohua
^ permalink raw reply
* Re: Bad raid0 bio too large problem
From: Jes Sorensen @ 2015-09-24 12:59 UTC (permalink / raw)
To: Neil Brown; +Cc: Xiao Ni, linux-raid, yizhan
In-Reply-To: <87io70wmst.fsf@notabene.neil.brown.name>
Neil Brown <neilb@suse.de> writes:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>
>> Neil Brown <neilb@suse.de> writes:
>>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>>
>>>> Hi Neil,
>>>>
>>>> I think we have some bad side effects with this patch:
>>>>
>>>> commit 199dc6ed5179251fa6158a461499c24bdd99c836
>>>> Author: NeilBrown <neilb@suse.com>
>>>> Date: Mon Aug 3 13:11:47 2015 +1000
>>>>
>>>> md/raid0: update queue parameter in a safer location.
>>>>
>>>> When a (e.g.) RAID5 array is reshaped to RAID0, the updating
>>>> of queue parameters (e.g. max number of sectors per bio) is
>>>> done in the wrong place.
>>>> It should be part of ->run, but it is actually part of ->takeover.
>>>> This means it happens before level_store() calls:
>>>>
>>>> blk_set_stacking_limits(&mddev->queue->limits);
>>>>
>>>> Running the '03r0assem' test suite fills my kernel log with output like
>>>> below. Yi Zhang also had issues where writes failed too.
>>>>
>>>> robably something we need to resolve for 4.2-final or revert the
>>>> offending patch.
>>>>
>>>> Cheers,
>>>> Jes
>>>>
>>>> md: bind<loop0>
>>>> md: bind<loop1>
>>>> md: bind<loop2>
>>>> md/raid0:md2: md_size is 116736 sectors.
>>>> md: RAID0 configuration for md2 - 1 zone
>>>> md: zone0=[loop0/loop1/loop2]
>>>> zone-offset= 0KB, device-offset= 0KB, size= 58368KB
>>>>
>>>> md2: detected capacity change from 0 to 59768832
>>>> bio too big device loop0 (296 > 255)
>>>> bio too big device loop0 (272 > 255)
>>>
>>> 1/ Why do you blame that particular patch?
>>>
>>> 2/ Where is that error message coming from? I cannot find "bio too big"
>>> in the kernel (except in a comment).
>>> Commit: 54efd50bfd87 ("block: make generic_make_request handle
>>> arbitrarily sized bios")
>>> removed the only instance of the error message that I know of.
>>>
>>> Which kernel exactly are you testing?
>>
>> I blame it because of bisect - I revert that patch and the issue goes
>> away.
>>
>> I checked out 199dc6ed5179251fa6158a461499c24bdd99c836 in Linus' tree,
>> see the bio too large. I revert it and it goes away.
>
> Well that's pretty convincing - thanks.
> And as you say - it is tagged for -stable so really needs to be fixed.
>
> Stares at the code again. And again.
>
> Ahhh. that patch moved the
> blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
> to after
> disk_stack_limits(...);
>
> That is wrong.
>
> Could you confirm that this fixes your test?
I refuse to do that! .... since Xiao beat me to it! Thanks!
I was half way bisecting my way through it last night. For some reason
the problem was reproducible in 4.2 if I applied the offending patch,
but not in 4.3-rc2.
Any chance you'll push these to your git tree in the near future?
Thanks!
Jes
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox