* [PATCH v2 0/6] a caching layer for raid5/6
From: Shaohua Li @ 2015-05-19 2:57 UTC (permalink / raw)
To: linux-raid; +Cc: Kernel-team, songliubraving, hch, dan.j.williams, neilb
Hi,
This is the second version of the raid5/6 caching layer patches. The patches add a
caching layer for raid5/6. The caching layer uses a SSD as a cache for a raid
5/6. It works like the similar way of a hardware raid controller. The purpose
is to improve raid performance (reduce read-modify-write) and fix write hole
issue. The main patch is patch 3 and the description has all details about the
implementation.
Main changes of V2 are to improve performance. Meta data write doesn't use FUA
any more. Discard request is only dispatched when discard range is big enough.
Also have some bug fixing and code cleanup. Please review!
Thanks,
Shaohua
Shaohua Li (5):
raid5: directly use mddev->queue
raid5: A caching layer for RAID5/6
raid5: add some sysfs entries
md: don't allow resize/reshape with cache support
raid5: skip resync if caching is enabled
Song Liu (1):
MD: add a new disk role to present cache device
drivers/md/Makefile | 2 +-
drivers/md/md.c | 14 +-
drivers/md/md.h | 4 +
drivers/md/raid5-cache.c | 3519 ++++++++++++++++++++++++++++++++++++++++
drivers/md/raid5.c | 97 +-
drivers/md/raid5.h | 16 +-
include/uapi/linux/raid/md_p.h | 73 +
7 files changed, 3705 insertions(+), 20 deletions(-)
create mode 100644 drivers/md/raid5-cache.c
--
1.8.1
^ permalink raw reply
* 20 disks, fastest possible mostly-sequential read speeds
From: Jon Nelson @ 2015-05-19 2:37 UTC (permalink / raw)
To: LinuxRaid
I'm looking for some advise on tuning.
I have a server with 20 disks behind an LSI 9271-something.
They are currently exposed as 20 individual raid0 with a "strip" size
of 1MB, and assembled into an mdraid, meta 1.2, layout 10 format f2,
with a 1MB chunk size and formatted using ext4 -T largefile.
To date, this has given me the best numbers when reading some 10,000
files (total size: about 2.5TB) sequentially or in parallel.
I can't seem to get better than about 1,800 MB/s read speeds though. I
*should* be able to get closer to 3,000 based on what the drives are
capable of. Quite some time ago on this very hardware I saw a
sustained 2,750 MB/s but I don't remember how I got there.
readahead values have been adjusted, I/O scheduler, etc... all played
with with some benefit but nothing huge. What should I be looking at
here if I want the best possible read performance?
I don't want to give up some measure of redundancy.
--
Jon
^ permalink raw reply
* Re: [PATCH v2] mdadm: monitor: fix nullptr dereference when get_md_name() returns NULL
From: David F. @ 2015-05-19 1:42 UTC (permalink / raw)
To: Sergey Vidishev; +Cc: linux-raid@vger.kernel.org, NeilBrown
In-Reply-To: <1591364.ylu2LGHCpE@sergeyv>
not sure if that causes a memory leak since st not freed ?
On Mon, May 18, 2015 at 4:33 PM, Sergey Vidishev <sergeyv@yandex-team.ru> wrote:
> From c1e59424bfabee349aa7b8b903833475a56cf145 Mon Sep 17 00:00:00 2001
> From: Sergey Vidishev <sergeyv@yandex-team.ru>
> Date: Wed, 8 Oct 2014 21:51:03 +0400
> Subject: [PATCH] mdadm: monitor: fix nullptr dereference when get_md_name()
> returns NULL
>
> Function add_new_arrays() expects that function get_md_name() should
> return pointer to devname, but also get_md_name() may return NULL. So
> check the pointer before use it in add_new_arrays().
>
> Signed-off-by: Sergey Vidishev <sergeyv@yandex-team.ru>
> ---
>
> v1 -> v2: more verbose commit message
>
> This patch against fresh git://neil.brown.name/mdadm.
> I'm not subscribed to the list, please CC me in replies.
>
> Monitor.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/Monitor.c b/Monitor.c
> index 1cd378b..1bbaf89 100644
> --- a/Monitor.c
> +++ b/Monitor.c
> @@ -687,6 +687,7 @@ static int add_new_arrays(struct mdstat_ent *mdstat, struct state **statelist,
> {
> struct mdstat_ent *mse;
> int new_found = 0;
> + char *name;
>
> for (mse=mdstat; mse; mse=mse->next)
> if (mse->devnm[0] &&
> @@ -697,7 +698,12 @@ static int add_new_arrays(struct mdstat_ent *mdstat, struct state **statelist,
> struct state *st = xcalloc(1, sizeof *st);
> mdu_array_info_t array;
> int fd;
> - st->devname = xstrdup(get_md_name(mse->devnm));
> +
> + name = get_md_name(mse->devnm);
> + if (!name)
> + return 0;
> +
> + st->devname = xstrdup(name);
> if ((fd = open(st->devname, O_RDONLY)) < 0 ||
> ioctl(fd, GET_ARRAY_INFO, &array)< 0) {
> /* no such array */
> --
> 1.9.1
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH v2] mdadm: monitor: fix nullptr dereference when get_md_name() returns NULL
From: Sergey Vidishev @ 2015-05-18 23:33 UTC (permalink / raw)
To: linux-raid; +Cc: NeilBrown
In-Reply-To: <1531453.K804EMet0W@sergeyv_box>
From c1e59424bfabee349aa7b8b903833475a56cf145 Mon Sep 17 00:00:00 2001
From: Sergey Vidishev <sergeyv@yandex-team.ru>
Date: Wed, 8 Oct 2014 21:51:03 +0400
Subject: [PATCH] mdadm: monitor: fix nullptr dereference when get_md_name()
returns NULL
Function add_new_arrays() expects that function get_md_name() should
return pointer to devname, but also get_md_name() may return NULL. So
check the pointer before use it in add_new_arrays().
Signed-off-by: Sergey Vidishev <sergeyv@yandex-team.ru>
---
v1 -> v2: more verbose commit message
This patch against fresh git://neil.brown.name/mdadm.
I'm not subscribed to the list, please CC me in replies.
Monitor.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/Monitor.c b/Monitor.c
index 1cd378b..1bbaf89 100644
--- a/Monitor.c
+++ b/Monitor.c
@@ -687,6 +687,7 @@ static int add_new_arrays(struct mdstat_ent *mdstat, struct state **statelist,
{
struct mdstat_ent *mse;
int new_found = 0;
+ char *name;
for (mse=mdstat; mse; mse=mse->next)
if (mse->devnm[0] &&
@@ -697,7 +698,12 @@ static int add_new_arrays(struct mdstat_ent *mdstat, struct state **statelist,
struct state *st = xcalloc(1, sizeof *st);
mdu_array_info_t array;
int fd;
- st->devname = xstrdup(get_md_name(mse->devnm));
+
+ name = get_md_name(mse->devnm);
+ if (!name)
+ return 0;
+
+ st->devname = xstrdup(name);
if ((fd = open(st->devname, O_RDONLY)) < 0 ||
ioctl(fd, GET_ARRAY_INFO, &array)< 0) {
/* no such array */
--
1.9.1
^ permalink raw reply related
* RAID1 working correctly, error messages during boot
From: Hans Malissa @ 2015-05-18 20:29 UTC (permalink / raw)
To: linux-raid
I have a software-RAID1 that seems to be working correctly:
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
976629568 blocks super 1.2 [2/2] [UU]
unused devices: <none>
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sun May 17 15:21:30 2015
Raid Level : raid1
Array Size : 976629568 (931.39 GiB 1000.07 GB)
Used Dev Size : 976629568 (931.39 GiB 1000.07 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Mon May 18 10:28:36 2015
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : eprb21:0 (local to host eprb21)
UUID : 0901fe50:444a29b6:d3caff14:e45ef9cc
Events : 19
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
But, on the other hand, when the system boots, I briefly see the following messages:
doing fast boot
Creating device nodes with udev
udevd[174]: failed to execute ‘/sbin/mdadm’ ‘/sbin/mdadm --incremental /dev/sdb1
udevd[175]: failed to execute ‘/sbin/mdadm’ ‘/sbin/mdadm --incremental /dev/sdc1 --offroot’: No such file or directory
But otherwise the system appears to run normally. After booting, /dev/md0 seems to be working correctly.
What does it mean, and should I worry about it? What can I do about it? My system is openSUSE 12.2.
Thanks a lot,
Hans Malissa--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Breaks LSI RAID on C600 chipset
From: David F. @ 2015-05-18 17:58 UTC (permalink / raw)
To: NeilBrown; +Cc: John Stoffel, linux-raid@vger.kernel.org
In-Reply-To: <CAGRSmLvS3kFFGwEWGE8H7smi+uN-EowPGYJxS-dM4+tezFUuVw@mail.gmail.com>
Do you think the latest version at
http://git.neil.brown.name/?p=mdadm.git;a=summary is stable enough to
use? It definitely fixes the major issue (losing the raid config)
from the prior version.
On Wed, May 13, 2015 at 7:26 AM, David F. <df7729@gmail.com> wrote:
> Thanks.
>
> 1) The update did fix the broken RAID. I can reboot and the RAID
> mirror still exists after reboots.
>
> 2) I do want to report that on the first test, the reboot was clean.
> On the second reboot there was a warning message about the BIOS
> detecting "Inconsistent Timestamps" on one of the drives and something
> about which CONFIG the BIOS would use. I didn't have time to get it
> all jotted down. Whatever it used was still okay since the RAID
> mirror was still there after that message.
>
>
> On Tue, May 12, 2015 at 5:35 PM, NeilBrown <neilb@suse.de> wrote:
>> On Mon, 11 May 2015 09:31:26 -0700 "David F." <df7729@gmail.com> wrote:
>>
>>> any progress or more information needed on this?
>>>
>>> On Mon, Apr 20, 2015 at 10:39 PM, David F. <df7729@gmail.com> wrote:
>>> > If you need access to the system, a network kvm is available - or if
>>> > you have a C600 based system, you should see the same problem there.
>>> >
>>> > On Fri, Apr 10, 2015 at 1:07 PM, David F. <df7729@gmail.com> wrote:
>>> >> Okay, and to confirm, this is happening on other C600 based systems.
>>> >> Boot to Linux with MDADM raid support and the raid is gone after
>>> >> reboot.
>>> >>
>>> >> On Thu, Apr 9, 2015 at 5:50 PM, NeilBrown <neilb@suse.de> wrote:
>>> >>> On Thu, 9 Apr 2015 14:13:05 -0700 "David F." <df7729@gmail.com> wrote:
>>> >>>
>>> >>>> Hello,
>>> >>>>
>>> >>>> I built a new system, installed new drives (no partitioning, just raw
>>> >>>> new drives), configured RAID 1, boot to linux, created reports
>>> >>>> attached. Rebooted and the system doesn't see any configured raid
>>> >>>> drives (just the two raw drives).
>>> >>>
>>> >>> Thanks. I might take a look, but I would greatly prefer it if you kept the
>>> >>> linux-raid list on the Cc.....
>>> >>>
>>> >>> NeilBrown
>>> >>>
>>
>> I must confess that I haven't even looked at it.
>> But I just saw and email on linux-raid from Martin Wilck:
>>
>> Subject: [PATCH] DDF: _write_super_to_disk: fix anchor header type
>>
>> which very likely fixes your problem. I've applied it and pushed out to
>> git://neil.brown.name/mdadm/
>>
>> NeilBrown
^ permalink raw reply
* Re: [PATCH V2 01/11] Create n bitmaps for clustered mode
From: Guoqing Jiang @ 2015-05-18 2:21 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, rgoldwyn
In-Reply-To: <20150515154527.7751f765@notabene.brown>
NeilBrown wrote:
>> diff --git a/mdadm.c b/mdadm.c
>> index 3e8c49b..bd9382e 100644
>> --- a/mdadm.c
>> +++ b/mdadm.c
>> @@ -1097,6 +1097,15 @@ int main(int argc, char *argv[])
>> s.bitmap_file = optarg;
>> continue;
>> }
>> + if (strcmp(optarg, "clustered")== 0) {
>> + s.bitmap_file = optarg;
>> + /* Set the default number of cluster nodes
>> + * to 4 if not already set by user
>> + */
>> + if (c.nodes < 1)
>> + c.nodes = 4;
>> + continue;
>> + }
>>
>
>
> Please make sure than mdadm compiles after each patch is applied.
> You don't defined that 'nodes' field until a later patch.
>
>
Oops, I will fix it in next version.
Thanks,
Guoqing
> NeilBrown
>
>
^ permalink raw reply
* Re: [PATCH] md: convert to kstrto*()
From: NeilBrown @ 2015-05-18 2:12 UTC (permalink / raw)
To: Alexey Dobriyan; +Cc: linux-raid
In-Reply-To: <20150516110238.GA32480@p183.telecom.by>
[-- Attachment #1: Type: text/plain, Size: 8253 bytes --]
On Sat, 16 May 2015 14:02:38 +0300 Alexey Dobriyan <adobriyan@gmail.com>
wrote:
> Convert away from deprecated simple_strto*() functions.
>
> Add "fit into sector_t" checks.
>
> Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> ---
>
> drivers/md/md.c | 149 ++++++++++++++++++++++++++++++--------------------------
> 1 file changed, 81 insertions(+), 68 deletions(-)
>
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -2630,13 +2630,14 @@ errors_show(struct md_rdev *rdev, char *page)
> static ssize_t
> errors_store(struct md_rdev *rdev, const char *buf, size_t len)
> {
> - char *e;
> - unsigned long n = simple_strtoul(buf, &e, 10);
> - if (*buf && (*e == 0 || *e == '\n')) {
> - atomic_set(&rdev->corrected_errors, n);
> - return len;
> - }
> - return -EINVAL;
> + unsigned int n;
> + int rv;
> +
> + rv = kstrtouint(buf, 10, &n);
> + if (rv < 0)
> + return rv;
> + atomic_set(&rdev->corrected_errors, n);
> + return len;
> }
> static struct rdev_sysfs_entry rdev_errors =
> __ATTR(errors, S_IRUGO|S_IWUSR, errors_show, errors_store);
> @@ -2653,13 +2654,16 @@ slot_show(struct md_rdev *rdev, char *page)
> static ssize_t
> slot_store(struct md_rdev *rdev, const char *buf, size_t len)
> {
> - char *e;
> + int slot;
> int err;
> - int slot = simple_strtoul(buf, &e, 10);
> +
> if (strncmp(buf, "none", 4)==0)
> slot = -1;
> - else if (e==buf || (*e && *e!= '\n'))
> - return -EINVAL;
> + else {
> + err = kstrtouint(buf, 10, (unsigned int *)&slot);
> + if (err < 0)
> + return err;
> + }
> if (rdev->mddev->pers && slot == -1) {
> /* Setting 'slot' on an active array requires also
> * updating the 'rd%d' link, and communicating
> @@ -3544,12 +3548,12 @@ layout_show(struct mddev *mddev, char *page)
> static ssize_t
> layout_store(struct mddev *mddev, const char *buf, size_t len)
> {
> - char *e;
> - unsigned long n = simple_strtoul(buf, &e, 10);
> + unsigned int n;
> int err;
>
> - if (!*buf || (*e && *e != '\n'))
> - return -EINVAL;
> + err = kstrtouint(buf, 10, &n);
> + if (err < 0)
> + return err;
> err = mddev_lock(mddev);
> if (err)
> return err;
> @@ -3593,12 +3597,12 @@ static int update_raid_disks(struct mddev *mddev, int raid_disks);
> static ssize_t
> raid_disks_store(struct mddev *mddev, const char *buf, size_t len)
> {
> - char *e;
> + unsigned int n;
> int err;
> - unsigned long n = simple_strtoul(buf, &e, 10);
>
> - if (!*buf || (*e && *e != '\n'))
> - return -EINVAL;
> + err = kstrtouint(buf, 10, &n);
> + if (err < 0)
> + return err;
>
> err = mddev_lock(mddev);
> if (err)
> @@ -3645,12 +3649,12 @@ chunk_size_show(struct mddev *mddev, char *page)
> static ssize_t
> chunk_size_store(struct mddev *mddev, const char *buf, size_t len)
> {
> + unsigned long n;
> int err;
> - char *e;
> - unsigned long n = simple_strtoul(buf, &e, 10);
>
> - if (!*buf || (*e && *e != '\n'))
> - return -EINVAL;
> + err = kstrtoul(buf, 10, &n);
> + if (err < 0)
> + return err;
>
> err = mddev_lock(mddev);
> if (err)
> @@ -3688,19 +3692,24 @@ resync_start_show(struct mddev *mddev, char *page)
> static ssize_t
> resync_start_store(struct mddev *mddev, const char *buf, size_t len)
> {
> + unsigned long long n;
> int err;
> - char *e;
> - unsigned long long n = simple_strtoull(buf, &e, 10);
> +
> + if (cmd_match(buf, "none"))
> + n = MaxSector;
> + else {
> + err = kstrtoull(buf, 10, &n);
> + if (err < 0)
> + return err;
> + if (n != (sector_t)n)
> + return -EINVAL;
> + }
>
> err = mddev_lock(mddev);
> if (err)
> return err;
> if (mddev->pers && !test_bit(MD_RECOVERY_FROZEN, &mddev->recovery))
> err = -EBUSY;
> - else if (cmd_match(buf, "none"))
> - n = MaxSector;
> - else if (!*buf || (*e && *e != '\n'))
> - err = -EINVAL;
>
> if (!err) {
> mddev->recovery_cp = n;
> @@ -3936,14 +3945,14 @@ max_corrected_read_errors_show(struct mddev *mddev, char *page) {
> static ssize_t
> max_corrected_read_errors_store(struct mddev *mddev, const char *buf, size_t len)
> {
> - char *e;
> - unsigned long n = simple_strtoul(buf, &e, 10);
> + unsigned int n;
> + int rv;
>
> - if (*buf && (*e == 0 || *e == '\n')) {
> - atomic_set(&mddev->max_corr_read_errors, n);
> - return len;
> - }
> - return -EINVAL;
> + rv = kstrtouint(buf, 10, &n);
> + if (rv < 0)
> + return rv;
> + atomic_set(&mddev->max_corr_read_errors, n);
> + return len;
> }
>
> static struct md_sysfs_entry max_corr_read_errors =
> @@ -4297,15 +4306,18 @@ sync_min_show(struct mddev *mddev, char *page)
> static ssize_t
> sync_min_store(struct mddev *mddev, const char *buf, size_t len)
> {
> - int min;
> - char *e;
> + unsigned int min;
> + int rv;
> +
> if (strncmp(buf, "system", 6)==0) {
> - mddev->sync_speed_min = 0;
> - return len;
> + min = 0;
> + } else {
> + rv = kstrtouint(buf, 10, &min);
> + if (rv < 0)
> + return rv;
> + if (min == 0)
> + return -EINVAL;
> }
> - min = simple_strtoul(buf, &e, 10);
> - if (buf == e || (*e && *e != '\n') || min <= 0)
> - return -EINVAL;
> mddev->sync_speed_min = min;
> return len;
> }
> @@ -4323,15 +4335,18 @@ sync_max_show(struct mddev *mddev, char *page)
> static ssize_t
> sync_max_store(struct mddev *mddev, const char *buf, size_t len)
> {
> - int max;
> - char *e;
> + unsigned int max;
> + int rv;
> +
> if (strncmp(buf, "system", 6)==0) {
> - mddev->sync_speed_max = 0;
> - return len;
> + max = 0;
> + } else {
> + rv = kstrtouint(buf, 10, &max);
> + if (rv < 0)
> + return rv;
> + if (max == 0)
> + return -EINVAL;
> }
> - max = simple_strtoul(buf, &e, 10);
> - if (buf == e || (*e && *e != '\n') || max <= 0)
> - return -EINVAL;
> mddev->sync_speed_max = max;
> return len;
> }
> @@ -4514,12 +4529,13 @@ suspend_lo_show(struct mddev *mddev, char *page)
> static ssize_t
> suspend_lo_store(struct mddev *mddev, const char *buf, size_t len)
> {
> - char *e;
> - unsigned long long new = simple_strtoull(buf, &e, 10);
> - unsigned long long old;
> + unsigned long long old, new;
> int err;
>
> - if (buf == e || (*e && *e != '\n'))
> + err = kstrtoull(buf, 10, &new);
> + if (err < 0)
> + return err;
> + if (new != (sector_t)new)
> return -EINVAL;
>
> err = mddev_lock(mddev);
> @@ -4556,12 +4572,13 @@ suspend_hi_show(struct mddev *mddev, char *page)
> static ssize_t
> suspend_hi_store(struct mddev *mddev, const char *buf, size_t len)
> {
> - char *e;
> - unsigned long long new = simple_strtoull(buf, &e, 10);
> - unsigned long long old;
> + unsigned long long old, new;
> int err;
>
> - if (buf == e || (*e && *e != '\n'))
> + err = kstrtoull(buf, 10, &new);
> + if (err < 0)
> + return err;
> + if (new != (sector_t)new)
> return -EINVAL;
>
> err = mddev_lock(mddev);
> @@ -4603,11 +4620,13 @@ static ssize_t
> reshape_position_store(struct mddev *mddev, const char *buf, size_t len)
> {
> struct md_rdev *rdev;
> - char *e;
> + unsigned long long new;
> int err;
> - unsigned long long new = simple_strtoull(buf, &e, 10);
>
> - if (buf == e || (*e && *e != '\n'))
> + err = kstrtoull(buf, 10, &new);
> + if (err < 0)
> + return err;
> + if (new != (sector_t)new)
> return -EINVAL;
> err = mddev_lock(mddev);
> if (err)
> @@ -9009,13 +9028,7 @@ static int get_ro(char *buffer, struct kernel_param *kp)
> }
> static int set_ro(const char *val, struct kernel_param *kp)
> {
> - char *e;
> - int num = simple_strtoul(val, &e, 10);
> - if (*val && (*e == '\0' || *e == '\n')) {
> - start_readonly = num;
> - return 0;
> - }
> - return -EINVAL;
> + return kstrtouint(val, 10, (unsigned int *)&start_readonly);
> }
>
> module_param_call(start_ro, set_ro, get_ro, NULL, S_IRUSR|S_IWUSR);
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Looks good - thanks.
I'll queue for the next merge window.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: RAID5 assemble fails after reboot while reshaping
From: NeilBrown @ 2015-05-18 0:03 UTC (permalink / raw)
To: Marco Fuckner; +Cc: linux-raid
In-Reply-To: <5558C551.80203@die-fuckners.de>
[-- Attachment #1: Type: text/plain, Size: 5998 bytes --]
On Sun, 17 May 2015 18:44:01 +0200 Marco Fuckner <marco@die-fuckners.de>
wrote:
> Hi everybody,
>
> first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
> partitioned with the same geometry.
>
> I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
> disks and initiating the grow, the reshape didn't seem to start:
>
> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
> [>....................] reshape = 0.0% (0/3906681600)
> finish=166847860.0min speed=0K/sec
> bitmap: 0/30 pages [0KB], 65536KB chunk
>
> I waited about three hours and checked again:
>
> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
> [>....................] reshape = 0.0% (0/3906681600)
> finish=9599856140.0min speed=0K/sec
> bitmap: 0/30 pages [0KB], 65536KB chunk
This looks very much like the reshape has not done anything at all.
i.e. your data is still exactly where you left it, it is just a case of
getting hold of it.
It's not impossible that running the --assemble with --update=revert-reshape
would work, but I'm far from certain. If you backed up the first gigabyte of
each device (sd?1) first then it would probably be safe enough to try.
Another option is to add --invalid-backup to the --assemble command.
This has a reasonable chance of allowing the reshape to continue, but also
has a reasonable chance of corrupting the first few megabytes of your array
(the part that it things should be backed up).
If you "make test_stripe" in the mdadm source code, you can use that to extra
the first few megabytes of array data so you could restore it if it gets
corrupted.
Something like
test_stripe save /root/thing 4 262144 5 2 0 \
$BIGNUM /dev/sdc1:262144 /dev/sdh1:262144 ......
Check the source to make sure you get the args right.
Make sure the order of the devices and their data_offsets are correct. check
the "Device Role:" for each and order them by that number.
Another option is to recreate the array as 4-drive RAID5. Again you need to
make sure the device order and data offsets are correct, along with all the
other data.
I might be able to dig into the code and find out what happened and maybe
offer an "easier" solution, but that won't be for a day or two at least.
NeilBrown
>
> Unfortunately, I forgot to save the output of the grow command, but it
> exited with 0.
> /mdadm --misc --detail /dev/md0/ didn't show anything suspicious to me:
>
> /dev/md0:
> Version : 1.2
> Creation Time : Sun Nov 9 02:38:25 2014
> Raid Level : raid5
> Array Size : 11720044800 (11177.11 GiB 12001.33 GB)
> Used Dev Size : 3906681600 (3725.70 GiB 4000.44 GB)
> Raid Devices : 7
> Total Devices : 7
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Mon May 11 11:55:07 2015
> State : clean, reshaping
> Active Devices : 7
> Working Devices : 7
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Reshape Status : 0% complete
> Delta Devices : 3, (4->7)
>
> Name : anaNAS:0 (local to host anaNAS)
> UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
> Events : 51839
>
> Number Major Minor RaidDevice State
> 0 8 33 0 active sync
> /dev/sdc1
> 1 8 113 1 active sync
> /dev/sdh1
> 3 8 97 2 active sync
> /dev/sdg1
> 4 8 17 3 active sync
> /dev/sdb1
> 7 8 81 4 active sync
> /dev/sdf1
> 6 8 65 5 active sync
> /dev/sde1
> 5 8 49 6 active sync
> /dev/sdd1
>
> As it looked like it wouldn't be ready until long after my death and I
> also wrote a backup file, somehow restarting and continuing afterwards
> seemed reasonable to me.
> The source I was reading suggested running /mdadm /dev/md0 --continue
> --backup-file=$FILE/. Apparently this command was wrong, and I couldn't
> reassamble the array:
>
> # mdadm --assemble /dev/md0 --verbose /dev/sd[b-h]1
> --backup-file=/root/grow7backup.bak
>
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
> mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
> mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
> mdadm: :/dev/md0 has an active reshape - checking if critical
> section needs to be restored
> mdadm: No backup metadata on /root/grow7backup.bak
> mdadm: No backup metadata on device-4
> mdadm: No backup metadata on device-5
> mdadm: No backup metadata on device-6
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
>
> I started searching for answers but didn't find anything helpful except
> the hint on the raid.wiki.kernel.org page to send an email here. The
> last sentence from mdadm sounds a bit pessimistic, but I hope someone in
> here can help me. The output of /mdadm --examine /dev/sd[bh]1 /is in the
> attachment.
>
> Thanks in advance,
>
> Marco
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: RAID5 assemble fails after reboot while reshaping
From: Marco Fuckner @ 2015-05-17 23:14 UTC (permalink / raw)
To: Phil Turmel, linux-raid
In-Reply-To: <5558E34F.4060708@turmel.org>
Hey Phil,
Am 17.05.2015 um 20:51 schrieb Phil Turmel:
> Hi Marco,
>
> On 05/17/2015 12:44 PM, Marco Fuckner wrote:
>> Hi everybody,
>>
>> first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
>> partitioned with the same geometry.
>>
>> I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
>> disks and initiating the grow, the reshape didn't seem to start:
>>
>> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
>> sdc1[0]
>> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
>> [7/7] [UUUUUUU]
>> [>....................] reshape = 0.0% (0/3906681600)
>> finish=166847860.0min speed=0K/sec
>> bitmap: 0/30 pages [0KB], 65536KB chunk
> Do you have the exact command you used to start the grow available? Did
> you include a backup file? Was it on a device outside the raid?
I used mdadm --grow --raid-devices=7 /dev/md0
--backup-file=/root/grow7backup.bak, the system is on a single disk
outside of the RAID.
> So nothing (or garbage) was written to your backup in the first place.
> Try again with the "--invalid-backup" option to skip trying to read the
> supposedly backed up critical section. You may have corruption to fix
> for that small section.
With the --invalid-backup switch it looks like this:
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
mdadm: :/dev/md0 has an active reshape - checking if critical
section needs to be restored
mdadm: No backup metadata on device-4
mdadm: No backup metadata on device-5
mdadm: No backup metadata on device-6
mdadm: Failed to find backup of critical section
mdadm: continuing without restoring backup
mdadm: added /dev/sdh1 to /dev/md0 as 1
mdadm: added /dev/sdg1 to /dev/md0 as 2
mdadm: added /dev/sdb1 to /dev/md0 as 3
mdadm: added /dev/sdf1 to /dev/md0 as 4
mdadm: added /dev/sde1 to /dev/md0 as 5
mdadm: added /dev/sdd1 to /dev/md0 as 6
mdadm: added /dev/sdc1 to /dev/md0 as 0
mdadm: failed to RUN_ARRAY /dev/md0: Invalid argument
This was the case with either my normal system or using a rescue CD
(mdadm 3.3.1-r2 and linux 3.14.35).
> Good report. Unfortunately, it sounds like a bug.
>
> If the --invalid-backup option doesn't help, my next suggestion would be
> to temporarily boot with a system rescue CD and continuing the --grow
> operation with a more stable kernel. If your backup file isn't empty,
> put it on a thumb drive or somewhere accessible to a rescue boot.
>
> If it works with a slightly older kernel, we'll need Neil.
>
> Phil
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
The backup file itself is about 1,6M in size. Commands with --grow fail
as the array is marked as inactive:
mdadm: /dev/md0 is not an active md array - aborting
I forgot to include the current status in the last mail:
md0 : inactive sdf1[7](S) sdg1[3](S) sdc1[0](S) sde1[6](S)
sdh1[1](S) sdd1[5](S) sdb1[4](S)
27346771700 blocks super 1.2
It seems like every disk is falsely recognized as a spare?
Regards,
Marco
^ permalink raw reply
* Re: RAID5 assemble fails after reboot while reshaping
From: Phil Turmel @ 2015-05-17 18:51 UTC (permalink / raw)
To: Marco Fuckner, linux-raid
In-Reply-To: <5558C551.80203@die-fuckners.de>
Hi Marco,
On 05/17/2015 12:44 PM, Marco Fuckner wrote:
> Hi everybody,
>
> first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
> partitioned with the same geometry.
>
> I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
> disks and initiating the grow, the reshape didn't seem to start:
>
> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
> [>....................] reshape = 0.0% (0/3906681600)
> finish=166847860.0min speed=0K/sec
> bitmap: 0/30 pages [0KB], 65536KB chunk
Do you have the exact command you used to start the grow available? Did
you include a backup file? Was it on a device outside the raid?
> I waited about three hours and checked again:
>
> md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
> sdc1[0]
> 11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
> [7/7] [UUUUUUU]
> [>....................] reshape = 0.0% (0/3906681600)
> finish=9599856140.0min speed=0K/sec
> bitmap: 0/30 pages [0KB], 65536KB chunk
That's not good. Looks like it is choking on the very first critical
section backup.
> Unfortunately, I forgot to save the output of the grow command, but it
> exited with 0.
[trim /]
> As it looked like it wouldn't be ready until long after my death and I
> also wrote a backup file, somehow restarting and continuing afterwards
> seemed reasonable to me.
> The source I was reading suggested running /mdadm /dev/md0 --continue
> --backup-file=$FILE/. Apparently this command was wrong, and I couldn't
> reassamble the array:
>
> # mdadm --assemble /dev/md0 --verbose /dev/sd[b-h]1
> --backup-file=/root/grow7backup.bak
Ah. That looks like a backup file in an appropriate location. :-)
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
> mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
> mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
> mdadm: :/dev/md0 has an active reshape - checking if critical
> section needs to be restored
> mdadm: No backup metadata on /root/grow7backup.bak
> mdadm: No backup metadata on device-4
> mdadm: No backup metadata on device-5
> mdadm: No backup metadata on device-6
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
So nothing (or garbage) was written to your backup in the first place.
Try again with the "--invalid-backup" option to skip trying to read the
supposedly backed up critical section. You may have corruption to fix
for that small section.
> I started searching for answers but didn't find anything helpful except
> the hint on the raid.wiki.kernel.org page to send an email here. The
> last sentence from mdadm sounds a bit pessimistic, but I hope someone in
> here can help me. The output of /mdadm --examine /dev/sd[bh]1 /is in the
> attachment.
Good report. Unfortunately, it sounds like a bug.
If the --invalid-backup option doesn't help, my next suggestion would be
to temporarily boot with a system rescue CD and continuing the --grow
operation with a more stable kernel. If your backup file isn't empty,
put it on a thumb drive or somewhere accessible to a rescue boot.
If it works with a slightly older kernel, we'll need Neil.
Phil
^ permalink raw reply
* RAID5 assemble fails after reboot while reshaping
From: Marco Fuckner @ 2015-05-17 16:44 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 4085 bytes --]
Hi everybody,
first of all, I'm using mdadm 3.3.2 on linux 4.0.1, all of my disks are
partitioned with the same geometry.
I wanted to grow my 4 disk RAID5 array to 7 disks. After adding the
disks and initiating the grow, the reshape didn't seem to start:
md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
sdc1[0]
11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
[7/7] [UUUUUUU]
[>....................] reshape = 0.0% (0/3906681600)
finish=166847860.0min speed=0K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk
I waited about three hours and checked again:
md0 : active raid5 sdf1[7] sde1[6] sdd1[5] sdg1[3] sdb1[4] sdh1[1]
sdc1[0]
11720044800 blocks super 1.2 level 5, 256k chunk, algorithm 2
[7/7] [UUUUUUU]
[>....................] reshape = 0.0% (0/3906681600)
finish=9599856140.0min speed=0K/sec
bitmap: 0/30 pages [0KB], 65536KB chunk
Unfortunately, I forgot to save the output of the grow command, but it
exited with 0.
/mdadm --misc --detail /dev/md0/ didn't show anything suspicious to me:
/dev/md0:
Version : 1.2
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Array Size : 11720044800 (11177.11 GiB 12001.33 GB)
Used Dev Size : 3906681600 (3725.70 GiB 4000.44 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon May 11 11:55:07 2015
State : clean, reshaping
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Reshape Status : 0% complete
Delta Devices : 3, (4->7)
Name : anaNAS:0 (local to host anaNAS)
UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Events : 51839
Number Major Minor RaidDevice State
0 8 33 0 active sync
/dev/sdc1
1 8 113 1 active sync
/dev/sdh1
3 8 97 2 active sync
/dev/sdg1
4 8 17 3 active sync
/dev/sdb1
7 8 81 4 active sync
/dev/sdf1
6 8 65 5 active sync
/dev/sde1
5 8 49 6 active sync
/dev/sdd1
As it looked like it wouldn't be ready until long after my death and I
also wrote a backup file, somehow restarting and continuing afterwards
seemed reasonable to me.
The source I was reading suggested running /mdadm /dev/md0 --continue
--backup-file=$FILE/. Apparently this command was wrong, and I couldn't
reassamble the array:
# mdadm --assemble /dev/md0 --verbose /dev/sd[b-h]1
--backup-file=/root/grow7backup.bak
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0.
mdadm: :/dev/md0 has an active reshape - checking if critical
section needs to be restored
mdadm: No backup metadata on /root/grow7backup.bak
mdadm: No backup metadata on device-4
mdadm: No backup metadata on device-5
mdadm: No backup metadata on device-6
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
I started searching for answers but didn't find anything helpful except
the hint on the raid.wiki.kernel.org page to send an email here. The
last sentence from mdadm sounds a bit pessimistic, but I hope someone in
here can help me. The output of /mdadm --examine /dev/sd[bh]1 /is in the
attachment.
Thanks in advance,
Marco
[-- Attachment #2: raid.status --]
[-- Type: text/plain, Size: 7434 bytes --]
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 721f47b6:4374452b:8cf4d3c6:672caeb3
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : c6c20176 - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 3
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 528ea570:3578ca82:4e86d330:bf30634c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : db5a0a5d - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 0
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : b6dfe4a2:6db19975:36b5c400:f470ac28
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : aca3041b - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 6
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : e233293b:919abecc:d5d7f0d4:7c1b748b
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d3000e94 - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 5
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 7a51397d:9785430a:bb4e502d:c07c6357
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 76e3ef5c - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 4
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 7b0832a7:da8108cd:0bd25ee8:c6bc0ba6
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6d5865f4 - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 2
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 33f0604f:46e80f5e:11b1a694:608fd9b3
Name : anaNAS:0 (local to host anaNAS)
Creation Time : Sun Nov 9 02:38:25 2014
Raid Level : raid5
Raid Devices : 7
Avail Dev Size : 7813363343 (3725.70 GiB 4000.44 GB)
Array Size : 23440089600 (22354.21 GiB 24002.65 GB)
Used Dev Size : 7813363200 (3725.70 GiB 4000.44 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : d15d44f2:1c4ed6dc:0726a874:b0422ef3
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 3 (4->7)
Update Time : Mon May 11 14:34:33 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : a1a46170 - correct
Events : 51840
Layout : left-symmetric
Chunk Size : 256K
Device Role : Active device 1
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply
* Problems with bdev_write_page().
From: NeilBrown @ 2015-05-17 7:02 UTC (permalink / raw)
To: Matthew Wilcox, Charles Bertsch; +Cc: linux-raid, lkml
[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]
Hi Matthew,
I've just been looking at bdev_write_page().
You can read about why here:
http://marc.info/?t=142984068300001&r=1&w=2
it ends with a "git bisect" which points the finger at you.
If I look at bdev_write_page() it says:
* On entry, the page should be locked and not currently under writeback.
* On exit, if the write started successfully, the page will be unlocked and
* under writeback. If the write failed already (eg the driver failed to
* queue the page to the device), the page will still be locked. If the
* caller is a ->writepage implementation, it will need to unlock the page.
So the page is unlocked on success.
In __mpage_writepage() I find
if (!bdev_write_page(bdev, blocks[0] << (blkbits - 9),
page, wbc)) {
clean_buffers(page, first_unmapped);
so if bdev_write_page() succeeds, i.e. if it returns '0', then
clean_buffers() is called. At this point the page is unlocked remember.
clean_buffers may call
try_to_free_buffers(page);
(without first locking the page, so still unlocked)..
try_to_free_buffers starts:
BUG_ON(!PageLocked(page));
Opps.
Can you propose a fix for Charles, who can trigger this bug and nicely
bisected it for us - thanks Charles!!!
Also while looking at the code, I notice that brd_rw_page() unconditionally
calls page_endio() and, in the WRITE case, page_endio unconditionally calls
end_page_writeback(), which has
if (!test_clear_page_writeback(page))
BUG();
and so cannot tolerate being called twice in a row.
So if brd_rw_page() ever returned an error (which seems possible though not
likely), end_page_writeback() would be called once by page_endio() and once
in the error path of bdev_write_page(), and the BUG above would be triggered.
I'll leave that for you to sort out too :-)
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH] md-cluster: avoid deadlock on MESSAGE lock resource
From: Goldwyn Rodrigues @ 2015-05-16 20:58 UTC (permalink / raw)
To: Abhijit Bhopatkar, linux-raid, Lidong Zhong; +Cc: Reese Faucette (rfaucett)
In-Reply-To: <554CB6B1.3030206@cisco.com>
On 05/08/2015 08:14 AM, Abhijit Bhopatkar wrote:
> On 08/05/15 6:40 pm, Abhijit Bhopatkar wrote:
>>
>> Every receiver has CR lock on MESSAGE while processing the message. When
>> every receiver releases ACK lock and for some reason fails to grab EX on
>> MESSAGE resource in time, a waiting sender could queue an EX on MESSAGE
>> instead. Now when receiver queues its up convert request on MESSAGE it
>> will end up in a deadlock situation.
>>
>> Setting NOQUEUE flag on MESSAGE lock resource while grabbing the EX on
>> MESSAGE on sender will avoid this deadlock. If sender can not grab
>> MESSAGE lock immediately it should retry until the lock is granted.
>>
>> Signed-off-by: Abhijit Bhopatkar <abhopatk@cisco.com>
>> ---
>> This has been minimally tested on a three node cluster.
>>
>
> I have tested standard mdadm operations (create, assemble etc).
> What more testing would you want me to do on this before its considered
> ready?
I am not sure how using LKF_NOQUEUE will help in this situation here.
LKF_NOQUEUE primarily means do not queue if you can't grant it right
away. Besides, I don't like the idea of goto loop.
The sender can still creep in between the ack and the message locks. A
situation would be where the "disrupting" sender is the lock owner of
all the locks and hence will not have to pay communication costs and
will manage to attain the locks faster.
Perhaps DLM_LKF_HEADQUEUE or DLM_LKF_NOORDER is what you are looking
for, but that again is not the complete solution.
Another idea I could think of is for the sender to downconvert TOKEN to
a shared lock such as CR halfway in the communication (say after
message CR), and all receivers take the TOKEN in CR mode and release it
once the communication is finally over.
Regards,
>
> Regards,
> Abhijit
>
>> drivers/md/md-cluster.c | 14 ++++++++++++--
>> 1 file changed, 12 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
>> index fcfc4b9..04ac309 100644
>> --- a/drivers/md/md-cluster.c
>> +++ b/drivers/md/md-cluster.c
>> @@ -512,7 +512,10 @@ static void unlock_comm(struct md_cluster_info *cinfo)
>> * This function performs the actual sending of the message. This function is
>> * usually called after performing the encompassing operation
>> * The function:
>> - * 1. Grabs the message lockresource in EX mode
>> + * 1. Grabs the message lockresource in EX. Do not queue the request if not granted
>> + immediately. This avoids deadlock with receivers when receivers try to
>> + upconvert CR to EX of message lockresource. The thread will retry until the
>> + request is granted.
>> * 2. Copies the message to the message LVB
>> * 3. Downconverts message lockresource to CR
>> * 4. Upconverts ack lock resource from CR to EX. This forces the BAST on other nodes
>> @@ -526,12 +529,19 @@ static int __sendmsg(struct md_cluster_info *cinfo, struct cluster_msg *cmsg)
>> int slot = cinfo->slot_number - 1;
>>
>> cmsg->slot = cpu_to_le32(slot);
>> - /*get EX on Message*/
>> +
>> + /* get EX on Message with noqueue flag */
>> + cinfo->message_lockres->flags |= DLM_LKF_NOQUEUE;
>> +
>> +retry:
>> error = dlm_lock_sync(cinfo->message_lockres, DLM_LOCK_EX);
>> if (error) {
>> + if (error == -EAGAIN)
>> + goto retry;
>> pr_err("md-cluster: failed to get EX on MESSAGE (%d)\n", error);
>> goto failed_message;
>> }
>> + cinfo->message_lockres->flags &= ~DLM_LKF_NOQUEUE;
>>
>> memcpy(cinfo->message_lockres->lksb.sb_lvbptr, (void *)cmsg,
>> sizeof(struct cluster_msg));
>> -- 2.1.0
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
--
Goldwyn
^ permalink raw reply
* [PATCH] md: convert to kstrto*()
From: Alexey Dobriyan @ 2015-05-16 11:02 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
Convert away from deprecated simple_strto*() functions.
Add "fit into sector_t" checks.
Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
---
drivers/md/md.c | 149 ++++++++++++++++++++++++++++++--------------------------
1 file changed, 81 insertions(+), 68 deletions(-)
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2630,13 +2630,14 @@ errors_show(struct md_rdev *rdev, char *page)
static ssize_t
errors_store(struct md_rdev *rdev, const char *buf, size_t len)
{
- char *e;
- unsigned long n = simple_strtoul(buf, &e, 10);
- if (*buf && (*e == 0 || *e == '\n')) {
- atomic_set(&rdev->corrected_errors, n);
- return len;
- }
- return -EINVAL;
+ unsigned int n;
+ int rv;
+
+ rv = kstrtouint(buf, 10, &n);
+ if (rv < 0)
+ return rv;
+ atomic_set(&rdev->corrected_errors, n);
+ return len;
}
static struct rdev_sysfs_entry rdev_errors =
__ATTR(errors, S_IRUGO|S_IWUSR, errors_show, errors_store);
@@ -2653,13 +2654,16 @@ slot_show(struct md_rdev *rdev, char *page)
static ssize_t
slot_store(struct md_rdev *rdev, const char *buf, size_t len)
{
- char *e;
+ int slot;
int err;
- int slot = simple_strtoul(buf, &e, 10);
+
if (strncmp(buf, "none", 4)==0)
slot = -1;
- else if (e==buf || (*e && *e!= '\n'))
- return -EINVAL;
+ else {
+ err = kstrtouint(buf, 10, (unsigned int *)&slot);
+ if (err < 0)
+ return err;
+ }
if (rdev->mddev->pers && slot == -1) {
/* Setting 'slot' on an active array requires also
* updating the 'rd%d' link, and communicating
@@ -3544,12 +3548,12 @@ layout_show(struct mddev *mddev, char *page)
static ssize_t
layout_store(struct mddev *mddev, const char *buf, size_t len)
{
- char *e;
- unsigned long n = simple_strtoul(buf, &e, 10);
+ unsigned int n;
int err;
- if (!*buf || (*e && *e != '\n'))
- return -EINVAL;
+ err = kstrtouint(buf, 10, &n);
+ if (err < 0)
+ return err;
err = mddev_lock(mddev);
if (err)
return err;
@@ -3593,12 +3597,12 @@ static int update_raid_disks(struct mddev *mddev, int raid_disks);
static ssize_t
raid_disks_store(struct mddev *mddev, const char *buf, size_t len)
{
- char *e;
+ unsigned int n;
int err;
- unsigned long n = simple_strtoul(buf, &e, 10);
- if (!*buf || (*e && *e != '\n'))
- return -EINVAL;
+ err = kstrtouint(buf, 10, &n);
+ if (err < 0)
+ return err;
err = mddev_lock(mddev);
if (err)
@@ -3645,12 +3649,12 @@ chunk_size_show(struct mddev *mddev, char *page)
static ssize_t
chunk_size_store(struct mddev *mddev, const char *buf, size_t len)
{
+ unsigned long n;
int err;
- char *e;
- unsigned long n = simple_strtoul(buf, &e, 10);
- if (!*buf || (*e && *e != '\n'))
- return -EINVAL;
+ err = kstrtoul(buf, 10, &n);
+ if (err < 0)
+ return err;
err = mddev_lock(mddev);
if (err)
@@ -3688,19 +3692,24 @@ resync_start_show(struct mddev *mddev, char *page)
static ssize_t
resync_start_store(struct mddev *mddev, const char *buf, size_t len)
{
+ unsigned long long n;
int err;
- char *e;
- unsigned long long n = simple_strtoull(buf, &e, 10);
+
+ if (cmd_match(buf, "none"))
+ n = MaxSector;
+ else {
+ err = kstrtoull(buf, 10, &n);
+ if (err < 0)
+ return err;
+ if (n != (sector_t)n)
+ return -EINVAL;
+ }
err = mddev_lock(mddev);
if (err)
return err;
if (mddev->pers && !test_bit(MD_RECOVERY_FROZEN, &mddev->recovery))
err = -EBUSY;
- else if (cmd_match(buf, "none"))
- n = MaxSector;
- else if (!*buf || (*e && *e != '\n'))
- err = -EINVAL;
if (!err) {
mddev->recovery_cp = n;
@@ -3936,14 +3945,14 @@ max_corrected_read_errors_show(struct mddev *mddev, char *page) {
static ssize_t
max_corrected_read_errors_store(struct mddev *mddev, const char *buf, size_t len)
{
- char *e;
- unsigned long n = simple_strtoul(buf, &e, 10);
+ unsigned int n;
+ int rv;
- if (*buf && (*e == 0 || *e == '\n')) {
- atomic_set(&mddev->max_corr_read_errors, n);
- return len;
- }
- return -EINVAL;
+ rv = kstrtouint(buf, 10, &n);
+ if (rv < 0)
+ return rv;
+ atomic_set(&mddev->max_corr_read_errors, n);
+ return len;
}
static struct md_sysfs_entry max_corr_read_errors =
@@ -4297,15 +4306,18 @@ sync_min_show(struct mddev *mddev, char *page)
static ssize_t
sync_min_store(struct mddev *mddev, const char *buf, size_t len)
{
- int min;
- char *e;
+ unsigned int min;
+ int rv;
+
if (strncmp(buf, "system", 6)==0) {
- mddev->sync_speed_min = 0;
- return len;
+ min = 0;
+ } else {
+ rv = kstrtouint(buf, 10, &min);
+ if (rv < 0)
+ return rv;
+ if (min == 0)
+ return -EINVAL;
}
- min = simple_strtoul(buf, &e, 10);
- if (buf == e || (*e && *e != '\n') || min <= 0)
- return -EINVAL;
mddev->sync_speed_min = min;
return len;
}
@@ -4323,15 +4335,18 @@ sync_max_show(struct mddev *mddev, char *page)
static ssize_t
sync_max_store(struct mddev *mddev, const char *buf, size_t len)
{
- int max;
- char *e;
+ unsigned int max;
+ int rv;
+
if (strncmp(buf, "system", 6)==0) {
- mddev->sync_speed_max = 0;
- return len;
+ max = 0;
+ } else {
+ rv = kstrtouint(buf, 10, &max);
+ if (rv < 0)
+ return rv;
+ if (max == 0)
+ return -EINVAL;
}
- max = simple_strtoul(buf, &e, 10);
- if (buf == e || (*e && *e != '\n') || max <= 0)
- return -EINVAL;
mddev->sync_speed_max = max;
return len;
}
@@ -4514,12 +4529,13 @@ suspend_lo_show(struct mddev *mddev, char *page)
static ssize_t
suspend_lo_store(struct mddev *mddev, const char *buf, size_t len)
{
- char *e;
- unsigned long long new = simple_strtoull(buf, &e, 10);
- unsigned long long old;
+ unsigned long long old, new;
int err;
- if (buf == e || (*e && *e != '\n'))
+ err = kstrtoull(buf, 10, &new);
+ if (err < 0)
+ return err;
+ if (new != (sector_t)new)
return -EINVAL;
err = mddev_lock(mddev);
@@ -4556,12 +4572,13 @@ suspend_hi_show(struct mddev *mddev, char *page)
static ssize_t
suspend_hi_store(struct mddev *mddev, const char *buf, size_t len)
{
- char *e;
- unsigned long long new = simple_strtoull(buf, &e, 10);
- unsigned long long old;
+ unsigned long long old, new;
int err;
- if (buf == e || (*e && *e != '\n'))
+ err = kstrtoull(buf, 10, &new);
+ if (err < 0)
+ return err;
+ if (new != (sector_t)new)
return -EINVAL;
err = mddev_lock(mddev);
@@ -4603,11 +4620,13 @@ static ssize_t
reshape_position_store(struct mddev *mddev, const char *buf, size_t len)
{
struct md_rdev *rdev;
- char *e;
+ unsigned long long new;
int err;
- unsigned long long new = simple_strtoull(buf, &e, 10);
- if (buf == e || (*e && *e != '\n'))
+ err = kstrtoull(buf, 10, &new);
+ if (err < 0)
+ return err;
+ if (new != (sector_t)new)
return -EINVAL;
err = mddev_lock(mddev);
if (err)
@@ -9009,13 +9028,7 @@ static int get_ro(char *buffer, struct kernel_param *kp)
}
static int set_ro(const char *val, struct kernel_param *kp)
{
- char *e;
- int num = simple_strtoul(val, &e, 10);
- if (*val && (*e == '\0' || *e == '\n')) {
- start_readonly = num;
- return 0;
- }
- return -EINVAL;
+ return kstrtouint(val, 10, (unsigned int *)&start_readonly);
}
module_param_call(start_ro, set_ro, get_ro, NULL, S_IRUSR|S_IWUSR);
^ permalink raw reply
* Re: PROBLEM: write to jbod with 3TB and 160GB drives hits BUG/oops
From: Charles Bertsch @ 2015-05-16 3:46 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, BertschC@acm.org
In-Reply-To: <LpCG1q00n1hjKLY01pCHNM>
[-- Attachment #1: Type: text/plain, Size: 2622 bytes --]
On 04/26/2015 06:11 PM, NeilBrown wrote:
> Thanks for the details.
> On the whole, I don't think it is likely that your problem is directly
> related to md - just a coincidence that it happened when you were using md
> things. But one never knows until that actual cause is found.
>
>> >
>> > > Is there any chance you could use "git bisect" to find out exactly which
>> > > commit introduced the problem? That is the mostly likely path to a
>> >solution.
>> > >
>> >
Neil,
I went through the steps, and at the last git bisect got this report --
a72132c31d580969a38972aaf925915e861cd342 is the first bad commit
commit a72132c31d580969a38972aaf925915e861cd342
Author: Matthew Wilcox <matthew.r.wilcox@intel.com>
Date: Wed Jun 4 16:07:49 2014 -0700
brd: add support for rw_page()
Signed-off-by: Matthew Wilcox <matthew.r.wilcox@intel.com>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Dheeraj Reddy <dheeraj.reddy@intel.com>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
:040000 040000 181578588b2fd2a8d59f56f9bcb945169db9bbea
f294ca33674881ab6bc0c43ac24b43a251204793 M drivers
Attached are the stack-traces from the three instances built with this
step. After begin of the command -- dd bs=16384 if=/dev/zero
of=/dev/md0 count=32M -- the failure would occur between 3 and 6
minutes. Other test builds in the sequence hit the failure as quickly
as a few seconds and as long as 10 minutes. Two of these instances had
a little script monirw running, checking /proc/diskstats for reads,
writes, and in-progress values. In one, monirw is not running, to show
that its presence is independent of the problem.
The linear raid-group was built using linux-4.0.0 using two of the five
attached drives, and used throughout, and worked fine with lin-3.3.5,
3.10, 3.14, and 3.15 in tests up to 16 and 17 hours. Those drives were
used as part of a test script that created striped (5 drives), linear (1
drive), linear (2 drives), mirror (2 drives), raid-5 (5 drives), raid-6
(5 drives), raid-10-2X (5 drives), and raid-10-3X (5 drives). The
linear raid-groups were created with ext3 but otherwise unused. All of
the others had ext3 file-systems with 20GB of data written, and multiple
operations such as file-system check performed. Only the linear-2-drive
test ran into this problem.
I still have not figured out what part of this system would be writing
to an ext2 file-system.
What else can I do to help with this problem ?
Thanks
Charles Bertsch
[-- Attachment #2: linprob.stktrace.1.txt --]
[-- Type: text/plain, Size: 9303 bytes --]
#
#
# cat /proc/version
Linux version 3.15.0-rc8+ (cbertsch@haxe) (gcc version 4.8.2 (GCC) ) #1 SMP Thu May 14 17:42:51 MST 2015
#
# cat /etc/zerv*
1.1.10x 2015-May-14 17:49
#
# cat /proc/partitions
major minor #blocks name
3 0 156290904 hda
8 0 312571224 sda
8 16 126976 sdb
8 17 126852 sdb1
8 32 2930266584 sdc
8 33 2930265591 sdc1
8 48 156290904 sdd
8 49 156289927 sdd1
8 64 2930266584 sde
9 0 3086293374 md0
#
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 34816 34816 0 100% /
devtmpfs 1002784 0 1002784 0% /dev
/dev/ram1 7745 343 7402 5% /hd
tmpfs 1021396 0 1021396 0% /dev/shm
/dev/sdb1 122708 106436 16272 87% /flash
#
# cat /proc/mdstat
Personalities : [linear]
md0 : active linear sdc1[0] sdd1[1]
3086293374 blocks super 1.2 0k rounding
unused devices: <none>
#
#
# dd bs=16384 if=/dev/zero of=/dev/md0 count=32M &
#
# monirw 10 md0 sdc sdd
/sbin/monirw sec=10 file=/proc/diskstats itemset=md0 sdc sdd
md0,33,148473,0,sdc,109,147532,147,sdd,74,0,0,1431651487,2015-05-14,17:58:07
md0,0,156467,0,sdc,0,156550,134,sdd,0,0,0,1431651497,2015-05-14,17:58:17
md0,0,154009,0,sdc,0,153990,137,sdd,0,0,0,1431651507,2015-05-14,17:58:27
md0,0,155219,0,sdc,0,155270,129,sdd,0,0,0,1431651518,2015-05-14,17:58:38
md0,0,156672,0,sdc,0,156614,138,sdd,0,0,0,1431651528,2015-05-14,17:58:48
md0,0,155852,0,sdc,0,155846,139,sdd,0,0,0,1431651538,2015-05-14,17:58:58
md0,0,153804,0,sdc,0,153817,137,sdd,0,0,0,1431651548,2015-05-14,17:59:08
md0,0,154009,0,sdc,0,154022,135,sdd,0,0,0,1431651558,2015-05-14,17:59:18
md0,0,153395,0,sdc,0,153408,133,sdd,0,0,0,1431651568,2015-05-14,17:59:28
md0,0,154763,0,sdc,0,154777,131,sdd,0,0,0,1431651578,2015-05-14,17:59:38
md0,0,156122,0,sdc,0,156102,134,sdd,0,0,0,1431651588,2015-05-14,17:59:48
md0,0,155648,0,sdc,0,155660,132,sdd,0,0,0,1431651598,2015-05-14,17:59:58
md0,0,153395,0,sdc,0,153369,136,sdd,0,0,0,1431651609,2015-05-14,18:00:09
md0,0,154009,0,sdc,0,154041,131,sdd,0,0,0,1431651619,2015-05-14,18:00:19
md0,0,155033,0,sdc,0,154976,140,sdd,0,0,0,1431651629,2015-05-14,18:00:29
md0,0,153804,0,sdc,0,153824,137,sdd,0,0,0,1431651639,2015-05-14,18:00:39
md0,0,156262,0,sdc,0,156217,144,sdd,0,0,0,1431651649,2015-05-14,18:00:49
md0,0,155648,0,sdc,0,155648,144,sdd,0,0,0,1431651659,2015-05-14,18:00:59
md0,0,155648,0,sdc,0,155641,145,sdd,0,0,0,1431651669,2015-05-14,18:01:09
------------[ cut here ]------------
kernel BUG at /home/cbertsch/NZdev/nz218/othersrc/linux-git/fs/buffer.c:3220!
invalid opcode: 0000 [#1] SMP
Modules linked in: linear md_mod ipv6 nfsd auth_rpcgss oid_registry exportfs lockd sunrpc af_packet sd_mod ide_gd_mod ata_generic pata_amd sata_nv libata i2c_dev amd74xx k8temp ide_pci_generic forcedeth ide_core skge ehci_pci ohci_pci ehci_hcd ohci_hcd i2c_nforce2 rtc_cmos
CPU: 0 PID: 140 Comm: kworker/u8:2 Not tainted 3.15.0-rc8+ #1
Hardware name: BIOSTAR Group N61PB-M2S/N61PB-M2S, BIOS 6.00 PG 02/27/2009
Workqueue: writeback bdi_writeback_workfn (flush-1:1)
task: ffff88007d342150 ti: ffff88007d4a0000 task.ti: ffff88007d4a0000
RIP: 0010:[<ffffffff8110f17d>] [<ffffffff8110f17d>] try_to_free_buffers+0x9d/0xa0
RSP: 0018:ffff88007d4a3998 EFLAGS: 00010246
RAX: 001000000000086c RBX: ffffea0000210480 RCX: ffff88000c7e2028
RDX: 0000000000000004 RSI: 0000000000000001 RDI: ffffea0000210480
RBP: ffff88007d89a5f8 R08: 2400000000000000 R09: a800008412000000
R10: 57ffe97bee210480 R11: 000000000000001a R12: ffffea0000210480
R13: 0000000000001000 R14: ffff88007d4a3bf0 R15: 0000000000000456
FS: 00007f301703c740(0000) GS:ffff88007ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fea47ba5ffc CR3: 000000007d237000 CR4: 00000000000007f0
Stack:
0000000000000000 0000000000000000 ffff88007a003380 ffffea0000210480
ffffffff81118767 000000048103b333 0000000000000004 0000000000000004
0000000000000000 ffff88007d4a3cc0 ffff88007d89a5f8 0000000000000001
Call Trace:
[<ffffffff81118767>] ? __mpage_writepage+0x4d7/0x5c0
[<ffffffff810cf756>] ? page_mkclean_one+0x66/0x80
[<ffffffff810d0c12>] ? rmap_walk+0x272/0x2a0
[<ffffffff810d0da5>] ? page_mkclean+0x65/0x80
[<ffffffff810cf6f0>] ? page_referenced_one+0xb0/0xb0
[<ffffffff810afee3>] ? write_cache_pages+0x1c3/0x380
[<ffffffff81118290>] ? clean_buffers+0x60/0x60
[<ffffffff81069f58>] ? __dequeue_entity+0x28/0x40
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff811181ec>] ? mpage_writepages+0x5c/0xa0
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff81107778>] ? __writeback_single_inode+0x28/0x100
[<ffffffff811081bb>] ? writeback_sb_inodes+0x19b/0x390
[<ffffffff8110843e>] ? __writeback_inodes_wb+0x8e/0xc0
[<ffffffff8110863b>] ? wb_writeback+0x1cb/0x1e0
[<ffffffff81108bf7>] ? bdi_writeback_workfn+0x157/0x340
[<ffffffff810558f2>] ? process_one_work+0x132/0x380
[<ffffffff81056536>] ? worker_thread+0x116/0x3a0
[<ffffffff81056420>] ? manage_workers.isra.24+0x290/0x290
[<ffffffff8105c251>] ? kthread+0xc1/0xe0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
[<ffffffff813b93fc>] ? ret_from_fork+0x7c/0xb0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
Code: 44 89 e0 48 83 c4 08 5b 5d 41 5c c3 66 90 be 00 10 00 00 48 89 df e8 33 40 fa ff eb bc 90 48 89 e6 e8 48 ec ff ff 41 89 c4 eb b6 <0f> 0b 90 41 57 41 56 41 89 d6 41 55 41 54 49 89 f4 55 48 89 fd
RIP [<ffffffff8110f17d>] try_to_free_buffers+0x9d/0xa0
RSP <ffff88007d4a3998>
---[ end trace a1fddf154ab6e51e ]---
May 14 18:01:17 ZigZag kernel: kernel BUG at /home/cbertsch/NZdev/nz218/othersrcBUG: unable to handle kernel paging request at ffffffffffffffd8
IP: [<ffffffff8105c737>] kthread_data+0x7/0x10
PGD 1611067 PUD 1613067 PMD 0
Oops: 0000 [#2] SMP
Modules linked in: linear md_mod ipv6 nfsd auth_rpcgss oid_registry exportfs lockd sunrpc af_packet sd_mod ide_gd_mod ata_generic pata_amd sata_nv libata i2c_dev amd74xx k8temp ide_pci_generic forcedeth ide_core skge ehci_pci ohci_pci ehci_hcd ohci_hcd i2c_nforce2 rtc_cmos
CPU: 0 PID: 140 Comm: kworker/u8:2 Tainted: G D 3.15.0-rc8+ #1
Hardware name: BIOSTAR Group N61PB-M2S/N61PB-M2S, BIOS 6.00 PG 02/27/2009
task: ffff88007d342150 ti: ffff88007d4a0000 task.ti: ffff88007d4a0000
RIP: 0010:[<ffffffff8105c737>] [<ffffffff8105c737>] kthread_data+0x7/0x10
RSP: 0018:ffff88007d4a36b0 EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000f067272f1
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88007d342150
RBP: ffff88007d4a37b8 R08: ffff88007d349950 R09: 0000000000000001
R10: 000000000000bd41 R11: 0000000000000000 R12: ffff88007d3428f0
R13: 0000000000000000 R14: 0000000000000000 R15: ffff88007d342150
FS: 00007f301703c740(0000) GS:ffff88007ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000028 CR3: 000000007d237000 CR4: 00000000000007f0
Stack:
ffffffff810569a8 ffff88007ca11740 ffffffff813b5ed3 ffff88007d342150
0000000000011740 ffff88007d4a3fd8 0000000000011740 000000000000008c
ffff8800775a31c0 ffff88007a192b58 ffffffff810facf0 ffff88007d4a373b
Call Trace:
[<ffffffff810569a8>] ? wq_worker_sleeping+0x8/0x80
[<ffffffff813b5ed3>] ? __schedule+0x3a3/0x6f0
[<ffffffff810facf0>] ? d_lookup+0x20/0x40
[<ffffffff81041b17>] ? do_exit+0x677/0x990
[<ffffffff81005a56>] ? oops_end+0x66/0x90
[<ffffffff81002cd1>] ? do_invalid_op+0x81/0xa0
[<ffffffff8110f17d>] ? try_to_free_buffers+0x9d/0xa0
[<ffffffff811f75fb>] ? radix_tree_insert+0x2b/0xe0
[<ffffffff813ba4a8>] ? invalid_op+0x18/0x20
[<ffffffff8110f17d>] ? try_to_free_buffers+0x9d/0xa0
[<ffffffff81118767>] ? __mpage_writepage+0x4d7/0x5c0
[<ffffffff810cf756>] ? page_mkclean_one+0x66/0x80
[<ffffffff810d0c12>] ? rmap_walk+0x272/0x2a0
[<ffffffff810d0da5>] ? page_mkclean+0x65/0x80
[<ffffffff810cf6f0>] ? page_referenced_one+0xb0/0xb0
[<ffffffff810afee3>] ? write_cache_pages+0x1c3/0x380
[<ffffffff81118290>] ? clean_buffers+0x60/0x60
[<ffffffff81069f58>] ? __dequeue_entity+0x28/0x40
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff811181ec>] ? mpage_writepages+0x5c/0xa0
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff81107778>] ? __writeback_single_inode+0x28/0x100
[<ffffffff811081bb>] ? writeback_sb_inodes+0x19b/0x390
[<ffffffff8110843e>] ? __writeback_inodes_wb+0x8e/0xc0
[<ffffffff8110863b>] ? wb_writeback+0x1cb/0x1e0
[<ffffffff81108bf7>] ? bdi_writeback_workfn+0x157/0x340
[<ffffffff810558f2>] ? process_one_work+0x132/0x380
[<ffffffff81056536>] ? worker_thread+0x116/0x3a0
[<ffffffff81056420>] ? manage_workers.isra.24+0x290/0x290
[<ffffffff8105c251>] ? kthread+0xc1/0xe0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
[<ffffffff813b93fc>] ? ret_from_fork+0x7c/0xb0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
Code: 00 00 00 00 65 48 8b 04 25 80 b8 00 00 48 8b 80 48 07 00 00 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 0f 1f 40 00 48 8b 87 48 07 00 00 <48> 8b 40 d8 c3 0f 1f 40 00 48 83 ec 08 48 8b b7 48 07 00 00 ba
RIP [<ffffffff8105c737>] kthread_data+0x7/0x10
RSP <ffff88007d4a36b0>
CR2: ffffffffffffffd8
---[ end trace a1fddf154ab6e51f ]---
Fixing recursive fault but reboot is needed!
[-- Attachment #3: linprob.stktrace.2.txt --]
[-- Type: text/plain, Size: 7849 bytes --]
#
# date
Thu May 14 18:05:44 MST 2015
#
# cat /proc/version
Linux version 3.15.0-rc8+ (cbertsch@haxe) (gcc version 4.8.2 (GCC) ) #1 SMP Thu May 14 17:42:51 MST 2015
#
# cat /etc/zerv*
1.1.10x 2015-May-14 17:49
#
# cat /proc/partitions
major minor #blocks name
3 0 156290904 hda
8 0 312571224 sda
8 16 126976 sdb
8 17 126852 sdb1
8 32 2930266584 sdc
8 33 2930265591 sdc1
8 48 156290904 sdd
8 49 156289927 sdd1
8 64 2930266584 sde
9 0 3086293374 md0
#
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 34816 34816 0 100% /
devtmpfs 1002784 0 1002784 0% /dev
/dev/ram1 7745 343 7402 5% /hd
tmpfs 1021396 0 1021396 0% /dev/shm
/dev/sdb1 122708 106436 16272 87% /flash
#
# cat /proc/mdstat
Personalities : [linear]
md0 : active linear sdc1[0] sdd1[1]
3086293374 blocks super 1.2 0k rounding
unused devices: <none>
#
#
# dd bs=16384 if=/dev/zero of=/dev/md0 count=32M &
#
# ------------[ cut here ]------------
kernel BUG at /home/cbertsch/NZdev/nz218/othersrc/linux-git/fs/buffer.c:3220!
invalid opcode: 0000 [#1] SMP
Modules linked in: linear md_mod ipv6 nfsd auth_rpcgss oid_registry exportfs lockd sunrpc af_packet sd_mod ide_gd_mod ata_generic pata_amd sata_nv libata i2c_dev amd74xx k8temp ide_pci_generic forcedeth ide_core skge ehci_pci ohci_pci ehci_hcd ohci_hcd i2c_nforce2 rtc_cmos
CPU: 0 PID: 111 Comm: kworker/u8:2 Not tainted 3.15.0-rc8+ #1
Hardware name: BIOSTAR Group N61PB-M2S/N61PB-M2S, BIOS 6.00 PG 02/27/2009
Workqueue: writeback bdi_writeback_workfn (flush-1:1)
task: ffff8800775a5270 ti: ffff88007d160000 task.ti: ffff88007d160000
RIP: 0010:[<ffffffff8110f17d>] [<ffffffff8110f17d>] try_to_free_buffers+0x9d/0xa0
RSP: 0018:ffff88007d163998 EFLAGS: 00010246
RAX: 001000000000086c RBX: ffffea000005dc68 RCX: ffff88004b38bdf8
RDX: 0000000000000004 RSI: 0000000000000001 RDI: ffffea000005dc68
RBP: ffff88007d4afbd8 R08: e340000000000000 R09: a800001771a00000
R10: 57ffe9e88e65dc68 R11: 000000000000001a R12: ffffea000005dc68
R13: 0000000000001000 R14: ffff88007d163bf0 R15: 0000000000000c38
FS: 00007fbebd993740(0000) GS:ffff88007ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fa7d2b9effc CR3: 00000000774c1000 CR4: 00000000000007f0
Stack:
0000000000000000 0000000000000000 ffff88007a003380 ffffea000005dc68
ffffffff81118767 000000048103b333 0000000000000004 0000000000000004
0000000000000000 ffff88007d163cc0 ffff88007d4afbd8 0000000000000001
Call Trace:
[<ffffffff81118767>] ? __mpage_writepage+0x4d7/0x5c0
[<ffffffff810cf756>] ? page_mkclean_one+0x66/0x80
[<ffffffff810d0c12>] ? rmap_walk+0x272/0x2a0
[<ffffffff810d0da5>] ? page_mkclean+0x65/0x80
[<ffffffff810cf6f0>] ? page_referenced_one+0xb0/0xb0
[<ffffffff810afee3>] ? write_cache_pages+0x1c3/0x380
[<ffffffff81118290>] ? clean_buffers+0x60/0x60
[<ffffffff81069f58>] ? __dequeue_entity+0x28/0x40
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff811181ec>] ? mpage_writepages+0x5c/0xa0
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff81107778>] ? __writeback_single_inode+0x28/0x100
[<ffffffff811081bb>] ? writeback_sb_inodes+0x19b/0x390
[<ffffffff8110843e>] ? __writeback_inodes_wb+0x8e/0xc0
[<ffffffff8110863b>] ? wb_writeback+0x1cb/0x1e0
[<ffffffff81108bf7>] ? bdi_writeback_workfn+0x157/0x340
[<ffffffff810558f2>] ? process_one_work+0x132/0x380
[<ffffffff81056536>] ? worker_thread+0x116/0x3a0
[<ffffffff81056420>] ? manage_workers.isra.24+0x290/0x290
[<ffffffff8105c251>] ? kthread+0xc1/0xe0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
[<ffffffff813b93fc>] ? ret_from_fork+0x7c/0xb0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
Code: 44 89 e0 48 83 c4 08 5b 5d 41 5c c3 66 90 be 00 10 00 00 48 89 df e8 33 40 fa ff eb bc 90 48 89 e6 e8 48 ec ff ff 41 89 c4 eb b6 <0f> 0b 90 41 57 41 56 41 89 d6 41 55 41 54 49 89 f4 55 48 89 fd
RIP [<ffffffff8110f17d>] try_to_free_buffers+0x9d/0xa0
RSP <ffff88007d163998>
---[ end trace f925040d8cafe7bb ]---
May 14 18:13:23 ZigZag kernel: kernel BUG at /home/cbertsch/NZdeBUG: unable to handle kernel paging request at ffffffffffffffd8
IP: [<ffffffff8105c737>] kthread_data+0x7/0x10
PGD 1611067 PUD 1613067 PMD 0
Oops: 0000 [#2] SMP
Modules linked in: linear md_mod ipv6 nfsd auth_rpcgss oid_registry exportfs lockd sunrpc af_packet sd_mod ide_gd_mod ata_generic pata_amd sata_nv libata i2c_dev amd74xx k8temp ide_pci_generic forcedeth ide_core skge ehci_pci ohci_pci ehci_hcd ohci_hcd i2c_nforce2 rtc_cmos
CPU: 0 PID: 111 Comm: kworker/u8:2 Tainted: G D 3.15.0-rc8+ #1
Hardware name: BIOSTAR Group N61PB-M2S/N61PB-M2S, BIOS 6.00 PG 02/27/2009
task: ffff8800775a5270 ti: ffff88007d160000 task.ti: ffff88007d160000
RIP: 0010:[<ffffffff8105c737>] [<ffffffff8105c737>] kthread_data+0x7/0x10
RSP: 0018:ffff88007d1636b0 EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000001b3c047296
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8800775a5270
RBP: ffff88007d1637b8 R08: ffff8800775a7590 R09: 0000000000000001
R10: 000000000000bc4e R11: 0000000000000000 R12: ffff8800775a5a10
R13: 0000000000000000 R14: 0000000000000000 R15: ffff8800775a5270
FS: 00007fbebd993740(0000) GS:ffff88007ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000028 CR3: 00000000774c1000 CR4: 00000000000007f0
Stack:
ffffffff810569a8 ffff88007ca11740 ffffffff813b5ed3 ffff8800775a5270
0000000000011740 ffff88007d163fd8 0000000000011740 000000000000006f
ffff8800793701c0 ffff88007a35c918 ffffffff810facf0 ffff88007d16373b
Call Trace:
[<ffffffff810569a8>] ? wq_worker_sleeping+0x8/0x80
[<ffffffff813b5ed3>] ? __schedule+0x3a3/0x6f0
[<ffffffff810facf0>] ? d_lookup+0x20/0x40
[<ffffffff81041b17>] ? do_exit+0x677/0x990
[<ffffffff81005a56>] ? oops_end+0x66/0x90
[<ffffffff81002cd1>] ? do_invalid_op+0x81/0xa0
[<ffffffff8110f17d>] ? try_to_free_buffers+0x9d/0xa0
[<ffffffff811f75fb>] ? radix_tree_insert+0x2b/0xe0
[<ffffffff813ba4a8>] ? invalid_op+0x18/0x20
[<ffffffff8110f17d>] ? try_to_free_buffers+0x9d/0xa0
[<ffffffff81118767>] ? __mpage_writepage+0x4d7/0x5c0
[<ffffffff810cf756>] ? page_mkclean_one+0x66/0x80
[<ffffffff810d0c12>] ? rmap_walk+0x272/0x2a0
[<ffffffff810d0da5>] ? page_mkclean+0x65/0x80
[<ffffffff810cf6f0>] ? page_referenced_one+0xb0/0xb0
[<ffffffff810afee3>] ? write_cache_pages+0x1c3/0x380
[<ffffffff81118290>] ? clean_buffers+0x60/0x60
[<ffffffff81069f58>] ? __dequeue_entity+0x28/0x40
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff811181ec>] ? mpage_writepages+0x5c/0xa0
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff81107778>] ? __writeback_single_inode+0x28/0x100
[<ffffffff811081bb>] ? writeback_sb_inodes+0x19b/0x390
[<ffffffff8110843e>] ? __writeback_inodes_wb+0x8e/0xc0
[<ffffffff8110863b>] ? wb_writeback+0x1cb/0x1e0
[<ffffffff81108bf7>] ? bdi_writeback_workfn+0x157/0x340
[<ffffffff810558f2>] ? process_one_work+0x132/0x380
[<ffffffff81056536>] ? worker_thread+0x116/0x3a0
[<ffffffff81056420>] ? manage_workers.isra.24+0x290/0x290
[<ffffffff8105c251>] ? kthread+0xc1/0xe0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
[<ffffffff813b93fc>] ? ret_from_fork+0x7c/0xb0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
Code: 00 00 00 00 65 48 8b 04 25 80 b8 00 00 48 8b 80 48 07 00 00 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 0f 1f 40 00 48 8b 87 48 07 00 00 <48> 8b 40 d8 c3 0f 1f 40 00 48 83 ec 08 48 8b b7 48 07 00 00 ba
RIP [<ffffffff8105c737>] kthread_data+0x7/0x10
RSP <ffff88007d1636b0>
CR2: ffffffffffffffd8
---[ end trace f925040d8cafe7bc ]---
Fixing recursive fault but reboot is needed!
[-- Attachment #4: linprob.stktrace.3.txt --]
[-- Type: text/plain, Size: 10768 bytes --]
#
# date
Thu May 14 18:17:52 MST 2015
#
# cat /proc/version
Linux version 3.15.0-rc8+ (cbertsch@haxe) (gcc version 4.8.2 (GCC) ) #1 SMP Thu May 14 17:42:51 MST 2015
#
# cat /etc/zerv*
1.1.10x 2015-May-14 17:49
#
# cat /proc/partitions
major minor #blocks name
3 0 156290904 hda
8 0 312571224 sda
8 16 126976 sdb
8 17 126852 sdb1
8 32 2930266584 sdc
8 33 2930265591 sdc1
8 48 156290904 sdd
8 49 156289927 sdd1
8 64 2930266584 sde
9 0 3086293374 md0
#
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 34816 34816 0 100% /
devtmpfs 1002784 0 1002784 0% /dev
/dev/ram1 7745 343 7402 5% /hd
tmpfs 1021396 0 1021396 0% /dev/shm
/dev/sdb1 122708 106436 16272 87% /flash
#
# cat /proc/mdstat
Personalities : [linear]
md0 : active linear sdc1[0] sdd1[1]
3086293374 blocks super 1.2 0k rounding
unused devices: <none>
#
#
# dd bs=16384 if=/dev/zero of=/dev/md0 count=32M &
#
# monirw 10 md0 sdc sdd
/sbin/monirw sec=10 file=/proc/diskstats itemset=md0 sdc sdd
md0,33,143353,0,sdc,109,142444,142,sdd,74,0,0,1431652769,2015-05-14,18:19:29
md0,0,154828,0,sdc,0,154860,137,sdd,0,0,0,1431652779,2015-05-14,18:19:39
md0,0,155033,0,sdc,0,155084,129,sdd,0,0,0,1431652789,2015-05-14,18:19:49
md0,0,154624,0,sdc,0,154534,143,sdd,0,0,0,1431652799,2015-05-14,18:19:59
md0,0,156876,0,sdc,0,156812,153,sdd,0,0,0,1431652809,2015-05-14,18:20:09
md0,0,155443,0,sdc,0,155577,132,sdd,0,0,0,1431652819,2015-05-14,18:20:19
md0,0,155238,0,sdc,0,155244,131,sdd,0,0,0,1431652830,2015-05-14,18:20:30
md0,0,153804,0,sdc,0,153728,143,sdd,0,0,0,1431652840,2015-05-14,18:20:40
md0,0,154624,0,sdc,0,154720,128,sdd,0,0,0,1431652850,2015-05-14,18:20:50
md0,0,154624,0,sdc,0,154489,149,sdd,0,0,0,1431652860,2015-05-14,18:21:00
md0,0,156262,0,sdc,0,156403,127,sdd,0,0,0,1431652870,2015-05-14,18:21:10
md0,0,156262,0,sdc,0,156230,132,sdd,0,0,0,1431652880,2015-05-14,18:21:20
md0,0,153395,0,sdc,0,153356,138,sdd,0,0,0,1431652890,2015-05-14,18:21:30
md0,0,153804,0,sdc,0,153817,136,sdd,0,0,0,1431652900,2015-05-14,18:21:40
md0,0,155852,0,sdc,0,155750,152,sdd,0,0,0,1431652910,2015-05-14,18:21:50
md0,0,154419,0,sdc,0,154451,147,sdd,0,0,0,1431652920,2015-05-14,18:22:00
md0,0,156262,0,sdc,0,156236,151,sdd,0,0,0,1431652931,2015-05-14,18:22:11
md0,0,156262,0,sdc,0,156352,137,sdd,0,0,0,1431652941,2015-05-14,18:22:21
md0,0,156262,0,sdc,0,156243,140,sdd,0,0,0,1431652951,2015-05-14,18:22:31
md0,0,152166,0,sdc,0,152185,137,sdd,0,0,0,1431652961,2015-05-14,18:22:41
md0,0,154828,0,sdc,0,154873,130,sdd,0,0,0,1431652971,2015-05-14,18:22:51
md0,0,154214,0,sdc,0,154150,140,sdd,0,0,0,1431652981,2015-05-14,18:23:01
md0,0,155033,0,sdc,0,155084,132,sdd,0,0,0,1431652991,2015-05-14,18:23:11
md0,0,155852,0,sdc,0,155859,131,sdd,0,0,0,1431653001,2015-05-14,18:23:21
md0,0,156057,0,sdc,0,156038,134,sdd,0,0,0,1431653011,2015-05-14,18:23:31
md0,0,154828,0,sdc,0,154771,143,sdd,0,0,0,1431653021,2015-05-14,18:23:41
md0,0,152371,0,sdc,0,152384,141,sdd,0,0,0,1431653032,2015-05-14,18:23:52
md0,0,155648,0,sdc,0,155660,139,sdd,0,0,0,1431653042,2015-05-14,18:24:02
md0,0,153600,0,sdc,0,153568,144,sdd,0,0,0,1431653052,2015-05-14,18:24:12
md0,0,155443,0,sdc,0,155462,141,sdd,0,0,0,1431653062,2015-05-14,18:24:22
md0,0,155648,0,sdc,0,155712,131,sdd,0,0,0,1431653072,2015-05-14,18:24:32
md0,0,155238,0,sdc,0,155251,129,sdd,0,0,0,1431653082,2015-05-14,18:24:42
md0,0,154009,0,sdc,0,153932,141,sdd,0,0,0,1431653092,2015-05-14,18:24:52
md0,0,153395,0,sdc,0,153376,144,sdd,0,0,0,1431653102,2015-05-14,18:25:02
md0,0,155648,0,sdc,0,155724,132,sdd,0,0,0,1431653112,2015-05-14,18:25:12
md0,0,154009,0,sdc,0,153971,138,sdd,0,0,0,1431653122,2015-05-14,18:25:22
------------[ cut here ]------------
kernel BUG at /home/cbertsch/NZdev/nz218/othersrc/linux-git/fs/buffer.c:3220!
invalid opcode: 0000 [#1] SMP
Modules linked in: linear md_mod ipv6 nfsd auth_rpcgss oid_registry exportfs lockd sunrpc af_packet sd_mod ide_gd_mod ata_generic pata_amd sata_nv libata i2c_dev amd74xx k8temp ide_pci_generic forcedeth ide_core skge ehci_pci ohci_pci ehci_hcd ohci_hcd i2c_nforce2 rtc_cmos
CPU: 0 PID: 146 Comm: kworker/u8:5 Not tainted 3.15.0-rc8+ #1
Hardware name: BIOSTAR Group N61PB-M2S/N61PB-M2S, BIOS 6.00 PG 02/27/2009
Workqueue: writeback bdi_writeback_workfn (flush-1:1)
task: ffff88007d4a4d70 ti: ffff88007d528000 task.ti: ffff88007d528000
RIP: 0010:[<ffffffff8110f17d>] [<ffffffff8110f17d>] try_to_free_buffers+0x9d/0xa0
RSP: 0018:ffff88007d52b998 EFLAGS: 00010246
RAX: 001000000000086c RBX: ffffea00013645e0 RCX: ffff880056989e60
RDX: 0000000000000004 RSI: 0000000000000001 RDI: ffffea00013645e0
RBP: ffff88007d8b6bd8 R08: 2f00000000000000 R09: a80004d917800000
R10: 57ffe526e9b645e0 R11: 000000000000001a R12: ffffea00013645e0
R13: 0000000000001000 R14: ffff88007d52bbf0 R15: 000000000000045a
FS: 00007ff7ddf96740(0000) GS:ffff88007ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f66423647b0 CR3: 000000007d285000 CR4: 00000000000007f0
Stack:
0000000000000000 0000000000000000 ffff88007a003380 ffffea00013645e0
ffffffff81118767 000000048103b333 0000000000000004 0000000000000004
0000000000000000 ffff88007d52bcc0 ffff88007d8b6bd8 0000000000000001
Call Trace:
[<ffffffff81118767>] ? __mpage_writepage+0x4d7/0x5c0
[<ffffffff810cf756>] ? page_mkclean_one+0x66/0x80
[<ffffffff810d0c12>] ? rmap_walk+0x272/0x2a0
[<ffffffff810d0da5>] ? page_mkclean+0x65/0x80
[<ffffffff810cf6f0>] ? page_referenced_one+0xb0/0xb0
[<ffffffff810afee3>] ? write_cache_pages+0x1c3/0x380
[<ffffffff81118290>] ? clean_buffers+0x60/0x60
[<ffffffff81069f58>] ? __dequeue_entity+0x28/0x40
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff811181ec>] ? mpage_writepages+0x5c/0xa0
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff81107778>] ? __writeback_single_inode+0x28/0x100
[<ffffffff811081bb>] ? writeback_sb_inodes+0x19b/0x390
[<ffffffff8110843e>] ? __writeback_inodes_wb+0x8e/0xc0
[<ffffffff8110863b>] ? wb_writeback+0x1cb/0x1e0
[<ffffffff81108bf7>] ? bdi_writeback_workfn+0x157/0x340
[<ffffffff810558f2>] ? process_one_work+0x132/0x380
[<ffffffff81056536>] ? worker_thread+0x116/0x3a0
[<ffffffff81056420>] ? manage_workers.isra.24+0x290/0x290
[<ffffffff8105c251>] ? kthread+0xc1/0xe0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
[<ffffffff813b93fc>] ? ret_from_fork+0x7c/0xb0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
Code: 44 89 e0 48 83 c4 08 5b 5d 41 5c c3 66 90 be 00 10 00 00 48 89 df e8 33 40 fa ff eb bc 90 48 89 e6 e8 48 ec ff ff 41 89 c4 eb b6 <0f> 0b 90 41 57 41 56 41 89 d6 41 55 41 54 49 89 f4 55 48 89 fd
RIP [<ffffffff8110f17d>] try_to_free_buffers+0x9d/0xa0
RSP <ffff88007d52b998>
---[ end trace 80fdc66ff52e7e58 ]---
May 14 18:25:27 ZigZag kernel: kernel BUG at /home/cbertsch/NZdeBUG: unable to handle kernel paging request at ffffffffffffffd8
IP: [<ffffffff8105c737>] kthread_data+0x7/0x10
PGD 1611067 PUD 1613067 PMD 0
Oops: 0000 [#2] SMP
Modules linked in: linear md_mod ipv6 nfsd auth_rpcgss oid_registry exportfs lockd sunrpc af_packet sd_mod ide_gd_mod ata_generic pata_amd sata_nv libata i2c_dev amd74xx k8temp ide_pci_generic forcedeth ide_core skge ehci_pci ohci_pci ehci_hcd ohci_hcd i2c_nforce2 rtc_cmos
CPU: 0 PID: 146 Comm: kworker/u8:5 Tainted: G D 3.15.0-rc8+ #1
Hardware name: BIOSTAR Group N61PB-M2S/N61PB-M2S, BIOS 6.00 PG 02/27/2009
task: ffff88007d4a4d70 ti: ffff88007d528000 task.ti: ffff88007d528000
RIP: 0010:[<ffffffff8105c737>] [<ffffffff8105c737>] kthread_data+0x7/0x10
RSP: 0018:ffff88007d52b6b0 EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000019f42e6541
RDX: ffffffffffd23940 RSI: 0000000000000000 RDI: ffff88007d4a4d70
RBP: ffff88007d52b7b8 R08: 0000000000000000 R09: 0000000000000001
R10: 000000000000bc0c R11: 0000000000000000 R12: ffff88007d4a5510
R13: 0000000000000000 R14: 0000000000000000 R15: ffff88007d4a4d70
FS: 00007ff7ddf96740(0000) GS:ffff88007ca00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000028 CR3: 000000007d285000 CR4: 00000000000007f0
Stack:
ffffffff810569a8 ffff88007ca11740 ffffffff813b5ed3 ffff88007d4a4d70
0000000000011740 ffff88007d52bfd8 0000000000011740 ffffffff8106470d
ffff88007ca11740 ffffffff8106472d ffff88007d1389d0 0000000000000046
Call Trace:
[<ffffffff810569a8>] ? wq_worker_sleeping+0x8/0x80
[<ffffffff813b5ed3>] ? __schedule+0x3a3/0x6f0
[<ffffffff8106470d>] ? check_preempt_curr+0x7d/0x90
[<ffffffff8106472d>] ? ttwu_do_wakeup+0xd/0x80
[<ffffffff81066740>] ? try_to_wake_up+0x1d0/0x270
[<ffffffff81054e0b>] ? __queue_work+0x10b/0x2a0
[<ffffffff81041b17>] ? do_exit+0x677/0x990
[<ffffffff81005a56>] ? oops_end+0x66/0x90
[<ffffffff81002cd1>] ? do_invalid_op+0x81/0xa0
[<ffffffff8110f17d>] ? try_to_free_buffers+0x9d/0xa0
[<ffffffff813b5d6d>] ? __schedule+0x23d/0x6f0
[<ffffffff811f75fb>] ? radix_tree_insert+0x2b/0xe0
[<ffffffff813ba4a8>] ? invalid_op+0x18/0x20
[<ffffffff8110f17d>] ? try_to_free_buffers+0x9d/0xa0
[<ffffffff81118767>] ? __mpage_writepage+0x4d7/0x5c0
[<ffffffff810cf756>] ? page_mkclean_one+0x66/0x80
[<ffffffff810d0c12>] ? rmap_walk+0x272/0x2a0
[<ffffffff810d0da5>] ? page_mkclean+0x65/0x80
[<ffffffff810cf6f0>] ? page_referenced_one+0xb0/0xb0
[<ffffffff810afee3>] ? write_cache_pages+0x1c3/0x380
[<ffffffff81118290>] ? clean_buffers+0x60/0x60
[<ffffffff81069f58>] ? __dequeue_entity+0x28/0x40
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff811181ec>] ? mpage_writepages+0x5c/0xa0
[<ffffffff8115adf0>] ? ext2_get_blocks+0x970/0x970
[<ffffffff81107778>] ? __writeback_single_inode+0x28/0x100
[<ffffffff811081bb>] ? writeback_sb_inodes+0x19b/0x390
[<ffffffff8110843e>] ? __writeback_inodes_wb+0x8e/0xc0
[<ffffffff8110863b>] ? wb_writeback+0x1cb/0x1e0
[<ffffffff81108bf7>] ? bdi_writeback_workfn+0x157/0x340
[<ffffffff810558f2>] ? process_one_work+0x132/0x380
[<ffffffff81056536>] ? worker_thread+0x116/0x3a0
[<ffffffff81056420>] ? manage_workers.isra.24+0x290/0x290
[<ffffffff8105c251>] ? kthread+0xc1/0xe0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
[<ffffffff813b93fc>] ? ret_from_fork+0x7c/0xb0
[<ffffffff8105c190>] ? kthread_create_on_node+0x170/0x170
Code: 00 00 00 00 65 48 8b 04 25 80 b8 00 00 48 8b 80 48 07 00 00 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 0f 1f 40 00 48 8b 87 48 07 00 00 <48> 8b 40 d8 c3 0f 1f 40 00 48 83 ec 08 48 8b b7 48 07 00 00 ba
RIP [<ffffffff8105c737>] kthread_data+0x7/0x10
RSP <ffff88007d52b6b0>
CR2: ffffffffffffffd8
---[ end trace 80fdc66ff52e7e59 ]---
Fixing recursive fault but reboot is needed!
[-- Attachment #5: Note.bisect.log.txt --]
[-- Type: text/plain, Size: 2696 bytes --]
git bisect start
# good: [1860e379875dfe7271c649058aeddffe5afd9d0d] Linux 3.15
git bisect good 1860e379875dfe7271c649058aeddffe5afd9d0d
# bad: [19583ca584d6f574384e17fe7613dfaeadcdc4a6] Linux 3.16
git bisect bad 19583ca584d6f574384e17fe7613dfaeadcdc4a6
# bad: [7b215de3d0abbc4f6daf2efd19e8809af0564490] Merge branch 'i2c/for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux into next
git bisect bad 7b215de3d0abbc4f6daf2efd19e8809af0564490
# good: [5142c33ed86acbcef5c63a63d2b7384b9210d39f] Merge tag 'staging-3.16-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging into next
git bisect good 5142c33ed86acbcef5c63a63d2b7384b9210d39f
# good: [15b588303155b22edd559672905db8e59a44ef9a] Merge tag 'fbdev-omap-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux into next
git bisect good 15b588303155b22edd559672905db8e59a44ef9a
# bad: [d62cf81524304396276f6aaa5cd7ce62f6f65110] init/main.c: don't use pr_debug()
git bisect bad d62cf81524304396276f6aaa5cd7ce62f6f65110
# good: [a58bdba749b36069ec372da9c9fd16017b6c0b47] Merge branch 'topic/firewire' into for-next
git bisect good a58bdba749b36069ec372da9c9fd16017b6c0b47
# good: [367464362591d89b371e2a690638e9bc899d8ebb] intel-iommu: integrate DMA CMA
git bisect good 367464362591d89b371e2a690638e9bc899d8ebb
# bad: [cfc47a2803db42140167b92d991ef04018e162c7] mm: page_alloc: lookup pageblock migratetype with IRQs enabled during free
git bisect bad cfc47a2803db42140167b92d991ef04018e162c7
# bad: [7c8e0181e6e0b8079c4c2ce902bf52d7a2c6fa5d] mm: replace __get_cpu_var uses with this_cpu_ptr
git bisect bad 7c8e0181e6e0b8079c4c2ce902bf52d7a2c6fa5d
# good: [c96b9e508f3d06ddb601dcc9792d62c044ab359e] mm/compaction: cleanup isolate_freepages()
git bisect good c96b9e508f3d06ddb601dcc9792d62c044ab359e
# good: [11de9927f9dd3cb0a0f18064fa4b6976fc37e79c] mm: numa: add migrated transhuge pages to LRU the same way as base pages
git bisect good 11de9927f9dd3cb0a0f18064fa4b6976fc37e79c
# bad: [a72132c31d580969a38972aaf925915e861cd342] brd: add support for rw_page()
git bisect bad a72132c31d580969a38972aaf925915e861cd342
# good: [90768eee4565adb28ea28b4ac5081c676a8fe1f2] fs/mpage.c: factor clean_buffers() out of __mpage_writepage()
git bisect good 90768eee4565adb28ea28b4ac5081c676a8fe1f2
# good: [47a191fd38ebddb1bd1510ec2bc1085c578c8868] fs/block_dev.c: add bdev_read_page() and bdev_write_page()
git bisect good 47a191fd38ebddb1bd1510ec2bc1085c578c8868
# good: [dd6bd0d9c7dbb3951005db4e424abbd37c4ff801] swap: use bdev_read_page() / bdev_write_page()
git bisect good dd6bd0d9c7dbb3951005db4e424abbd37c4ff801
# first bad commit: [a72132c31d580969a38972aaf925915e861cd342] brd: add support for rw_page()
^ permalink raw reply
* [PATCH v2 5/4] dm-raid: add merge method to target
From: heinzm @ 2015-05-15 14:04 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
Patch series
"[PATCH v2 0/4] dm-raid: Add support for the MD RAID0 personality"
is missing a merge function which can lead to data corruption
on read ahead.
This patch introduces it
It inquires the MD raid0 personalities mergeable_bvec
to make sure, that read ahead payload gets limited properly.
The problem did not occur with the other raid levels,
because it either did not apply without striping or got
avoided via stripe caching.
Signed-of-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/dm-raid.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
index 97e1651..06f9d63 100644
--- a/drivers/md/dm-raid.c
+++ b/drivers/md/dm-raid.c
@@ -1717,6 +1717,24 @@ static void raid_resume(struct dm_target *ti)
mddev_resume(&rs->md);
}
+static int raid_merge(struct dm_target *ti, struct bvec_merge_data *bvm,
+ struct bio_vec *biovec, int max_size)
+{
+ struct raid_set *rs = ti->private;
+ struct md_personality *pers = rs->md.pers;
+
+ if (pers && pers->mergeable_bvec)
+ return min(max_size, pers->mergeable_bvec(&rs->md, bvm, biovec));
+
+ /*
+ * In case we can't request the personality because
+ * the raid set is not running yet
+ *
+ * -> return safe minimum
+ */
+ return rs->md.chunk_sectors;
+}
+
static struct target_type raid_target = {
.name = "raid",
.version = {1, 7, 0},
@@ -1731,6 +1749,7 @@ static struct target_type raid_target = {
.presuspend = raid_presuspend,
.postsuspend = raid_postsuspend,
.resume = raid_resume,
+ .merge = raid_merge
};
static int __init dm_raid_init(void)
--
2.1.0
^ permalink raw reply related
* raid5 reshape is stuck
From: Xiao Ni @ 2015-05-15 7:00 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1612858661.15347659.1431671671467.JavaMail.zimbra@redhat.com>
Hi Neil
I encounter the problem when I reshape a 4-disks raid5 to raid5. It just can
appear with loop devices.
The steps are:
[root@dhcp-12-158 mdadm-3.3.2]# mdadm -CR /dev/md0 -l5 -n5 /dev/loop[0-4] --assume-clean
mdadm: /dev/loop0 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop1 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop2 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop3 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: /dev/loop4 appears to be part of a raid array:
level=raid5 devices=6 ctime=Fri May 15 13:47:17 2015
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@dhcp-12-158 mdadm-3.3.2]# mdadm /dev/md0 -a /dev/loop5
mdadm: added /dev/loop5
[root@dhcp-12-158 mdadm-3.3.2]# mdadm --grow /dev/md0 --raid-devices 6
mdadm: Need to backup 10240K of critical section..
[root@dhcp-12-158 mdadm-3.3.2]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
8187904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
[>....................] reshape = 0.0% (0/2046976) finish=6396.8min speed=0K/sec
unused devices: <none>
It because the sync_max is set to 0 when run the command --grow
[root@dhcp-12-158 mdadm-3.3.2]# cd /sys/block/md0/md/
[root@dhcp-12-158 md]# cat sync_max
0
I tried reproduce with normal sata devices. The progress of reshape is no problem. Then
I checked the Grow.c. If I use sata devices, in function reshape_array, the return value
of set_new_data_offset is 0. But if I used loop devices, it return 1. Then it call the function
start_reshape.
In the function start_reshape it set the sync_max to reshape_progress. But in sysfs_read it
doesn't read reshape_progress. So it's 0 and the sync_max is set to 0. Why it need to set the
sync_max at this? I'm not sure about this.
I tried to fix this but I'm not sure whether it's the right way. I'll send the patches in
other mails.
Best Regards
Xiao
^ permalink raw reply
* Re: [PATCH V2 01/11] Create n bitmaps for clustered mode
From: NeilBrown @ 2015-05-15 5:45 UTC (permalink / raw)
To: gqjiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1431075029-5484-2-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 7004 bytes --]
On Fri, 8 May 2015 16:50:19 +0800 gqjiang@suse.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> For a clustered MD, create bitmaps equal to number of nodes so
> each node has an independent bitmap.
>
> Only the first bitmap is has the bits set so that the first node
> that assembles the device also performs the sync.
>
> The bitmaps are aligned to 4k boundaries.
>
> On-disk format:
>
> 0 4k 8k 12k
> -------------------------------------------------------------------
> | idle | md super | bm super [0] + bits |
> | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
> | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
> | bm bits [3, contd] | | |
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> Create.c | 3 ++-
> bitmap.c | 2 ++
> bitmap.h | 7 +++++--
> mdadm.8.in | 7 ++++++-
> mdadm.c | 9 +++++++++
> super1.c | 59 +++++++++++++++++++++++++++++++++++++++++------------------
> 6 files changed, 65 insertions(+), 22 deletions(-)
>
> diff --git a/Create.c b/Create.c
> index ef28da0..69f5432 100644
> --- a/Create.c
> +++ b/Create.c
> @@ -750,7 +750,8 @@ int Create(struct supertype *st, char *mddev,
> #endif
> }
>
> - if (s->bitmap_file && strcmp(s->bitmap_file, "internal")==0) {
> + if (s->bitmap_file && (strcmp(s->bitmap_file, "internal")==0
> + || strcmp(s->bitmap_file, "clustered")==0)) {
> if ((vers%100) < 2) {
> pr_err("internal bitmaps not supported by this kernel.\n");
> goto abort_locked;
> diff --git a/bitmap.c b/bitmap.c
> index b1d54a6..920033a 100644
> --- a/bitmap.c
> +++ b/bitmap.c
> @@ -32,6 +32,8 @@ inline void sb_le_to_cpu(bitmap_super_t *sb)
> sb->daemon_sleep = __le32_to_cpu(sb->daemon_sleep);
> sb->sync_size = __le64_to_cpu(sb->sync_size);
> sb->write_behind = __le32_to_cpu(sb->write_behind);
> + sb->nodes = __le32_to_cpu(sb->nodes);
> + sb->sectors_reserved = __le32_to_cpu(sb->sectors_reserved);
> }
>
> inline void sb_cpu_to_le(bitmap_super_t *sb)
> diff --git a/bitmap.h b/bitmap.h
> index c8725a3..adbf0b4 100644
> --- a/bitmap.h
> +++ b/bitmap.h
> @@ -154,8 +154,11 @@ typedef struct bitmap_super_s {
> __u32 chunksize; /* 52 the bitmap chunk size in bytes */
> __u32 daemon_sleep; /* 56 seconds between disk flushes */
> __u32 write_behind; /* 60 number of outstanding write-behind writes */
> -
> - __u8 pad[256 - 64]; /* set to zero */
> + __u32 sectors_reserved; /* 64 number of 512-byte sectors that are
> + * reserved for the bitmap. */
> + __u32 nodes; /* 68 the maximum number of nodes in cluster. */
> + __u8 cluster_name[64]; /* 72 cluster name to which this md belongs */
> + __u8 pad[256 - 136]; /* set to zero */
> } bitmap_super_t;
>
> /* notes:
> diff --git a/mdadm.8.in b/mdadm.8.in
> index a630310..4aec0db 100644
> --- a/mdadm.8.in
> +++ b/mdadm.8.in
> @@ -694,7 +694,12 @@ and so is replicated on all devices. If the word
> .B "none"
> is given with
> .B \-\-grow
> -mode, then any bitmap that is present is removed.
> +mode, then any bitmap that is present is removed. If the word
> +.B "clustered"
> +is given, the array is created for a clustered environment. One bitmap
> +is created for each node as defined by the
> +.B \-\-nodes
> +parameter and are stored internally.
>
> To help catch typing errors, the filename must contain at least one
> slash ('/') if it is a real file (not 'internal' or 'none').
> diff --git a/mdadm.c b/mdadm.c
> index 3e8c49b..bd9382e 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -1097,6 +1097,15 @@ int main(int argc, char *argv[])
> s.bitmap_file = optarg;
> continue;
> }
> + if (strcmp(optarg, "clustered")== 0) {
> + s.bitmap_file = optarg;
> + /* Set the default number of cluster nodes
> + * to 4 if not already set by user
> + */
> + if (c.nodes < 1)
> + c.nodes = 4;
> + continue;
> + }
Please make sure than mdadm compiles after each patch is applied.
You don't defined that 'nodes' field until a later patch.
NeilBrown
> /* probable typo */
> pr_err("bitmap file must contain a '/', or be 'internal', or 'none'\n"
> " not '%s'\n", optarg);
> diff --git a/super1.c b/super1.c
> index f0508fe..57b1526 100644
> --- a/super1.c
> +++ b/super1.c
> @@ -2144,6 +2144,10 @@ add_internal_bitmap1(struct supertype *st,
> bms->daemon_sleep = __cpu_to_le32(delay);
> bms->sync_size = __cpu_to_le64(size);
> bms->write_behind = __cpu_to_le32(write_behind);
> + bms->nodes = __cpu_to_le32(st->nodes);
> + if (st->cluster_name)
> + strncpy((char *)bms->cluster_name,
> + st->cluster_name, strlen(st->cluster_name));
>
> *chunkp = chunk;
> return 1;
> @@ -2177,6 +2181,7 @@ static int write_bitmap1(struct supertype *st, int fd)
> void *buf;
> int towrite, n;
> struct align_fd afd;
> + unsigned int i;
>
> init_afd(&afd, fd);
>
> @@ -2185,27 +2190,45 @@ static int write_bitmap1(struct supertype *st, int fd)
> if (posix_memalign(&buf, 4096, 4096))
> return -ENOMEM;
>
> - memset(buf, 0xff, 4096);
> - memcpy(buf, (char *)bms, sizeof(bitmap_super_t));
> -
> - towrite = __le64_to_cpu(bms->sync_size) / (__le32_to_cpu(bms->chunksize)>>9);
> - towrite = (towrite+7) >> 3; /* bits to bytes */
> - towrite += sizeof(bitmap_super_t);
> - towrite = ROUND_UP(towrite, 512);
> - while (towrite > 0) {
> - n = towrite;
> - if (n > 4096)
> - n = 4096;
> - n = awrite(&afd, buf, n);
> - if (n > 0)
> - towrite -= n;
> + /* We use bms->nodes as opposed to st->nodes to
> + * be compatible with write-after-reads such as
> + * the GROW operation.
> + */
> + for (i = 0; i < __le32_to_cpu(bms->nodes); i++) {
> + /* Only the bitmap[0] should resync
> + * whole device on initial assembly
> + */
> + if (i)
> + memset(buf, 0x00, 4096);
> else
> + memset(buf, 0xff, 4096);
> + memcpy(buf, (char *)bms, sizeof(bitmap_super_t));
> +
> + towrite = __le64_to_cpu(bms->sync_size) / (__le32_to_cpu(bms->chunksize)>>9);
> + towrite = (towrite+7) >> 3; /* bits to bytes */
> + towrite += sizeof(bitmap_super_t);
> + /* we need the bitmaps to be at 4k boundary */
> + towrite = ROUND_UP(towrite, 4096);
> + while (towrite > 0) {
> + n = towrite;
> + if (n > 4096)
> + n = 4096;
> + n = awrite(&afd, buf, n);
> + if (n > 0)
> + towrite -= n;
> + else
> + break;
> + if (i)
> + memset(buf, 0x00, 4096);
> + else
> + memset(buf, 0xff, 4096);
> + }
> + fsync(fd);
> + if (towrite) {
> + rv = -2;
> break;
> - memset(buf, 0xff, 4096);
> + }
> }
> - fsync(fd);
> - if (towrite)
> - rv = -2;
>
> free(buf);
> return rv;
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* RE: [PATCH 0/7] mdadm support for caching layer in raid 5/6
From: Song Liu @ 2015-05-14 18:01 UTC (permalink / raw)
To: Jason Keltz, linux-raid@vger.kernel.org
In-Reply-To: <14d5230fde0.276a.cf392d89e21755b689a9b33faa59c92f@cse.yorku.ca>
Hi Jason,
As Shaohua discussed earlier in another thread, in this write cache approach,
we are trying to solve two problems together:
1. RAID-5/6 write hole;
2. Slow sync writes to RAID-5/6 (due to read-modify-write).
For other RAID types, there is no write hole issue. To get better sync write
performance on RAID-10, you can use other caching solutions, like bcache
or flashcache.
The write cache can also be a MD device.
Thanks,
Song
> -----Original Message-----
> From: Jason Keltz [mailto:jas@cse.yorku.ca]
> Sent: Thursday, May 14, 2015 4:31 AM
> To: Song Liu; linux-raid@vger.kernel.org
> Subject: Re: [PATCH 0/7] mdadm support for caching layer in raid 5/6
>
> Hi..
> I'm curious - why not support write cache for other raid levels? If the write
> cache becomes a fast SSD (with high endurance) for a pool that is all standard
> HD then wouldn't the write cache benefit there as well? (Eg.
> Raid10) . Can the write cache also be an MD device? I would think you could
> use that to improve performance and redundancy of the cache?
>
> I'm using RHEL 7.1 on an NFS server for a virtualization back end. Because of
> sync writes the write performance is poor. Its really no surprise.. I've got a 22
> disk raid10. At the moment I could try dm-cache in rhel7 in an effort to
> improve write performance but its a technology preview and not sure I'd want
> to rely on that for production. It's not really just a simple write cache either
> which is all that I'm really after. I like simple! I could also enable async writes
> on my NFS export and not sleep at night (only 3 virtualization nodes.. UPS
> protection everywhere protected..
> But what if the file server crashes? You want eerie? I just got an email that
> power is off on one node of my test setup....UPS failed... Aie! That could just as
> easily have been the FS.) ..I could stop using MD for NFS backend storage
> altogether and rely on a raid card with battery backed cache as well and then
> when there's some kind of card firmware issue wait for the hardware vendor to
> close my support ticket due to inactivity before they even respond.... which has
> happened several times with a common 3 letter storage company that has
> more than 3 letters in their name now... I could wait for someone to create a
> magical hardware device that would do for MD software raid what the battery
> backed cache does for hardware raid.
> To this day I really don't understand why that hasn't been done. It would be a
> whole lot cheaper than SSD. . it would have higher longevity.... I'd buy it... And I
> suspect a billion other people as well... But I guess simple write caching will
> help as well....
>
> Since I don't have any of the above goodies I've decided to try a switch
> from ext4 to xfs. Then test with HD based external log... Then Ill try
> SSD based external log ... And finally potentially external log to an MD
> raid1 SSD log device.... Though I can't find too many details, I've got a hunch
> that if the log fails I'd be in trouble...
>
> J.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
>
> On May 14, 2015 2:44:04 AM Song Liu <songliubraving@fb.com> wrote:
>
> > Hi,
> >
> > These are mdadm patches to support cache layer in raid 5/6. Shaohua
> > has sent the kernel patch earlier with subject "a caching layer for
> > raid 5/6".
> >
> > These patches add write cache support for the following commands:
> >
> > mdadm --detail
> > mdadm --create
> > mdadm --assemble
> > mdadm --incremental
> > mdadm --examine
> > mdadm --zero-super
> >
> > Cache device is assigned with dev_role 0xFFFD (where 0xFFFF is for
> > spare and 0xFFFE is for failed). Note that there is compatibility
> > issue that older mdadm will show cache device as spare in --detail:
> >
> > Number Major Minor RaidDevice State
> > 0 8 32 0 active sync /dev/sdc
> > 1 8 48 1 active sync /dev/sdd
> > 2 8 64 2 active sync /dev/sde
> > 3 8 80 3 active sync /dev/sdf
> >
> > 4 8 17 - spare /dev/sdb1
> >
> > Also, older mdadm will show cache device as "Active device 65533"
> > in --examine:
> >
> > Device Role : Active device 65533
> > Array State : AAAA ('A' == active, '.' == missing, 'R' ==
> > replacing)
> >
> >
> > Song Liu (7):
> > Show device as cache in --detail
> > Enable create array with write cache (--write-cache DEVICE).
> > Create write-cache superblock in mdadm --create
> > Assemble array with writecache
> > Check write cache in incremental
> > Zero write-cache superblock in --zero-super
> > Add information about write-cache superblock to --examine
> >
> > Assemble.c | 52 +++++++++++++----
> > Create.c | 21 +++++--
> > Detail.c | 3 +-
> > Examine.c | 29 +++++++++-
> > Incremental.c | 37 ++++++++++--
> > Kill.c | 19 ++++++
> > ReadMe.c | 1 +
> > md_p.h | 74 ++++++++++++++++++++++++
> > mdadm.c | 19 ++++++
> > mdadm.h | 11 +++-
> > super1.c | 182
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
> > util.c | 3 +-
> > 12 files changed, 422 insertions(+), 29 deletions(-)
> >
> > --
> > 1.8.1
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: [PATCH 0/7] mdadm support for caching layer in raid 5/6
From: Jason Keltz @ 2015-05-14 11:30 UTC (permalink / raw)
To: Song Liu, linux-raid
In-Reply-To: <14d521c83b0.276a.cf392d89e21755b689a9b33faa59c92f@cse.yorku.ca>
Hi..
I'm curious - why not support write cache for other raid levels? If the
write cache becomes a fast SSD (with high endurance) for a pool that is all
standard HD then wouldn't the write cache benefit there as well? (Eg.
Raid10) . Can the write cache also be an MD device? I would think you
could use that to improve performance and redundancy of the cache?
I'm using RHEL 7.1 on an NFS server for a virtualization back end. Because
of sync writes the write performance is poor. Its really no surprise.. I've
got a 22 disk raid10. At the moment I could try dm-cache in rhel7 in an
effort to improve write performance but its a technology preview and not
sure I'd want to rely on that for production. It's not really just a
simple write cache either which is all that I'm really after. I like
simple! I could also enable async writes on my NFS export and not sleep at
night (only 3 virtualization nodes.. UPS protection everywhere protected..
But what if the file server crashes? You want eerie? I just got an email
that power is off on one node of my test setup....UPS failed... Aie! That
could just as easily have been the FS.) ..I could stop using MD for NFS
backend storage altogether and rely on a raid card with battery backed
cache as well and then when there's some kind of card firmware issue wait
for the hardware vendor to close my support ticket due to inactivity before
they even respond.... which has happened several times with a common 3
letter storage company that has more than 3 letters in their name now... I
could wait for someone to create a magical hardware device that would do
for MD software raid what the battery backed cache does for hardware raid.
To this day I really don't understand why that hasn't been done. It would
be a whole lot cheaper than SSD. . it would have higher longevity.... I'd
buy it... And I suspect a billion other people as well... But I guess
simple write caching will help as well....
Since I don't have any of the above goodies I've decided to try a switch
from ext4 to xfs. Then test with HD based external log... Then Ill try
SSD based external log ... And finally potentially external log to an MD
raid1 SSD log device.... Though I can't find too many details, I've got a
hunch that if the log fails I'd be in trouble...
J.
Sent with AquaMail for Android
http://www.aqua-mail.com
On May 14, 2015 2:44:04 AM Song Liu <songliubraving@fb.com> wrote:
> Hi,
>
> These are mdadm patches to support cache layer in raid 5/6. Shaohua
> has sent the kernel patch earlier with subject "a caching layer
> for raid 5/6".
>
> These patches add write cache support for the following commands:
>
> mdadm --detail
> mdadm --create
> mdadm --assemble
> mdadm --incremental
> mdadm --examine
> mdadm --zero-super
>
> Cache device is assigned with dev_role 0xFFFD (where 0xFFFF is for
> spare and 0xFFFE is for failed). Note that there is compatibility
> issue that older mdadm will show cache device as spare in --detail:
>
> Number Major Minor RaidDevice State
> 0 8 32 0 active sync /dev/sdc
> 1 8 48 1 active sync /dev/sdd
> 2 8 64 2 active sync /dev/sde
> 3 8 80 3 active sync /dev/sdf
>
> 4 8 17 - spare /dev/sdb1
>
> Also, older mdadm will show cache device as "Active device 65533"
> in --examine:
>
> Device Role : Active device 65533
> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>
>
> Song Liu (7):
> Show device as cache in --detail
> Enable create array with write cache (--write-cache DEVICE).
> Create write-cache superblock in mdadm --create
> Assemble array with writecache
> Check write cache in incremental
> Zero write-cache superblock in --zero-super
> Add information about write-cache superblock to --examine
>
> Assemble.c | 52 +++++++++++++----
> Create.c | 21 +++++--
> Detail.c | 3 +-
> Examine.c | 29 +++++++++-
> Incremental.c | 37 ++++++++++--
> Kill.c | 19 ++++++
> ReadMe.c | 1 +
> md_p.h | 74 ++++++++++++++++++++++++
> mdadm.c | 19 ++++++
> mdadm.h | 11 +++-
> super1.c | 182 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
> util.c | 3 +-
> 12 files changed, 422 insertions(+), 29 deletions(-)
>
> --
> 1.8.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 7/7] Add information about write-cache superblock to --examine
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
In-Reply-To: <1431585836-4103033-1-git-send-email-songliubraving@fb.com>
Example output:
./mdadm -E /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 261d25c2:7264d13e:670f0307:71441559
Name : 0
Creation Time : Wed May 13 11:04:07 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 706560 (345.00 MiB 361.76 MB)
Array Size : 11720662464 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774976 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=18446744065896483200 sectors
State : clean
Device UUID : f89a0182:10bc963c:3e6a2735:16d306fa
Internal Bitmap : 8 sectors from superblock
Update Time : Wed May 13 11:54:37 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 74417070 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 32K
Device Role : cache
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
Write Cache Info:
Magic : 6433c509
Superblock Version : 1
Stripe Cache Size : 4096
Block Size : 8 sectors
Stripe Data Size : 192 sectors
Chunk Size : 64 sectors
Stripe Size : 256 sectors
Total Blocks : 88319 blocks
First Block : 1
Last Checkpoint : 58641
Update Time : Wed May 13 11:54:37 2015
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Examine.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/Examine.c b/Examine.c
index 953b8ee..f6086a5 100644
--- a/Examine.c
+++ b/Examine.c
@@ -141,7 +141,8 @@ int Examine(struct mddev_dev *devlist,
} else
st->ss->getinfo_super(st, &ap->info, NULL);
if (!have_container &&
- !(ap->info.disk.state & (1<<MD_DISK_SYNC)))
+ !(ap->info.disk.state & (1<<MD_DISK_SYNC)) &&
+ !(ap->info.disk.state & (1<<MD_DISK_WRITECACHE)))
ap->spares++;
d = dl_strdup(devlist->devname);
dl_add(ap->devs, d);
@@ -152,6 +153,9 @@ int Examine(struct mddev_dev *devlist,
} else {
printf("%s:\n",devlist->devname);
st->ss->examine_super(st, c->homehost);
+ if (st->r5l_sb) {
+ ExamineR5LSuper(st->r5l_sb);
+ }
st->ss->free_super(st);
}
}
@@ -223,3 +227,26 @@ out:
}
return err;
}
+
+int ExamineR5LSuper(struct r5l_super_block *sb_blk)
+{
+ time_t atime;
+
+ if (sb_blk == NULL)
+ return 1;
+ printf("\n\nWrite Cache Info:\n");
+ printf(" Magic : %08x\n", __le32_to_cpu(sb_blk->header.magic));
+ printf("Superblock Version : %d\n", sb_blk->version);
+ printf(" Stripe Cache Size : %d\n", sb_blk->stripe_cache_size);
+ printf(" Block Size : %d sectors\n", sb_blk->block_size >> 9);
+ printf(" Stripe Data Size : %d sectors\n", sb_blk->stripe_data_size >> 9);
+ printf(" Chunk Size : %d sectors\n", sb_blk->chunk_size >> 9);
+ printf(" Stripe Size : %d sectors\n", sb_blk->stripe_size >> 9);
+ printf(" Total Blocks : %lld blocks\n", sb_blk->total_blocks);
+ printf(" Last Checkpoint : %lld\n", sb_blk->last_checkpoint);
+
+ atime = __le64_to_cpu(sb_blk->update_time_sec) & 0xFFFFFFFFFFULL;
+ printf(" Update Time : %.24s\n", ctime(&atime));
+
+ return 0;
+}
--
1.8.1
^ permalink raw reply related
* [PATCH 6/7] Zero write-cache superblock in --zero-super
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
In-Reply-To: <1431585836-4103033-1-git-send-email-songliubraving@fb.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Kill.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/Kill.c b/Kill.c
index f2fdb85..9dc566b 100644
--- a/Kill.c
+++ b/Kill.c
@@ -29,6 +29,18 @@
#include "md_u.h"
#include "md_p.h"
+int KillR5LSuper(struct supertype *st, int fd)
+{
+ memset(st->r5l_sb, 0, LOG_BLOCK_SIZE);
+
+ if (lseek64(fd, 512 * (st->data_offset), 0) < 0LL)
+ return 1;
+ if (write(fd, st->r5l_sb, LOG_BLOCK_SIZE))
+ return 1;
+ fsync(fd);
+ return 0;
+}
+
int Kill(char *dev, struct supertype *st, int force, int verbose, int noexcl)
{
/*
@@ -60,8 +72,15 @@ int Kill(char *dev, struct supertype *st, int force, int verbose, int noexcl)
return 2;
}
st->ignore_hw_compat = 1;
+ st->data_offset = INVALID_SECTORS; /* load st->data_offset from sb for KillR5LSuper */
rv = st->ss->load_super(st, fd, dev);
if (rv == 0 || (force && rv >= 2)) {
+ if (st->r5l_sb) {
+ if (verbose > 0)
+ pr_err("zero write_cache superblock\n");
+ KillR5LSuper(st, fd);
+ }
+
st->ss->free_super(st);
st->ss->init_super(st, NULL, 0, "", NULL, NULL,
INVALID_SECTORS);
--
1.8.1
^ permalink raw reply related
* [PATCH 5/7] Check write cache in incremental
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
In-Reply-To: <1431585836-4103033-1-git-send-email-songliubraving@fb.com>
If cache device is missing, do not start the array, and shows:
./mdadm -I /dev/sdf
mdadm: Cache device is missing, not safe to start yet.
The array will be started when the cache device is attached with -I
./mdadm -I /dev/sdb1
mdadm: /dev/sdb1 attached to /dev/md/0_0, which has been started.
To force start without cache device:
./mdadm -I /dev/sdf --run
mdadm: Trying to run with missing cache device
mdadm: /dev/sdf attached to /dev/md/0_0, which has been started.
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Incremental.c | 37 +++++++++++++++++++++++++++++++++----
super1.c | 2 ++
2 files changed, 35 insertions(+), 4 deletions(-)
diff --git a/Incremental.c b/Incremental.c
index 0c9a9a4..500fd9e 100644
--- a/Incremental.c
+++ b/Incremental.c
@@ -35,7 +35,7 @@
static int count_active(struct supertype *st, struct mdinfo *sra,
int mdfd, char **availp,
- struct mdinfo *info);
+ struct mdinfo *info, int *cache_device_missing);
static void find_reject(int mdfd, struct supertype *st, struct mdinfo *sra,
int number, __u64 events, int verbose,
char *array_name);
@@ -104,6 +104,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
struct map_ent target_array;
int have_target;
char *devname = devlist->devname;
+ int cache_device_missing = 0;
struct createinfo *ci = conf_get_create_info();
@@ -216,6 +217,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
free(st);
goto out;
}
+
close (dfd); dfd = -1;
st->ss->getinfo_super(st, &info, NULL);
@@ -470,6 +472,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
info.array.working_disks ++;
}
+
if (strncmp(chosen_name, "/dev/md/", 8) == 0)
md_devname = chosen_name+8;
else
@@ -511,10 +514,13 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
* state. Eventually this state should be kept up-to-date as
* things change.
*/
+
sysfs_free(sra);
sra = sysfs_read(mdfd, NULL, (GET_DEVS | GET_STATE |
GET_OFFSET | GET_SIZE));
- active_disks = count_active(st, sra, mdfd, &avail, &info);
+
+ active_disks = count_active(st, sra, mdfd, &avail, &info, &cache_device_missing);
+
if (enough(info.array.level, info.array.raid_disks,
info.array.layout, info.array.state & 1,
avail) == 0) {
@@ -544,10 +550,13 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
}
map_unlock(&map);
- if (c->runstop > 0 || active_disks >= info.array.working_disks) {
+ if (c->runstop > 0 || (!cache_device_missing && active_disks >= info.array.working_disks)) {
struct mdinfo *dsk;
/* Let's try to start it */
+ if (cache_device_missing)
+ pr_err("Trying to run with missing cache device\n");
+
if (info.reshape_active && !(info.reshape_active & RESHAPE_NO_BACKUP)) {
pr_err("%s: This array is being reshaped and cannot be started\n",
chosen_name);
@@ -614,6 +623,8 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
} else {
if (c->export) {
printf("MD_STARTED=unsafe\n");
+ } else if (cache_device_missing) {
+ pr_err("Cache device is missing, not safe to start yet.\n");
} else if (c->verbose >= 0)
pr_err("%s attached to %s, not enough to start safely.\n",
devname, chosen_name);
@@ -680,7 +691,8 @@ static void find_reject(int mdfd, struct supertype *st, struct mdinfo *sra,
static int count_active(struct supertype *st, struct mdinfo *sra,
int mdfd, char **availp,
- struct mdinfo *bestinfo)
+ struct mdinfo *bestinfo,
+ int *cache_device_missing)
{
/* count how many devices in sra think they are active */
struct mdinfo *d;
@@ -694,6 +706,8 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
int devnum;
int b, i;
int raid_disks = 0;
+ int require_cache_dev = 0;
+ int has_cache_dev = 0;
if (!sra)
return 0;
@@ -714,8 +728,19 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
close(dfd);
if (ok != 0)
continue;
+
+ if (st->ss->require_cache) {
+ require_cache_dev = st->ss->require_cache(st);
+ if (require_cache_dev == 2) {
+ pr_err("BUG: Superblock not loaded in Incremental.c:count_active\n");
+ return 0;
+ }
+ }
+
info.array.raid_disks = raid_disks;
st->ss->getinfo_super(st, &info, devmap + raid_disks * devnum);
+ if (info.disk.raid_disk == 0xFFFD)
+ has_cache_dev = 1;
if (!avail) {
raid_disks = info.array.raid_disks;
avail = xcalloc(raid_disks, 1);
@@ -765,6 +790,10 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
replcnt++;
st->ss->free_super(st);
}
+
+ if (require_cache_dev && !has_cache_dev)
+ *cache_device_missing = 1;
+
if (!avail)
return 0;
/* We need to reject any device that thinks the best device is
diff --git a/super1.c b/super1.c
index c345a40..e229efe 100644
--- a/super1.c
+++ b/super1.c
@@ -144,6 +144,7 @@ static int require_cache1(struct supertype *st)
{
struct mdp_superblock_1 *sb = st->sb;
int i;
+
if (sb)
for (i=0; i<MAX_DEVS; i++) {
if (0xFFFD == sb->dev_roles[i])
@@ -151,6 +152,7 @@ static int require_cache1(struct supertype *st)
}
else
return 2; /* no sb loaded */
+
return 0;
}
--
1.8.1
^ permalink raw reply related
* [PATCH 4/7] Assemble array with writecache
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
In-Reply-To: <1431585836-4103033-1-git-send-email-songliubraving@fb.com>
Example output:
./mdadm --assemble /dev/md0 /dev/sd[c-f] /dev/sdb1
mdadm: /dev/md0 has been started with 4 drives and 1 cache.
mdadm checks superblock for cache devices. If the
array appears to have a cache device, but it is not given,
it will complain as
./mdadm --assemble /dev/md0 /dev/sd[c-f]
mdadm: Not safe to assemble with cache device missing, consider --force.
This can be overwritten with --force
./mdadm --assemble /dev/md0 /dev/sd[c-f] --force
mdadm: Force start with missing cache device...
mdadm: /dev/md0 has been started with 4 drives.
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Assemble.c | 52 ++++++++++++++++++++++++++++++++++++++++++----------
mdadm.h | 2 ++
super1.c | 22 ++++++++++++++++++++++
util.c | 3 ++-
4 files changed, 68 insertions(+), 11 deletions(-)
diff --git a/Assemble.c b/Assemble.c
index 25a103d..ccf142e 100644
--- a/Assemble.c
+++ b/Assemble.c
@@ -718,7 +718,8 @@ static int load_devices(struct devs *devices, char *devmap,
i = devcnt;
else
i = devices[devcnt].i.disk.raid_disk;
- if (i+1 == 0) {
+
+ if (i+1 == 0 || i == 0xfffd) {
if (nextspare < content->array.raid_disks*2)
nextspare = content->array.raid_disks*2;
i = nextspare++;
@@ -772,8 +773,9 @@ static int load_devices(struct devs *devices, char *devmap,
}
if (best[i] == -1
|| (devices[best[i]].i.events
- < devices[devcnt].i.events))
+ < devices[devcnt].i.events)) {
best[i] = devcnt;
+ }
}
devcnt++;
}
@@ -927,6 +929,7 @@ static int start_array(int mdfd,
unsigned int okcnt,
unsigned int sparecnt,
unsigned int rebuilding_cnt,
+ unsigned int cachecnt,
struct context *c,
int clean, char *avail,
int start_partial_ok,
@@ -938,6 +941,22 @@ static int start_array(int mdfd,
int i;
unsigned int req_cnt;
+ if (st->ss->require_cache) {
+ rv = st->ss->require_cache(st);
+ if (rv == 2) {
+ pr_err("BUG: Superblock not loaded in Assemble.c:start_array\n");
+ return 1;
+ }
+
+ if (cachecnt == 0 && rv == 1) {
+ if (!(c->force)) {
+ pr_err("Not safe to assemble with cache device missing, consider --force.\n");
+ return 1;
+ } else
+ pr_err("Force start with missing cache device...\n");
+ }
+ }
+
rv = set_array_info(mdfd, st, content);
if (rv && !err_ok) {
pr_err("failed to set array info for %s: %s\n",
@@ -1015,7 +1034,8 @@ static int start_array(int mdfd,
if (content->array.level == LEVEL_CONTAINER) {
if (c->verbose >= 0) {
pr_err("Container %s has been assembled with %d drive%s",
- mddev, okcnt+sparecnt, okcnt+sparecnt==1?"":"s");
+ mddev, okcnt+sparecnt+cachecnt,
+ okcnt+sparecnt+cachecnt==1?"":"s");
if (okcnt < (unsigned)content->array.raid_disks)
fprintf(stderr, " (out of %d)",
content->array.raid_disks);
@@ -1101,6 +1121,8 @@ static int start_array(int mdfd,
fprintf(stderr, "%s %d rebuilding", sparecnt?",":" and", rebuilding_cnt);
if (sparecnt)
fprintf(stderr, " and %d spare%s", sparecnt, sparecnt==1?"":"s");
+ if (cachecnt == 1)
+ fprintf(stderr, " and 1 cache");
fprintf(stderr, ".\n");
}
if (content->reshape_active &&
@@ -1268,7 +1290,7 @@ int Assemble(struct supertype *st, char *mddev,
int *best = NULL; /* indexed by raid_disk */
int bestcnt = 0;
int devcnt;
- unsigned int okcnt, sparecnt, rebuilding_cnt, replcnt;
+ unsigned int okcnt, sparecnt, rebuilding_cnt, replcnt, cachecnt;
int i;
int was_forced = 0;
int most_recent = 0;
@@ -1479,6 +1501,7 @@ try_again:
devcnt = load_devices(devices, devmap, ident, &st, devlist,
c, content, mdfd, mddev,
&most_recent, &bestcnt, &best, inargv);
+
if (devcnt < 0)
return 1;
@@ -1506,7 +1529,9 @@ try_again:
okcnt = 0;
replcnt = 0;
sparecnt=0;
+ cachecnt=0;
rebuilding_cnt=0;
+
for (i=0; i< bestcnt; i++) {
int j = best[i];
int event_margin = 1; /* always allow a difference of '1'
@@ -1516,8 +1541,10 @@ try_again:
/* note: we ignore error flags in multipath arrays
* as they don't make sense
*/
- if (content->array.level != LEVEL_MULTIPATH)
- if (!(devices[j].i.disk.state & (1<<MD_DISK_ACTIVE))) {
+ if (content->array.level != LEVEL_MULTIPATH) {
+ if (devices[j].i.disk.state & (1<<MD_DISK_WRITECACHE)) {
+ cachecnt++;
+ } else if (!(devices[j].i.disk.state & (1<<MD_DISK_ACTIVE))) {
if (!(devices[j].i.disk.state
& (1<<MD_DISK_FAULTY))) {
devices[j].uptodate = 1;
@@ -1525,6 +1552,7 @@ try_again:
}
continue;
}
+ }
/* If this device thinks that 'most_recent' has failed, then
* we must reject this device.
*/
@@ -1559,10 +1587,11 @@ try_again:
replcnt++;
} else
rebuilding_cnt++;
- } else
+ } else if (devices[j].i.disk.raid_disk != 0xfffd)
sparecnt++;
}
}
+
free(devmap);
if (c->force) {
int force_ok = force_array(content, devices, best, bestcnt,
@@ -1583,8 +1612,9 @@ try_again:
int j = best[i];
int fd;
- if (j<0)
+ if (j<0) {
continue;
+ }
if (!devices[j].uptodate)
continue;
if (devices[j].i.events < devices[most_recent].i.events)
@@ -1623,7 +1653,9 @@ try_again:
int j = best[i];
unsigned int desired_state;
- if (i >= content->array.raid_disks * 2)
+ if (devices[j].i.disk.raid_disk == 0xfffd)
+ desired_state = (1<<MD_DISK_WRITECACHE);
+ else if (i >= content->array.raid_disks * 2)
desired_state = 0;
else if (i & 1)
desired_state = (1<<MD_DISK_ACTIVE) | (1<<MD_DISK_REPLACEMENT);
@@ -1770,7 +1802,7 @@ try_again:
rv = start_array(mdfd, mddev, content,
st, ident, best, bestcnt,
chosen_drive, devices, okcnt, sparecnt,
- rebuilding_cnt,
+ rebuilding_cnt, cachecnt,
c,
clean, avail, start_partial_ok,
pre_exist != NULL,
diff --git a/mdadm.h b/mdadm.h
index d7a205c..62a0293 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -971,6 +971,8 @@ extern struct superswitch {
/* write super block of raid5-cache*/
int (*write_r5l_super)(struct supertype *st, int fd);
+ /* whether the array require a cache device */
+ int (*require_cache)(struct supertype *st);
int swapuuid; /* true if uuid is bigending rather than hostendian */
int external;
const char *name; /* canonical metadata name */
diff --git a/super1.c b/super1.c
index f2697a8..c345a40 100644
--- a/super1.c
+++ b/super1.c
@@ -135,6 +135,27 @@ struct misc_dev_info {
)
static int write_r5l_super1(struct supertype *st, int fd);
+/* return value:
+ * 0, cache not required
+ * 1, cache required
+ * 2, no superblock loated (st->sb == NULL)
+ */
+static int require_cache1(struct supertype *st)
+{
+ struct mdp_superblock_1 *sb = st->sb;
+ int i;
+ if (sb)
+ for (i=0; i<MAX_DEVS; i++) {
+ if (0xFFFD == sb->dev_roles[i])
+ return 1;
+ }
+ else
+ return 2; /* no sb loaded */
+ return 0;
+}
+
+static int write_r5l_super1(struct supertype *st, int fd);
+
static int role_from_sb(struct mdp_superblock_1 *sb)
{
unsigned int d;
@@ -2556,6 +2577,7 @@ struct superswitch super1 = {
.write_bitmap = write_bitmap1,
.free_super = free_super1,
.write_r5l_super = write_r5l_super1,
+ .require_cache = require_cache1,
#if __BYTE_ORDER == BIG_ENDIAN
.swapuuid = 0,
#else
diff --git a/util.c b/util.c
index cc98d3b..f314748 100644
--- a/util.c
+++ b/util.c
@@ -334,8 +334,9 @@ int enough(int level, int raid_disks, int layout, int clean, char *avail)
int i;
int avail_disks = 0;
- for (i = 0; i < raid_disks; i++)
+ for (i = 0; i < raid_disks; i++) {
avail_disks += !!avail[i];
+ }
switch (level) {
case 10:
--
1.8.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox