* Re: [PATCH 3/3] Safeguard against writing to an active device of another node
From: Guoqing Jiang @ 2015-07-30 3:16 UTC (permalink / raw)
To: NeilBrown; +Cc: rgoldwyn, linux-raid
In-Reply-To: <20150730080247.20f03d23@noble>
Hi Neil,
>
> Probably something like that - yes. Unfortunately your mailer messed
> that up more that usual make it rather difficult to apply.
>
>
Sorry, it is just for review.
> I was wondering if we could make it a run-time dependency though .. a
> bit like get_cluster_name. We can still do that later I guess. I'm
> not really sure what is best at the moment.
>
>
It could be, I am changing it and run some test, then the new version patch
will be send.
Thanks,
Guoqing
^ permalink raw reply
* Re: [PATCH 3/3] Safeguard against writing to an active device of another node
From: Guoqing Jiang @ 2015-07-30 3:18 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: NeilBrown, linux-raid
In-Reply-To: <55B96474.6080604@suse.de>
Hi Goldwyn,
>> Probably something like that - yes. Unfortunately your mailer messed
>> that up more that usual make it rather difficult to apply.
>>
>> I was wondering if we could make it a run-time dependency though .. a
>> bit like get_cluster_name. We can still do that later I guess. I'm
>> not really sure what is best at the moment.
>>
>
>
> Yes, I would second that: Making these functions a run-time
> dependency. We should have the ability for it to work by just
> installing libdlm (with the proper flags set), rather than recompiling
> the package for cluster features.
>
NP, I should take a look at get_cluster_name first.
Thanks,
Guoqing
^ permalink raw reply
* Re: [PATCH 3/3] Safeguard against writing to an active device of another node
From: Guoqing Jiang @ 2015-07-30 8:22 UTC (permalink / raw)
To: NeilBrown; +Cc: rgoldwyn, linux-raid
In-Reply-To: <55B99709.2050609@suse.com>
Guoqing Jiang wrote:
> Hi Neil,
>
>> Probably something like that - yes. Unfortunately your mailer messed
>> that up more that usual make it rather difficult to apply.
>>
>>
>>
> Sorry, it is just for review.
>
>> I was wondering if we could make it a run-time dependency though .. a
>> bit like get_cluster_name. We can still do that later I guess. I'm
>> not really sure what is best at the moment.
>>
>>
>>
> It could be, I am changing it and run some test, then the new version patch
> will be send.
>
After change to dynamic load dlm library, if compile mdadm without
related dlm library,
then some problems appeared, such as unlock dlm would fail when create
cluster-md,
the errno is 52 (EBADE: Invalid exchange).
But, if compile mdadm with previous code, then nothing wrong happened
whether
the dlm library is installed or not.
I will post the new patch which depends on run-time library anyway.
Thanks,
Guoqing
^ permalink raw reply
* Re: [PATCH 3/3] Safeguard against writing to an active device of another node
From: Guoqing Jiang @ 2015-07-30 8:38 UTC (permalink / raw)
To: NeilBrown; +Cc: rgoldwyn, linux-raid
In-Reply-To: <55B9DEBB.4040403@suse.com>
Guoqing Jiang wrote:
> Guoqing Jiang wrote:
>
>> Hi Neil,
>>
>>
>>> Probably something like that - yes. Unfortunately your mailer messed
>>> that up more that usual make it rather difficult to apply.
>>>
>>>
>>>
>>>
>> Sorry, it is just for review.
>>
>>
>>> I was wondering if we could make it a run-time dependency though .. a
>>> bit like get_cluster_name. We can still do that later I guess. I'm
>>> not really sure what is best at the moment.
>>>
>>>
>>>
>>>
>> It could be, I am changing it and run some test, then the new version patch
>> will be send.
>>
>>
> After change to dynamic load dlm library, if compile mdadm without
> related dlm library,
> then some problems appeared, such as unlock dlm would fail when create
> cluster-md,
> the errno is 52 (EBADE: Invalid exchange).
>
> But, if compile mdadm with previous code, then nothing wrong happened
> whether
> the dlm library is installed or not.
>
>
Oops, if dlm library is not existed then the dlm code is not even
compiled in previous code,
so it doesn't complain about anything about dlm.
Thanks,
Guoqing
^ permalink raw reply
* [V2 PATCH] Safeguard against writing to an active device of another node
From: Guoqing Jiang @ 2015-07-30 8:49 UTC (permalink / raw)
To: neilb; +Cc: rgoldwyn, linux-raid, Guoqing Jiang, Goldwyn Rodrigues
Modifying an exiting device's superblock or creating a new superblock
on an existing device needs to be checked because the device could be
in use by another node in another array. So, we check this by taking
all superblock locks in userspace so that we don't step onto an active
device used by another node and safeguard against accidental edits.
After the edit is complete, we release all locks and the lockspace so
that it can be used by the kernel space.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
V2 Changes: use run-time check for dlm library
Makefile | 6 +-
mdadm.h | 3 +
super1.c | 51 ++++++++++++++++
util.c | 204 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 263 insertions(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index c189279..a76ae13 100644
--- a/Makefile
+++ b/Makefile
@@ -81,11 +81,12 @@ FAILED_SLOTS_DIR = $(RUN_DIR)/failed-slots
SYSTEMD_DIR=/lib/systemd/system
COROSYNC:=$(shell [ -d /usr/include/corosync ] || echo -DNO_COROSYNC)
+DLM:=$(shell [ -f /usr/include/libdlm.h ] || echo -DNO_DLM)
DIRFLAGS = -DMAP_DIR=\"$(MAP_DIR)\" -DMAP_FILE=\"$(MAP_FILE)\"
DIRFLAGS += -DMDMON_DIR=\"$(MDMON_DIR)\"
DIRFLAGS += -DFAILED_SLOTS_DIR=\"$(FAILED_SLOTS_DIR)\"
-CFLAGS = $(CWFLAGS) $(CXFLAGS) -DSendmail=\""$(MAILCMD)"\" $(CONFFILEFLAGS) $(DIRFLAGS) $(COROSYNC)
+CFLAGS = $(CWFLAGS) $(CXFLAGS) -DSendmail=\""$(MAILCMD)"\" $(CONFFILEFLAGS) $(DIRFLAGS) $(COROSYNC) $(DLM)
VERSION = $(shell [ -d .git ] && git describe HEAD | sed 's/mdadm-//')
VERS_DATE = $(shell [ -d .git ] && date --date="`git log -n1 --format=format:%cd --date=short`" '+%0dth %B %Y' | sed -e 's/1th/1st/' -e 's/2th/2nd/' -e 's/11st/11th/' -e 's/12nd/12th/')
@@ -105,6 +106,9 @@ endif
# LDFLAGS = -static
# STRIP = -s
LDLIBS=-ldl
+ifneq ($(DLM), -DNO_DLM)
+LDLIBS += -ldlm_lt
+endif
INSTALL = /usr/bin/install
DESTDIR =
diff --git a/mdadm.h b/mdadm.h
index 97892e6..59f851e 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -1429,6 +1429,9 @@ extern char *fd2devnm(int fd);
extern int in_initrd(void);
extern int get_cluster_name(char **name);
+extern int is_clustered(struct supertype *st);
+extern int cluster_get_dlmlock(struct supertype *st, int *lockid);
+extern int cluster_release_dlmlock(struct supertype *st, int lockid);
#define _ROUND_UP(val, base) (((val) + (base) - 1) & ~(base - 1))
#define ROUND_UP(val, base) _ROUND_UP(val, (typeof(val))(base))
diff --git a/super1.c b/super1.c
index fda71e3..bd88c36 100644
--- a/super1.c
+++ b/super1.c
@@ -1072,8 +1072,18 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
* ignored.
*/
int rv = 0;
+ int lockid;
struct mdp_superblock_1 *sb = st->sb;
+ if (is_clustered(st)) {
+ rv = cluster_get_dlmlock(st, &lockid);
+ if (rv) {
+ pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
+ cluster_release_dlmlock(st, lockid);
+ return rv;
+ }
+ }
+
if (strcmp(update, "homehost") == 0 &&
homehost) {
/* Note that 'homehost' is special as it is really
@@ -1330,6 +1340,9 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
rv = -1;
sb->sb_csum = calc_sb_1_csum(sb);
+ if (is_clustered(st))
+ cluster_release_dlmlock(st, lockid);
+
return rv;
}
@@ -1433,6 +1446,16 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
struct mdp_superblock_1 *sb = st->sb;
__u16 *rp = sb->dev_roles + dk->number;
struct devinfo *di, **dip;
+ int rv, lockid;
+
+ if (is_clustered(st)) {
+ rv = cluster_get_dlmlock(st, &lockid);
+ if (rv) {
+ pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
+ cluster_release_dlmlock(st, lockid);
+ return rv;
+ }
+ }
if ((dk->state & 6) == 6) /* active, sync */
*rp = __cpu_to_le16(dk->raid_disk);
@@ -1460,6 +1483,9 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
di->next = NULL;
*dip = di;
+ if (is_clustered(st))
+ cluster_release_dlmlock(st, lockid);
+
return 0;
}
#endif
@@ -1473,6 +1499,16 @@ static int store_super1(struct supertype *st, int fd)
struct align_fd afd;
int sbsize;
unsigned long long dsize;
+ int rv, lockid;
+
+ if (is_clustered(st)) {
+ rv = cluster_get_dlmlock(st, &lockid);
+ if (rv) {
+ pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
+ cluster_release_dlmlock(st, lockid);
+ return rv;
+ }
+ }
if (!get_dev_size(fd, NULL, &dsize))
return 1;
@@ -1533,6 +1569,9 @@ static int store_super1(struct supertype *st, int fd)
}
}
fsync(fd);
+ if (is_clustered(st))
+ cluster_release_dlmlock(st, lockid);
+
return 0;
}
@@ -2282,6 +2321,16 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
static void free_super1(struct supertype *st)
{
+ int rv, lockid;
+ if (is_clustered(st)) {
+ rv = cluster_get_dlmlock(st, &lockid);
+ if (rv) {
+ pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
+ cluster_release_dlmlock(st, lockid);
+ return;
+ }
+ }
+
if (st->sb)
free(st->sb);
while (st->info) {
@@ -2292,6 +2341,8 @@ static void free_super1(struct supertype *st)
free(di);
}
st->sb = NULL;
+ if (is_clustered(st))
+ cluster_release_dlmlock(st, lockid);
}
#ifndef MDASSEMBLE
diff --git a/util.c b/util.c
index ea6e688..3d01219 100644
--- a/util.c
+++ b/util.c
@@ -24,6 +24,7 @@
#include "mdadm.h"
#include "md_p.h"
+#include <sys/poll.h>
#include <sys/socket.h>
#include <sys/utsname.h>
#include <sys/wait.h>
@@ -42,6 +43,25 @@
#else
#include <corosync/cmap.h>
#endif
+#ifndef NO_DLM
+#include <libdlm.h>
+#include <errno.h>
+#else
+#define LKF_NOQUEUE 0x00000001
+#define LKF_CONVERT 0x00000004
+#define LKM_PWMODE 4
+#define EUNLOCK 0x10002
+
+typedef unsigned int uint32_t;
+typedef void *dlm_lshandle_t;
+
+struct dlm_lksb {
+ int sb_status;
+ uint32_t sb_lkid;
+ char sb_flags;
+ char *sb_lvbptr;
+};
+#endif
/*
@@ -88,6 +108,190 @@ struct blkpg_partition {
aren't permitted). */
#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
+struct dlm_lock_resource {
+ dlm_lshandle_t *ls;
+ struct dlm_lksb lksb;
+};
+
+struct dlm_lock_resource *dlm_lock_res = NULL;
+static int ast_called = 0;
+
+int is_clustered(struct supertype *st)
+{
+ /* is it a cluster md or not */
+ return st->cluster_name ? 1 :0;
+}
+
+/* Using poll(2) to wait for and dispatch ASTs */
+static int poll_for_ast(dlm_lshandle_t ls)
+{
+ void *lib_handle = NULL;
+ struct pollfd pfd;
+
+ static int (*ls_get_fd)(dlm_lshandle_t ls);
+ static int (*dispatch)(int fd);
+
+ lib_handle = dlopen("libdlm_lt.so.3", RTLD_NOW | RTLD_LOCAL);
+ if (!lib_handle)
+ return -1;
+
+ ls_get_fd = dlsym(lib_handle, "dlm_ls_get_fd");
+ if (!ls_get_fd)
+ goto out;
+
+ dispatch = dlsym(lib_handle, "dlm_dispatch");
+ if (!dispatch)
+ goto out;
+
+ pfd.fd = ls_get_fd(ls);
+ pfd.events = POLLIN;
+
+ while (!ast_called)
+ {
+ if (poll(&pfd, 1, 0) < 0)
+ {
+ perror("poll");
+ dlclose(lib_handle);
+ return -1;
+ }
+ dispatch(ls_get_fd(ls));
+ }
+ ast_called = 0;
+
+out:
+ dlclose(lib_handle);
+ return 0;
+}
+
+static void dlm_ast(void *arg)
+{
+ ast_called = 1;
+}
+
+/* Create the lockspace, take bitmapXXX locks on all the bitmaps. */
+int cluster_get_dlmlock(struct supertype *st, int *lockid)
+{
+ int ret = -1;
+ void *lib_handle = NULL;
+ char str[64];
+ int flags = LKF_NOQUEUE;
+
+ static dlm_lshandle_t (*create_lockspace)(const char *name,
+ unsigned int mode);
+ static int (*ls_lock)(dlm_lshandle_t lockspace, uint32_t mode,
+ struct dlm_lksb *lksb, uint32_t flags,
+ const void *name, unsigned int namelen,
+ uint32_t parent, void (*astaddr) (void *astarg),
+ void *astarg, void (*bastaddr) (void *astarg),
+ void *range);
+
+ lib_handle = dlopen("libdlm_lt.so.3", RTLD_NOW | RTLD_LOCAL);
+ if (!lib_handle)
+ return ret;
+
+ create_lockspace = dlsym(lib_handle, "dlm_create_lockspace");
+ if (!create_lockspace)
+ goto out;
+
+ ls_lock = dlsym(lib_handle, "dlm_ls_lock");
+ if (!ls_lock)
+ goto out;
+
+ dlm_lock_res = xmalloc(sizeof(struct dlm_lock_resource));
+ if (!dlm_lock_res)
+ goto out;
+
+ dlm_lock_res->ls = create_lockspace(st->cluster_name, O_RDWR);
+ if (!dlm_lock_res->ls) {
+ pr_err("%s failed to create lockspace\n", st->cluster_name);
+ goto out;
+ }
+
+ /* Conversions need the lockid in the LKSB */
+ if (flags & LKF_CONVERT)
+ dlm_lock_res->lksb.sb_lkid = *lockid;
+
+ snprintf(str, 64, "bitmap%04d", st->nodes);
+ /* if flags with LKF_CONVERT causes below return ENOENT which means
+ * "No such file or directory" */
+ ret = ls_lock(dlm_lock_res->ls, LKM_PWMODE, &dlm_lock_res->lksb,
+ flags, str, strlen(str), 0, dlm_ast,
+ dlm_lock_res, NULL, NULL);
+ if (ret) {
+ pr_err("error %d when get PW mode on lock %s\n", errno, str);
+ goto out;
+ }
+
+ /* Wait for it to complete */
+ poll_for_ast(dlm_lock_res->ls);
+ *lockid = dlm_lock_res->lksb.sb_lkid;
+
+ errno = dlm_lock_res->lksb.sb_status;
+ if (errno) {
+ pr_err("error %d happened in ast with lock %s\n", errno, str);
+ goto out;
+ }
+
+out:
+ dlclose(lib_handle);
+ return ret;
+}
+
+int cluster_release_dlmlock(struct supertype *st, int lockid)
+{
+ int ret = -1;
+ void *lib_handle = NULL;
+
+ static int (*ls_unlock)(dlm_lshandle_t lockspace, uint32_t lkid,
+ uint32_t flags, struct dlm_lksb *lksb,
+ void *astarg);
+ static int (*release_lockspace)(const char *name, dlm_lshandle_t ls,
+ int force);
+
+ lib_handle = dlopen("libdlm_lt.so.3", RTLD_NOW | RTLD_LOCAL);
+ if (!lib_handle)
+ return ret;
+
+ ls_unlock = dlsym(lib_handle, "dlm_ls_unlock");
+ if (!ls_unlock)
+ goto out;
+
+ release_lockspace = dlsym(lib_handle, "dlm_release_lockspace");
+ if (!release_lockspace)
+ goto out;
+
+ /* if flags with LKF_CONVERT causes below return EINVAL which means
+ * "Invalid argument" */
+ ret = ls_unlock(dlm_lock_res->ls, lockid, 0, &dlm_lock_res->lksb, dlm_lock_res);
+ if (ret) {
+ pr_err("error %d happened when unlock\n", errno);
+ /* XXX make sure the lock is unlocked eventually */
+ goto out;
+ }
+
+ /* Wait for it to complete */
+ poll_for_ast(dlm_lock_res->ls);
+
+ errno = dlm_lock_res->lksb.sb_status;
+ if (errno != EUNLOCK) {
+ pr_err("error %d happened in ast when unlock lockspace\n", errno);
+ /* XXX make sure the lockspace is unlocked eventually */
+ goto out;
+ }
+
+ ret = release_lockspace(st->cluster_name, dlm_lock_res->ls, 1);
+ if (ret) {
+ pr_err("error %d happened when release lockspace\n", errno);
+ /* XXX make sure the lockspace is released eventually */
+ goto out;
+ }
+ free(dlm_lock_res);
+
+out:
+ dlclose(lib_handle);
+ return ret;
+}
+
/*
* Parse a 128 bit uuid in 4 integers
* format is 32 hexx nibbles with options :.<space> separator
--
1.7.12.4
^ permalink raw reply related
* kernel BUG at drivers/scsi/scsi_lib.c:1101! observed during md5sum for one file on (RAID4->RAID0) device
From: Yi Zhang @ 2015-07-30 9:03 UTC (permalink / raw)
To: linux-raid, linux-scsi; +Cc: xni, Jes.Sorensen, Yi Zhang
In-Reply-To: <286306267.839569.1438244765784.JavaMail.zimbra@redhat.com>
Hi SCSI/RAID maintainer
During raid test with 4.2.0-rc3, I observed below kernel BUG, pls check below info for the test log/environment/test steps.
Log:
[ 306.741662] md: bind<sdb1>
[ 306.750865] md: bind<sdc1>
[ 306.753993] md: bind<sdd1>
[ 306.764475] md: bind<sde1>
[ 306.786156] md: bind<sdf1>
[ 306.789362] md: bind<sdh1>
[ 306.792555] md: bind<sdg1>
[ 306.868166] raid6: sse2x1 gen() 10589 MB/s
[ 306.889143] raid6: sse2x1 xor() 8218 MB/s
[ 306.910121] raid6: sse2x2 gen() 13453 MB/s
[ 306.931102] raid6: sse2x2 xor() 8990 MB/s
[ 306.952079] raid6: sse2x4 gen() 15539 MB/s
[ 306.973063] raid6: sse2x4 xor() 10771 MB/s
[ 306.994039] raid6: avx2x1 gen() 20582 MB/s
[ 307.015017] raid6: avx2x2 gen() 24019 MB/s
[ 307.035998] raid6: avx2x4 gen() 27824 MB/s
[ 307.040755] raid6: using algorithm avx2x4 gen() 27824 MB/s
[ 307.046869] raid6: using avx2x2 recovery algorithm
[ 307.058793] async_tx: api initialized (async)
[ 307.075428] xor: automatically using best checksumming function:
[ 307.091942] avx : 32008.000 MB/sec
[ 307.147662] md: raid6 personality registered for level 6
[ 307.153584] md: raid5 personality registered for level 5
[ 307.159505] md: raid4 personality registered for level 4
[ 307.165698] md/raid:md0: device sdf1 operational as raid disk 4
[ 307.172300] md/raid:md0: device sde1 operational as raid disk 3
[ 307.178899] md/raid:md0: device sdd1 operational as raid disk 2
[ 307.185497] md/raid:md0: device sdc1 operational as raid disk 1
[ 307.192093] md/raid:md0: device sdb1 operational as raid disk 0
[ 307.199052] md/raid:md0: allocated 6482kB
[ 307.203573] md/raid:md0: raid level 4 active with 5 out of 6 devices, algorithm 0
[ 307.211958] md0: detected capacity change from 0 to 53645148160
[ 307.218658] md: recovery of RAID array md0
[ 307.223226] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 307.229729] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 307.240427] md: using 128k window, over a total of 10477568k.
[ 374.670951] md: md0: recovery done.
[ 375.722806] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
[ 447.553364] md: unbind<sdh1>
[ 447.559905] md: export_rdev(sdh1)
[ 447.572684] md: cannot remove active disk sdg1 from md0 ...
[ 447.578909] md/raid:md0: Disk failure on sdg1, disabling device.
[ 447.578909] md/raid:md0: Operation continuing on 5 devices.
[ 447.594850] md: unbind<sdg1>
[ 447.601834] md: export_rdev(sdg1)
[ 447.615446] md: raid0 personality registered for level 0
[ 447.629275] md/raid0:md0: md_size is 104775680 sectors.
[ 447.635094] md: RAID0 configuration for md0 - 1 zone
[ 447.640627] md: zone0=[sdb1/sdc1/sdd1/sde1/sdf1]
[ 447.645833] zone-offset= 0KB, device-offset= 0KB, size= 52387840KB
[ 447.654949]
[ 447.739443] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
[ 447.749258] bio too big device sde1 (768 > 512)
[ 447.754824] bio too big device sdf1 (1024 > 512)
[ 447.759989] bio too big device sdb1 (768 > 512)
[ 447.771102] bio too big device sdc1 (1024 > 512)
[ 447.776276] bio too big device sdd1 (1024 > 512)
[ 447.781459] bio too big device sde1 (1024 > 512)
[ 447.786635] bio too big device sdf1 (768 > 512)
[ 447.811156] bio too big device sdb1 (1024 > 512)
[ 447.816329] bio too big device sdc1 (1024 > 512)
[ 447.821513] bio too big device sdd1 (1024 > 512)
[ 447.826681] bio too big device sde1 (768 > 512)
[ 447.886106] bio too big device sdf1 (1024 > 512)
[ 447.891269] bio too big device sdb1 (1024 > 512)
[ 447.896452] bio too big device sdc1 (1024 > 512)
[ 447.901628] bio too big device sdd1 (768 > 512)
[ 447.930647] bio too big device sde1 (1024 > 512)
[ 447.935820] bio too big device sdf1 (1024 > 512)
[ 447.941003] bio too big device sdb1 (1024 > 512)
[ 447.946179] bio too big device sdc1 (768 > 512)
[ 447.976196] bio too big device sdd1 (1024 > 512)
[ 447.981367] bio too big device sde1 (1024 > 512)
[ 447.986549] bio too big device sdf1 (1024 > 512)
[ 447.991728] bio too big device sdb1 (768 > 512)
[ 448.033614] bio too big device sdc1 (1024 > 512)
[ 448.038786] bio too big device sdd1 (1024 > 512)
[ 448.043968] bio too big device sde1 (1024 > 512)
[ 448.049145] bio too big device sdf1 (768 > 512)
[ 448.083273] bio too big device sdb1 (1024 > 512)
[ 448.088444] bio too big device sdc1 (1024 > 512)
[ 448.093626] bio too big device sdd1 (1024 > 512)
[ 448.098804] bio too big device sde1 (768 > 512)
[ 448.128357] bio too big device sdf1 (1024 > 512)
[ 448.133536] bio too big device sdb1 (1024 > 512)
[ 448.138720] bio too big device sdc1 (1024 > 512)
[ 448.143897] bio too big device sdd1 (768 > 512)
[ 448.173456] bio too big device sde1 (1024 > 512)
[ 448.178627] bio too big device sdf1 (1024 > 512)
[ 448.183811] bio too big device sdb1 (1024 > 512)
[ 448.188985] bio too big device sdc1 (768 > 512)
[ 448.231050] bio too big device sdd1 (1024 > 512)
[ 448.236221] bio too big device sde1 (1024 > 512)
[ 448.241405] bio too big device sdf1 (1024 > 512)
[ 448.246583] bio too big device sdb1 (768 > 512)
[ 448.282548] bio too big device sdc1 (1024 > 512)
[ 448.287719] bio too big device sdd1 (1024 > 512)
[ 448.292904] bio too big device sde1 (1024 > 512)
[ 448.298082] bio too big device sdf1 (768 > 512)
[ 448.328300] bio too big device sdb1 (1024 > 512)
[ 448.333471] bio too big device sdc1 (1024 > 512)
[ 448.338654] bio too big device sdd1 (1024 > 512)
[ 448.343830] bio too big device sde1 (768 > 512)
[ 448.374081] bio too big device sdf1 (1024 > 512)
[ 448.379250] bio too big device sdb1 (1024 > 512)
[ 448.384433] bio too big device sdc1 (1024 > 512)
[ 448.389609] bio too big device sdd1 (768 > 512)
[ 448.394690] ------------[ cut here ]------------
[ 448.399832] kernel BUG at drivers/scsi/scsi_lib.c:1095!
[ 448.405653] invalid opcode: 0000 [#1] SMP
[ 448.410232] Modules linked in: raid0 ext4 mbcache jbd2 raid456 async_raid6_recov async_memcpy async_pq async_xor xor asyd
[ 448.491371] CPU: 1 PID: 11918 Comm: md5sum Not tainted 4.2.0-rc3 #2
[ 448.498354] Hardware name: Dell Inc. PowerEdge R730/0599V5, BIOS 1.2.10 03/09/2015
[ 448.506791] task: ffff880461f28000 ti: ffff880462e08000 task.ti: ffff880462e08000
[ 448.515130] RIP: 0010:[<ffffffff8146aaf2>] [<ffffffff8146aaf2>] scsi_init_sgtable+0x72/0x80
[ 448.524548] RSP: 0018:ffff880462e0b8f8 EFLAGS: 00010002
[ 448.530465] RAX: 0000000000000003 RBX: ffff8803fc03f980 RCX: 0000000000001000
[ 448.538417] RDX: 0000000000000000 RSI: ffff8803fbb78040 RDI: 0000000000000000
[ 448.546369] RBP: ffff880462e0b918 R08: ffff8803fbb78040 R09: 0000000000000000
[ 448.554320] R10: 00000000000001f0 R11: ffffea000feede00 R12: ffff8803fba3b860
[ 448.562272] R13: 0000000000000000 R14: ffff880461edc000 R15: ffff8803fc03f980
[ 448.570224] FS: 00007f41ce7cc740(0000) GS:ffff88046d240000(0000) knlGS:0000000000000000
[ 448.579242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 448.585644] CR2: 0000000000e0226f CR3: 0000000467aa3000 CR4: 00000000001406e0
[ 448.593597] Stack:
[ 448.595834] ffff880462e0b918 ffff8803fba3b780 ffff88046072a200 ffff880461edc000
[ 448.604113] ffff880462e0b968 ffffffff8146ab4a ffff88046072aaf8 ffff8803fbb78000
[ 448.612392] ffff8803fbb78000 ffff8803fc03f980 ffff88046072a260 ffff88046064ec00
[ 448.620669] Call Trace:
[ 448.623393] [<ffffffff8146ab4a>] scsi_init_io+0x4a/0x1c0
[ 448.629410] [<ffffffffa004ed67>] sd_setup_read_write_cmnd+0x47/0xa40 [sd_mod]
[ 448.637460] [<ffffffff81462a8b>] ? scsi_host_alloc_command+0x4b/0xc0
[ 448.644638] [<ffffffffa00527d7>] sd_init_command+0x27/0xa0 [sd_mod]
[ 448.651720] [<ffffffff8146adb1>] scsi_setup_cmnd+0xf1/0x160
[ 448.658026] [<ffffffff8146af71>] scsi_prep_fn+0xd1/0x170
[ 448.664042] [<ffffffff81309dac>] ? deadline_dispatch_requests+0xac/0x160
[ 448.671609] [<ffffffff812ed683>] blk_peek_request+0x153/0x260
[ 448.678110] [<ffffffff8146ca7f>] scsi_request_fn+0x3f/0x610
[ 448.684416] [<ffffffff812e8c57>] __blk_run_queue+0x37/0x50
[ 448.690626] [<ffffffff812e8cee>] queue_unplugged+0x2e/0xa0
[ 448.696836] [<ffffffff812eda65>] blk_flush_plug_list+0x1b5/0x200
[ 448.703626] [<ffffffff812ede14>] blk_finish_plug+0x34/0x50
[ 448.709836] [<ffffffff8118cdfd>] __do_page_cache_readahead+0x1cd/0x240
[ 448.717207] [<ffffffff8118cfb5>] ondemand_readahead+0x145/0x270
[ 448.723903] [<ffffffff812209ba>] ? inode_congested+0xaa/0x110
[ 448.730402] [<ffffffff8118d14c>] page_cache_async_readahead+0x6c/0x70
[ 448.737677] [<ffffffff811811d3>] generic_file_read_iter+0x3c3/0x5e0
[ 448.744760] [<ffffffff811f7569>] __vfs_read+0xc9/0x100
[ 448.750582] [<ffffffff811f7b86>] vfs_read+0x86/0x130
[ 448.756211] [<ffffffff811f8a15>] SyS_read+0x55/0xc0
[ 448.761742] [<ffffffff81681b2e>] entry_SYSCALL_64_fastpath+0x12/0x71
[ 448.768920] Code: ff 41 3b 44 24 08 77 23 41 89 44 24 08 8b 43 5c 41 89 44 24 10 48 83 c4 08 44 89 e8 5b 41 5c 41 5d 5d
[ 448.790490] RIP [<ffffffff8146aaf2>] scsi_init_sgtable+0x72/0x80
[ 448.797287] RSP <ffff880462e0b8f8>
[ 448.801171] ---[ end trace fa7203c8f83678c8 ]---
[ 448.853171] Kernel panic - not syncing: Fatal exception
[ 448.859020] Kernel Offset: disabled
[ 448.862904] drm_kms_helper: panic occurred, switching back to text console
[ 448.920805] ---[ end Kernel panic - not syncing: Fatal exception
[ 448.927513] ------------[ cut here ]------------
[ 448.932661] WARNING: CPU: 1 PID: 11918 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5d/0x60()
[ 448.943423] Modules linked in: raid0 ext4 mbcache jbd2 raid456 async_raid6_recov async_memcpy async_pq async_xor xor asyd
[ 449.024578] CPU: 1 PID: 11918 Comm: md5sum Tainted: G D 4.2.0-rc3 #2
[ 449.032918] Hardware name: Dell Inc. PowerEdge R730/0599V5, BIOS 1.2.10 03/09/2015
[ 449.041353] 0000000000000000 00000000429195bb ffff88046d243d68 ffffffff8167acdd
[ 449.049635] 0000000000000000 0000000000000000 ffff88046d243da8 ffffffff81081a4a
[ 449.057917] ffff88046d243da8 0000000000000000 ffff88046d216780 0000000000000001
[ 449.066198] Call Trace:
[ 449.068918] <IRQ> [<ffffffff8167acdd>] dump_stack+0x45/0x57
[ 449.075336] [<ffffffff81081a4a>] warn_slowpath_common+0x8a/0xc0
[ 449.082030] [<ffffffff81081b7a>] warn_slowpath_null+0x1a/0x20
[ 449.088530] [<ffffffff8104d56d>] native_smp_send_reschedule+0x5d/0x60
[ 449.095805] [<ffffffff810be8e5>] trigger_load_balance+0x145/0x1f0
[ 449.102693] [<ffffffff810ad486>] scheduler_tick+0xa6/0xe0
[ 449.108807] [<ffffffff810f9bb0>] ? tick_sched_do_timer+0x50/0x50
[ 449.115599] [<ffffffff810ea651>] update_process_times+0x51/0x60
[ 449.122293] [<ffffffff810f9965>] tick_sched_handle.isra.17+0x25/0x60
[ 449.129471] [<ffffffff810f9bf4>] tick_sched_timer+0x44/0x80
[ 449.135779] [<ffffffff810eb1e3>] __hrtimer_run_queues+0xf3/0x220
[ 449.142570] [<ffffffff810eb648>] hrtimer_interrupt+0xa8/0x1a0
[ 449.149069] [<ffffffff810500b9>] local_apic_timer_interrupt+0x39/0x60
[ 449.156345] [<ffffffff81684835>] smp_apic_timer_interrupt+0x45/0x60
[ 449.163427] [<ffffffff816829cb>] apic_timer_interrupt+0x6b/0x70
[ 449.170118] <EOI> [<ffffffff816755e3>] ? panic+0x1cc/0x20d
[ 449.176435] [<ffffffff816755dc>] ? panic+0x1c5/0x20d
[ 449.182065] [<ffffffff81019428>] oops_end+0xc8/0xe0
[ 449.187595] [<ffffffff8101994b>] die+0x4b/0x70
[ 449.192643] [<ffffffff81015e6d>] do_trap+0x13d/0x150
[ 449.198272] [<ffffffff81016338>] do_error_trap+0xa8/0x170
[ 449.204386] [<ffffffff8146aaf2>] ? scsi_init_sgtable+0x72/0x80
[ 449.210983] [<ffffffff811821b5>] ? mempool_alloc_slab+0x15/0x20
[ 449.217675] [<ffffffff811822f9>] ? mempool_alloc+0x69/0x170
[ 449.223980] [<ffffffff81016850>] do_invalid_op+0x20/0x30
[ 449.229996] [<ffffffff8168348e>] invalid_op+0x1e/0x30
[ 449.235721] [<ffffffff8146aaf2>] ? scsi_init_sgtable+0x72/0x80
[ 449.242317] [<ffffffff8146aac8>] ? scsi_init_sgtable+0x48/0x80
[ 449.248912] [<ffffffff8146ab4a>] scsi_init_io+0x4a/0x1c0
[ 449.254930] [<ffffffffa004ed67>] sd_setup_read_write_cmnd+0x47/0xa40 [sd_mod]
[ 449.262979] [<ffffffff81462a8b>] ? scsi_host_alloc_command+0x4b/0xc0
[ 449.270157] [<ffffffffa00527d7>] sd_init_command+0x27/0xa0 [sd_mod]
[ 449.277239] [<ffffffff8146adb1>] scsi_setup_cmnd+0xf1/0x160
[ 449.283544] [<ffffffff8146af71>] scsi_prep_fn+0xd1/0x170
[ 449.289561] [<ffffffff81309dac>] ? deadline_dispatch_requests+0xac/0x160
[ 449.297128] [<ffffffff812ed683>] blk_peek_request+0x153/0x260
[ 449.303628] [<ffffffff8146ca7f>] scsi_request_fn+0x3f/0x610
[ 449.309933] [<ffffffff812e8c57>] __blk_run_queue+0x37/0x50
[ 449.316142] [<ffffffff812e8cee>] queue_unplugged+0x2e/0xa0
[ 449.322351] [<ffffffff812eda65>] blk_flush_plug_list+0x1b5/0x200
[ 449.329142] [<ffffffff812ede14>] blk_finish_plug+0x34/0x50
[ 449.335351] [<ffffffff8118cdfd>] __do_page_cache_readahead+0x1cd/0x240
[ 449.342722] [<ffffffff8118cfb5>] ondemand_readahead+0x145/0x270
[ 449.349416] [<ffffffff812209ba>] ? inode_congested+0xaa/0x110
[ 449.355916] [<ffffffff8118d14c>] page_cache_async_readahead+0x6c/0x70
[ 449.363190] [<ffffffff811811d3>] generic_file_read_iter+0x3c3/0x5e0
[ 449.370273] [<ffffffff811f7569>] __vfs_read+0xc9/0x100
[ 449.376094] [<ffffffff811f7b86>] vfs_read+0x86/0x130
[ 449.381723] [<ffffffff811f8a15>] SyS_read+0x55/0xc0
[ 449.387254] [<ffffffff81681b2e>] entry_SYSCALL_64_fastpath+0x12/0x71
[ 449.394432] ---[ end trace fa7203c8f83678c9 ]---
Environment: 4.2.0-rc3
[root@storageqe-09 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 931.5G 0 disk
└─sdb1 8:17 0 10G 0 part
sdc 8:32 0 931.5G 0 disk
└─sdc1 8:33 0 10G 0 part
sdd 8:48 0 931.5G 0 disk
└─sdd1 8:49 0 10G 0 part
sde 8:64 0 931.5G 0 disk
└─sde1 8:65 0 10G 0 part
sdf 8:80 0 931.5G 0 disk
└─sdf1 8:81 0 10G 0 part
sdg 8:96 0 3.7T 0 disk
└─sdg1 8:97 0 10G 0 part
sdh 8:112 0 3.7T 0 disk
└─sdh1 8:113 0 10G 0 part
Reproduce-steps:
While [ 1 ]
do
mdadm --create --run /dev/md0 --level 4 --metadata 1.2 --raid-devices 6 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 --spare-devices 1 /dev/sdh1 --chunk 512
mdadm --wait /dev/md0
mkfs -t ext4 /dev/md0
mkdir /mnt/md_test
mount /dev/md0 /mnt/md_test
dd if=/dev/urandom of=/mnt/md_test/testfile bs=1M count=1000
md5sum /mnt/md_test/testfile > md5.old
umount /dev/md0
mdadm --grow -l0 /dev/md0 --backup-file=tmp0
mdadm --wait /dev/md0
mount /dev/md0 /mnt/md_test
md5sum /mnt/md_test/testfile >md5.new // kernel BUG at drivers/scsi/scsi_lib.c:1101!
umount /dev/md0
mdadm -Ss
mdadm --zero-superblock /dev/sd[bcdefgh]1
done
Best Regards,
Yi Zhang
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [V2 PATCH] Safeguard against writing to an active device of another node
From: Goldwyn Rodrigues @ 2015-07-30 11:24 UTC (permalink / raw)
To: Guoqing Jiang, neilb; +Cc: rgoldwyn, linux-raid
In-Reply-To: <1438246190-24079-1-git-send-email-gqjiang@suse.com>
On 07/30/2015 03:49 AM, Guoqing Jiang wrote:
> Modifying an exiting device's superblock or creating a new superblock
> on an existing device needs to be checked because the device could be
> in use by another node in another array. So, we check this by taking
> all superblock locks in userspace so that we don't step onto an active
> device used by another node and safeguard against accidental edits.
> After the edit is complete, we release all locks and the lockspace so
> that it can be used by the kernel space.
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> V2 Changes: use run-time check for dlm library
This may be better done using a structure of function pointers which is
setup and resolved one-time as opposed to performing symbol resolution
every time the function is called. You could include get_cluster_name as
well in it.
>
> Makefile | 6 +-
> mdadm.h | 3 +
> super1.c | 51 ++++++++++++++++
> util.c | 204 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 4 files changed, 263 insertions(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index c189279..a76ae13 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -81,11 +81,12 @@ FAILED_SLOTS_DIR = $(RUN_DIR)/failed-slots
> SYSTEMD_DIR=/lib/systemd/system
>
> COROSYNC:=$(shell [ -d /usr/include/corosync ] || echo -DNO_COROSYNC)
> +DLM:=$(shell [ -f /usr/include/libdlm.h ] || echo -DNO_DLM)
>
> DIRFLAGS = -DMAP_DIR=\"$(MAP_DIR)\" -DMAP_FILE=\"$(MAP_FILE)\"
> DIRFLAGS += -DMDMON_DIR=\"$(MDMON_DIR)\"
> DIRFLAGS += -DFAILED_SLOTS_DIR=\"$(FAILED_SLOTS_DIR)\"
> -CFLAGS = $(CWFLAGS) $(CXFLAGS) -DSendmail=\""$(MAILCMD)"\" $(CONFFILEFLAGS) $(DIRFLAGS) $(COROSYNC)
> +CFLAGS = $(CWFLAGS) $(CXFLAGS) -DSendmail=\""$(MAILCMD)"\" $(CONFFILEFLAGS) $(DIRFLAGS) $(COROSYNC) $(DLM)
>
> VERSION = $(shell [ -d .git ] && git describe HEAD | sed 's/mdadm-//')
> VERS_DATE = $(shell [ -d .git ] && date --date="`git log -n1 --format=format:%cd --date=short`" '+%0dth %B %Y' | sed -e 's/1th/1st/' -e 's/2th/2nd/' -e 's/11st/11th/' -e 's/12nd/12th/')
> @@ -105,6 +106,9 @@ endif
> # LDFLAGS = -static
> # STRIP = -s
> LDLIBS=-ldl
> +ifneq ($(DLM), -DNO_DLM)
> +LDLIBS += -ldlm_lt
> +endif
>
> INSTALL = /usr/bin/install
> DESTDIR =
> diff --git a/mdadm.h b/mdadm.h
> index 97892e6..59f851e 100644
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -1429,6 +1429,9 @@ extern char *fd2devnm(int fd);
>
> extern int in_initrd(void);
> extern int get_cluster_name(char **name);
> +extern int is_clustered(struct supertype *st);
> +extern int cluster_get_dlmlock(struct supertype *st, int *lockid);
> +extern int cluster_release_dlmlock(struct supertype *st, int lockid);
>
> #define _ROUND_UP(val, base) (((val) + (base) - 1) & ~(base - 1))
> #define ROUND_UP(val, base) _ROUND_UP(val, (typeof(val))(base))
> diff --git a/super1.c b/super1.c
> index fda71e3..bd88c36 100644
> --- a/super1.c
> +++ b/super1.c
> @@ -1072,8 +1072,18 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
> * ignored.
> */
> int rv = 0;
> + int lockid;
> struct mdp_superblock_1 *sb = st->sb;
>
> + if (is_clustered(st)) {
> + rv = cluster_get_dlmlock(st, &lockid);
> + if (rv) {
> + pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
> + cluster_release_dlmlock(st, lockid);
> + return rv;
> + }
> + }
> +
> if (strcmp(update, "homehost") == 0 &&
> homehost) {
> /* Note that 'homehost' is special as it is really
> @@ -1330,6 +1340,9 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
> rv = -1;
>
> sb->sb_csum = calc_sb_1_csum(sb);
> + if (is_clustered(st))
> + cluster_release_dlmlock(st, lockid);
> +
> return rv;
> }
>
> @@ -1433,6 +1446,16 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
> struct mdp_superblock_1 *sb = st->sb;
> __u16 *rp = sb->dev_roles + dk->number;
> struct devinfo *di, **dip;
> + int rv, lockid;
> +
> + if (is_clustered(st)) {
> + rv = cluster_get_dlmlock(st, &lockid);
> + if (rv) {
> + pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
> + cluster_release_dlmlock(st, lockid);
> + return rv;
> + }
> + }
>
> if ((dk->state & 6) == 6) /* active, sync */
> *rp = __cpu_to_le16(dk->raid_disk);
> @@ -1460,6 +1483,9 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
> di->next = NULL;
> *dip = di;
>
> + if (is_clustered(st))
> + cluster_release_dlmlock(st, lockid);
> +
> return 0;
> }
> #endif
> @@ -1473,6 +1499,16 @@ static int store_super1(struct supertype *st, int fd)
> struct align_fd afd;
> int sbsize;
> unsigned long long dsize;
> + int rv, lockid;
> +
> + if (is_clustered(st)) {
> + rv = cluster_get_dlmlock(st, &lockid);
> + if (rv) {
> + pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
> + cluster_release_dlmlock(st, lockid);
> + return rv;
> + }
> + }
>
> if (!get_dev_size(fd, NULL, &dsize))
> return 1;
> @@ -1533,6 +1569,9 @@ static int store_super1(struct supertype *st, int fd)
> }
> }
> fsync(fd);
> + if (is_clustered(st))
> + cluster_release_dlmlock(st, lockid);
> +
> return 0;
> }
>
> @@ -2282,6 +2321,16 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
>
> static void free_super1(struct supertype *st)
> {
> + int rv, lockid;
> + if (is_clustered(st)) {
> + rv = cluster_get_dlmlock(st, &lockid);
> + if (rv) {
> + pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
> + cluster_release_dlmlock(st, lockid);
> + return;
> + }
> + }
> +
> if (st->sb)
> free(st->sb);
> while (st->info) {
> @@ -2292,6 +2341,8 @@ static void free_super1(struct supertype *st)
> free(di);
> }
> st->sb = NULL;
> + if (is_clustered(st))
> + cluster_release_dlmlock(st, lockid);
> }
>
> #ifndef MDASSEMBLE
> diff --git a/util.c b/util.c
> index ea6e688..3d01219 100644
> --- a/util.c
> +++ b/util.c
> @@ -24,6 +24,7 @@
>
> #include "mdadm.h"
> #include "md_p.h"
> +#include <sys/poll.h>
> #include <sys/socket.h>
> #include <sys/utsname.h>
> #include <sys/wait.h>
> @@ -42,6 +43,25 @@
> #else
> #include <corosync/cmap.h>
> #endif
> +#ifndef NO_DLM
> +#include <libdlm.h>
> +#include <errno.h>
> +#else
> +#define LKF_NOQUEUE 0x00000001
> +#define LKF_CONVERT 0x00000004
> +#define LKM_PWMODE 4
> +#define EUNLOCK 0x10002
> +
> +typedef unsigned int uint32_t;
> +typedef void *dlm_lshandle_t;
> +
> +struct dlm_lksb {
> + int sb_status;
> + uint32_t sb_lkid;
> + char sb_flags;
> + char *sb_lvbptr;
> +};
> +#endif
>
>
> /*
> @@ -88,6 +108,190 @@ struct blkpg_partition {
> aren't permitted). */
> #define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
>
> +struct dlm_lock_resource {
> + dlm_lshandle_t *ls;
> + struct dlm_lksb lksb;
> +};
> +
> +struct dlm_lock_resource *dlm_lock_res = NULL;
> +static int ast_called = 0;
> +
> +int is_clustered(struct supertype *st)
> +{
> + /* is it a cluster md or not */
> + return st->cluster_name ? 1 :0;
> +}
> +
> +/* Using poll(2) to wait for and dispatch ASTs */
> +static int poll_for_ast(dlm_lshandle_t ls)
> +{
> + void *lib_handle = NULL;
> + struct pollfd pfd;
> +
> + static int (*ls_get_fd)(dlm_lshandle_t ls);
> + static int (*dispatch)(int fd);
> +
> + lib_handle = dlopen("libdlm_lt.so.3", RTLD_NOW | RTLD_LOCAL);
> + if (!lib_handle)
> + return -1;
> +
> + ls_get_fd = dlsym(lib_handle, "dlm_ls_get_fd");
> + if (!ls_get_fd)
> + goto out;
> +
> + dispatch = dlsym(lib_handle, "dlm_dispatch");
> + if (!dispatch)
> + goto out;
> +
> + pfd.fd = ls_get_fd(ls);
> + pfd.events = POLLIN;
> +
> + while (!ast_called)
> + {
> + if (poll(&pfd, 1, 0) < 0)
> + {
> + perror("poll");
> + dlclose(lib_handle);
> + return -1;
> + }
> + dispatch(ls_get_fd(ls));
> + }
> + ast_called = 0;
> +
> +out:
> + dlclose(lib_handle);
> + return 0;
> +}
> +
> +static void dlm_ast(void *arg)
> +{
> + ast_called = 1;
> +}
> +
> +/* Create the lockspace, take bitmapXXX locks on all the bitmaps. */
> +int cluster_get_dlmlock(struct supertype *st, int *lockid)
> +{
> + int ret = -1;
> + void *lib_handle = NULL;
> + char str[64];
> + int flags = LKF_NOQUEUE;
> +
> + static dlm_lshandle_t (*create_lockspace)(const char *name,
> + unsigned int mode);
> + static int (*ls_lock)(dlm_lshandle_t lockspace, uint32_t mode,
> + struct dlm_lksb *lksb, uint32_t flags,
> + const void *name, unsigned int namelen,
> + uint32_t parent, void (*astaddr) (void *astarg),
> + void *astarg, void (*bastaddr) (void *astarg),
> + void *range);
> +
> + lib_handle = dlopen("libdlm_lt.so.3", RTLD_NOW | RTLD_LOCAL);
> + if (!lib_handle)
> + return ret;
> +
> + create_lockspace = dlsym(lib_handle, "dlm_create_lockspace");
> + if (!create_lockspace)
> + goto out;
> +
> + ls_lock = dlsym(lib_handle, "dlm_ls_lock");
> + if (!ls_lock)
> + goto out;
> +
> + dlm_lock_res = xmalloc(sizeof(struct dlm_lock_resource));
> + if (!dlm_lock_res)
> + goto out;
> +
> + dlm_lock_res->ls = create_lockspace(st->cluster_name, O_RDWR);
> + if (!dlm_lock_res->ls) {
> + pr_err("%s failed to create lockspace\n", st->cluster_name);
> + goto out;
> + }
> +
> + /* Conversions need the lockid in the LKSB */
> + if (flags & LKF_CONVERT)
> + dlm_lock_res->lksb.sb_lkid = *lockid;
> +
> + snprintf(str, 64, "bitmap%04d", st->nodes);
> + /* if flags with LKF_CONVERT causes below return ENOENT which means
> + * "No such file or directory" */
> + ret = ls_lock(dlm_lock_res->ls, LKM_PWMODE, &dlm_lock_res->lksb,
> + flags, str, strlen(str), 0, dlm_ast,
> + dlm_lock_res, NULL, NULL);
> + if (ret) {
> + pr_err("error %d when get PW mode on lock %s\n", errno, str);
> + goto out;
> + }
> +
> + /* Wait for it to complete */
> + poll_for_ast(dlm_lock_res->ls);
> + *lockid = dlm_lock_res->lksb.sb_lkid;
> +
> + errno = dlm_lock_res->lksb.sb_status;
> + if (errno) {
> + pr_err("error %d happened in ast with lock %s\n", errno, str);
> + goto out;
> + }
> +
> +out:
> + dlclose(lib_handle);
> + return ret;
> +}
> +
> +int cluster_release_dlmlock(struct supertype *st, int lockid)
> +{
> + int ret = -1;
> + void *lib_handle = NULL;
> +
> + static int (*ls_unlock)(dlm_lshandle_t lockspace, uint32_t lkid,
> + uint32_t flags, struct dlm_lksb *lksb,
> + void *astarg);
> + static int (*release_lockspace)(const char *name, dlm_lshandle_t ls,
> + int force);
> +
> + lib_handle = dlopen("libdlm_lt.so.3", RTLD_NOW | RTLD_LOCAL);
> + if (!lib_handle)
> + return ret;
> +
> + ls_unlock = dlsym(lib_handle, "dlm_ls_unlock");
> + if (!ls_unlock)
> + goto out;
> +
> + release_lockspace = dlsym(lib_handle, "dlm_release_lockspace");
> + if (!release_lockspace)
> + goto out;
> +
> + /* if flags with LKF_CONVERT causes below return EINVAL which means
> + * "Invalid argument" */
> + ret = ls_unlock(dlm_lock_res->ls, lockid, 0, &dlm_lock_res->lksb, dlm_lock_res);
> + if (ret) {
> + pr_err("error %d happened when unlock\n", errno);
> + /* XXX make sure the lock is unlocked eventually */
> + goto out;
> + }
> +
> + /* Wait for it to complete */
> + poll_for_ast(dlm_lock_res->ls);
> +
> + errno = dlm_lock_res->lksb.sb_status;
> + if (errno != EUNLOCK) {
> + pr_err("error %d happened in ast when unlock lockspace\n", errno);
> + /* XXX make sure the lockspace is unlocked eventually */
> + goto out;
> + }
> +
> + ret = release_lockspace(st->cluster_name, dlm_lock_res->ls, 1);
> + if (ret) {
> + pr_err("error %d happened when release lockspace\n", errno);
> + /* XXX make sure the lockspace is released eventually */
> + goto out;
> + }
> + free(dlm_lock_res);
> +
> +out:
> + dlclose(lib_handle);
> + return ret;
> +}
> +
> /*
> * Parse a 128 bit uuid in 4 integers
> * format is 32 hexx nibbles with options :.<space> separator
>
--
Goldwyn
^ permalink raw reply
* raid1 (re)-add recovery data corruption
From: Yi Zhang @ 2015-07-30 11:35 UTC (permalink / raw)
To: linux-raid; +Cc: Xiao Ni, jes sorensen, Yi Zhang
In-Reply-To: <543799034.898896.1438255427047.JavaMail.zimbra@redhat.com>
Hi Neil
I observed raid1 data corruption on raid1 test, below is the test env/reproduce steps/log, pls check it.
Kernel-verison: 4.2.0-rc3
Test-steps:
1. First create one 2GB file bigfile
2. Execute below script
#!/bin/bash
Create_Loop()
{
for i in `seq 0 7`;do
dd if=/dev/zero of=/tmp/$i.tmp bs=1M count=3000 &
done
wait
for i in `seq 0 7`;do
losetup /dev/loop$i /tmp/$i.tmp
done
}
Prepare()
{
mdadm --create --run /dev/md0 --level 1 --metadata 1.2 --raid-devices 8 /dev/loop[0-7] --chunk 512 --bitmap=internal --bitmap-chunk=64M
mdadm --wait /dev/md0
mkfs.ext4 /dev/md0
mkdir /mnt/fortest
mount /dev/md0 /mnt/fortest
md5sum bigfile >md5sum1
}
Create_Loop
Prepare
cnt=0
while [ 1 ]; do
echo "-----------------------------------------------------$cnt"
cp bigfile /mnt/fortest &
sleep 10
mdadm /dev/md0 -f /dev/loop0
sleep 5
mdadm /dev/md0 -r /dev/loop0
while [ 1 ]; do
if [ $? -ne 0 ];then
sleep 5
mdadm /dev/md0 -r /dev/loop0
else
break
fi
done
sleep 30
mdadm /dev/md0 -a /dev/loop0
wait
echo "cp done"
mdadm --wait /dev/md0
echo "recovery done"
md5sum /mnt/fortest/bigfile > md5sum2
tmp1=`awk '{print $1}' ./md5sum1`
tmp2=`awk '{print $1}' ./md5sum2`
echo $tmp1 > a
echo $tmp2 > b
diff a b //data corruption observed
if [ $? -ne 0 ]; then
echo "There are some date corruption, cnt is $cnt"
exit 1
fi
((cnt++))
rm -rf /mnt/fortest/bigfile
done
Kernel-Log:
[ 1113.577378] loop: module loaded
[ 1290.190065] md: bind<loop0>
[ 1290.193214] md: bind<loop1>
[ 1290.196387] md: bind<loop2>
[ 1290.199542] md: bind<loop3>
[ 1290.202704] md: bind<loop4>
[ 1290.205854] md: bind<loop5>
[ 1290.209003] md: bind<loop6>
[ 1290.212170] md: bind<loop7>
[ 1290.229799] md: raid1 personality registered for level 1
[ 1290.235946] md/raid1:md0: not clean -- starting background reconstruction
[ 1290.243515] md/raid1:md0: active with 8 out of 8 mirrors
[ 1290.249449] created bitmap (1 pages) for device md0
[ 1290.254927] md0: bitmap initialized from disk: read 1 pages, set 47 of 47 bits
[ 1290.328736] md0: detected capacity change from 0 to 3143630848
[ 1290.335316] md: resync of RAID array md0
[ 1290.339689] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 1290.346192] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 1290.356702] md: using 128k window, over a total of 3069952k.
[ 1640.101181] md: md0: resync done.
[ 1668.352287] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
[ 1681.845966] md/raid1:md0: Disk failure on loop0, disabling device.
[ 1681.845966] md/raid1:md0: Operation continuing on 7 devices.
[ 1844.296614] md: unbind<loop0>
[ 1844.302013] md: export_rdev(loop0)
[ 1874.363488] md: bind<loop0>
[ 1874.566435] md: recovery of RAID array md0
[ 1874.571006] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 1874.577514] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 1874.588224] md: using 128k window, over a total of 3069952k.
[ 1889.487210] md: md0: recovery done.
Test Log:
-----------------------------------------------------0
mdadm: set /dev/loop0 faulty in /dev/md0
mdadm: hot removed /dev/loop0 from /dev/md0
mdadm: re-added /dev/loop0
cp done
recovery done
-----------------------------------------------------1
mdadm: set /dev/loop0 faulty in /dev/md0
mdadm: hot removed /dev/loop0 from /dev/md0
mdadm: re-added /dev/loop0
cp done
recovery done
1c1
< c4eddcf325ba5741d37f164750412619
---
> 4444f8bbfb1d22f1731fb5b0c846ef8a
There are some date corruption, cnt is 1
Best Regards,
Yi Zhang
^ permalink raw reply
* Re: kernel BUG at drivers/scsi/scsi_lib.c:1101! observed during md5sum for one file on (RAID4->RAID0) device
From: James Bottomley @ 2015-07-30 13:28 UTC (permalink / raw)
To: Yi Zhang; +Cc: linux-raid, linux-scsi, xni, Jes.Sorensen, dm-devel
In-Reply-To: <1710310402.852769.1438246982906.JavaMail.zimbra@redhat.com>
On Thu, 2015-07-30 at 05:03 -0400, Yi Zhang wrote:
> Hi SCSI/RAID maintainer
>
> During raid test with 4.2.0-rc3, I observed below kernel BUG, pls check below info for the test log/environment/test steps.
>
> Log:
> [ 306.741662] md: bind<sdb1>
> [ 306.750865] md: bind<sdc1>
> [ 306.753993] md: bind<sdd1>
> [ 306.764475] md: bind<sde1>
> [ 306.786156] md: bind<sdf1>
> [ 306.789362] md: bind<sdh1>
> [ 306.792555] md: bind<sdg1>
> [ 306.868166] raid6: sse2x1 gen() 10589 MB/s
> [ 306.889143] raid6: sse2x1 xor() 8218 MB/s
> [ 306.910121] raid6: sse2x2 gen() 13453 MB/s
> [ 306.931102] raid6: sse2x2 xor() 8990 MB/s
> [ 306.952079] raid6: sse2x4 gen() 15539 MB/s
> [ 306.973063] raid6: sse2x4 xor() 10771 MB/s
> [ 306.994039] raid6: avx2x1 gen() 20582 MB/s
> [ 307.015017] raid6: avx2x2 gen() 24019 MB/s
> [ 307.035998] raid6: avx2x4 gen() 27824 MB/s
> [ 307.040755] raid6: using algorithm avx2x4 gen() 27824 MB/s
> [ 307.046869] raid6: using avx2x2 recovery algorithm
> [ 307.058793] async_tx: api initialized (async)
> [ 307.075428] xor: automatically using best checksumming function:
> [ 307.091942] avx : 32008.000 MB/sec
> [ 307.147662] md: raid6 personality registered for level 6
> [ 307.153584] md: raid5 personality registered for level 5
> [ 307.159505] md: raid4 personality registered for level 4
> [ 307.165698] md/raid:md0: device sdf1 operational as raid disk 4
> [ 307.172300] md/raid:md0: device sde1 operational as raid disk 3
> [ 307.178899] md/raid:md0: device sdd1 operational as raid disk 2
> [ 307.185497] md/raid:md0: device sdc1 operational as raid disk 1
> [ 307.192093] md/raid:md0: device sdb1 operational as raid disk 0
> [ 307.199052] md/raid:md0: allocated 6482kB
> [ 307.203573] md/raid:md0: raid level 4 active with 5 out of 6 devices, algorithm 0
> [ 307.211958] md0: detected capacity change from 0 to 53645148160
> [ 307.218658] md: recovery of RAID array md0
> [ 307.223226] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 307.229729] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> [ 307.240427] md: using 128k window, over a total of 10477568k.
> [ 374.670951] md: md0: recovery done.
> [ 375.722806] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
> [ 447.553364] md: unbind<sdh1>
> [ 447.559905] md: export_rdev(sdh1)
> [ 447.572684] md: cannot remove active disk sdg1 from md0 ...
> [ 447.578909] md/raid:md0: Disk failure on sdg1, disabling device.
> [ 447.578909] md/raid:md0: Operation continuing on 5 devices.
> [ 447.594850] md: unbind<sdg1>
> [ 447.601834] md: export_rdev(sdg1)
> [ 447.615446] md: raid0 personality registered for level 0
> [ 447.629275] md/raid0:md0: md_size is 104775680 sectors.
> [ 447.635094] md: RAID0 configuration for md0 - 1 zone
> [ 447.640627] md: zone0=[sdb1/sdc1/sdd1/sde1/sdf1]
> [ 447.645833] zone-offset= 0KB, device-offset= 0KB, size= 52387840KB
> [ 447.654949]
> [ 447.739443] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
> [ 447.749258] bio too big device sde1 (768 > 512)
This is the actual error. It looks like an md problem (md list copied).
> [ 447.754824] bio too big device sdf1 (1024 > 512)
> [ 447.759989] bio too big device sdb1 (768 > 512)
> [ 447.771102] bio too big device sdc1 (1024 > 512)
> [ 447.776276] bio too big device sdd1 (1024 > 512)
> [ 447.781459] bio too big device sde1 (1024 > 512)
> [ 447.786635] bio too big device sdf1 (768 > 512)
> [ 447.811156] bio too big device sdb1 (1024 > 512)
> [ 447.816329] bio too big device sdc1 (1024 > 512)
> [ 447.821513] bio too big device sdd1 (1024 > 512)
> [ 447.826681] bio too big device sde1 (768 > 512)
> [ 447.886106] bio too big device sdf1 (1024 > 512)
> [ 447.891269] bio too big device sdb1 (1024 > 512)
> [ 447.896452] bio too big device sdc1 (1024 > 512)
> [ 447.901628] bio too big device sdd1 (768 > 512)
> [ 447.930647] bio too big device sde1 (1024 > 512)
> [ 447.935820] bio too big device sdf1 (1024 > 512)
> [ 447.941003] bio too big device sdb1 (1024 > 512)
> [ 447.946179] bio too big device sdc1 (768 > 512)
> [ 447.976196] bio too big device sdd1 (1024 > 512)
> [ 447.981367] bio too big device sde1 (1024 > 512)
> [ 447.986549] bio too big device sdf1 (1024 > 512)
> [ 447.991728] bio too big device sdb1 (768 > 512)
> [ 448.033614] bio too big device sdc1 (1024 > 512)
> [ 448.038786] bio too big device sdd1 (1024 > 512)
> [ 448.043968] bio too big device sde1 (1024 > 512)
> [ 448.049145] bio too big device sdf1 (768 > 512)
> [ 448.083273] bio too big device sdb1 (1024 > 512)
> [ 448.088444] bio too big device sdc1 (1024 > 512)
> [ 448.093626] bio too big device sdd1 (1024 > 512)
> [ 448.098804] bio too big device sde1 (768 > 512)
> [ 448.128357] bio too big device sdf1 (1024 > 512)
> [ 448.133536] bio too big device sdb1 (1024 > 512)
> [ 448.138720] bio too big device sdc1 (1024 > 512)
> [ 448.143897] bio too big device sdd1 (768 > 512)
> [ 448.173456] bio too big device sde1 (1024 > 512)
> [ 448.178627] bio too big device sdf1 (1024 > 512)
> [ 448.183811] bio too big device sdb1 (1024 > 512)
> [ 448.188985] bio too big device sdc1 (768 > 512)
> [ 448.231050] bio too big device sdd1 (1024 > 512)
> [ 448.236221] bio too big device sde1 (1024 > 512)
> [ 448.241405] bio too big device sdf1 (1024 > 512)
> [ 448.246583] bio too big device sdb1 (768 > 512)
> [ 448.282548] bio too big device sdc1 (1024 > 512)
> [ 448.287719] bio too big device sdd1 (1024 > 512)
> [ 448.292904] bio too big device sde1 (1024 > 512)
> [ 448.298082] bio too big device sdf1 (768 > 512)
> [ 448.328300] bio too big device sdb1 (1024 > 512)
> [ 448.333471] bio too big device sdc1 (1024 > 512)
> [ 448.338654] bio too big device sdd1 (1024 > 512)
> [ 448.343830] bio too big device sde1 (768 > 512)
> [ 448.374081] bio too big device sdf1 (1024 > 512)
> [ 448.379250] bio too big device sdb1 (1024 > 512)
> [ 448.384433] bio too big device sdc1 (1024 > 512)
> [ 448.389609] bio too big device sdd1 (768 > 512)
> [ 448.394690] ------------[ cut here ]------------
> [ 448.399832] kernel BUG at drivers/scsi/scsi_lib.c:1095!
This bug on is here:
BUG_ON(count > sdb->table.nents);
It's merely enforcing with a BUG_ON what the warning was complaining
about.
James
> [ 448.405653] invalid opcode: 0000 [#1] SMP
> [ 448.410232] Modules linked in: raid0 ext4 mbcache jbd2 raid456 async_raid6_recov async_memcpy async_pq async_xor xor asyd
> [ 448.491371] CPU: 1 PID: 11918 Comm: md5sum Not tainted 4.2.0-rc3 #2
> [ 448.498354] Hardware name: Dell Inc. PowerEdge R730/0599V5, BIOS 1.2.10 03/09/2015
> [ 448.506791] task: ffff880461f28000 ti: ffff880462e08000 task.ti: ffff880462e08000
> [ 448.515130] RIP: 0010:[<ffffffff8146aaf2>] [<ffffffff8146aaf2>] scsi_init_sgtable+0x72/0x80
> [ 448.524548] RSP: 0018:ffff880462e0b8f8 EFLAGS: 00010002
> [ 448.530465] RAX: 0000000000000003 RBX: ffff8803fc03f980 RCX: 0000000000001000
> [ 448.538417] RDX: 0000000000000000 RSI: ffff8803fbb78040 RDI: 0000000000000000
> [ 448.546369] RBP: ffff880462e0b918 R08: ffff8803fbb78040 R09: 0000000000000000
> [ 448.554320] R10: 00000000000001f0 R11: ffffea000feede00 R12: ffff8803fba3b860
> [ 448.562272] R13: 0000000000000000 R14: ffff880461edc000 R15: ffff8803fc03f980
> [ 448.570224] FS: 00007f41ce7cc740(0000) GS:ffff88046d240000(0000) knlGS:0000000000000000
> [ 448.579242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 448.585644] CR2: 0000000000e0226f CR3: 0000000467aa3000 CR4: 00000000001406e0
> [ 448.593597] Stack:
> [ 448.595834] ffff880462e0b918 ffff8803fba3b780 ffff88046072a200 ffff880461edc000
> [ 448.604113] ffff880462e0b968 ffffffff8146ab4a ffff88046072aaf8 ffff8803fbb78000
> [ 448.612392] ffff8803fbb78000 ffff8803fc03f980 ffff88046072a260 ffff88046064ec00
> [ 448.620669] Call Trace:
> [ 448.623393] [<ffffffff8146ab4a>] scsi_init_io+0x4a/0x1c0
> [ 448.629410] [<ffffffffa004ed67>] sd_setup_read_write_cmnd+0x47/0xa40 [sd_mod]
> [ 448.637460] [<ffffffff81462a8b>] ? scsi_host_alloc_command+0x4b/0xc0
> [ 448.644638] [<ffffffffa00527d7>] sd_init_command+0x27/0xa0 [sd_mod]
> [ 448.651720] [<ffffffff8146adb1>] scsi_setup_cmnd+0xf1/0x160
> [ 448.658026] [<ffffffff8146af71>] scsi_prep_fn+0xd1/0x170
> [ 448.664042] [<ffffffff81309dac>] ? deadline_dispatch_requests+0xac/0x160
> [ 448.671609] [<ffffffff812ed683>] blk_peek_request+0x153/0x260
> [ 448.678110] [<ffffffff8146ca7f>] scsi_request_fn+0x3f/0x610
> [ 448.684416] [<ffffffff812e8c57>] __blk_run_queue+0x37/0x50
> [ 448.690626] [<ffffffff812e8cee>] queue_unplugged+0x2e/0xa0
> [ 448.696836] [<ffffffff812eda65>] blk_flush_plug_list+0x1b5/0x200
> [ 448.703626] [<ffffffff812ede14>] blk_finish_plug+0x34/0x50
> [ 448.709836] [<ffffffff8118cdfd>] __do_page_cache_readahead+0x1cd/0x240
> [ 448.717207] [<ffffffff8118cfb5>] ondemand_readahead+0x145/0x270
> [ 448.723903] [<ffffffff812209ba>] ? inode_congested+0xaa/0x110
> [ 448.730402] [<ffffffff8118d14c>] page_cache_async_readahead+0x6c/0x70
> [ 448.737677] [<ffffffff811811d3>] generic_file_read_iter+0x3c3/0x5e0
> [ 448.744760] [<ffffffff811f7569>] __vfs_read+0xc9/0x100
> [ 448.750582] [<ffffffff811f7b86>] vfs_read+0x86/0x130
> [ 448.756211] [<ffffffff811f8a15>] SyS_read+0x55/0xc0
> [ 448.761742] [<ffffffff81681b2e>] entry_SYSCALL_64_fastpath+0x12/0x71
> [ 448.768920] Code: ff 41 3b 44 24 08 77 23 41 89 44 24 08 8b 43 5c 41 89 44 24 10 48 83 c4 08 44 89 e8 5b 41 5c 41 5d 5d
> [ 448.790490] RIP [<ffffffff8146aaf2>] scsi_init_sgtable+0x72/0x80
> [ 448.797287] RSP <ffff880462e0b8f8>
> [ 448.801171] ---[ end trace fa7203c8f83678c8 ]---
> [ 448.853171] Kernel panic - not syncing: Fatal exception
> [ 448.859020] Kernel Offset: disabled
> [ 448.862904] drm_kms_helper: panic occurred, switching back to text console
> [ 448.920805] ---[ end Kernel panic - not syncing: Fatal exception
> [ 448.927513] ------------[ cut here ]------------
> [ 448.932661] WARNING: CPU: 1 PID: 11918 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5d/0x60()
> [ 448.943423] Modules linked in: raid0 ext4 mbcache jbd2 raid456 async_raid6_recov async_memcpy async_pq async_xor xor asyd
> [ 449.024578] CPU: 1 PID: 11918 Comm: md5sum Tainted: G D 4.2.0-rc3 #2
> [ 449.032918] Hardware name: Dell Inc. PowerEdge R730/0599V5, BIOS 1.2.10 03/09/2015
> [ 449.041353] 0000000000000000 00000000429195bb ffff88046d243d68 ffffffff8167acdd
> [ 449.049635] 0000000000000000 0000000000000000 ffff88046d243da8 ffffffff81081a4a
> [ 449.057917] ffff88046d243da8 0000000000000000 ffff88046d216780 0000000000000001
> [ 449.066198] Call Trace:
> [ 449.068918] <IRQ> [<ffffffff8167acdd>] dump_stack+0x45/0x57
> [ 449.075336] [<ffffffff81081a4a>] warn_slowpath_common+0x8a/0xc0
> [ 449.082030] [<ffffffff81081b7a>] warn_slowpath_null+0x1a/0x20
> [ 449.088530] [<ffffffff8104d56d>] native_smp_send_reschedule+0x5d/0x60
> [ 449.095805] [<ffffffff810be8e5>] trigger_load_balance+0x145/0x1f0
> [ 449.102693] [<ffffffff810ad486>] scheduler_tick+0xa6/0xe0
> [ 449.108807] [<ffffffff810f9bb0>] ? tick_sched_do_timer+0x50/0x50
> [ 449.115599] [<ffffffff810ea651>] update_process_times+0x51/0x60
> [ 449.122293] [<ffffffff810f9965>] tick_sched_handle.isra.17+0x25/0x60
> [ 449.129471] [<ffffffff810f9bf4>] tick_sched_timer+0x44/0x80
> [ 449.135779] [<ffffffff810eb1e3>] __hrtimer_run_queues+0xf3/0x220
> [ 449.142570] [<ffffffff810eb648>] hrtimer_interrupt+0xa8/0x1a0
> [ 449.149069] [<ffffffff810500b9>] local_apic_timer_interrupt+0x39/0x60
> [ 449.156345] [<ffffffff81684835>] smp_apic_timer_interrupt+0x45/0x60
> [ 449.163427] [<ffffffff816829cb>] apic_timer_interrupt+0x6b/0x70
> [ 449.170118] <EOI> [<ffffffff816755e3>] ? panic+0x1cc/0x20d
> [ 449.176435] [<ffffffff816755dc>] ? panic+0x1c5/0x20d
> [ 449.182065] [<ffffffff81019428>] oops_end+0xc8/0xe0
> [ 449.187595] [<ffffffff8101994b>] die+0x4b/0x70
> [ 449.192643] [<ffffffff81015e6d>] do_trap+0x13d/0x150
> [ 449.198272] [<ffffffff81016338>] do_error_trap+0xa8/0x170
> [ 449.204386] [<ffffffff8146aaf2>] ? scsi_init_sgtable+0x72/0x80
> [ 449.210983] [<ffffffff811821b5>] ? mempool_alloc_slab+0x15/0x20
> [ 449.217675] [<ffffffff811822f9>] ? mempool_alloc+0x69/0x170
> [ 449.223980] [<ffffffff81016850>] do_invalid_op+0x20/0x30
> [ 449.229996] [<ffffffff8168348e>] invalid_op+0x1e/0x30
> [ 449.235721] [<ffffffff8146aaf2>] ? scsi_init_sgtable+0x72/0x80
> [ 449.242317] [<ffffffff8146aac8>] ? scsi_init_sgtable+0x48/0x80
> [ 449.248912] [<ffffffff8146ab4a>] scsi_init_io+0x4a/0x1c0
> [ 449.254930] [<ffffffffa004ed67>] sd_setup_read_write_cmnd+0x47/0xa40 [sd_mod]
> [ 449.262979] [<ffffffff81462a8b>] ? scsi_host_alloc_command+0x4b/0xc0
> [ 449.270157] [<ffffffffa00527d7>] sd_init_command+0x27/0xa0 [sd_mod]
> [ 449.277239] [<ffffffff8146adb1>] scsi_setup_cmnd+0xf1/0x160
> [ 449.283544] [<ffffffff8146af71>] scsi_prep_fn+0xd1/0x170
> [ 449.289561] [<ffffffff81309dac>] ? deadline_dispatch_requests+0xac/0x160
> [ 449.297128] [<ffffffff812ed683>] blk_peek_request+0x153/0x260
> [ 449.303628] [<ffffffff8146ca7f>] scsi_request_fn+0x3f/0x610
> [ 449.309933] [<ffffffff812e8c57>] __blk_run_queue+0x37/0x50
> [ 449.316142] [<ffffffff812e8cee>] queue_unplugged+0x2e/0xa0
> [ 449.322351] [<ffffffff812eda65>] blk_flush_plug_list+0x1b5/0x200
> [ 449.329142] [<ffffffff812ede14>] blk_finish_plug+0x34/0x50
> [ 449.335351] [<ffffffff8118cdfd>] __do_page_cache_readahead+0x1cd/0x240
> [ 449.342722] [<ffffffff8118cfb5>] ondemand_readahead+0x145/0x270
> [ 449.349416] [<ffffffff812209ba>] ? inode_congested+0xaa/0x110
> [ 449.355916] [<ffffffff8118d14c>] page_cache_async_readahead+0x6c/0x70
> [ 449.363190] [<ffffffff811811d3>] generic_file_read_iter+0x3c3/0x5e0
> [ 449.370273] [<ffffffff811f7569>] __vfs_read+0xc9/0x100
> [ 449.376094] [<ffffffff811f7b86>] vfs_read+0x86/0x130
> [ 449.381723] [<ffffffff811f8a15>] SyS_read+0x55/0xc0
> [ 449.387254] [<ffffffff81681b2e>] entry_SYSCALL_64_fastpath+0x12/0x71
> [ 449.394432] ---[ end trace fa7203c8f83678c9 ]---
>
>
> Environment: 4.2.0-rc3
> [root@storageqe-09 ~]# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sdb 8:16 0 931.5G 0 disk
> └─sdb1 8:17 0 10G 0 part
> sdc 8:32 0 931.5G 0 disk
> └─sdc1 8:33 0 10G 0 part
> sdd 8:48 0 931.5G 0 disk
> └─sdd1 8:49 0 10G 0 part
> sde 8:64 0 931.5G 0 disk
> └─sde1 8:65 0 10G 0 part
> sdf 8:80 0 931.5G 0 disk
> └─sdf1 8:81 0 10G 0 part
> sdg 8:96 0 3.7T 0 disk
> └─sdg1 8:97 0 10G 0 part
> sdh 8:112 0 3.7T 0 disk
> └─sdh1 8:113 0 10G 0 part
>
> Reproduce-steps:
> While [ 1 ]
> do
> mdadm --create --run /dev/md0 --level 4 --metadata 1.2 --raid-devices 6 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 --spare-devices 1 /dev/sdh1 --chunk 512
> mdadm --wait /dev/md0
> mkfs -t ext4 /dev/md0
> mkdir /mnt/md_test
> mount /dev/md0 /mnt/md_test
> dd if=/dev/urandom of=/mnt/md_test/testfile bs=1M count=1000
> md5sum /mnt/md_test/testfile > md5.old
> umount /dev/md0
> mdadm --grow -l0 /dev/md0 --backup-file=tmp0
> mdadm --wait /dev/md0
> mount /dev/md0 /mnt/md_test
> md5sum /mnt/md_test/testfile >md5.new // kernel BUG at drivers/scsi/scsi_lib.c:1101!
> umount /dev/md0
> mdadm -Ss
> mdadm --zero-superblock /dev/sd[bcdefgh]1
> done
>
>
> Best Regards,
> Yi Zhang
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: raid5 regression in 4.1.x
From: NeilBrown @ 2015-07-30 22:57 UTC (permalink / raw)
To: Tomas Papan; +Cc: linux-raid
In-Reply-To: <CAMGsXDSySVJA3PFo=v+_hWS+4K9VmuHMUEb3hSNtxqRG0P4=8A@mail.gmail.com>
On Wed, 29 Jul 2015 09:21:22 +0200 Tomas Papan <tomas.papan@gmail.com>
wrote:
> Hello there,
>
> I found a bug on my system. I do not know if this was already
> reported. I can't find it anywhere.
> Would you please have a look at this trace (attached)
>
>
>
> Unfortunately it happens completely random (after few days in this
> case, with 4.1.0 it was in matter of hours) and I'm not able to bisect
> this.
>
> Regards
> Tomas
Thanks for the report.
The crash is happening in get_free_stripe() and the 'hash' argument is
stored in register RSI and has (32bit) value 0xFFFFFFFF. i.e. -1.
That shouldn't happen - it should be from 0 to 7.
get_free_stripe was called from drop_one_stripe():
int hash = (conf->max_nr_stripes - 1) % NR_STRIPE_HASH_LOCKS;
spin_lock_irq(conf->hash_locks + hash);
sh = get_free_stripe(conf, hash);
so it seems that conf->max-nr_stripes was 0, which it really should
never be.
There is a patch submitted for 4.2 which should get into -stable a
couple of weeks later which might help.
If you would like to try just applying that patch and see if the
problem goes away, it is
http://git.neil.brown.name/?p=md.git;a=commitdiff;h=2d5b569b665ea6d0b15c52529ff06300de81a7ce
or
http://git.neil.brown.name/?p=md.git;a=commitdiff_plain;h=2d5b569b665ea6d0b15c52529ff06300de81a7ce
I'll have another look at the code a bit later and see if I can
convince myself what happened.
Thanks,
NeilBrown
^ permalink raw reply
* Re: Subject: ANNOUNCE: mdadm 3.3.3 - A tool for managing md Soft RAID under Linux
From: NeilBrown @ 2015-07-30 23:34 UTC (permalink / raw)
To: Piergiorgio Sartor; +Cc: linux-raid
In-Reply-To: <20150727162906.GA3051@lazy.lzy>
On Mon, 27 Jul 2015 18:29:06 +0200 Piergiorgio Sartor
<piergiorgio.sartor@nexgo.de> wrote:
> On Fri, Jul 24, 2015 at 04:35:30PM +1000, NeilBrown wrote:
> >
> >
> > I am pleased to announce the availability of
> > mdadm version 3.3.3
> >
> > It is available at the usual places:
> > http://www.kernel.org/pub/linux/utils/raid/mdadm/
> > and via git at
> > git://github.com/neilbrown/mdadm
> > git://neil.brown.name/mdadm
> > http://git.neil.brown.name/git/mdadm.git
> >
> > The 100 changes since 3.3.3 are mostly little bugfixes and some improvements
> > to the selftests.
> > raid6check now handle all RAID6 layouts including DDF correctly.
> > See git log for the rest.
> >
> > NeilBrown 24th July 2015
>
> Hi Neil,
>
> I would like to thank you for clean up and improving
> "raid6check.c".
> I personally appreciate your contribution very much,
> and I think it is good to have your expert eye
> looking at the code.
>
> Thanks again for your time,
>
> bye,
>
Happy to help, but I should confess that the motivation was purely
selfish.
I was doing some testing and the 01raid6integ test (which fails
different devices in a RAID6 and checks the data is still correct) was
occasionally reporting errors - a bit of a worry.
It eventually turned out to be a hardware issue - I was running in VMs
and after power-cycling my workstation the problem doesn't appear any
more.
But before I got to the stage I wanted to get a clearer idea of what
sort of corruption was happening, so I added a call to raid6check to
the test and it reported really strange results for some layouts.
So in order to get the data I needed, I fixed the code to work properly
for all layout. i.e. write a test script to test all layouts, then fix
things until the script stopped complaining :-)
So despite that fact that it didn't actually lead me to the cause:
thank you for providing the program in the first place :-)
Thanks,
NeilBrown
^ permalink raw reply
* Re: raid1_end_read_request does not retry failed READ from a recovering drive
From: NeilBrown @ 2015-07-31 0:12 UTC (permalink / raw)
To: Alexander Lyakas; +Cc: linux-raid
In-Reply-To: <CAGRgLy5rE6Yuhcw2jA+4ymv5U489vX_NvSpNAuy0OPbcKnatFA@mail.gmail.com>
On Mon, 27 Jul 2015 10:07:37 +0200 Alexander Lyakas
<alex.bolshoy@gmail.com> wrote:
> Hi Neil,
> Thanks for the comments. Hopefully now we have the complete fix.
> I am posting what we have applied on top of 3.8.13 (now we are moving
> to 3.18.19, which already has part of the fix, so I will need to apply
> a delta, until both your latest patches reach Mr. Stable). Locking the
> spinlock in "error" function is not there (but I will apply it to
> 3.18.19).
>
> Below patch is a bit ugly:
> - CONFIG_MD_ZADARA is a define that we add, so that every engineer can
> clearly distinguish our changes vs the original code. I know it's
> ugly.
> - zklog is a macro that eventually ends up in printk. This is only to
> have a bit more prints, and is not functionally needed. zklog also
> prints current->pid, function name etc, to have more context.
>
> Hopefully, gmail will not make the patch even uglier.
>
> Thanks for your help!
> Alex.
>
Thanks for that Alex - I think we are one the same page now :-)
Thanks,
NeilBrown
^ permalink raw reply
* Re: [dm-devel] kernel BUG at drivers/scsi/scsi_lib.c:1101! observed during md5sum for one file on (RAID4->RAID0) device
From: NeilBrown @ 2015-07-31 1:20 UTC (permalink / raw)
To: James Bottomley
Cc: Yi Zhang, linux-raid, Jes.Sorensen, xni, dm-devel, linux-scsi
In-Reply-To: <1438262886.2229.1.camel@HansenPartnership.com>
On Thu, 30 Jul 2015 06:28:06 -0700 James Bottomley
<James.Bottomley@HansenPartnership.com> wrote:
> On Thu, 2015-07-30 at 05:03 -0400, Yi Zhang wrote:
> > Hi SCSI/RAID maintainer
> >
> > During raid test with 4.2.0-rc3, I observed below kernel BUG, pls check below info for the test log/environment/test steps.
> >
> > Log:
> > [ 306.741662] md: bind<sdb1>
> > [ 306.750865] md: bind<sdc1>
> > [ 306.753993] md: bind<sdd1>
> > [ 306.764475] md: bind<sde1>
> > [ 306.786156] md: bind<sdf1>
> > [ 306.789362] md: bind<sdh1>
> > [ 306.792555] md: bind<sdg1>
> > [ 306.868166] raid6: sse2x1 gen() 10589 MB/s
> > [ 306.889143] raid6: sse2x1 xor() 8218 MB/s
> > [ 306.910121] raid6: sse2x2 gen() 13453 MB/s
> > [ 306.931102] raid6: sse2x2 xor() 8990 MB/s
> > [ 306.952079] raid6: sse2x4 gen() 15539 MB/s
> > [ 306.973063] raid6: sse2x4 xor() 10771 MB/s
> > [ 306.994039] raid6: avx2x1 gen() 20582 MB/s
> > [ 307.015017] raid6: avx2x2 gen() 24019 MB/s
> > [ 307.035998] raid6: avx2x4 gen() 27824 MB/s
> > [ 307.040755] raid6: using algorithm avx2x4 gen() 27824 MB/s
> > [ 307.046869] raid6: using avx2x2 recovery algorithm
> > [ 307.058793] async_tx: api initialized (async)
> > [ 307.075428] xor: automatically using best checksumming function:
> > [ 307.091942] avx : 32008.000 MB/sec
> > [ 307.147662] md: raid6 personality registered for level 6
> > [ 307.153584] md: raid5 personality registered for level 5
> > [ 307.159505] md: raid4 personality registered for level 4
> > [ 307.165698] md/raid:md0: device sdf1 operational as raid disk 4
> > [ 307.172300] md/raid:md0: device sde1 operational as raid disk 3
> > [ 307.178899] md/raid:md0: device sdd1 operational as raid disk 2
> > [ 307.185497] md/raid:md0: device sdc1 operational as raid disk 1
> > [ 307.192093] md/raid:md0: device sdb1 operational as raid disk 0
> > [ 307.199052] md/raid:md0: allocated 6482kB
> > [ 307.203573] md/raid:md0: raid level 4 active with 5 out of 6 devices, algorithm 0
> > [ 307.211958] md0: detected capacity change from 0 to 53645148160
> > [ 307.218658] md: recovery of RAID array md0
> > [ 307.223226] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> > [ 307.229729] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> > [ 307.240427] md: using 128k window, over a total of 10477568k.
> > [ 374.670951] md: md0: recovery done.
> > [ 375.722806] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
> > [ 447.553364] md: unbind<sdh1>
> > [ 447.559905] md: export_rdev(sdh1)
> > [ 447.572684] md: cannot remove active disk sdg1 from md0 ...
> > [ 447.578909] md/raid:md0: Disk failure on sdg1, disabling device.
> > [ 447.578909] md/raid:md0: Operation continuing on 5 devices.
> > [ 447.594850] md: unbind<sdg1>
> > [ 447.601834] md: export_rdev(sdg1)
> > [ 447.615446] md: raid0 personality registered for level 0
> > [ 447.629275] md/raid0:md0: md_size is 104775680 sectors.
> > [ 447.635094] md: RAID0 configuration for md0 - 1 zone
> > [ 447.640627] md: zone0=[sdb1/sdc1/sdd1/sde1/sdf1]
> > [ 447.645833] zone-offset= 0KB, device-offset= 0KB, size= 52387840KB
> > [ 447.654949]
> > [ 447.739443] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
> > [ 447.749258] bio too big device sde1 (768 > 512)
>
> This is the actual error. It looks like an md problem (md list copied).
Thanks. It certainly does look like an md problem.... ah, found it.
level_store in drivers/md/md.c calls blk_set_stacking_limits after
calling ->takeover and before calling ->run.
->run should impose the limits from the underlying device, but for
RAID0, ->takeover is doing that.
I can fix that... hopefully it will become irrelevant soon when the
immutable-bio patches go in.
This patch isn't quite right, but it should be pretty close.
Can you test and confirm?
Thanks,
NeilBrown
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index efb654eb5399..17804f374709 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -83,7 +83,6 @@ static int create_strip_zones(struct mddev *mddev,
struct r0conf **private_conf) char b[BDEVNAME_SIZE];
char b2[BDEVNAME_SIZE];
struct r0conf *conf = kzalloc(sizeof(*conf), GFP_KERNEL);
- bool discard_supported = false;
if (!conf)
return -ENOMEM;
@@ -188,19 +187,12 @@ static int create_strip_zones(struct mddev
*mddev, struct r0conf **private_conf) }
dev[j] = rdev1;
- if (mddev->queue)
- disk_stack_limits(mddev->gendisk, rdev1->bdev,
- rdev1->data_offset << 9);
-
if (rdev1->bdev->bd_disk->queue->merge_bvec_fn)
conf->has_merge_bvec = 1;
if (!smallest || (rdev1->sectors < smallest->sectors))
smallest = rdev1;
cnt++;
-
- if (blk_queue_discard(bdev_get_queue(rdev1->bdev)))
- discard_supported = true;
}
if (cnt != mddev->raid_disks) {
printk(KERN_ERR "md/raid0:%s: too few disks (%d of %d)
- " @@ -272,17 +264,6 @@ static int create_strip_zones(struct mddev
*mddev, struct r0conf **private_conf) goto abort;
}
- if (mddev->queue) {
- blk_queue_io_min(mddev->queue, mddev->chunk_sectors <<
9);
- blk_queue_io_opt(mddev->queue,
- (mddev->chunk_sectors << 9) *
mddev->raid_disks); -
- if (!discard_supported)
- queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD,
mddev->queue);
- else
- queue_flag_set_unlocked(QUEUE_FLAG_DISCARD,
mddev->queue);
- }
-
pr_debug("md/raid0:%s: done.\n", mdname(mddev));
*private_conf = conf;
@@ -433,12 +414,6 @@ static int raid0_run(struct mddev *mddev)
if (md_check_no_bitmap(mddev))
return -EINVAL;
- if (mddev->queue) {
- blk_queue_max_hw_sectors(mddev->queue,
mddev->chunk_sectors);
- blk_queue_max_write_same_sectors(mddev->queue,
mddev->chunk_sectors);
- blk_queue_max_discard_sectors(mddev->queue,
mddev->chunk_sectors);
- }
-
/* if private is not null, we are here after takeover */
if (mddev->private == NULL) {
ret = create_strip_zones(mddev, &conf);
@@ -447,6 +422,29 @@ static int raid0_run(struct mddev *mddev)
mddev->private = conf;
}
conf = mddev->private;
+ if (mddev->queue) {
+ struct md_rdev *rdev;
+ bool discard_supported = false;
+
+ rdev_for_each(rdev, mddev) {
+ disk_stack_limits(mddev->gendisk, rdev->bdev,
+ rdev->data_offset << 9);
+ if
(blk_queue_discard(bdev_get_queue(rdev->bdev)))
+ discard_supported = true;
+ }
+ blk_queue_max_hw_sectors(mddev->queue,
mddev->chunk_sectors);
+ blk_queue_max_write_same_sectors(mddev->queue,
mddev->chunk_sectors);
+ blk_queue_max_discard_sectors(mddev->queue,
mddev->chunk_sectors); +
+ blk_queue_io_min(mddev->queue, mddev->chunk_sectors <<
9);
+ blk_queue_io_opt(mddev->queue,
+ (mddev->chunk_sectors << 9) *
mddev->raid_disks); +
+ if (!discard_supported)
+ queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD,
mddev->queue);
+ else
+ queue_flag_set_unlocked(QUEUE_FLAG_DISCARD,
mddev->queue);
+ }
/* calculate array device size */
md_set_array_sectors(mddev, raid0_size(mddev, 0, 0));
^ permalink raw reply related
* [PATCH 06/15] drivers: md: Drop unlikely before IS_ERR(_OR_NULL)
From: Viresh Kumar @ 2015-07-31 8:38 UTC (permalink / raw)
To: akpm
Cc: linaro-kernel, linux-kernel, Viresh Kumar, Alasdair Kergon,
maintainer:DEVICE-MAPPER LVM,
open list:SOFTWARE RAID Multiple Disks SUPPORT, Mike Snitzer,
Neil Brown
In-Reply-To: <cover.1438331416.git.viresh.kumar@linaro.org>
IS_ERR(_OR_NULL) already contain an 'unlikely' compiler flag and there
is no need to do that again from its callers. Drop it.
Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
---
drivers/md/dm-snap-persistent.c | 2 +-
drivers/md/dm-verity.c | 2 +-
drivers/md/persistent-data/dm-block-manager.c | 8 ++++----
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persistent.c
index 808b8419bc48..bf71583296f7 100644
--- a/drivers/md/dm-snap-persistent.c
+++ b/drivers/md/dm-snap-persistent.c
@@ -533,7 +533,7 @@ static int read_exceptions(struct pstore *ps,
chunk = area_location(ps, ps->current_area);
area = dm_bufio_read(client, chunk, &bp);
- if (unlikely(IS_ERR(area))) {
+ if (IS_ERR(area)) {
r = PTR_ERR(area);
goto ret_destroy_bufio;
}
diff --git a/drivers/md/dm-verity.c b/drivers/md/dm-verity.c
index bb9c6a00e4b0..1d131519ed21 100644
--- a/drivers/md/dm-verity.c
+++ b/drivers/md/dm-verity.c
@@ -271,7 +271,7 @@ static int verity_verify_level(struct dm_verity_io *io, sector_t block,
verity_hash_at_level(v, block, level, &hash_block, &offset);
data = dm_bufio_read(v->bufio, hash_block, &buf);
- if (unlikely(IS_ERR(data)))
+ if (IS_ERR(data))
return PTR_ERR(data);
aux = dm_bufio_get_aux_data(buf);
diff --git a/drivers/md/persistent-data/dm-block-manager.c b/drivers/md/persistent-data/dm-block-manager.c
index 4d6c9b689eaa..88dbe7b97c2c 100644
--- a/drivers/md/persistent-data/dm-block-manager.c
+++ b/drivers/md/persistent-data/dm-block-manager.c
@@ -454,7 +454,7 @@ int dm_bm_read_lock(struct dm_block_manager *bm, dm_block_t b,
int r;
p = dm_bufio_read(bm->bufio, b, (struct dm_buffer **) result);
- if (unlikely(IS_ERR(p)))
+ if (IS_ERR(p))
return PTR_ERR(p);
aux = dm_bufio_get_aux_data(to_buffer(*result));
@@ -490,7 +490,7 @@ int dm_bm_write_lock(struct dm_block_manager *bm,
return -EPERM;
p = dm_bufio_read(bm->bufio, b, (struct dm_buffer **) result);
- if (unlikely(IS_ERR(p)))
+ if (IS_ERR(p))
return PTR_ERR(p);
aux = dm_bufio_get_aux_data(to_buffer(*result));
@@ -523,7 +523,7 @@ int dm_bm_read_try_lock(struct dm_block_manager *bm,
int r;
p = dm_bufio_get(bm->bufio, b, (struct dm_buffer **) result);
- if (unlikely(IS_ERR(p)))
+ if (IS_ERR(p))
return PTR_ERR(p);
if (unlikely(!p))
return -EWOULDBLOCK;
@@ -559,7 +559,7 @@ int dm_bm_write_lock_zero(struct dm_block_manager *bm,
return -EPERM;
p = dm_bufio_new(bm->bufio, b, (struct dm_buffer **) result);
- if (unlikely(IS_ERR(p)))
+ if (IS_ERR(p))
return PTR_ERR(p);
memset(p, 0, dm_bm_block_size(bm));
--
2.4.0
^ permalink raw reply related
* Re: raid5 regression in 4.1.x
From: Tomas Papan @ 2015-07-31 8:52 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150731085724.025f3b2b@noble>
Hi Neil,
Thanks for the explanation, I've installed the 4.2.0-rc4 (which has
your patch in it).
It booted and so far dmesg looks clean. I'll report back in couple of days.
Thanks
Tomas
On Fri, Jul 31, 2015 at 12:57 AM, NeilBrown <neilb@suse.com> wrote:
> On Wed, 29 Jul 2015 09:21:22 +0200 Tomas Papan <tomas.papan@gmail.com>
> wrote:
>
>> Hello there,
>>
>> I found a bug on my system. I do not know if this was already
>> reported. I can't find it anywhere.
>> Would you please have a look at this trace (attached)
>>
>>
>>
>> Unfortunately it happens completely random (after few days in this
>> case, with 4.1.0 it was in matter of hours) and I'm not able to bisect
>> this.
>>
>> Regards
>> Tomas
>
> Thanks for the report.
>
> The crash is happening in get_free_stripe() and the 'hash' argument is
> stored in register RSI and has (32bit) value 0xFFFFFFFF. i.e. -1.
> That shouldn't happen - it should be from 0 to 7.
>
> get_free_stripe was called from drop_one_stripe():
>
> int hash = (conf->max_nr_stripes - 1) % NR_STRIPE_HASH_LOCKS;
>
> spin_lock_irq(conf->hash_locks + hash);
> sh = get_free_stripe(conf, hash);
>
> so it seems that conf->max-nr_stripes was 0, which it really should
> never be.
>
> There is a patch submitted for 4.2 which should get into -stable a
> couple of weeks later which might help.
>
> If you would like to try just applying that patch and see if the
> problem goes away, it is
>
> http://git.neil.brown.name/?p=md.git;a=commitdiff;h=2d5b569b665ea6d0b15c52529ff06300de81a7ce
>
> or
>
> http://git.neil.brown.name/?p=md.git;a=commitdiff_plain;h=2d5b569b665ea6d0b15c52529ff06300de81a7ce
>
> I'll have another look at the code a bit later and see if I can
> convince myself what happened.
>
> Thanks,
> NeilBrown
^ permalink raw reply
* Re: [V2 PATCH] Safeguard against writing to an active device of another node
From: Guoqing Jiang @ 2015-07-31 9:04 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: neilb, rgoldwyn, linux-raid
In-Reply-To: <55BA097A.3090400@suse.com>
Goldwyn Rodrigues wrote:
>
>
> On 07/30/2015 03:49 AM, Guoqing Jiang wrote:
>> Modifying an exiting device's superblock or creating a new superblock
>> on an existing device needs to be checked because the device could be
>> in use by another node in another array. So, we check this by taking
>> all superblock locks in userspace so that we don't step onto an active
>> device used by another node and safeguard against accidental edits.
>> After the edit is complete, we release all locks and the lockspace so
>> that it can be used by the kernel space.
>>
>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
>> ---
>> V2 Changes: use run-time check for dlm library
>
> This may be better done using a structure of function pointers which
> is setup and resolved one-time as opposed to performing symbol
> resolution every time the function is called. You could include
> get_cluster_name as well in it.
Good point, I will send another incremental patch for this.
Thanks,
Guoqing
^ permalink raw reply
* Thoughts on big SSD arrays?
From: Matt Garman @ 2015-07-31 15:23 UTC (permalink / raw)
To: Mdadm
Every few years I reprise this topic on this mailing list[1], [2].
Basically I'm just brainstorming what is possible on the DIY front
versus purchased solutions from a traditional "big iron" storage
vendor. Our particular use case is "ultra-high parallel sequential
read throughput". Our workload is effectively WORM: we do a small
daily incremental write, and then the rest of the time it's constant
re-reading of the data. Literally 99:1 read:write
I continue to be inspired by the "Dirt Cheap Data Warehouse (DCDW)"
[3]. SSD are getting bigger and prices are dropping rapidly (2 TB
SSDs available now for $800). With our WORM-like workload, I believe
we can safely get away with consumer drives, as durability shouldn't
be an issue.
So at this point I'm just putting out a feeler---has anyone out there
actually built a massive SSD array, using either Linux software raid
or hardware raid (technically off-topic for this list, though I hope
the discussion is interesting enough to let it slide). If so, how big
of an array (i.e. drives/capacity)? What was the target versus actual
performance? Any particularly challenging issues that came up?
FWIW, I'm thinking of something along the lines of a 24-disk chassis,
with 2 disks for OS (raid1), 2 disks as hot spares, and the remaining
20 in raid-6. The 22 data disks (raid + hot spares) would be 2 TB
SSDs.
The "problem" with SSDs is that they're just so seductive:
back-of-the-envelope numbers are wonderful, so it's easy to get
overly-optimistic about builds that use them. But as with most
things, the devil's in the details.
Off the top of my head, potential issues I can think of:
- Subtle PCIe latency/timing issues of the motherboard
- High variation in SSD latency
- Software stacks still making assumptions based on spinning
drives (i.e. not adequately tuned for SSDs)
- Non-parallel RAID implementation (i.e. single CPU bottleneck potential)
- Potential bandwidth bottlenecks at various stages: SATA/SAS
interface, SAS expander/backplane, SATA/SAS controller (or HBA), PCIe
bus, CPU memory bus, network card, etc
- I forget the exact number, but the DCDW guy told me with Linux
he was only able to get about 30% of the predicted throughput in his
SSD array
- Wacky TRIM related issues (seem to be drive dependent)
Not asking any particular question here, just hoping to start an
open-ended discussion. Of course I'd love to hear from anyone with
actual SSD RAID experience!
Thanks,
Matt
[1] "high throughput storage server?", Feb 14, 2011
http://marc.info/?l=linux-raid&m=129772818924753&w=2
[2] "high read throughput storage server, take 2"
http://marc.info/?l=linux-raid&m=138359009013781&w=2
[3] "The Dirt Cheap Data Warehouse"
http://www.openida.com/the-dirt-cheap-data-warehouse-an-introduction/
^ permalink raw reply
* Re: Thoughts on big SSD arrays?
From: Pasi Kärkkäinen @ 2015-08-01 8:34 UTC (permalink / raw)
To: Matt Garman; +Cc: Mdadm
In-Reply-To: <CAJvUf-AW08pcRkADn4Fgh46_vVAxgofiuB93+pzQYk99hx1nbw@mail.gmail.com>
On Fri, Jul 31, 2015 at 10:23:26AM -0500, Matt Garman wrote:
> Every few years I reprise this topic on this mailing list[1], [2].
> Basically I'm just brainstorming what is possible on the DIY front
> versus purchased solutions from a traditional "big iron" storage
> vendor. Our particular use case is "ultra-high parallel sequential
> read throughput". Our workload is effectively WORM: we do a small
> daily incremental write, and then the rest of the time it's constant
> re-reading of the data. Literally 99:1 read:write
>
> I continue to be inspired by the "Dirt Cheap Data Warehouse (DCDW)"
> [3]. SSD are getting bigger and prices are dropping rapidly (2 TB
> SSDs available now for $800). With our WORM-like workload, I believe
> we can safely get away with consumer drives, as durability shouldn't
> be an issue.
>
> So at this point I'm just putting out a feeler---has anyone out there
> actually built a massive SSD array, using either Linux software raid
> or hardware raid (technically off-topic for this list, though I hope
> the discussion is interesting enough to let it slide). If so, how big
> of an array (i.e. drives/capacity)? What was the target versus actual
> performance? Any particularly challenging issues that came up?
>
> FWIW, I'm thinking of something along the lines of a 24-disk chassis,
> with 2 disks for OS (raid1), 2 disks as hot spares, and the remaining
> 20 in raid-6. The 22 data disks (raid + hot spares) would be 2 TB
> SSDs.
>
Also remember raid rebuilds after SSD failures.. with 20 disks in the same raid6-set,
you'll have a lot of reads going on during rebuild :)
-- Pasi
> The "problem" with SSDs is that they're just so seductive:
> back-of-the-envelope numbers are wonderful, so it's easy to get
> overly-optimistic about builds that use them. But as with most
> things, the devil's in the details.
>
> Off the top of my head, potential issues I can think of:
>
> - Subtle PCIe latency/timing issues of the motherboard
> - High variation in SSD latency
> - Software stacks still making assumptions based on spinning
> drives (i.e. not adequately tuned for SSDs)
> - Non-parallel RAID implementation (i.e. single CPU bottleneck potential)
> - Potential bandwidth bottlenecks at various stages: SATA/SAS
> interface, SAS expander/backplane, SATA/SAS controller (or HBA), PCIe
> bus, CPU memory bus, network card, etc
> - I forget the exact number, but the DCDW guy told me with Linux
> he was only able to get about 30% of the predicted throughput in his
> SSD array
> - Wacky TRIM related issues (seem to be drive dependent)
>
> Not asking any particular question here, just hoping to start an
> open-ended discussion. Of course I'd love to hear from anyone with
> actual SSD RAID experience!
>
> Thanks,
> Matt
>
>
> [1] "high throughput storage server?", Feb 14, 2011
> http://marc.info/?l=linux-raid&m=129772818924753&w=2
>
> [2] "high read throughput storage server, take 2"
> http://marc.info/?l=linux-raid&m=138359009013781&w=2
>
> [3] "The Dirt Cheap Data Warehouse"
> http://www.openida.com/the-dirt-cheap-data-warehouse-an-introduction/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 3/3] Safeguard against writing to an active device of another node
From: Wols Lists @ 2015-08-01 15:50 UTC (permalink / raw)
To: Goldwyn Rodrigues, NeilBrown, Guoqing Jiang; +Cc: linux-raid
In-Reply-To: <55B96474.6080604@suse.de>
On 30/07/15 00:40, Goldwyn Rodrigues wrote:
>>
>> I was wondering if we could make it a run-time dependency though .. a
>> bit like get_cluster_name. We can still do that later I guess. I'm
>> not really sure what is best at the moment.
>>
>
>
> Yes, I would second that: Making these functions a run-time dependency.
> We should have the ability for it to work by just installing libdlm
> (with the proper flags set), rather than recompiling the package for
> cluster features.
Or - maybe unlikely but maybe not - what happens if somebody installs a
binary that was built on a system WITH dlm, but is meant to run on a
system WITHOUT dlm?
Actually, wouldn't that be the typical case for a typical distro?
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH 3/3] Safeguard against writing to an active device of another node
From: Goldwyn Rodrigues @ 2015-08-01 15:54 UTC (permalink / raw)
To: Wols Lists, NeilBrown, Guoqing Jiang; +Cc: linux-raid
In-Reply-To: <55BCEABF.6060505@youngman.org.uk>
On 08/01/2015 10:50 AM, Wols Lists wrote:
> On 30/07/15 00:40, Goldwyn Rodrigues wrote:
>>>
>>> I was wondering if we could make it a run-time dependency though .. a
>>> bit like get_cluster_name. We can still do that later I guess. I'm
>>> not really sure what is best at the moment.
>>>
>>
>>
>> Yes, I would second that: Making these functions a run-time dependency.
>> We should have the ability for it to work by just installing libdlm
>> (with the proper flags set), rather than recompiling the package for
>> cluster features.
>
> Or - maybe unlikely but maybe not - what happens if somebody installs a
> binary that was built on a system WITH dlm, but is meant to run on a
> system WITHOUT dlm?
>
> Actually, wouldn't that be the typical case for a typical distro?
>
That's exactly the case we are trying to cover in the above explanation.
IOw, we would be using -ldl for runtime compatibility. If libdlm is not
present, cluster functions will be resolved to null and will not be
executed/allowed.
--
Goldwyn
^ permalink raw reply
* AW: Thoughts on big SSD arrays?
From: Markus Stockhausen @ 2015-08-03 10:52 UTC (permalink / raw)
To: Matt Garman, Mdadm
In-Reply-To: <CAJvUf-AW08pcRkADn4Fgh46_vVAxgofiuB93+pzQYk99hx1nbw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2848 bytes --]
> Von: linux-raid-owner@vger.kernel.org [linux-raid-owner@vger.kernel.org]" im Auftrag von "Matt Garman [matthew.garman@gmail.com]
> Gesendet: Freitag, 31. Juli 2015 17:23
> An: Mdadm
> Betreff: Thoughts on big SSD arrays?
>
> Every few years I reprise this topic on this mailing list[1], [2].
> Basically I'm just brainstorming what is possible on the DIY front
> versus purchased solutions from a traditional "big iron" storage
> vendor. Our particular use case is "ultra-high parallel sequential
> read throughput". Our workload is effectively WORM: we do a small
> daily incremental write, and then the rest of the time it's constant
> re-reading of the data. Literally 99:1 read:write
>
> I continue to be inspired by the "Dirt Cheap Data Warehouse (DCDW)"
> [3]. SSD are getting bigger and prices are dropping rapidly (2 TB
> SSDs available now for $800). With our WORM-like workload, I believe
> we can safely get away with consumer drives, as durability shouldn't
> be an issue.
>
> So at this point I'm just putting out a feeler---has anyone out there
> actually built a massive SSD array, using either Linux software raid
> or hardware raid (technically off-topic for this list, though I hope
> the discussion is interesting enough to let it slide). If so, how big
> of an array (i.e. drives/capacity)? What was the target versus actual
> performance? Any particularly challenging issues that came up?
Hi Matt,
maybe not 100% matching... We wanted to start a 22 HDD md raid 6 last
year for one of our storage servers. Doing some tests we experienced
slow random write I/Os and got discouraged. So we headed over to
hardware raid.
To get a clearer picture we did further analysis. The bootleneck arised
from two sources. md issued only 4K I/Os and it always did a reconstruct
write. Thus massive write amplification. Several patches in 4.1 mitigate
the situation. In between I like a lot of things in md raid 4/5/6. Especially
the parity calucation massively benefits from high power CPUs.
Back to the reason for my explanation: The biggest "area under construction"
in md raid 4/5/6 is the internal stripe cache handling. Allocating, flushing
and freeing stripes is based on solid but not tuned algorithms (like LRU).
IIRC it does not even have a scheduling or queueing. That could become
your bottleneck. Maybe massive CPU power easily compensates the gap.
That said. Your read mostly setup might avoid a lot of headaches but you
need to push it to the limits yourself. I do not see benchmarks in the net
that closely fit your case. So I advise to enable ramdisks in the Linux
kernel and build a raid 6 on top of /dev/ramX. A synthetic test according
to your needs should give an idea of scalability and overhead of md raid.
Good luck.
Markus
=
[-- Attachment #2: InterScan_Disclaimer.txt --]
[-- Type: text/plain, Size: 1650 bytes --]
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
^ permalink raw reply
* [PATCH 1/2] Safeguard against writing to an active device of another node
From: Guoqing Jiang @ 2015-08-03 11:31 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
Modifying an exiting device's superblock or creating a new superblock
on an existing device needs to be checked because the device could be
in use by another node in another array. So, we check this by taking
all superblock locks in userspace so that we don't step onto an active
device used by another node and safeguard against accidental edits.
After the edit is complete, we release all locks and the lockspace so
that it can be used by the kernel space.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Makefile | 3 +-
mdadm.c | 11 +++++
mdadm.h | 46 ++++++++++++++++++
super1.c | 51 ++++++++++++++++++++
util.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 275 insertions(+), 1 deletion(-)
diff --git a/Makefile b/Makefile
index c189279..370ef75 100644
--- a/Makefile
+++ b/Makefile
@@ -81,11 +81,12 @@ FAILED_SLOTS_DIR = $(RUN_DIR)/failed-slots
SYSTEMD_DIR=/lib/systemd/system
COROSYNC:=$(shell [ -d /usr/include/corosync ] || echo -DNO_COROSYNC)
+DLM:=$(shell [ -f /usr/include/libdlm.h ] || echo -DNO_DLM)
DIRFLAGS = -DMAP_DIR=\"$(MAP_DIR)\" -DMAP_FILE=\"$(MAP_FILE)\"
DIRFLAGS += -DMDMON_DIR=\"$(MDMON_DIR)\"
DIRFLAGS += -DFAILED_SLOTS_DIR=\"$(FAILED_SLOTS_DIR)\"
-CFLAGS = $(CWFLAGS) $(CXFLAGS) -DSendmail=\""$(MAILCMD)"\" $(CONFFILEFLAGS) $(DIRFLAGS) $(COROSYNC)
+CFLAGS = $(CWFLAGS) $(CXFLAGS) -DSendmail=\""$(MAILCMD)"\" $(CONFFILEFLAGS) $(DIRFLAGS) $(COROSYNC) $(DLM)
VERSION = $(shell [ -d .git ] && git describe HEAD | sed 's/mdadm-//')
VERS_DATE = $(shell [ -d .git ] && date --date="`git log -n1 --format=format:%cd --date=short`" '+%0dth %B %Y' | sed -e 's/1th/1st/' -e 's/2th/2nd/' -e 's/11st/11th/' -e 's/12nd/12th/')
diff --git a/mdadm.c b/mdadm.c
index c4daf25..d8032c1 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -1322,6 +1322,8 @@ int main(int argc, char *argv[])
/* --scan implied --brief unless -vv */
c.brief = 1;
+ set_dlm_hookers(); /* get dlm funcs from libdlm_lt.so.3 */
+
rv = 0;
switch(mode) {
case MANAGE:
@@ -1362,10 +1364,12 @@ int main(int argc, char *argv[])
else if (devs_found > 0) {
if (c.update && devs_found > 1) {
pr_err("can only update a single array at a time\n");
+ free_dlm_hookers(); /* close dlm stuffs */
exit(1);
}
if (c.backup_file && devs_found > 1) {
pr_err("can only assemble a single array when providing a backup file.\n");
+ free_dlm_hookers(); /* close dlm stuffs */
exit(1);
}
for (dv = devlist ; dv ; dv=dv->next) {
@@ -1384,10 +1388,12 @@ int main(int argc, char *argv[])
} else {
if (c.update) {
pr_err("--update not meaningful with a --scan assembly.\n");
+ free_dlm_hookers(); /* close dlm stuffs */
exit(1);
}
if (c.backup_file) {
pr_err("--backup_file not meaningful with a --scan assembly.\n");
+ free_dlm_hookers(); /* close dlm stuffs */
exit(1);
}
rv = scan_assemble(ss, &c, &ident);
@@ -1455,12 +1461,14 @@ int main(int argc, char *argv[])
if (devmode == 'E') {
if (devlist == NULL && !c.scan) {
pr_err("No devices to examine\n");
+ free_dlm_hookers(); /* close dlm stuffs */
exit(2);
}
if (devlist == NULL)
devlist = conf_get_devs();
if (devlist == NULL) {
pr_err("No devices listed in %s\n", configfile?configfile:DefaultConfFile);
+ free_dlm_hookers(); /* close dlm stuffs */
exit(1);
}
rv = Examine(devlist, &c, ss);
@@ -1477,6 +1485,7 @@ int main(int argc, char *argv[])
rv = Write_rules(udev_filename);
else {
pr_err("No devices given.\n");
+ free_dlm_hookers(); /* close dlm stuffs */
exit(2);
}
} else
@@ -1615,6 +1624,8 @@ int main(int argc, char *argv[])
autodetect();
break;
}
+
+ free_dlm_hookers(); /* close dlm stuffs */
exit(rv);
}
diff --git a/mdadm.h b/mdadm.h
index 97892e6..c53adc5 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -35,6 +35,7 @@ extern __off64_t lseek64 __P ((int __fd, __off64_t __offset, int __whence));
#include <sys/types.h>
#include <sys/stat.h>
+#include <stdint.h>
#include <stdlib.h>
#include <time.h>
#include <sys/time.h>
@@ -51,6 +52,25 @@ extern __off64_t lseek64 __P ((int __fd, __off64_t __offset, int __whence));
#define srandom srand
#endif
+#ifndef NO_DLM
+#include <libdlm.h>
+#include <errno.h>
+#else
+#define LKF_NOQUEUE 0x00000001
+#define LKF_CONVERT 0x00000004
+#define LKM_PWMODE 4
+#define EUNLOCK 0x10002
+
+typedef void *dlm_lshandle_t;
+
+struct dlm_lksb {
+ int sb_status;
+ uint32_t sb_lkid;
+ char sb_flags;
+ char *sb_lvbptr;
+};
+#endif
+
#include <linux/kdev_t.h>
/*#include <linux/fs.h> */
#include <sys/mount.h>
@@ -1428,7 +1448,33 @@ extern char *stat2devnm(struct stat *st);
extern char *fd2devnm(int fd);
extern int in_initrd(void);
+
+struct dlm_hookers {
+ void *dlm_handle; /* dlm lib related */
+
+ dlm_lshandle_t (*create_lockspace)(const char *name,
+ unsigned int mode);
+ int (*release_lockspace)(const char *name, dlm_lshandle_t ls,
+ int force);
+ int (*ls_lock)(dlm_lshandle_t lockspace, uint32_t mode,
+ struct dlm_lksb *lksb, uint32_t flags,
+ const void *name, unsigned int namelen,
+ uint32_t parent, void (*astaddr) (void *astarg),
+ void *astarg, void (*bastaddr) (void *astarg),
+ void *range);
+ int (*ls_unlock)(dlm_lshandle_t lockspace, uint32_t lkid,
+ uint32_t flags, struct dlm_lksb *lksb,
+ void *astarg);
+ int (*ls_get_fd)(dlm_lshandle_t ls);
+ int (*dispatch)(int fd);
+};
+
extern int get_cluster_name(char **name);
+extern int is_clustered(struct supertype *st);
+extern int cluster_get_dlmlock(struct supertype *st, int *lockid);
+extern int cluster_release_dlmlock(struct supertype *st, int lockid);
+extern void set_dlm_hookers(void);
+extern void free_dlm_hookers(void);
#define _ROUND_UP(val, base) (((val) + (base) - 1) & ~(base - 1))
#define ROUND_UP(val, base) _ROUND_UP(val, (typeof(val))(base))
diff --git a/super1.c b/super1.c
index fda71e3..bd88c36 100644
--- a/super1.c
+++ b/super1.c
@@ -1072,8 +1072,18 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
* ignored.
*/
int rv = 0;
+ int lockid;
struct mdp_superblock_1 *sb = st->sb;
+ if (is_clustered(st)) {
+ rv = cluster_get_dlmlock(st, &lockid);
+ if (rv) {
+ pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
+ cluster_release_dlmlock(st, lockid);
+ return rv;
+ }
+ }
+
if (strcmp(update, "homehost") == 0 &&
homehost) {
/* Note that 'homehost' is special as it is really
@@ -1330,6 +1340,9 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
rv = -1;
sb->sb_csum = calc_sb_1_csum(sb);
+ if (is_clustered(st))
+ cluster_release_dlmlock(st, lockid);
+
return rv;
}
@@ -1433,6 +1446,16 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
struct mdp_superblock_1 *sb = st->sb;
__u16 *rp = sb->dev_roles + dk->number;
struct devinfo *di, **dip;
+ int rv, lockid;
+
+ if (is_clustered(st)) {
+ rv = cluster_get_dlmlock(st, &lockid);
+ if (rv) {
+ pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
+ cluster_release_dlmlock(st, lockid);
+ return rv;
+ }
+ }
if ((dk->state & 6) == 6) /* active, sync */
*rp = __cpu_to_le16(dk->raid_disk);
@@ -1460,6 +1483,9 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
di->next = NULL;
*dip = di;
+ if (is_clustered(st))
+ cluster_release_dlmlock(st, lockid);
+
return 0;
}
#endif
@@ -1473,6 +1499,16 @@ static int store_super1(struct supertype *st, int fd)
struct align_fd afd;
int sbsize;
unsigned long long dsize;
+ int rv, lockid;
+
+ if (is_clustered(st)) {
+ rv = cluster_get_dlmlock(st, &lockid);
+ if (rv) {
+ pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
+ cluster_release_dlmlock(st, lockid);
+ return rv;
+ }
+ }
if (!get_dev_size(fd, NULL, &dsize))
return 1;
@@ -1533,6 +1569,9 @@ static int store_super1(struct supertype *st, int fd)
}
}
fsync(fd);
+ if (is_clustered(st))
+ cluster_release_dlmlock(st, lockid);
+
return 0;
}
@@ -2282,6 +2321,16 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
static void free_super1(struct supertype *st)
{
+ int rv, lockid;
+ if (is_clustered(st)) {
+ rv = cluster_get_dlmlock(st, &lockid);
+ if (rv) {
+ pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
+ cluster_release_dlmlock(st, lockid);
+ return;
+ }
+ }
+
if (st->sb)
free(st->sb);
while (st->info) {
@@ -2292,6 +2341,8 @@ static void free_super1(struct supertype *st)
free(di);
}
st->sb = NULL;
+ if (is_clustered(st))
+ cluster_release_dlmlock(st, lockid);
}
#ifndef MDASSEMBLE
diff --git a/util.c b/util.c
index ea6e688..19ecf9f 100644
--- a/util.c
+++ b/util.c
@@ -24,6 +24,7 @@
#include "mdadm.h"
#include "md_p.h"
+#include <sys/poll.h>
#include <sys/socket.h>
#include <sys/utsname.h>
#include <sys/wait.h>
@@ -88,6 +89,135 @@ struct blkpg_partition {
aren't permitted). */
#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
+static struct dlm_hookers *dlm_hookers = NULL;
+static int is_dlm_hookers_ready = 0;
+static struct dlm_lock_resource *dlm_lock_res = NULL;
+static int ast_called = 0;
+
+struct dlm_lock_resource {
+ dlm_lshandle_t *ls;
+ struct dlm_lksb lksb;
+};
+
+int is_clustered(struct supertype *st)
+{
+ /* is it a cluster md or not */
+ if (is_dlm_hookers_ready && st->cluster_name)
+ return 1;
+ else
+ return 0;
+}
+
+/* Using poll(2) to wait for and dispatch ASTs */
+static int poll_for_ast(dlm_lshandle_t ls)
+{
+ struct pollfd pfd;
+
+ pfd.fd = dlm_hookers->ls_get_fd(ls);
+ pfd.events = POLLIN;
+
+ while (!ast_called)
+ {
+ if (poll(&pfd, 1, 0) < 0)
+ {
+ perror("poll");
+ return -1;
+ }
+ dlm_hookers->dispatch(dlm_hookers->ls_get_fd(ls));
+ }
+ ast_called = 0;
+
+ return 0;
+}
+
+static void dlm_ast(void *arg)
+{
+ ast_called = 1;
+}
+
+/* Create the lockspace, take bitmapXXX locks on all the bitmaps. */
+int cluster_get_dlmlock(struct supertype *st, int *lockid)
+{
+ int ret = -1;
+ char str[64];
+ int flags = LKF_NOQUEUE;
+
+ dlm_lock_res = xmalloc(sizeof(struct dlm_lock_resource));
+ if (!dlm_lock_res)
+ goto out;
+
+ dlm_lock_res->ls = dlm_hookers->create_lockspace(st->cluster_name, O_RDWR);
+ if (!dlm_lock_res->ls) {
+ pr_err("%s failed to create lockspace\n", st->cluster_name);
+ goto out;
+ }
+
+ /* Conversions need the lockid in the LKSB */
+ if (flags & LKF_CONVERT)
+ dlm_lock_res->lksb.sb_lkid = *lockid;
+
+ snprintf(str, 64, "bitmap%04d", st->nodes);
+ /* if flags with LKF_CONVERT causes below return ENOENT which means
+ * "No such file or directory" */
+ ret = dlm_hookers->ls_lock(dlm_lock_res->ls, LKM_PWMODE, &dlm_lock_res->lksb,
+ flags, str, strlen(str), 0, dlm_ast,
+ dlm_lock_res, NULL, NULL);
+ if (ret) {
+ pr_err("error %d when get PW mode on lock %s\n", errno, str);
+ goto out;
+ }
+
+ /* Wait for it to complete */
+ poll_for_ast(dlm_lock_res->ls);
+ *lockid = dlm_lock_res->lksb.sb_lkid;
+
+ errno = dlm_lock_res->lksb.sb_status;
+ if (errno) {
+ pr_err("error %d happened in ast with lock %s\n", errno, str);
+ goto out;
+ }
+
+out:
+ return ret;
+}
+
+int cluster_release_dlmlock(struct supertype *st, int lockid)
+{
+ int ret = -1;
+
+ /* if flags with LKF_CONVERT causes below return EINVAL which means
+ * "Invalid argument" */
+ ret = dlm_hookers->ls_unlock(dlm_lock_res->ls, lockid, 0,
+ &dlm_lock_res->lksb, dlm_lock_res);
+ if (ret) {
+ pr_err("error %d happened when unlock\n", errno);
+ /* XXX make sure the lock is unlocked eventually */
+ goto out;
+ }
+
+ /* Wait for it to complete */
+ poll_for_ast(dlm_lock_res->ls);
+
+ errno = dlm_lock_res->lksb.sb_status;
+ if (errno != EUNLOCK) {
+ pr_err("error %d happened in ast when unlock lockspace\n", errno);
+ /* XXX make sure the lockspace is unlocked eventually */
+ goto out;
+ }
+
+ ret = dlm_hookers->release_lockspace(st->cluster_name, dlm_lock_res->ls, 1);
+ if (ret) {
+ pr_err("error %d happened when release lockspace\n", errno);
+ /* XXX make sure the lockspace is released eventually */
+ goto out;
+ }
+ free(dlm_lock_res);
+
+out:
+ return ret;
+}
+
+
/*
* Parse a 128 bit uuid in 4 integers
* format is 32 hexx nibbles with options :.<space> separator
@@ -2043,3 +2173,38 @@ out:
dlclose(lib_handle);
return rv;
}
+
+void set_dlm_hookers(void)
+{
+ dlm_hookers = xmalloc(sizeof(struct dlm_hookers));
+ if (!dlm_hookers)
+ return;
+
+ dlm_hookers->dlm_handle = dlopen("libdlm_lt.so.3", RTLD_NOW | RTLD_LOCAL);
+ if (!dlm_hookers->dlm_handle)
+ return;
+
+ dlm_hookers->create_lockspace = dlsym(dlm_hookers->dlm_handle, "dlm_create_lockspace");
+ dlm_hookers->release_lockspace = dlsym(dlm_hookers->dlm_handle, "dlm_release_lockspace");
+ dlm_hookers->ls_lock = dlsym(dlm_hookers->dlm_handle, "dlm_ls_lock");
+ dlm_hookers->ls_unlock = dlsym(dlm_hookers->dlm_handle, "dlm_ls_unlock");
+ dlm_hookers->ls_get_fd = dlsym(dlm_hookers->dlm_handle, "dlm_ls_get_fd");
+ dlm_hookers->dispatch = dlsym(dlm_hookers->dlm_handle, "dlm_dispatch");
+
+ if (!dlm_hookers->create_lockspace || !dlm_hookers->ls_lock ||
+ !dlm_hookers->ls_unlock || !dlm_hookers->release_lockspace ||
+ !dlm_hookers->ls_get_fd || !dlm_hookers->dispatch)
+ dlclose(dlm_hookers->dlm_handle);
+ else
+ is_dlm_hookers_ready = 1;
+}
+
+void free_dlm_hookers(void)
+{
+ if (is_dlm_hookers_ready) {
+ dlclose(dlm_hookers->dlm_handle);
+ is_dlm_hookers_ready = 0;
+ }
+ if (dlm_hookers)
+ free(dlm_hookers);
+}
--
1.7.12.4
^ permalink raw reply related
* [PATCH 2/2] Make cmap_* also has same policy as dlm_*
From: Guoqing Jiang @ 2015-08-03 11:31 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1438601519-17919-1-git-send-email-gqjiang@suse.com>
Let libcmap lib and related funs also only need one-time
setup during mdadm running period.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
mdadm.c | 23 +++++++++++--------
mdadm.h | 20 ++++++++++++++++
util.c | 81 +++++++++++++++++++++++++++++++++++++++--------------------------
3 files changed, 82 insertions(+), 42 deletions(-)
diff --git a/mdadm.c b/mdadm.c
index d8032c1..4812fc5 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -1293,18 +1293,22 @@ int main(int argc, char *argv[])
c.require_homehost = 0;
}
+ set_hookers(); /* set hookers from libs */
+
if (c.homecluster == NULL && (c.nodes > 0)) {
c.homecluster = conf_get_homecluster();
if (c.homecluster == NULL)
rv = get_cluster_name(&c.homecluster);
if (rv != 0) {
pr_err("The md can't get cluster name\n");
+ free_hookers();
exit(1);
}
}
if (c.backup_file && data_offset != INVALID_SECTORS) {
pr_err("--backup-file and --data-offset are incompatible\n");
+ free_hookers();
exit(2);
}
@@ -1313,6 +1317,7 @@ int main(int argc, char *argv[])
/* Anyone may try this */;
else if (geteuid() != 0) {
pr_err("must be super-user to perform this action\n");
+ free_hookers();
exit(1);
}
@@ -1322,8 +1327,6 @@ int main(int argc, char *argv[])
/* --scan implied --brief unless -vv */
c.brief = 1;
- set_dlm_hookers(); /* get dlm funcs from libdlm_lt.so.3 */
-
rv = 0;
switch(mode) {
case MANAGE:
@@ -1364,12 +1367,12 @@ int main(int argc, char *argv[])
else if (devs_found > 0) {
if (c.update && devs_found > 1) {
pr_err("can only update a single array at a time\n");
- free_dlm_hookers(); /* close dlm stuffs */
+ free_hookers(); /* close dlm stuffs */
exit(1);
}
if (c.backup_file && devs_found > 1) {
pr_err("can only assemble a single array when providing a backup file.\n");
- free_dlm_hookers(); /* close dlm stuffs */
+ free_hookers(); /* close dlm stuffs */
exit(1);
}
for (dv = devlist ; dv ; dv=dv->next) {
@@ -1388,12 +1391,12 @@ int main(int argc, char *argv[])
} else {
if (c.update) {
pr_err("--update not meaningful with a --scan assembly.\n");
- free_dlm_hookers(); /* close dlm stuffs */
+ free_hookers(); /* close dlm stuffs */
exit(1);
}
if (c.backup_file) {
pr_err("--backup_file not meaningful with a --scan assembly.\n");
- free_dlm_hookers(); /* close dlm stuffs */
+ free_hookers(); /* close dlm stuffs */
exit(1);
}
rv = scan_assemble(ss, &c, &ident);
@@ -1461,14 +1464,14 @@ int main(int argc, char *argv[])
if (devmode == 'E') {
if (devlist == NULL && !c.scan) {
pr_err("No devices to examine\n");
- free_dlm_hookers(); /* close dlm stuffs */
+ free_hookers(); /* close dlm stuffs */
exit(2);
}
if (devlist == NULL)
devlist = conf_get_devs();
if (devlist == NULL) {
pr_err("No devices listed in %s\n", configfile?configfile:DefaultConfFile);
- free_dlm_hookers(); /* close dlm stuffs */
+ free_hookers(); /* close dlm stuffs */
exit(1);
}
rv = Examine(devlist, &c, ss);
@@ -1485,7 +1488,7 @@ int main(int argc, char *argv[])
rv = Write_rules(udev_filename);
else {
pr_err("No devices given.\n");
- free_dlm_hookers(); /* close dlm stuffs */
+ free_hookers(); /* close dlm stuffs */
exit(2);
}
} else
@@ -1625,7 +1628,7 @@ int main(int argc, char *argv[])
break;
}
- free_dlm_hookers(); /* close dlm stuffs */
+ free_hookers(); /* close dlm stuffs */
exit(rv);
}
diff --git a/mdadm.h b/mdadm.h
index c53adc5..dd7fef4 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -52,6 +52,12 @@ extern __off64_t lseek64 __P ((int __fd, __off64_t __offset, int __whence));
#define srandom srand
#endif
+#ifdef NO_COROSYNC
+#define CS_OK 1
+#else
+#include <corosync/cmap.h>
+#endif
+
#ifndef NO_DLM
#include <libdlm.h>
#include <errno.h>
@@ -1449,6 +1455,16 @@ extern char *fd2devnm(int fd);
extern int in_initrd(void);
+struct cmap_hookers {
+ void *cmap_handle; /* corosync lib related */
+
+ int (*initialize)(cmap_handle_t *handle);
+ int (*get_string)(cmap_handle_t handle,
+ const char *string,
+ char **name);
+ int (*finalize)(cmap_handle_t handle);
+};
+
struct dlm_hookers {
void *dlm_handle; /* dlm lib related */
@@ -1475,6 +1491,10 @@ extern int cluster_get_dlmlock(struct supertype *st, int *lockid);
extern int cluster_release_dlmlock(struct supertype *st, int lockid);
extern void set_dlm_hookers(void);
extern void free_dlm_hookers(void);
+extern void set_cmap_hookers(void);
+extern void free_cmap_hookers(void);
+extern void set_hookers(void);
+extern void free_hookers(void);
#define _ROUND_UP(val, base) (((val) + (base) - 1) & ~(base - 1))
#define ROUND_UP(val, base) _ROUND_UP(val, (typeof(val))(base))
diff --git a/util.c b/util.c
index 19ecf9f..1c87eb5 100644
--- a/util.c
+++ b/util.c
@@ -36,13 +36,6 @@
#include <dirent.h>
#include <signal.h>
#include <dlfcn.h>
-#include <stdint.h>
-#ifdef NO_COROSYNC
- typedef uint64_t cmap_handle_t;
- #define CS_OK 1
-#else
- #include <corosync/cmap.h>
-#endif
/*
@@ -2126,40 +2119,53 @@ void reopen_mddev(int mdfd)
dup2(fd, mdfd);
}
-int get_cluster_name(char **cluster_name)
+static struct cmap_hookers *cmap_hookers = NULL;
+static int is_cmap_hookers_ready = 0;
+
+void set_cmap_hookers(void)
{
- void *lib_handle = NULL;
- int rv = -1;
+ cmap_hookers = xmalloc(sizeof(struct cmap_hookers));
+ if (!cmap_hookers)
+ return;
- cmap_handle_t handle;
- static int (*initialize)(cmap_handle_t *handle);
- static int (*get_string)(cmap_handle_t handle,
- const char *string,
- char **name);
- static int (*finalize)(cmap_handle_t handle);
+ cmap_hookers->cmap_handle = dlopen("libcmap.so.4", RTLD_NOW | RTLD_LOCAL);
+ if (!cmap_hookers->cmap_handle)
+ return;
+ cmap_hookers->initialize = dlsym(cmap_hookers->cmap_handle, "cmap_initialize");
+ cmap_hookers->get_string = dlsym(cmap_hookers->cmap_handle, "cmap_get_string");
+ cmap_hookers->finalize = dlsym(cmap_hookers->cmap_handle, "cmap_finalize");
- lib_handle = dlopen("libcmap.so.4", RTLD_NOW | RTLD_LOCAL);
- if (!lib_handle)
- return rv;
+ if (!cmap_hookers->initialize || !cmap_hookers->get_string ||
+ !cmap_hookers->finalize)
+ dlclose(cmap_hookers->cmap_handle);
+ else
+ is_cmap_hookers_ready = 1;
+}
- initialize = dlsym(lib_handle, "cmap_initialize");
- if (!initialize)
- goto out;
+void free_cmap_hookers(void)
+{
+ if (is_cmap_hookers_ready) {
+ dlclose(cmap_hookers->cmap_handle);
+ is_cmap_hookers_ready = 0;
+ }
+ if (cmap_hookers)
+ free(cmap_hookers);
+}
- get_string = dlsym(lib_handle, "cmap_get_string");
- if (!get_string)
- goto out;
+int get_cluster_name(char **cluster_name)
+{
+ int rv = -1;
+ cmap_handle_t handle;
- finalize = dlsym(lib_handle, "cmap_finalize");
- if (!finalize)
- goto out;
+ if (!is_cmap_hookers_ready)
+ return rv;
- rv = initialize(&handle);
+ rv = cmap_hookers->initialize(&handle);
if (rv != CS_OK)
goto out;
- rv = get_string(handle, "totem.cluster_name", cluster_name);
+ rv = cmap_hookers->get_string(handle, "totem.cluster_name", cluster_name);
if (rv != CS_OK) {
free(*cluster_name);
rv = -1;
@@ -2168,9 +2174,8 @@ int get_cluster_name(char **cluster_name)
rv = 0;
name_err:
- finalize(handle);
+ cmap_hookers->finalize(handle);
out:
- dlclose(lib_handle);
return rv;
}
@@ -2208,3 +2213,15 @@ void free_dlm_hookers(void)
if (dlm_hookers)
free(dlm_hookers);
}
+
+void set_hookers(void)
+{
+ set_dlm_hookers();
+ set_cmap_hookers();
+}
+
+void free_hookers(void)
+{
+ free_dlm_hookers();
+ free_cmap_hookers();
+}
--
1.7.12.4
^ permalink raw reply related
* Re: Thoughts on big SSD arrays?
From: Adam Goryachev @ 2015-08-03 11:38 UTC (permalink / raw)
To: Matt Garman, Mdadm
In-Reply-To: <CAJvUf-AW08pcRkADn4Fgh46_vVAxgofiuB93+pzQYk99hx1nbw@mail.gmail.com>
On 1/08/2015 01:23, Matt Garman wrote:
> I continue to be inspired by the "Dirt Cheap Data Warehouse (DCDW)"
> [3]. SSD are getting bigger and prices are dropping rapidly (2 TB
> SSDs available now for $800). With our WORM-like workload, I believe
> we can safely get away with consumer drives, as durability shouldn't
> be an issue.
>
> So at this point I'm just putting out a feeler---has anyone out there
> actually built a massive SSD array, using either Linux software raid
> or hardware raid (technically off-topic for this list, though I hope
> the discussion is interesting enough to let it slide). If so, how big
> of an array (i.e. drives/capacity)? What was the target versus actual
> performance? Any particularly challenging issues that came up?
I have been using a 8x 480GB RAID5 linux md array for a iSCSI SAN for a
number of years, and it worked well after some careful tuning, and
careful (lucky) hardware selection (ie, motherboard was lucky to have
the right bandwidth memory/PCI bus/etc).
The main challenge I had was actually with DRBD on top of the array,
once I disabled the forced writes, then it all worked really well. The
forced writes were forcing a consistent on disk status for every write
since the SSD's I use did not have any ability to save the data during a
power outage.
> FWIW, I'm thinking of something along the lines of a 24-disk chassis,
> with 2 disks for OS (raid1), 2 disks as hot spares, and the remaining
> 20 in raid-6. The 22 data disks (raid + hot spares) would be 2 TB
> SSDs.
I'm not sure that sounds like a good idea. Personally, I'd probably
prefer to use 2 x RAID6 arrays at least, but then that is just what
advice I hear on the list. Using two arrays will also get you more
parallel processing (use more cpu cores), as I think you are limited to
one cpu per array.
> The "problem" with SSDs is that they're just so seductive:
> back-of-the-envelope numbers are wonderful, so it's easy to get
> overly-optimistic about builds that use them. But as with most
> things, the devil's in the details.
I was able to get 2.5GB/s read and 1.5GB/s write with (I think) only 6
SSD's in RAID5. However, eventually, when I did the correct test to
match my actual load, that dropped to abysmal values (well under
100MB/s). The reason is that my live load uses very small read/write
block size, so there were a massive number of small random read/writes,
leading to high IOPS. Using large block sizes can deliver massive
throughput, with very small number of IOPS,
> Off the top of my head, potential issues I can think of:
>
> - Subtle PCIe latency/timing issues of the motherboard
From memory, this can include the amount of bandwidth between
memory/CPU/PCI bus/SATA bus/etc... Including the speed of the RAM as
just one of the factors. I don't know all the tricky details, but I do
recall that while the bandwdith looks plenty fast enough at first, the
data moves over a number of bridges, and sometimes the same bridge more
than once (eg, the disk interface and network interface might be on the
same bridge).
> - High variation in SSD latency
> - Software stacks still making assumptions based on spinning
> drives (i.e. not adequately tuned for SSDs)
> - Non-parallel RAID implementation (i.e. single CPU bottleneck potential)
> - Potential bandwidth bottlenecks at various stages: SATA/SAS
> interface, SAS expander/backplane, SATA/SAS controller (or HBA), PCIe
> bus, CPU memory bus, network card, etc
> - I forget the exact number, but the DCDW guy told me with Linux
> he was only able to get about 30% of the predicted throughput in his
> SSD array
I got close to the theoretical maximum (from memory), but it depended on
the actual real life workload. Those theoretical performance values are
only achieved in "optimal" conditions, real life is often a lot more messy.
> - Wacky TRIM related issues (seem to be drive dependent)
If you are mostly read then TRIM shouldn't be much of an issue for you.
> Not asking any particular question here, just hoping to start an
> open-ended discussion. Of course I'd love to hear from anyone with
> actual SSD RAID experience!
>
My experience has been positive. BTW, I'm using the Intel 480GB SSD
(basically consumer grade 520/530 series). If you want any extra
information/details, let me know.
Regards,
Adam
^ permalink raw reply
* Re: [PATCH 1/2] Safeguard against writing to an active device of another node
From: Goldwyn Rodrigues @ 2015-08-03 11:56 UTC (permalink / raw)
To: Guoqing Jiang, neilb; +Cc: linux-raid
In-Reply-To: <1438601519-17919-1-git-send-email-gqjiang@suse.com>
On 08/03/2015 06:31 AM, Guoqing Jiang wrote:
> Modifying an exiting device's superblock or creating a new superblock
> on an existing device needs to be checked because the device could be
> in use by another node in another array. So, we check this by taking
> all superblock locks in userspace so that we don't step onto an active
> device used by another node and safeguard against accidental edits.
> After the edit is complete, we release all locks and the lockspace so
> that it can be used by the kernel space.
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> Makefile | 3 +-
> mdadm.c | 11 +++++
> mdadm.h | 46 ++++++++++++++++++
> super1.c | 51 ++++++++++++++++++++
> util.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 5 files changed, 275 insertions(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index c189279..370ef75 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -81,11 +81,12 @@ FAILED_SLOTS_DIR = $(RUN_DIR)/failed-slots
> SYSTEMD_DIR=/lib/systemd/system
>
> COROSYNC:=$(shell [ -d /usr/include/corosync ] || echo -DNO_COROSYNC)
> +DLM:=$(shell [ -f /usr/include/libdlm.h ] || echo -DNO_DLM)
>
> DIRFLAGS = -DMAP_DIR=\"$(MAP_DIR)\" -DMAP_FILE=\"$(MAP_FILE)\"
> DIRFLAGS += -DMDMON_DIR=\"$(MDMON_DIR)\"
> DIRFLAGS += -DFAILED_SLOTS_DIR=\"$(FAILED_SLOTS_DIR)\"
> -CFLAGS = $(CWFLAGS) $(CXFLAGS) -DSendmail=\""$(MAILCMD)"\" $(CONFFILEFLAGS) $(DIRFLAGS) $(COROSYNC)
> +CFLAGS = $(CWFLAGS) $(CXFLAGS) -DSendmail=\""$(MAILCMD)"\" $(CONFFILEFLAGS) $(DIRFLAGS) $(COROSYNC) $(DLM)
>
> VERSION = $(shell [ -d .git ] && git describe HEAD | sed 's/mdadm-//')
> VERS_DATE = $(shell [ -d .git ] && date --date="`git log -n1 --format=format:%cd --date=short`" '+%0dth %B %Y' | sed -e 's/1th/1st/' -e 's/2th/2nd/' -e 's/11st/11th/' -e 's/12nd/12th/')
> diff --git a/mdadm.c b/mdadm.c
> index c4daf25..d8032c1 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -1322,6 +1322,8 @@ int main(int argc, char *argv[])
> /* --scan implied --brief unless -vv */
> c.brief = 1;
>
> + set_dlm_hookers(); /* get dlm funcs from libdlm_lt.so.3 */
> +
Universal Comment: Let call it set_dlm_hooks as opposed to hookers.
> rv = 0;
> switch(mode) {
> case MANAGE:
> @@ -1362,10 +1364,12 @@ int main(int argc, char *argv[])
> else if (devs_found > 0) {
> if (c.update && devs_found > 1) {
> pr_err("can only update a single array at a time\n");
> + free_dlm_hookers(); /* close dlm stuffs */
> exit(1);
> }
> if (c.backup_file && devs_found > 1) {
> pr_err("can only assemble a single array when providing a backup file.\n");
> + free_dlm_hookers(); /* close dlm stuffs */
> exit(1);
> }
> for (dv = devlist ; dv ; dv=dv->next) {
> @@ -1384,10 +1388,12 @@ int main(int argc, char *argv[])
> } else {
> if (c.update) {
> pr_err("--update not meaningful with a --scan assembly.\n");
> + free_dlm_hookers(); /* close dlm stuffs */
> exit(1);
> }
> if (c.backup_file) {
> pr_err("--backup_file not meaningful with a --scan assembly.\n");
> + free_dlm_hookers(); /* close dlm stuffs */
> exit(1);
> }
> rv = scan_assemble(ss, &c, &ident);
> @@ -1455,12 +1461,14 @@ int main(int argc, char *argv[])
> if (devmode == 'E') {
> if (devlist == NULL && !c.scan) {
> pr_err("No devices to examine\n");
> + free_dlm_hookers(); /* close dlm stuffs */
> exit(2);
> }
> if (devlist == NULL)
> devlist = conf_get_devs();
> if (devlist == NULL) {
> pr_err("No devices listed in %s\n", configfile?configfile:DefaultConfFile);
> + free_dlm_hookers(); /* close dlm stuffs */
> exit(1);
> }
> rv = Examine(devlist, &c, ss);
> @@ -1477,6 +1485,7 @@ int main(int argc, char *argv[])
> rv = Write_rules(udev_filename);
> else {
> pr_err("No devices given.\n");
> + free_dlm_hookers(); /* close dlm stuffs */
> exit(2);
> }
> } else
> @@ -1615,6 +1624,8 @@ int main(int argc, char *argv[])
> autodetect();
> break;
> }
> +
> + free_dlm_hookers(); /* close dlm stuffs */
> exit(rv);
> }
>
> diff --git a/mdadm.h b/mdadm.h
> index 97892e6..c53adc5 100644
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -35,6 +35,7 @@ extern __off64_t lseek64 __P ((int __fd, __off64_t __offset, int __whence));
>
> #include <sys/types.h>
> #include <sys/stat.h>
> +#include <stdint.h>
> #include <stdlib.h>
> #include <time.h>
> #include <sys/time.h>
> @@ -51,6 +52,25 @@ extern __off64_t lseek64 __P ((int __fd, __off64_t __offset, int __whence));
> #define srandom srand
> #endif
>
> +#ifndef NO_DLM
> +#include <libdlm.h>
> +#include <errno.h>
> +#else
> +#define LKF_NOQUEUE 0x00000001
> +#define LKF_CONVERT 0x00000004
> +#define LKM_PWMODE 4
> +#define EUNLOCK 0x10002
> +
> +typedef void *dlm_lshandle_t;
> +
> +struct dlm_lksb {
> + int sb_status;
> + uint32_t sb_lkid;
> + char sb_flags;
> + char *sb_lvbptr;
> +};
> +#endif
> +
> #include <linux/kdev_t.h>
> /*#include <linux/fs.h> */
> #include <sys/mount.h>
> @@ -1428,7 +1448,33 @@ extern char *stat2devnm(struct stat *st);
> extern char *fd2devnm(int fd);
>
> extern int in_initrd(void);
> +
> +struct dlm_hookers {
> + void *dlm_handle; /* dlm lib related */
> +
> + dlm_lshandle_t (*create_lockspace)(const char *name,
> + unsigned int mode);
> + int (*release_lockspace)(const char *name, dlm_lshandle_t ls,
> + int force);
> + int (*ls_lock)(dlm_lshandle_t lockspace, uint32_t mode,
> + struct dlm_lksb *lksb, uint32_t flags,
> + const void *name, unsigned int namelen,
> + uint32_t parent, void (*astaddr) (void *astarg),
> + void *astarg, void (*bastaddr) (void *astarg),
> + void *range);
> + int (*ls_unlock)(dlm_lshandle_t lockspace, uint32_t lkid,
> + uint32_t flags, struct dlm_lksb *lksb,
> + void *astarg);
> + int (*ls_get_fd)(dlm_lshandle_t ls);
> + int (*dispatch)(int fd);
> +};
> +
> extern int get_cluster_name(char **name);
> +extern int is_clustered(struct supertype *st);
> +extern int cluster_get_dlmlock(struct supertype *st, int *lockid);
> +extern int cluster_release_dlmlock(struct supertype *st, int lockid);
> +extern void set_dlm_hookers(void);
> +extern void free_dlm_hookers(void);
>
> #define _ROUND_UP(val, base) (((val) + (base) - 1) & ~(base - 1))
> #define ROUND_UP(val, base) _ROUND_UP(val, (typeof(val))(base))
> diff --git a/super1.c b/super1.c
> index fda71e3..bd88c36 100644
> --- a/super1.c
> +++ b/super1.c
> @@ -1072,8 +1072,18 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
> * ignored.
> */
> int rv = 0;
> + int lockid;
> struct mdp_superblock_1 *sb = st->sb;
>
> + if (is_clustered(st)) {
> + rv = cluster_get_dlmlock(st, &lockid);
> + if (rv) {
> + pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
> + cluster_release_dlmlock(st, lockid);
> + return rv;
> + }
> + }
> +
> if (strcmp(update, "homehost") == 0 &&
> homehost) {
> /* Note that 'homehost' is special as it is really
> @@ -1330,6 +1340,9 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
> rv = -1;
>
> sb->sb_csum = calc_sb_1_csum(sb);
> + if (is_clustered(st))
> + cluster_release_dlmlock(st, lockid);
> +
> return rv;
> }
>
> @@ -1433,6 +1446,16 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
> struct mdp_superblock_1 *sb = st->sb;
> __u16 *rp = sb->dev_roles + dk->number;
> struct devinfo *di, **dip;
> + int rv, lockid;
> +
> + if (is_clustered(st)) {
> + rv = cluster_get_dlmlock(st, &lockid);
> + if (rv) {
> + pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
> + cluster_release_dlmlock(st, lockid);
> + return rv;
> + }
> + }
>
> if ((dk->state & 6) == 6) /* active, sync */
> *rp = __cpu_to_le16(dk->raid_disk);
> @@ -1460,6 +1483,9 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
> di->next = NULL;
> *dip = di;
>
> + if (is_clustered(st))
> + cluster_release_dlmlock(st, lockid);
> +
> return 0;
> }
> #endif
> @@ -1473,6 +1499,16 @@ static int store_super1(struct supertype *st, int fd)
> struct align_fd afd;
> int sbsize;
> unsigned long long dsize;
> + int rv, lockid;
> +
> + if (is_clustered(st)) {
> + rv = cluster_get_dlmlock(st, &lockid);
> + if (rv) {
> + pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
> + cluster_release_dlmlock(st, lockid);
> + return rv;
> + }
> + }
>
> if (!get_dev_size(fd, NULL, &dsize))
> return 1;
> @@ -1533,6 +1569,9 @@ static int store_super1(struct supertype *st, int fd)
> }
> }
> fsync(fd);
> + if (is_clustered(st))
> + cluster_release_dlmlock(st, lockid);
> +
> return 0;
> }
>
> @@ -2282,6 +2321,16 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
>
> static void free_super1(struct supertype *st)
> {
> + int rv, lockid;
> + if (is_clustered(st)) {
> + rv = cluster_get_dlmlock(st, &lockid);
> + if (rv) {
> + pr_err("Cannot get dlmlock in %s return %d\n", __func__, rv);
> + cluster_release_dlmlock(st, lockid);
> + return;
> + }
> + }
> +
> if (st->sb)
> free(st->sb);
> while (st->info) {
> @@ -2292,6 +2341,8 @@ static void free_super1(struct supertype *st)
> free(di);
> }
> st->sb = NULL;
> + if (is_clustered(st))
> + cluster_release_dlmlock(st, lockid);
> }
>
> #ifndef MDASSEMBLE
> diff --git a/util.c b/util.c
> index ea6e688..19ecf9f 100644
> --- a/util.c
> +++ b/util.c
> @@ -24,6 +24,7 @@
>
> #include "mdadm.h"
> #include "md_p.h"
> +#include <sys/poll.h>
> #include <sys/socket.h>
> #include <sys/utsname.h>
> #include <sys/wait.h>
> @@ -88,6 +89,135 @@ struct blkpg_partition {
> aren't permitted). */
> #define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
>
> +static struct dlm_hookers *dlm_hookers = NULL;
> +static int is_dlm_hookers_ready = 0;
This should not be required, just checking for dlm_hooks == NULL should
be enough. This needs to be set accordingly in set_dlm_hooks.
> +static struct dlm_lock_resource *dlm_lock_res = NULL;
> +static int ast_called = 0;
> +
> +struct dlm_lock_resource {
> + dlm_lshandle_t *ls;
> + struct dlm_lksb lksb;
> +};
> +
> +int is_clustered(struct supertype *st)
> +{
> + /* is it a cluster md or not */
> + if (is_dlm_hookers_ready && st->cluster_name)
> + return 1;
> + else
> + return 0;
> +}
> +
> +/* Using poll(2) to wait for and dispatch ASTs */
> +static int poll_for_ast(dlm_lshandle_t ls)
> +{
> + struct pollfd pfd;
Shouldn't you check dlm_hooks is NULL here? and starting of every
function which requires dlm_hooks.
Also, a return value from these functions do not mean an error, it means
the library is not present.
> +
> + pfd.fd = dlm_hookers->ls_get_fd(ls);
> + pfd.events = POLLIN;
> +
> + while (!ast_called)
> + {
> + if (poll(&pfd, 1, 0) < 0)
> + {
> + perror("poll");
> + return -1;
> + }
> + dlm_hookers->dispatch(dlm_hookers->ls_get_fd(ls));
> + }
> + ast_called = 0;
> +
> + return 0;
> +}
> +
> +static void dlm_ast(void *arg)
> +{
> + ast_called = 1;
> +}
> +
> +/* Create the lockspace, take bitmapXXX locks on all the bitmaps. */
> +int cluster_get_dlmlock(struct supertype *st, int *lockid)
> +{
> + int ret = -1;
> + char str[64];
> + int flags = LKF_NOQUEUE;
> +
> + dlm_lock_res = xmalloc(sizeof(struct dlm_lock_resource));
> + if (!dlm_lock_res)
> + goto out;
> +
> + dlm_lock_res->ls = dlm_hookers->create_lockspace(st->cluster_name, O_RDWR);
> + if (!dlm_lock_res->ls) {
> + pr_err("%s failed to create lockspace\n", st->cluster_name);
> + goto out;
> + }
> +
> + /* Conversions need the lockid in the LKSB */
> + if (flags & LKF_CONVERT)
> + dlm_lock_res->lksb.sb_lkid = *lockid;
> +
> + snprintf(str, 64, "bitmap%04d", st->nodes);
> + /* if flags with LKF_CONVERT causes below return ENOENT which means
> + * "No such file or directory" */
> + ret = dlm_hookers->ls_lock(dlm_lock_res->ls, LKM_PWMODE, &dlm_lock_res->lksb,
> + flags, str, strlen(str), 0, dlm_ast,
> + dlm_lock_res, NULL, NULL);
> + if (ret) {
> + pr_err("error %d when get PW mode on lock %s\n", errno, str);
> + goto out;
> + }
> +
> + /* Wait for it to complete */
> + poll_for_ast(dlm_lock_res->ls);
> + *lockid = dlm_lock_res->lksb.sb_lkid;
> +
> + errno = dlm_lock_res->lksb.sb_status;
> + if (errno) {
> + pr_err("error %d happened in ast with lock %s\n", errno, str);
> + goto out;
> + }
> +
> +out:
> + return ret;
> +}
> +
> +int cluster_release_dlmlock(struct supertype *st, int lockid)
> +{
> + int ret = -1;
> +
> + /* if flags with LKF_CONVERT causes below return EINVAL which means
> + * "Invalid argument" */
> + ret = dlm_hookers->ls_unlock(dlm_lock_res->ls, lockid, 0,
> + &dlm_lock_res->lksb, dlm_lock_res);
> + if (ret) {
> + pr_err("error %d happened when unlock\n", errno);
> + /* XXX make sure the lock is unlocked eventually */
> + goto out;
> + }
> +
> + /* Wait for it to complete */
> + poll_for_ast(dlm_lock_res->ls);
> +
> + errno = dlm_lock_res->lksb.sb_status;
> + if (errno != EUNLOCK) {
> + pr_err("error %d happened in ast when unlock lockspace\n", errno);
> + /* XXX make sure the lockspace is unlocked eventually */
> + goto out;
> + }
> +
> + ret = dlm_hookers->release_lockspace(st->cluster_name, dlm_lock_res->ls, 1);
> + if (ret) {
> + pr_err("error %d happened when release lockspace\n", errno);
> + /* XXX make sure the lockspace is released eventually */
> + goto out;
> + }
> + free(dlm_lock_res);
> +
> +out:
> + return ret;
> +}
> +
> +
> /*
> * Parse a 128 bit uuid in 4 integers
> * format is 32 hexx nibbles with options :.<space> separator
> @@ -2043,3 +2173,38 @@ out:
> dlclose(lib_handle);
> return rv;
> }
> +
> +void set_dlm_hookers(void)
> +{
> + dlm_hookers = xmalloc(sizeof(struct dlm_hookers));
> + if (!dlm_hookers)
> + return;
> +
> + dlm_hookers->dlm_handle = dlopen("libdlm_lt.so.3", RTLD_NOW | RTLD_LOCAL);
> + if (!dlm_hookers->dlm_handle)
> + return;
> +
> + dlm_hookers->create_lockspace = dlsym(dlm_hookers->dlm_handle, "dlm_create_lockspace");
> + dlm_hookers->release_lockspace = dlsym(dlm_hookers->dlm_handle, "dlm_release_lockspace");
> + dlm_hookers->ls_lock = dlsym(dlm_hookers->dlm_handle, "dlm_ls_lock");
> + dlm_hookers->ls_unlock = dlsym(dlm_hookers->dlm_handle, "dlm_ls_unlock");
> + dlm_hookers->ls_get_fd = dlsym(dlm_hookers->dlm_handle, "dlm_ls_get_fd");
> + dlm_hookers->dispatch = dlsym(dlm_hookers->dlm_handle, "dlm_dispatch");
> +
> + if (!dlm_hookers->create_lockspace || !dlm_hookers->ls_lock ||
> + !dlm_hookers->ls_unlock || !dlm_hookers->release_lockspace ||
> + !dlm_hookers->ls_get_fd || !dlm_hookers->dispatch)
> + dlclose(dlm_hookers->dlm_handle);
> + else
> + is_dlm_hookers_ready = 1;
> +}
> +
> +void free_dlm_hookers(void)
> +{
> + if (is_dlm_hookers_ready) {
> + dlclose(dlm_hookers->dlm_handle);
> + is_dlm_hookers_ready = 0;
> + }
> + if (dlm_hookers)
> + free(dlm_hookers);
> +}
>
--
Goldwyn
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox