* Re: Help needed recovering from raid failure
From: NeilBrown @ 2015-04-28 22:26 UTC (permalink / raw)
To: Peter van Es; +Cc: linux-raid
In-Reply-To: <4D8713B5-39E7-4EE2-898C-35DC0948B4CA@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4049 bytes --]
On Mon, 27 Apr 2015 11:35:09 +0200 Peter van Es <vanes.peter@gmail.com> wrote:
> Sorry for the long post...
>
> I am running Ubuntu LTS 14.04.02 Server edition, 64 bits, with 4x 2.0TB drives in a raid-5 array.
>
> The 4th drive was beginning to show read errors. Because it was weekend, I could not go out
> and buy a spare 2TB drive to replace the one that was beginning to fail.
>
> I first got a fail event:
>
> This is an automatically generated mail message from mdadm
> running on bali
>
> A Fail event had been detected on md device /dev/md/1.
>
> It could be related to component device /dev/sdd2.
>
> Faithfully yours, etc.
>
> P.S. The /proc/mdstat file currently contains the following:
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md1 : active raid5 sdc2[2] sdb2[1] sda2[0] sdd2[3](F)
> 5854290432 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
>
> md0 : active raid5 sdc1[2] sdd1[3] sdb1[1] sda1[0]
> 5850624 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
>
> unused devices: <none>
>
> And then subsequently, around 18 hours later:
>
> This is an automatically generated mail message from mdadm
> running on bali
>
> A DegradedArray event had been detected on md device /dev/md/1.
This isn't really reporting anything new.
There is probably a daily cron job which reports all degraded arrays. This
message is reported by that job.
>
> Faithfully yours, etc.
>
> P.S. The /proc/mdstat file currently contains the following:
>
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md1 : active raid5 sdc2[2] sdb2[1] sda2[0] sdd2[3](F)
> 5854290432 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
>
> md0 : active raid5 sdc1[2] sdd1[3] sdb1[1] sda1[0]
> 5850624 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
>
> unused devices: <none>
>
> The server had taken the array off line at that point.
Why do you think the array is off-line? The above message doesn't suggest
that.
>
> Needless to say, I can't boot the system anymore as the boot drive is /dev/md0, and GRUB can't
> get at it. I do need to recover data (I know, but there's stuf on there I have no backup for--yet).
You boot off a RAID5? Does grub support that? I didn't know.
But md0 hasn't failed, has it?
Confused.
>
> I booted Linux from a USB stick (which is on /dev/sdc1 hence changing the numbering),
> in recovery mode. Below is the output of /proc/mdstat and
> mdadm --examine. It looks like somehow the /dev/sdd2 and /dev/sde2 drives took on the
> super block of the /dev/md127 device (my swap file). May that have been done by the boot from
> the Ubuntu USB stick?
There is something VERY sick here. I suggest that you tread very carefully.
All your '1' partitions should be about 2GB and the '2' parititions about 2TB
But the --examine output suggests sda2 and sdb2 are 2TB, while sdd2 and sde2
are 2GB.
That really really shouldn't happen. Maybe check your partition table
(fdisk).
I really cannot see how this would happen.
>
> My plan... assemble a degraded array, with /dev/sde2 (the 4th drive, formerly known as /dev/sdd2) not in it.
> Because the fail event put the file system in RO mode, I expect /dev/sdd2 (formerly /dev/sdc2) to be ok.
> Then insert new 2TB drive in slot 4. Let system resync and recover.
>
> I'm running xfs on the /dev/md1 device.
>
> Questions:
>
> 1. is this the wise course of action ?
> 2. how exactly do I reassemble the array (/etc/mdadm.conf is inaccessible in recovery mode)
> 3. what command line options do I use exactly from the --examine output below without screwing things up
>
> And help or pointers gratefully accepted
Can you
mdadm -Ss
to stop all the arrays, then
fdisk -l /dev/sd?
then
mdadm -Esvv
and post all of that. Hopefully some of it will make sense.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: BUG?: RAID6 reshape hung in reshape_request
From: NeilBrown @ 2015-04-29 0:03 UTC (permalink / raw)
To: David Wahler; +Cc: linux-raid
In-Reply-To: <CAGivzjF-5VWBzE0M9FB9BGpYeed=0rxNvKPzkZJdVbUAwbqyJA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2044 bytes --]
On Mon, 27 Apr 2015 12:20:50 -0500 David Wahler <dwahler@gmail.com> wrote:
>
> I don't urgently need this array up and running, so I'm happy to leave
> it in its current state for the next few days in case there's anything
> else I can do to help track this down.
Thanks for the various status data.
I'm fairly easily able to reproduce the problem. I clearly never thought
about 'reshape' when I was writing the bad_block handling.
You can allow the reshape to complete by the following hack:
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 77dfd720aaa0..e6c68a450d4c 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -4306,7 +4306,7 @@ static void handle_stripe(struct stripe_head *sh)
*/
if (s.failed > conf->max_degraded) {
sh->check_state = 0;
- sh->reconstruct_state = 0;
+// sh->reconstruct_state = 0;
if (s.to_read+s.to_write+s.written)
handle_failed_stripe(conf, sh, &s, disks, &s.return_bi);
if (s.syncing + s.replacing)
It may not necessarily do exactly the right thing, but it won't be too bad.
I'm tempted to simply disable reshapes if there are bad blocks, but that
might not be necessary.
The presence of a 'bad block' can mean two things.
1/ The data is missing. If there are enough bad blocks in a stripe then
some data cannot be recovered. In that case we can only let the 'grow'
proceed if we record the destination blocks as 'bad', which isn't too hard.
2/ The media is faulty and writes fail. A 'bad block' doesn't always mean
this, but it can and it is hard to know if it does or not.
This case only really matters when writing. I could probably just
over-write anyway and handle failure as we normally would.
If the 'write' succeeds, I need to clear the 'bad block' record, but I
think I do that anyway.
So I should be able to make it work. I'll probably get mdadm to warn
strongly against reshaping an array with bad blocks though.
I'm going to have to study the code some more.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: PROBLEM: write to jbod with 3TB and 160GB drives hits BUG/oops
From: Charles Bertsch @ 2015-04-29 1:05 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, BertschC@acm.org
In-Reply-To: <LpCG1q00n1hjKLY01pCHNM>
On 04/26/2015 06:11 PM, NeilBrown wrote:
...
> Thanks for the details.
> On the whole, I don't think it is likely that your problem is directly
> related to md - just a coincidence that it happened when you were using md
> things. But one never knows until that actual cause is found.
>
>>
>> > Is there any chance you could use "git bisect" to find out exactly which
>> > commit introduced the problem? That is the mostly likely path to a
>> solution.
>> >
>>
>>....
> NeilBrown
>
Neil,
Thanks for your attention on this matter. I will attempt to narrow down
the release or commit which leads to the symptom.
Charles Bertsch
^ permalink raw reply
* Re: [PATCH 01/10] Add nodes option while creating md
From: NeilBrown @ 2015-04-29 1:30 UTC (permalink / raw)
To: gqjiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-2-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 4442 bytes --]
On Fri, 24 Apr 2015 15:30:32 +0800 gqjiang@suse.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> Specifies the maximum number of nodes in the cluster that may use
> this device simultaneously. This is equivalent to the number of
> bitmaps created in the internal superblock (patches to follow).
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
This doesn't really make much sense coming first in the series. It sets up a
value that is never used.
I would much rather 03/10 came first - which would just create 4 bitmaps.
Then have this patch to allow that '4' to be changed.
> ---
> Create.c | 1 +
> ReadMe.c | 1 +
> mdadm.8.in | 5 +++++
> mdadm.c | 20 +++++++++++++++++++-
> mdadm.h | 3 +++
> 5 files changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/Create.c b/Create.c
> index ef28da0..b73f6cb 100644
> --- a/Create.c
> +++ b/Create.c
> @@ -531,6 +531,7 @@ int Create(struct supertype *st, char *mddev,
> st->ss->name);
> warn = 1;
> }
> + st->nodes = c->nodes;
>
> if (warn) {
> if (c->runstop!= 1) {
> diff --git a/ReadMe.c b/ReadMe.c
> index 87a4916..30c569d 100644
> --- a/ReadMe.c
> +++ b/ReadMe.c
> @@ -140,6 +140,7 @@ struct option long_options[] = {
> {"homehost", 1, 0, HomeHost},
> {"symlinks", 1, 0, Symlinks},
> {"data-offset",1, 0, DataOffset},
> + {"nodes",1, 0, Nodes},
>
> /* For assemble */
> {"uuid", 1, 0, 'u'},
> diff --git a/mdadm.8.in b/mdadm.8.in
> index a630310..bd8d59e 100644
> --- a/mdadm.8.in
> +++ b/mdadm.8.in
> @@ -966,6 +966,11 @@ However for RAID0, it is not possible to add spares. So to increase
> the number of devices in a RAID0, it is necessary to set the new
> number of devices, and to add the new devices, in the same command.
>
> +.TP
> +.BR \-\-nodes
> +Specify the maximum number of nodes in the cluster that will use this
> +device simultaneously. If not specified, this defaults to 4.
> +
I think this should mention that it only makes sense with --bitmap=cluster
(which of course isn't available at this point....)
> .SH For assemble:
>
> .TP
> diff --git a/mdadm.c b/mdadm.c
> index 3e8c49b..bce6a76 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -588,7 +588,14 @@ int main(int argc, char *argv[])
> }
> ident.raid_disks = s.raiddisks;
> continue;
> -
> + case O(CREATE, Nodes):
> + c.nodes = parse_num(optarg);
> + if (c.nodes <= 0) {
> + pr_err("invalid number for the number of "
> + "cluster nodes: %s\n", optarg);
> + exit(2);
> + }
> + continue;
> case O(CREATE,'x'): /* number of spare (eXtra) disks */
> if (s.sparedisks) {
> pr_err("spare-devices set twice: %d and %s\n",
> @@ -1377,6 +1384,17 @@ int main(int argc, char *argv[])
> case CREATE:
> if (c.delay == 0)
> c.delay = DEFAULT_BITMAP_DELAY;
> +
> + if (!strncmp(s.bitmap_file, "internal", 9) ||
> + !strncmp(s.bitmap_file,"none", 4)) {
I'm sorry but I absolutely *hate* this construct.
The '!' at the front makes it seem like you are testing that the string does
*not* have that value, but it is exactly the reverse.
And why "strncmp" rather than "strcmp" ??
Please use
if (strcmp(s.bitmap_file, "internal") == 0 ||
strcmp(s.bitmap_file, "none) == 0) {
except.... what about if bitmap_file in /path/to/somewhere. Presumably you
want to exclude that case too.
May be
if (strcmp(s.bitmap_file, "cluster") != 0) {
??
Thanks,
NeilBrown
> + if (c.nodes) {
> + pr_err("--nodes argument is incompatible with --bitmap=%s.\n",
> + s.bitmap_file);
> + rv = 1;
> + break;
> + }
> + }
> +
> if (s.write_behind && !s.bitmap_file) {
> pr_err("write-behind mode requires a bitmap.\n");
> rv = 1;
> diff --git a/mdadm.h b/mdadm.h
> index 141f963..9d55801 100644
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -344,6 +344,7 @@ enum special_options {
> Dump,
> Restore,
> Action,
> + Nodes,
> };
>
> enum prefix_standard {
> @@ -418,6 +419,7 @@ struct context {
> char *backup_file;
> int invalid_backup;
> char *action;
> + int nodes;
> };
>
> struct shape {
> @@ -1029,6 +1031,7 @@ struct supertype {
> */
> int devcnt;
> int retry_soon;
> + int nodes;
>
> struct mdinfo *devs;
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 03/10] Create n bitmaps for clustered mode
From: NeilBrown @ 2015-04-29 1:36 UTC (permalink / raw)
To: gqjiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-4-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 7113 bytes --]
On Fri, 24 Apr 2015 15:30:34 +0800 gqjiang@suse.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> For a clustered MD, create bitmaps equal to number of nodes so
> each node has an independent bitmap.
>
> Only the first bitmap is has the bits set so that the first node
> that assembles the device also performs the sync.
>
> The bitmaps are aligned to 4k boundaries.
>
> On-disk format:
>
> 0 4k 8k 12k
> -------------------------------------------------------------------
> | idle | md super | bm super [0] + bits |
> | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
> | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
> | bm bits [3, contd] | | |
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> Create.c | 3 ++-
> bitmap.h | 7 +++++--
> mdadm.8.in | 7 ++++++-
> mdadm.c | 17 ++++++++++++++++-
> super1.c | 59 +++++++++++++++++++++++++++++++++++++++++------------------
> 5 files changed, 70 insertions(+), 23 deletions(-)
>
> diff --git a/Create.c b/Create.c
> index cd5485b..9663dc4 100644
> --- a/Create.c
> +++ b/Create.c
> @@ -752,7 +752,8 @@ int Create(struct supertype *st, char *mddev,
> #endif
> }
>
> - if (s->bitmap_file && strcmp(s->bitmap_file, "internal")==0) {
> + if (s->bitmap_file && (strcmp(s->bitmap_file, "internal")==0
> + || strcmp(s->bitmap_file, "clustered")==0)) {
> if ((vers%100) < 2) {
> pr_err("internal bitmaps not supported by this kernel.\n");
> goto abort_locked;
> diff --git a/bitmap.h b/bitmap.h
> index c8725a3..adbf0b4 100644
> --- a/bitmap.h
> +++ b/bitmap.h
> @@ -154,8 +154,11 @@ typedef struct bitmap_super_s {
> __u32 chunksize; /* 52 the bitmap chunk size in bytes */
> __u32 daemon_sleep; /* 56 seconds between disk flushes */
> __u32 write_behind; /* 60 number of outstanding write-behind writes */
> -
> - __u8 pad[256 - 64]; /* set to zero */
> + __u32 sectors_reserved; /* 64 number of 512-byte sectors that are
> + * reserved for the bitmap. */
> + __u32 nodes; /* 68 the maximum number of nodes in cluster. */
> + __u8 cluster_name[64]; /* 72 cluster name to which this md belongs */
> + __u8 pad[256 - 136]; /* set to zero */
> } bitmap_super_t;
>
> /* notes:
> diff --git a/mdadm.8.in b/mdadm.8.in
> index a0e8288..c015cbf 100644
> --- a/mdadm.8.in
> +++ b/mdadm.8.in
> @@ -700,7 +700,12 @@ and so is replicated on all devices. If the word
> .B "none"
> is given with
> .B \-\-grow
> -mode, then any bitmap that is present is removed.
> +mode, then any bitmap that is present is removed. If the word
> +.B "clustered"
> +is given, the array is created for a clustered environment. One bitmap
> +is created for each node as defined by the
> +.B \-\-nodes
> +parameter and are stored internally.
>
> To help catch typing errors, the filename must contain at least one
> slash ('/') if it is a real file (not 'internal' or 'none').
> diff --git a/mdadm.c b/mdadm.c
> index e4f8568..6963a09 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -1111,6 +1111,15 @@ int main(int argc, char *argv[])
> s.bitmap_file = optarg;
> continue;
> }
> + if (strcmp(optarg, "clustered")== 0) {
> + s.bitmap_file = optarg;
> + /* Set the default number of cluster nodes
> + * to 4 if not already set by user
> + */
> + if (c.nodes < 1)
> + c.nodes = 4;
> + continue;
> + }
> /* probable typo */
> pr_err("bitmap file must contain a '/', or be 'internal', or 'none'\n"
> " not '%s'\n", optarg);
> @@ -1404,7 +1413,13 @@ int main(int argc, char *argv[])
> if (c.delay == 0)
> c.delay = DEFAULT_BITMAP_DELAY;
>
> - if (!strncmp(s.bitmap_file, "internal", 9) ||
> + if (!strncmp(s.bitmap_file, "clustered", 9)) {
> + if (s.level != 1) {
> + pr_err("--bitmap=clustered is currently supported with RAID mirror only\n");
> + rv = 1;
> + break;
> + }
> + } else if (!strncmp(s.bitmap_file, "internal", 9) ||
> !strncmp(s.bitmap_file,"none", 4)) {
> if (c.nodes) {
> pr_err("--nodes argument is incompatible with --bitmap=%s.\n",
> diff --git a/super1.c b/super1.c
> index f0508fe..ac1b011 100644
> --- a/super1.c
> +++ b/super1.c
> @@ -2144,6 +2144,10 @@ add_internal_bitmap1(struct supertype *st,
> bms->daemon_sleep = __cpu_to_le32(delay);
> bms->sync_size = __cpu_to_le64(size);
> bms->write_behind = __cpu_to_le32(write_behind);
> + bms->nodes = __cpu_to_le32(st->nodes);
> + if (st->cluster_name)
> + strncpy((char *)bms->cluster_name,
> + st->cluster_name, strlen(st->cluster_name));
>
> *chunkp = chunk;
> return 1;
> @@ -2177,6 +2181,7 @@ static int write_bitmap1(struct supertype *st, int fd)
> void *buf;
> int towrite, n;
> struct align_fd afd;
> + unsigned int i;
>
> init_afd(&afd, fd);
>
> @@ -2185,27 +2190,45 @@ static int write_bitmap1(struct supertype *st, int fd)
> if (posix_memalign(&buf, 4096, 4096))
> return -ENOMEM;
>
> - memset(buf, 0xff, 4096);
> - memcpy(buf, (char *)bms, sizeof(bitmap_super_t));
> -
> - towrite = __le64_to_cpu(bms->sync_size) / (__le32_to_cpu(bms->chunksize)>>9);
> - towrite = (towrite+7) >> 3; /* bits to bytes */
> - towrite += sizeof(bitmap_super_t);
> - towrite = ROUND_UP(towrite, 512);
> - while (towrite > 0) {
> - n = towrite;
> - if (n > 4096)
> - n = 4096;
> - n = awrite(&afd, buf, n);
> - if (n > 0)
> - towrite -= n;
> + /* We use bms->nodes as opposed to st->nodes to
> + * be compatible with write-after-reads such as
> + * the GROW operation.
> + */
> + for (i = 0; i < __le32_to_cpu(bms->nodes); i++) {
> + /* Only the first bitmap should resync
> + * the whole device
> + */
> + if (i)
> + memset(buf, 0x00, 4096);
> else
> + memset(buf, 0xff, 4096);
Why is the first bitmap initialised to 0x00 and the others to 0xff?
If there is a good reason it should be documented either in a comment in the
code or in the changelog entry.
Thanks,
NeilBrown
> + memcpy(buf, (char *)bms, sizeof(bitmap_super_t));
> +
> + towrite = __le64_to_cpu(bms->sync_size) / (__le32_to_cpu(bms->chunksize)>>9);
> + towrite = (towrite+7) >> 3; /* bits to bytes */
> + towrite += sizeof(bitmap_super_t);
> + /* we need the bitmaps to be at 4k boundary */
> + towrite = ROUND_UP(towrite, 4096);
> + while (towrite > 0) {
> + n = towrite;
> + if (n > 4096)
> + n = 4096;
> + n = awrite(&afd, buf, n);
> + if (n > 0)
> + towrite -= n;
> + else
> + break;
> + if (i)
> + memset(buf, 0x00, 4096);
> + else
> + memset(buf, 0xff, 4096);
> + }
> + fsync(fd);
> + if (towrite) {
> + rv = -2;
> break;
> - memset(buf, 0xff, 4096);
> + }
> }
> - fsync(fd);
> - if (towrite)
> - rv = -2;
>
> free(buf);
> return rv;
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 03/10] Create n bitmaps for clustered mode
From: NeilBrown @ 2015-04-29 1:41 UTC (permalink / raw)
To: gqjiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-4-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 2586 bytes --]
On Fri, 24 Apr 2015 15:30:34 +0800 gqjiang@suse.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> For a clustered MD, create bitmaps equal to number of nodes so
> each node has an independent bitmap.
>
> Only the first bitmap is has the bits set so that the first node
> that assembles the device also performs the sync.
>
> The bitmaps are aligned to 4k boundaries.
>
> On-disk format:
>
> 0 4k 8k 12k
> -------------------------------------------------------------------
> | idle | md super | bm super [0] + bits |
> | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
> | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
> | bm bits [3, contd] | | |
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> Create.c | 3 ++-
> bitmap.h | 7 +++++--
> mdadm.8.in | 7 ++++++-
> mdadm.c | 17 ++++++++++++++++-
> super1.c | 59 +++++++++++++++++++++++++++++++++++++++++------------------
> 5 files changed, 70 insertions(+), 23 deletions(-)
>
> diff --git a/Create.c b/Create.c
> index cd5485b..9663dc4 100644
> --- a/Create.c
> +++ b/Create.c
> @@ -752,7 +752,8 @@ int Create(struct supertype *st, char *mddev,
> #endif
> }
>
> - if (s->bitmap_file && strcmp(s->bitmap_file, "internal")==0) {
> + if (s->bitmap_file && (strcmp(s->bitmap_file, "internal")==0
> + || strcmp(s->bitmap_file, "clustered")==0)) {
> if ((vers%100) < 2) {
> pr_err("internal bitmaps not supported by this kernel.\n");
> goto abort_locked;
> diff --git a/bitmap.h b/bitmap.h
> index c8725a3..adbf0b4 100644
> --- a/bitmap.h
> +++ b/bitmap.h
> @@ -154,8 +154,11 @@ typedef struct bitmap_super_s {
> __u32 chunksize; /* 52 the bitmap chunk size in bytes */
> __u32 daemon_sleep; /* 56 seconds between disk flushes */
> __u32 write_behind; /* 60 number of outstanding write-behind writes */
> -
> - __u8 pad[256 - 64]; /* set to zero */
> + __u32 sectors_reserved; /* 64 number of 512-byte sectors that are
> + * reserved for the bitmap. */
> + __u32 nodes; /* 68 the maximum number of nodes in cluster. */
> + __u8 cluster_name[64]; /* 72 cluster name to which this md belongs */
> + __u8 pad[256 - 136]; /* set to zero */
> } bitmap_super_t;
I missed this the first time, but these fields that you have added need to be
added to sb_le_to_cpu().
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 04/10] Show all bitmaps while examining bitmap
From: NeilBrown @ 2015-04-29 1:41 UTC (permalink / raw)
To: gqjiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-5-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 2941 bytes --]
On Fri, 24 Apr 2015 15:30:35 +0800 gqjiang@suse.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> This adds capability of exmining bitmaps corresponding to all
> nodes/slots on the device.
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> bitmap.c | 35 ++++++++++++++++++++++++++---------
> 1 file changed, 26 insertions(+), 9 deletions(-)
>
> diff --git a/bitmap.c b/bitmap.c
> index b1d54a6..ab83f4e 100644
> --- a/bitmap.c
> +++ b/bitmap.c
> @@ -258,7 +258,7 @@ int ExamineBitmap(char *filename, int brief, struct supertype *st)
> int rv = 1;
> char buf[64];
> int swap;
> - int fd;
> + int fd, i;
> __u32 uuid32[4];
>
> fd = bitmap_file_open(filename, &st);
> @@ -315,9 +315,6 @@ int ExamineBitmap(char *filename, int brief, struct supertype *st)
> uuid32[2],
> uuid32[3]);
>
> - printf(" Events : %llu\n", (unsigned long long)sb->events);
> - printf(" Events Cleared : %llu\n", (unsigned long long)sb->events_cleared);
> - printf(" State : %s\n", bitmap_state(sb->state));
Can we please leave this where it is in the case where there is only one node?
Only move it down if there are multiple nodes to report on.
NeilBrown
> printf(" Chunksize : %s\n", human_chunksize(sb->chunksize));
> printf(" Daemon : %ds flush period\n", sb->daemon_sleep);
> if (sb->write_behind)
> @@ -327,11 +324,31 @@ int ExamineBitmap(char *filename, int brief, struct supertype *st)
> printf(" Write Mode : %s\n", buf);
> printf(" Sync Size : %llu%s\n", (unsigned long long)sb->sync_size/2,
> human_size(sb->sync_size * 512));
> - if (brief)
> - goto free_info;
> - printf(" Bitmap : %llu bits (chunks), %llu dirty (%2.1f%%)\n",
> - info->total_bits, info->dirty_bits,
> - 100.0 * info->dirty_bits / (info->total_bits?:1));
> + if (sb->nodes > 0) {
> + printf(" Cluster nodes : %d\n", sb->nodes);
> + printf(" Cluster name : %s\n", sb->cluster_name);
> + }
> + i = 0;
> + do {
> + if (i) {
> + free(info);
> + info = bitmap_fd_read(fd, brief);
> + sb = &info->sb;
> + }
> + if (sb->magic != BITMAP_MAGIC) {
> + pr_err("invalid bitmap magic 0x%x, the bitmap file appears to be corrupted\n", sb->magic);
> + }
> + printf(" Node Slot : %d\n", i);
> + printf(" Events : %llu\n", (unsigned long long)sb->events);
> + printf(" Events Cleared : %llu\n", (unsigned long long)sb->events_cleared);
> + printf(" State : %s\n", bitmap_state(sb->state));
> + if (brief)
> + continue;
> + printf(" Bitmap : %llu bits (chunks), %llu dirty (%2.1f%%)\n",
> + info->total_bits, info->dirty_bits,
> + 100.0 * info->dirty_bits / (info->total_bits?:1));
> + } while (++i < (int)sb->nodes);
> +
> free_info:
> free(info);
> return rv;
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 1/3] wait: introduce wait_event_cmd_exclusive
From: Yuanhan Liu @ 2015-04-29 1:45 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: neilb, linux-raid, linux-kernel, Ingo Molnar
In-Reply-To: <20150428141315.GD23123@twins.programming.kicks-ass.net>
On Tue, Apr 28, 2015 at 04:13:15PM +0200, Peter Zijlstra wrote:
> On Mon, Apr 27, 2015 at 12:51:01PM +0800, Yuanhan Liu wrote:
> > It's just a variant of wait_event_cmd, with exclusive flag being set.
> >
> > For cases like RAID5, which puts many processes to sleep until 1/4
> > resources are free, a wake_up wakes up all processes to run, but
> > there is one process being able to get the resource as it's protected
> > by a spin lock. That ends up introducing heavy lock contentions, and
> > hurts performance badly.
> >
> > Here introduce wait_event_cmd_exclusive to relieve the lock contention
> > naturally by letting wake_up() just wake up one process.
> >
> > Cc: Ingo Molnar <mingo@redhat.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
> > ---
> > include/linux/wait.h | 14 +++++++++++---
> > 1 file changed, 11 insertions(+), 3 deletions(-)
> >
> > diff --git a/include/linux/wait.h b/include/linux/wait.h
> > index 2db8334..6c3b4de 100644
> > --- a/include/linux/wait.h
> > +++ b/include/linux/wait.h
> > @@ -358,10 +358,18 @@ do { \
> > __ret; \
> > })
> >
> > -#define __wait_event_cmd(wq, condition, cmd1, cmd2) \
> > - (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
> > +#define __wait_event_cmd(wq, condition, cmd1, cmd2, exclusive) \
> > + (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, exclusive, 0, \
> > cmd1; schedule(); cmd2)
> >
> > +
> > +#define wait_event_cmd_exclusive(wq, condition, cmd1, cmd2) \
> > +do { \
> > + if (condition) \
> > + break; \
> > + __wait_event_cmd(wq, condition, cmd1, cmd2, 1); \
> > +} while (0)
> > +
> > /**
> > * wait_event_cmd - sleep until a condition gets true
> > * @wq: the waitqueue to wait on
> > @@ -380,7 +388,7 @@ do { \
> > do { \
> > if (condition) \
> > break; \
> > - __wait_event_cmd(wq, condition, cmd1, cmd2); \
> > + __wait_event_cmd(wq, condition, cmd1, cmd2, 0); \
> > } while (0)
> >
>
> No, that's wrong, its assumed that wait*() and __wait*() have the same
> arguments.
Thanks. Will send an updated patch soon.
--yliu
^ permalink raw reply
* Re: [PATCH 05/10] Add a new clustered disk
From: NeilBrown @ 2015-04-29 1:45 UTC (permalink / raw)
To: gqjiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-6-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 9758 bytes --]
On Fri, 24 Apr 2015 15:30:36 +0800 gqjiang@suse.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> A clustered disk is added by the traditional --add sequence.
> However, other nodes need to acknowledge that they can "see"
> the device. This is done by --cluster-confirm:
>
> --cluster-confirm Y:/dev/whatever (if disk is found)
> or
> --cluster-confirm Y:missing (if disk is not found)
>
> The node initiating the --add, has the disk state tagged with
> MD_DISK_CLUSTER_ADD and the one confirming tag the disk with
> MD_DISK_CANDIDATE.
You haven't explained 'Y' here. It looks like it means 'Yes', but it doesn't.
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> Manage.c | 33 +++++++++++++++++++++++++++++----
> ReadMe.c | 1 +
> md_p.h | 7 +++++++
> md_u.h | 1 +
> mdadm.8.in | 9 +++++++++
> mdadm.c | 4 ++++
> mdadm.h | 2 ++
> util.c | 11 +++++++++++
> 8 files changed, 64 insertions(+), 4 deletions(-)
>
> diff --git a/Manage.c b/Manage.c
> index d3cfb55..4c3d451 100644
> --- a/Manage.c
> +++ b/Manage.c
> @@ -690,7 +690,8 @@ skip_re_add:
> int Manage_add(int fd, int tfd, struct mddev_dev *dv,
> struct supertype *tst, mdu_array_info_t *array,
> int force, int verbose, char *devname,
> - char *update, unsigned long rdev, unsigned long long array_size)
> + char *update, unsigned long rdev, unsigned long long array_size,
> + int raid_slot)
> {
> unsigned long long ldsize;
> struct supertype *dev_st = NULL;
> @@ -879,7 +880,10 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
> }
> disc.major = major(rdev);
> disc.minor = minor(rdev);
> - disc.number =j;
> + if (raid_slot < 0)
> + disc.number = j;
> + else
> + disc.number = raid_slot;
> disc.state = 0;
> if (array->not_persistent==0) {
> int dfd;
> @@ -920,6 +924,14 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
> }
> free(used);
> }
> +
> + if (array->state & (1 << MD_SB_CLUSTERED)) {
> + if (dv->disposition == 'c')
> + disc.state |= (1 << MD_DISK_CANDIDATE);
> + else
> + disc.state |= (1 << MD_DISK_CLUSTER_ADD);
> + }
> +
> if (dv->writemostly == 1)
> disc.state |= (1 << MD_DISK_WRITEMOSTLY);
> if (tst->ss->external) {
> @@ -1239,6 +1251,7 @@ int Manage_subdevs(char *devname, int fd,
> * variant on 'A'
> * 'F' - Another variant of 'A', where the device was faulty
> * so must be removed from the array first.
> + * 'c' - confirm the device as found (for clustered environments)
> *
> * For 'f' and 'r', the device can also be a kernel-internal
> * name such as 'sdb'.
> @@ -1254,6 +1267,7 @@ int Manage_subdevs(char *devname, int fd,
> struct mdinfo info;
> int frozen = 0;
> int busy = 0;
> + int raid_slot = -1;
>
> if (ioctl(fd, GET_ARRAY_INFO, &array)) {
> pr_err("Cannot get array info for %s\n",
> @@ -1282,6 +1296,11 @@ int Manage_subdevs(char *devname, int fd,
> int rv;
> int mj,mn;
>
> + raid_slot = -1;
> + if (dv->disposition == 'c')
> + parse_cluster_confirm_arg(dv->devname, &dv->devname,
> + &raid_slot);
> +
> if (strcmp(dv->devname, "failed") == 0 ||
> strcmp(dv->devname, "faulty") == 0) {
> if (dv->disposition != 'A'
> @@ -1307,6 +1326,11 @@ int Manage_subdevs(char *devname, int fd,
> if (strcmp(dv->devname, "missing") == 0) {
> struct mddev_dev *add_devlist = NULL;
> struct mddev_dev **dp;
> + if (dv->disposition == 'c') {
> + rv = ioctl(fd, CLUSTERED_DISK_NACK, NULL);
> + break;
> + }
> +
> if (dv->disposition != 'A') {
> pr_err("'missing' only meaningful with --re-add\n");
> goto abort;
> @@ -1399,7 +1423,7 @@ int Manage_subdevs(char *devname, int fd,
> else {
> int open_err = errno;
> if (stat(dv->devname, &stb) != 0) {
> - pr_err("Cannot find %s: %s\n",
> + pr_err("%s: %d Cannot find %s: %s\n", __func__, __LINE__,
> dv->devname, strerror(errno));
> goto abort;
> }
> @@ -1437,6 +1461,7 @@ int Manage_subdevs(char *devname, int fd,
> case 'A':
> case 'M': /* --re-add missing */
> case 'F': /* --re-add faulty */
> + case 'c': /* --cluster-confirm */
> /* add the device */
> if (subarray) {
> pr_err("Cannot add disks to a \'member\' array, perform this operation on the parent container\n");
> @@ -1470,7 +1495,7 @@ int Manage_subdevs(char *devname, int fd,
> }
> rv = Manage_add(fd, tfd, dv, tst, &array,
> force, verbose, devname, update,
> - rdev, array_size);
> + rdev, array_size, raid_slot);
> close(tfd);
> tfd = -1;
> if (rv < 0)
> diff --git a/ReadMe.c b/ReadMe.c
> index c6286ae..c854cd5 100644
> --- a/ReadMe.c
> +++ b/ReadMe.c
> @@ -169,6 +169,7 @@ struct option long_options[] = {
> {"wait", 0, 0, WaitOpt},
> {"wait-clean", 0, 0, Waitclean },
> {"action", 1, 0, Action },
> + {"cluster-confirm", 0, 0, ClusterConfirm},
>
> /* For Detail/Examine */
> {"brief", 0, 0, Brief},
> diff --git a/md_p.h b/md_p.h
> index c4846ba..e59504f 100644
> --- a/md_p.h
> +++ b/md_p.h
> @@ -78,6 +78,12 @@
> #define MD_DISK_ACTIVE 1 /* disk is running but may not be in sync */
> #define MD_DISK_SYNC 2 /* disk is in sync with the raid set */
> #define MD_DISK_REMOVED 3 /* disk is in sync with the raid set */
> +#define MD_DISK_CLUSTER_ADD 4 /* Initiate a disk add across the cluster
> + * For clustered enviroments only.
> + */
> +#define MD_DISK_CANDIDATE 5 /* disk is added as spare (local) until confirmed
> + * For clustered enviroments only.
> + */
>
> #define MD_DISK_WRITEMOSTLY 9 /* disk is "write-mostly" is RAID1 config.
> * read requests will only be sent here in
> @@ -106,6 +112,7 @@ typedef struct mdp_device_descriptor_s {
> #define MD_SB_BLOCK_CONTAINER_RESHAPE 3 /* block container wide reshapes */
> #define MD_SB_BLOCK_VOLUME 4 /* block activation of array, other arrays
> * in container can be activated */
> +#define MD_SB_CLUSTERED 5 /* MD is clustered */
> #define MD_SB_BITMAP_PRESENT 8 /* bitmap may be present nearby */
>
> typedef struct mdp_superblock_s {
> diff --git a/md_u.h b/md_u.h
> index be9868a..76068d6 100644
> --- a/md_u.h
> +++ b/md_u.h
> @@ -44,6 +44,7 @@
> #define STOP_ARRAY _IO (MD_MAJOR, 0x32)
> #define STOP_ARRAY_RO _IO (MD_MAJOR, 0x33)
> #define RESTART_ARRAY_RW _IO (MD_MAJOR, 0x34)
> +#define CLUSTERED_DISK_NACK _IO (MD_MAJOR, 0x35)
>
> typedef struct mdu_version_s {
> int major;
> diff --git a/mdadm.8.in b/mdadm.8.in
> index c015cbf..6873cc7 100644
> --- a/mdadm.8.in
> +++ b/mdadm.8.in
> @@ -1405,6 +1405,15 @@ will avoid reading from these devices if possible.
> .BR \-\-readwrite
> Subsequent devices that are added or re\-added will have the 'write-mostly'
> flag cleared.
> +.TP
> +.BR \-\-cluster\-confirm
> +Confirm the existence of the device. This is issued in response to an \-\-add
> +request by a node in a cluster. When a node adds a device it sends a message
> +to all nodes in the cluster to look for a device with a UUID. This translates
> +to a udev notification with the UUID of the device to be added and the slot
> +number. The receiving node must acknowledge this message
> +with \-\-cluster\-confirm. Valid arguments are <slot>:<devicename> in case
> +the device is found or <slot>:missing in case the device is not found.
>
> .P
> Each of these options requires that the first device listed is the array
> diff --git a/mdadm.c b/mdadm.c
> index 6963a09..5b4b3ef 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -196,6 +196,7 @@ int main(int argc, char *argv[])
> case 'f':
> case Fail:
> case ReAdd: /* re-add */
> + case ClusterConfirm:
> if (!mode) {
> newmode = MANAGE;
> shortopt = short_bitmap_options;
> @@ -933,6 +934,9 @@ int main(int argc, char *argv[])
> * remove the device */
> devmode = 'f';
> continue;
> + case O(MANAGE, ClusterConfirm):
> + devmode = 'c';
> + continue;
> case O(MANAGE,Replace):
> /* Mark these devices for replacement */
> devmode = 'R';
> diff --git a/mdadm.h b/mdadm.h
> index f56d9d6..00c726e 100644
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -346,6 +346,7 @@ enum special_options {
> Action,
> Nodes,
> ClusterName,
> + ClusterConfirm,
> };
>
> enum prefix_standard {
> @@ -1281,6 +1282,7 @@ extern int parse_uuid(char *str, int uuid[4]);
> extern int parse_layout_10(char *layout);
> extern int parse_layout_faulty(char *layout);
> extern long parse_num(char *num);
> +extern int parse_cluster_confirm_arg(char *inp, char **devname, int *slot);
> extern int check_ext2(int fd, char *name);
> extern int check_reiser(int fd, char *name);
> extern int check_raid(int fd, char *name);
> diff --git a/util.c b/util.c
> index ed9a745..1d82fc7 100644
> --- a/util.c
> +++ b/util.c
> @@ -273,6 +273,17 @@ long parse_num(char *num)
> }
> #endif
>
> +int parse_cluster_confirm_arg(char *input, char **devname, int *slot)
> +{
> + char *dev;
> + *slot = strtoul(input, &dev, 10);
> + if (dev[0] == ':')
> + *devname = dev+1;
> + else
> + return -1;
> + return 0;
> +}
The logic here hurts my brain :-(
*slot = strtoul(input, &dev, 10);
if (dev == input || dev[0] != ':')
return -1;
*devname = dev+1;
return 0;
> +
> void remove_partitions(int fd)
> {
> /* remove partitions from this block devices.
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 08/10] mdadm: add the ability to change cluster name
From: NeilBrown @ 2015-04-29 1:50 UTC (permalink / raw)
To: gqjiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-9-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 6994 bytes --]
On Fri, 24 Apr 2015 15:30:39 +0800 gqjiang@suse.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> To support change the cluster name, the commit do the followings:
>
> 1. extend original write_bitmap function for new scenario.
> 2. add the scenarion to handle the modification of cluster's name
> in write_bitmap1.
> 3. make update_super1 can change the name in mdp_superblock_1.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> Assemble.c | 5 +++++
> Grow.c | 2 +-
> mdadm.c | 3 +++
> mdadm.h | 7 ++++++-
> super0.c | 4 ++--
> super1.c | 46 ++++++++++++++++++++++++++++++++++++++++++++--
> 6 files changed, 61 insertions(+), 6 deletions(-)
>
> diff --git a/Assemble.c b/Assemble.c
> index 25a103d..e1b846c 100644
> --- a/Assemble.c
> +++ b/Assemble.c
> @@ -644,6 +644,11 @@ static int load_devices(struct devs *devices, char *devmap,
> *stp = st;
> return -1;
> }
> + if (strcmp(c->update, "home-cluster") == 0) {
> + err = tst->ss->update_super(tst, content, c->update,
> + devname, 0, 0, c->homecluster);
> + tst->ss->write_bitmap(tst, dfd, NameUpdate);
> + }
> if (strcmp(c->update, "uuid")==0 &&
> !ident->uuid_set) {
> ident->uuid_set = 1;
> diff --git a/Grow.c b/Grow.c
> index 1122cec..bf44e66 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -420,7 +420,7 @@ int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
> bitmapsize, offset_setable,
> major)
> )
> - st->ss->write_bitmap(st, fd2);
> + st->ss->write_bitmap(st, fd2, NoUpdate);
> else {
> pr_err("failed to create internal bitmap - chunksize problem.\n");
> close(fd2);
> diff --git a/mdadm.c b/mdadm.c
> index 5b4b3ef..20f195d 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -598,6 +598,7 @@ int main(int argc, char *argv[])
> }
> continue;
> case O(CREATE, ClusterName):
> + case O(ASSEMBLE, ClusterName):
> c.homecluster = optarg;
> if (strlen(c.homecluster) > 64) {
> pr_err("Cluster name too big.\n");
> @@ -741,6 +742,8 @@ int main(int argc, char *argv[])
> continue;
> if (strcmp(c.update, "homehost")==0)
> continue;
> + if (strcmp(c.update, "home-cluster")==0)
> + continue;
> if (strcmp(c.update, "devicesize")==0)
> continue;
> if (strcmp(c.update, "no-bitmap")==0)
> diff --git a/mdadm.h b/mdadm.h
> index 00c726e..d8b0749 100644
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -354,6 +354,11 @@ enum prefix_standard {
> IEC
> };
>
> +enum bitmap_update {
> + NoUpdate,
> + NameUpdate,
> +};
> +
> /* structures read from config file */
> /* List of mddevice names and identifiers
> * Identifiers can be:
> @@ -850,7 +855,7 @@ extern struct superswitch {
> /* if add_internal_bitmap succeeded for existing array, this
> * writes it out.
> */
> - int (*write_bitmap)(struct supertype *st, int fd);
> + int (*write_bitmap)(struct supertype *st, int fd, enum bitmap_update update);
> /* Free the superblock and any other allocated data */
> void (*free_super)(struct supertype *st);
>
> diff --git a/super0.c b/super0.c
> index deb5999..6ad9d39 100644
> --- a/super0.c
> +++ b/super0.c
> @@ -900,7 +900,7 @@ static int write_init_super0(struct supertype *st)
> rv = store_super0(st, di->fd);
>
> if (rv == 0 && (sb->state & (1<<MD_SB_BITMAP_PRESENT)))
> - rv = st->ss->write_bitmap(st, di->fd);
> + rv = st->ss->write_bitmap(st, di->fd, NoUpdate);
>
> if (rv)
> pr_err("failed to write superblock to %s\n",
> @@ -1175,7 +1175,7 @@ static void locate_bitmap0(struct supertype *st, int fd)
> lseek64(fd, offset, 0);
> }
>
> -static int write_bitmap0(struct supertype *st, int fd)
> +static int write_bitmap0(struct supertype *st, int fd, enum bitmap_update update)
> {
> unsigned long long dsize;
> unsigned long long offset;
> diff --git a/super1.c b/super1.c
> index 3c2fce2..e43bef1 100644
> --- a/super1.c
> +++ b/super1.c
> @@ -1073,6 +1073,25 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
> info->name[32] = 0;
> }
>
> + if (strcmp(update, "home-cluster") == 0 &&
> + homehost) {
> + /* Note that 'home-cluster' is to change the name of cluster,
> + * it is another "name" update.
> + */
> + char *new_name = xmalloc(sizeof(sb->set_name));
> + if (strrchr(sb->set_name, ':')) {
> + strcpy(new_name, strchr(sb->set_name, ':'));
> + }
> +
> + memset(sb->set_name, 0, sizeof(sb->set_name));
> + strcpy(sb->set_name, homehost);
> + if (new_name)
> + strcat(sb->set_name, new_name);
> +
> + free(new_name);
> + goto out;
> + }
> +
Please get rid of the 'goto out' and put an 'else' in here.
"homehost" is special because it translates to "name".
"home-cluster" is not special.
> if (strcmp(update, "force-one")==0) {
> /* Not enough devices for a working array,
> * so bring this one up-to-date
> @@ -1313,6 +1332,7 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
> else
> rv = -1;
>
> +out:
> sb->sb_csum = calc_sb_1_csum(sb);
> return rv;
> }
> @@ -1691,7 +1711,7 @@ static int write_init_super1(struct supertype *st)
> sb->sb_csum = calc_sb_1_csum(sb);
> rv = store_super1(st, di->fd);
> if (rv == 0 && (__le32_to_cpu(sb->feature_map) & 1))
> - rv = st->ss->write_bitmap(st, di->fd);
> + rv = st->ss->write_bitmap(st, di->fd, NoUpdate);
> close(di->fd);
> di->fd = -1;
> if (rv)
> @@ -2175,7 +2195,7 @@ static void locate_bitmap1(struct supertype *st, int fd)
> lseek64(fd, offset<<9, 0);
> }
>
> -static int write_bitmap1(struct supertype *st, int fd)
> +static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update)
> {
> struct mdp_superblock_1 *sb = st->sb;
> bitmap_super_t *bms = (bitmap_super_t*)(((char*)sb)+MAX_SB_SIZE);
> @@ -2185,6 +2205,28 @@ static int write_bitmap1(struct supertype *st, int fd)
> struct align_fd afd;
> unsigned int i;
>
> + switch (update) {
> + case NameUpdate:
> + {
> + char *new_name = xmalloc(sizeof(sb->set_name));
> +
> + strncpy(new_name, sb->set_name, sizeof(sb->set_name));
> + memset((char *)bms->cluster_name, 0, sizeof(bms->cluster_name));
> +
> + if (strtok(new_name, ":"))
> + strncpy((char *)bms->cluster_name, new_name, strlen(sb->set_name));
> + else
> + /* In case the original set_name doesn't like aaa:md* */
> + strncpy((char *)bms->cluster_name, sb->set_name, strlen(sb->set_name));
> +
> + free(new_name);
> + break;
> + }
I don't like the braces there - too confusing.
Just make 'new_name' a top-level variable and get rid of the {}
Thanks,
NeilBrown
> + case NoUpdate:
> + default:
> + break;
> + }
> +
> init_afd(&afd, fd);
>
> locate_bitmap1(st, fd);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 09/10] mdadm: change the num of cluster node
From: NeilBrown @ 2015-04-29 1:51 UTC (permalink / raw)
To: gqjiang; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-10-git-send-email-gqjiang@suse.com>
[-- Attachment #1: Type: text/plain, Size: 3319 bytes --]
On Fri, 24 Apr 2015 15:30:40 +0800 gqjiang@suse.com wrote:
> From: Guoqing Jiang <gqjiang@suse.com>
>
> This extends nodes option for assemble mode, make the num of
> cluster node could be change by user.
>
> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
> ---
> Assemble.c | 4 ++++
> ReadMe.c | 2 +-
> mdadm.c | 3 +++
> mdadm.h | 1 +
> super1.c | 6 ++++++
> 5 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/Assemble.c b/Assemble.c
> index e1b846c..22042a9 100644
> --- a/Assemble.c
> +++ b/Assemble.c
> @@ -649,6 +649,10 @@ static int load_devices(struct devs *devices, char *devmap,
> devname, 0, 0, c->homecluster);
> tst->ss->write_bitmap(tst, dfd, NameUpdate);
> }
> + if (strcmp(c->update, "nodes") == 0) {
> + tst->nodes = c->nodes;
> + tst->ss->write_bitmap(tst, dfd, NodeNumUpdate);
> + }
Doesn't there need to be some test that there is enough free space on all
devices to store the extra bitmaps (when nodes is increasing)??
NeilBrown
> if (strcmp(c->update, "uuid")==0 &&
> !ident->uuid_set) {
> ident->uuid_set = 1;
> diff --git a/ReadMe.c b/ReadMe.c
> index c854cd5..d1830e1 100644
> --- a/ReadMe.c
> +++ b/ReadMe.c
> @@ -140,7 +140,7 @@ struct option long_options[] = {
> {"homehost", 1, 0, HomeHost},
> {"symlinks", 1, 0, Symlinks},
> {"data-offset",1, 0, DataOffset},
> - {"nodes",1, 0, Nodes},
> + {"nodes",1, 0, Nodes}, /* also for --assemble */
> {"home-cluster",1, 0, ClusterName},
>
> /* For assemble */
> diff --git a/mdadm.c b/mdadm.c
> index 20f195d..344bde2 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -589,6 +589,7 @@ int main(int argc, char *argv[])
> }
> ident.raid_disks = s.raiddisks;
> continue;
> + case O(ASSEMBLE, Nodes):
> case O(CREATE, Nodes):
> c.nodes = parse_num(optarg);
> if (c.nodes <= 0) {
> @@ -744,6 +745,8 @@ int main(int argc, char *argv[])
> continue;
> if (strcmp(c.update, "home-cluster")==0)
> continue;
> + if (strcmp(c.update, "nodes")==0)
> + continue;
> if (strcmp(c.update, "devicesize")==0)
> continue;
> if (strcmp(c.update, "no-bitmap")==0)
> diff --git a/mdadm.h b/mdadm.h
> index d8b0749..97892e6 100644
> --- a/mdadm.h
> +++ b/mdadm.h
> @@ -357,6 +357,7 @@ enum prefix_standard {
> enum bitmap_update {
> NoUpdate,
> NameUpdate,
> + NodeNumUpdate,
> };
>
> /* structures read from config file */
> diff --git a/super1.c b/super1.c
> index e43bef1..047e799 100644
> --- a/super1.c
> +++ b/super1.c
> @@ -1329,6 +1329,9 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
> sb->devflags |= WriteMostly1;
> else if (strcmp(update, "readwrite")==0)
> sb->devflags &= ~WriteMostly1;
> + else if (strcmp(update, "nodes")==0)
> + /* Just a placeholder since no related member in mdp_superblock_1 */
> + ;
> else
> rv = -1;
>
> @@ -2222,6 +2225,9 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
> free(new_name);
> break;
> }
> + case NodeNumUpdate:
> + bms->nodes = __cpu_to_le32(st->nodes);
> + break;
> case NoUpdate:
> default:
> break;
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 03/10] Create n bitmaps for clustered mode
From: Goldwyn Rodrigues @ 2015-04-29 2:41 UTC (permalink / raw)
To: NeilBrown, gqjiang; +Cc: linux-raid
In-Reply-To: <20150429113632.0a211e3c@notabene.brown>
On 04/28/2015 08:36 PM, NeilBrown wrote:
> On Fri, 24 Apr 2015 15:30:34 +0800 gqjiang@suse.com wrote:
>
>> From: Guoqing Jiang <gqjiang@suse.com>
>>
>> For a clustered MD, create bitmaps equal to number of nodes so
>> each node has an independent bitmap.
>>
>> Only the first bitmap is has the bits set so that the first node
>> that assembles the device also performs the sync.
>>
>> The bitmaps are aligned to 4k boundaries.
>>
>> On-disk format:
>>
>> 0 4k 8k 12k
>> -------------------------------------------------------------------
>> | idle | md super | bm super [0] + bits |
>> | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
>> | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
>> | bm bits [3, contd] | | |
>>
>> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
>> Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
>> ---
>> Create.c | 3 ++-
>> bitmap.h | 7 +++++--
>> mdadm.8.in | 7 ++++++-
>> mdadm.c | 17 ++++++++++++++++-
>> super1.c | 59 +++++++++++++++++++++++++++++++++++++++++------------------
>> 5 files changed, 70 insertions(+), 23 deletions(-)
>>
>> diff --git a/Create.c b/Create.c
>> index cd5485b..9663dc4 100644
>> --- a/Create.c
>> +++ b/Create.c
>> @@ -752,7 +752,8 @@ int Create(struct supertype *st, char *mddev,
>> #endif
>> }
>>
>> - if (s->bitmap_file && strcmp(s->bitmap_file, "internal")==0) {
>> + if (s->bitmap_file && (strcmp(s->bitmap_file, "internal")==0
>> + || strcmp(s->bitmap_file, "clustered")==0)) {
>> if ((vers%100) < 2) {
>> pr_err("internal bitmaps not supported by this kernel.\n");
>> goto abort_locked;
>> diff --git a/bitmap.h b/bitmap.h
>> index c8725a3..adbf0b4 100644
>> --- a/bitmap.h
>> +++ b/bitmap.h
>> @@ -154,8 +154,11 @@ typedef struct bitmap_super_s {
>> __u32 chunksize; /* 52 the bitmap chunk size in bytes */
>> __u32 daemon_sleep; /* 56 seconds between disk flushes */
>> __u32 write_behind; /* 60 number of outstanding write-behind writes */
>> -
>> - __u8 pad[256 - 64]; /* set to zero */
>> + __u32 sectors_reserved; /* 64 number of 512-byte sectors that are
>> + * reserved for the bitmap. */
>> + __u32 nodes; /* 68 the maximum number of nodes in cluster. */
>> + __u8 cluster_name[64]; /* 72 cluster name to which this md belongs */
>> + __u8 pad[256 - 136]; /* set to zero */
>> } bitmap_super_t;
>>
>> /* notes:
>> diff --git a/mdadm.8.in b/mdadm.8.in
>> index a0e8288..c015cbf 100644
>> --- a/mdadm.8.in
>> +++ b/mdadm.8.in
>> @@ -700,7 +700,12 @@ and so is replicated on all devices. If the word
>> .B "none"
>> is given with
>> .B \-\-grow
>> -mode, then any bitmap that is present is removed.
>> +mode, then any bitmap that is present is removed. If the word
>> +.B "clustered"
>> +is given, the array is created for a clustered environment. One bitmap
>> +is created for each node as defined by the
>> +.B \-\-nodes
>> +parameter and are stored internally.
>>
>> To help catch typing errors, the filename must contain at least one
>> slash ('/') if it is a real file (not 'internal' or 'none').
>> diff --git a/mdadm.c b/mdadm.c
>> index e4f8568..6963a09 100644
>> --- a/mdadm.c
>> +++ b/mdadm.c
>> @@ -1111,6 +1111,15 @@ int main(int argc, char *argv[])
>> s.bitmap_file = optarg;
>> continue;
>> }
>> + if (strcmp(optarg, "clustered")== 0) {
>> + s.bitmap_file = optarg;
>> + /* Set the default number of cluster nodes
>> + * to 4 if not already set by user
>> + */
>> + if (c.nodes < 1)
>> + c.nodes = 4;
>> + continue;
>> + }
>> /* probable typo */
>> pr_err("bitmap file must contain a '/', or be 'internal', or 'none'\n"
>> " not '%s'\n", optarg);
>> @@ -1404,7 +1413,13 @@ int main(int argc, char *argv[])
>> if (c.delay == 0)
>> c.delay = DEFAULT_BITMAP_DELAY;
>>
>> - if (!strncmp(s.bitmap_file, "internal", 9) ||
>> + if (!strncmp(s.bitmap_file, "clustered", 9)) {
>> + if (s.level != 1) {
>> + pr_err("--bitmap=clustered is currently supported with RAID mirror only\n");
>> + rv = 1;
>> + break;
>> + }
>> + } else if (!strncmp(s.bitmap_file, "internal", 9) ||
>> !strncmp(s.bitmap_file,"none", 4)) {
>> if (c.nodes) {
>> pr_err("--nodes argument is incompatible with --bitmap=%s.\n",
>> diff --git a/super1.c b/super1.c
>> index f0508fe..ac1b011 100644
>> --- a/super1.c
>> +++ b/super1.c
>> @@ -2144,6 +2144,10 @@ add_internal_bitmap1(struct supertype *st,
>> bms->daemon_sleep = __cpu_to_le32(delay);
>> bms->sync_size = __cpu_to_le64(size);
>> bms->write_behind = __cpu_to_le32(write_behind);
>> + bms->nodes = __cpu_to_le32(st->nodes);
>> + if (st->cluster_name)
>> + strncpy((char *)bms->cluster_name,
>> + st->cluster_name, strlen(st->cluster_name));
>>
>> *chunkp = chunk;
>> return 1;
>> @@ -2177,6 +2181,7 @@ static int write_bitmap1(struct supertype *st, int fd)
>> void *buf;
>> int towrite, n;
>> struct align_fd afd;
>> + unsigned int i;
>>
>> init_afd(&afd, fd);
>>
>> @@ -2185,27 +2190,45 @@ static int write_bitmap1(struct supertype *st, int fd)
>> if (posix_memalign(&buf, 4096, 4096))
>> return -ENOMEM;
>>
>> - memset(buf, 0xff, 4096);
>> - memcpy(buf, (char *)bms, sizeof(bitmap_super_t));
>> -
>> - towrite = __le64_to_cpu(bms->sync_size) / (__le32_to_cpu(bms->chunksize)>>9);
>> - towrite = (towrite+7) >> 3; /* bits to bytes */
>> - towrite += sizeof(bitmap_super_t);
>> - towrite = ROUND_UP(towrite, 512);
>> - while (towrite > 0) {
>> - n = towrite;
>> - if (n > 4096)
>> - n = 4096;
>> - n = awrite(&afd, buf, n);
>> - if (n > 0)
>> - towrite -= n;
>> + /* We use bms->nodes as opposed to st->nodes to
>> + * be compatible with write-after-reads such as
>> + * the GROW operation.
>> + */
>> + for (i = 0; i < __le32_to_cpu(bms->nodes); i++) {
>> + /* Only the first bitmap should resync
>> + * the whole device
>> + */
>> + if (i)
>> + memset(buf, 0x00, 4096);
>> else
>> + memset(buf, 0xff, 4096);
>
> Why is the first bitmap initialised to 0x00 and the others to 0xff?
> If there is a good reason it should be documented either in a comment in the
> code or in the changelog entry.
Rather, it is the reverse. The first one is initialized to 0xff and the
rest are set to 0x00.
The reason is only the first node to assemble the device should perform
the resync (if --assume-clean is not provided). The comment is right
above the code. Perhaps I should be more elaborate with the comment.
--
Goldwyn
^ permalink raw reply
* [PATCH 1/3 v2] wait: introduce wait_event_exclusive_cmd
From: Yuanhan Liu @ 2015-04-29 2:48 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, linux-kernel, Yuanhan Liu, Ingo Molnar,
Peter Zijlstra
It's just a variant of wait_event_cmd(), with exclusive flag being set.
For cases like RAID5, which puts many processes to sleep until 1/4
resources are free, a wake_up wakes up all processes to run, but
there is one process being able to get the resource as it's protected
by a spin lock. That ends up introducing heavy lock contentions, and
hurts performance badly.
Here introduce wait_event_exclusive_cmd to relieve the lock contention
naturally by letting wake_up just wake up one process.
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
v2: its assumed that wait*() and __wait*() have the same arguments - peterz
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
include/linux/wait.h | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/include/linux/wait.h b/include/linux/wait.h
index 2db8334..db78c72 100644
--- a/include/linux/wait.h
+++ b/include/linux/wait.h
@@ -358,6 +358,19 @@ do { \
__ret; \
})
+#define __wait_event_exclusive_cmd(wq, condition, cmd1, cmd2) \
+ (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 1, 0, \
+ cmd1; schedule(); cmd2)
+/*
+ * Just like wait_event_cmd(), except it sets exclusive flag
+ */
+#define wait_event_exclusive_cmd(wq, condition, cmd1, cmd2) \
+do { \
+ if (condition) \
+ break; \
+ __wait_event_exclusive_cmd(wq, condition, cmd1, cmd2); \
+} while (0)
+
#define __wait_event_cmd(wq, condition, cmd1, cmd2) \
(void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
cmd1; schedule(); cmd2)
--
1.9.0
^ permalink raw reply related
* [PATCH 2/3] md/raid5: split wait_for_stripe and introduce wait_for_quiescent
From: Yuanhan Liu @ 2015-04-29 2:48 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, linux-kernel, Yuanhan Liu
In-Reply-To: <1430275735-20290-1-git-send-email-yuanhan.liu@linux.intel.com>
I noticed heavy spin lock contention at get_active_stripe(), introduced
at being wake up stage, where a bunch of processes try to re-hold the
spin lock again.
After giving some thoughts on this issue, I found the lock could be
relieved(and even avoided) if we turn the wait_for_stripe to per
waitqueue for each lock hash and make the wake up exclusive: wake up
one process each time, which avoids the lock contention naturally.
Before go hacking with wait_for_stripe, I found it actually has 2
usages: for the array to enter or leave the quiescent state, and also
to wait for an available stripe in each of the hash lists.
So this patch splits the first usage off into a separate wait_queue,
wait_for_quiescent, and the next patch will turn the second usage into
one waitqueue for each hash value, and make it exclusive, to relieve
the lock contention.
v2: wake_up(wait_for_quiescent) when (active_stripes == 0)
Commit log refactor suggestion from Neil.
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
drivers/md/raid5.c | 15 +++++++++------
drivers/md/raid5.h | 1 +
2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 77dfd72..64d5bea 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -374,6 +374,8 @@ static void release_inactive_stripe_list(struct r5conf *conf,
if (do_wakeup) {
wake_up(&conf->wait_for_stripe);
+ if (atomic_read(&conf->active_stripes) == 0)
+ wake_up(&conf->wait_for_quiescent);
if (conf->retry_read_aligned)
md_wakeup_thread(conf->mddev->thread);
}
@@ -667,7 +669,7 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
spin_lock_irq(conf->hash_locks + hash);
do {
- wait_event_lock_irq(conf->wait_for_stripe,
+ wait_event_lock_irq(conf->wait_for_quiescent,
conf->quiesce == 0 || noquiesce,
*(conf->hash_locks + hash));
sh = __find_stripe(conf, sector, conf->generation - previous);
@@ -4729,7 +4731,7 @@ static void raid5_align_endio(struct bio *bi, int error)
raid_bi, 0);
bio_endio(raid_bi, 0);
if (atomic_dec_and_test(&conf->active_aligned_reads))
- wake_up(&conf->wait_for_stripe);
+ wake_up(&conf->wait_for_quiescent);
return;
}
@@ -4824,7 +4826,7 @@ static int chunk_aligned_read(struct mddev *mddev, struct bio * raid_bio)
align_bi->bi_iter.bi_sector += rdev->data_offset;
spin_lock_irq(&conf->device_lock);
- wait_event_lock_irq(conf->wait_for_stripe,
+ wait_event_lock_irq(conf->wait_for_quiescent,
conf->quiesce == 0,
conf->device_lock);
atomic_inc(&conf->active_aligned_reads);
@@ -5668,7 +5670,7 @@ static int retry_aligned_read(struct r5conf *conf, struct bio *raid_bio)
bio_endio(raid_bio, 0);
}
if (atomic_dec_and_test(&conf->active_aligned_reads))
- wake_up(&conf->wait_for_stripe);
+ wake_up(&conf->wait_for_quiescent);
return handled;
}
@@ -6399,6 +6401,7 @@ static struct r5conf *setup_conf(struct mddev *mddev)
goto abort;
spin_lock_init(&conf->device_lock);
seqcount_init(&conf->gen_lock);
+ init_waitqueue_head(&conf->wait_for_quiescent);
init_waitqueue_head(&conf->wait_for_stripe);
init_waitqueue_head(&conf->wait_for_overlap);
INIT_LIST_HEAD(&conf->handle_list);
@@ -7422,7 +7425,7 @@ static void raid5_quiesce(struct mddev *mddev, int state)
* active stripes can drain
*/
conf->quiesce = 2;
- wait_event_cmd(conf->wait_for_stripe,
+ wait_event_cmd(conf->wait_for_quiescent,
atomic_read(&conf->active_stripes) == 0 &&
atomic_read(&conf->active_aligned_reads) == 0,
unlock_all_device_hash_locks_irq(conf),
@@ -7436,7 +7439,7 @@ static void raid5_quiesce(struct mddev *mddev, int state)
case 0: /* re-enable writes */
lock_all_device_hash_locks_irq(conf);
conf->quiesce = 0;
- wake_up(&conf->wait_for_stripe);
+ wake_up(&conf->wait_for_quiescent);
wake_up(&conf->wait_for_overlap);
unlock_all_device_hash_locks_irq(conf);
break;
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 7dc0dd8..4cc05ec 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -508,6 +508,7 @@ struct r5conf {
struct list_head inactive_list[NR_STRIPE_HASH_LOCKS];
atomic_t empty_inactive_list_nr;
struct llist_head released_stripes;
+ wait_queue_head_t wait_for_quiescent;
wait_queue_head_t wait_for_stripe;
wait_queue_head_t wait_for_overlap;
unsigned long cache_state;
--
1.9.0
^ permalink raw reply related
* [PATCH 3/3] md/raid5: per hash value and exclusive wait_for_stripe
From: Yuanhan Liu @ 2015-04-29 2:48 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, linux-kernel, Yuanhan Liu
In-Reply-To: <1430275735-20290-1-git-send-email-yuanhan.liu@linux.intel.com>
I noticed heavy spin lock contention at get_active_stripe() with fsmark
multiple thread write workloads.
Here is how this hot contention comes from. We have limited stripes, and
it's a multiple thread write workload. Hence, those stripes will be taken
soon, which puts later processes to sleep for waiting free stripes. When
enough stripes(>= 1/4 total stripes) are released, all process are woken,
trying to get the lock. But there is one only being able to get this lock
for each hash lock, making other processes spinning out there for acquiring
the lock.
Thus, it's effectiveless to wakeup all processes and let them battle for
a lock that permits one to access only each time. Instead, we could make
it be a exclusive wake up: wake up one process only. That avoids the heavy
spin lock contention naturally.
To do the exclusive wake up, we've to split wait_for_stripe into multiple
wait queues, to make it per hash value, just like the hash lock.
Here are some test results I have got with this patch applied(all test run
3 times):
`fsmark.files_per_sec'
=====================
next-20150317 this patch
------------------------- -------------------------
metric_value ±stddev metric_value ±stddev change testbox/benchmark/testcase-params
------------------------- ------------------------- -------- ------------------------------
25.600 ±0.0 92.700 ±2.5 262.1% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-btrfs-4M-30G-fsyncBeforeClose
25.600 ±0.0 77.800 ±0.6 203.9% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-btrfs-4M-30G-fsyncBeforeClose
32.000 ±0.0 93.800 ±1.7 193.1% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-ext4-4M-30G-fsyncBeforeClose
32.000 ±0.0 81.233 ±1.7 153.9% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-ext4-4M-30G-fsyncBeforeClose
48.800 ±14.5 99.667 ±2.0 104.2% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-xfs-4M-30G-fsyncBeforeClose
6.400 ±0.0 12.800 ±0.0 100.0% ivb44/fsmark/1x-64t-3HDD-RAID5-btrfs-4M-40G-fsyncBeforeClose
63.133 ±8.2 82.800 ±0.7 31.2% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-xfs-4M-30G-fsyncBeforeClose
245.067 ±0.7 306.567 ±7.9 25.1% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-f2fs-4M-30G-fsyncBeforeClose
17.533 ±0.3 21.000 ±0.8 19.8% ivb44/fsmark/1x-1t-3HDD-RAID5-xfs-4M-40G-fsyncBeforeClose
188.167 ±1.9 215.033 ±3.1 14.3% ivb44/fsmark/1x-1t-4BRD_12G-RAID5-btrfs-4M-30G-NoSync
254.500 ±1.8 290.733 ±2.4 14.2% ivb44/fsmark/1x-1t-9BRD_6G-RAID5-btrfs-4M-30G-NoSync
`time.system_time'
=====================
next-20150317 this patch
------------------------- -------------------------
metric_value ±stddev metric_value ±stddev change testbox/benchmark/testcase-params
------------------------- ------------------------- -------- ------------------------------
7235.603 ±1.2 185.163 ±1.9 -97.4% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-btrfs-4M-30G-fsyncBeforeClose
7666.883 ±2.9 202.750 ±1.0 -97.4% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-btrfs-4M-30G-fsyncBeforeClose
14567.893 ±0.7 421.230 ±0.4 -97.1% ivb44/fsmark/1x-64t-3HDD-RAID5-btrfs-4M-40G-fsyncBeforeClose
3697.667 ±14.0 148.190 ±1.7 -96.0% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-xfs-4M-30G-fsyncBeforeClose
5572.867 ±3.8 310.717 ±1.4 -94.4% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-ext4-4M-30G-fsyncBeforeClose
5565.050 ±0.5 313.277 ±1.5 -94.4% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-ext4-4M-30G-fsyncBeforeClose
2420.707 ±17.1 171.043 ±2.7 -92.9% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-xfs-4M-30G-fsyncBeforeClose
3743.300 ±4.6 379.827 ±3.5 -89.9% ivb44/fsmark/1x-64t-3HDD-RAID5-ext4-4M-40G-fsyncBeforeClose
3308.687 ±6.3 363.050 ±2.0 -89.0% ivb44/fsmark/1x-64t-3HDD-RAID5-xfs-4M-40G-fsyncBeforeClose
Where,
1x: where 'x' means iterations or loop, corresponding to the 'L' option of fsmark
1t, 64t: where 't' means thread
4M: means the single file size, corresponding to the '-s' option of fsmark
40G, 30G, 120G: means the total test size
4BRD_12G: BRD is the ramdisk, where '4' means 4 ramdisk, and where '12G' means
the size of one ramdisk. So, it would be 48G in total. And we made a
raid on those ramdisk
As you can see, though there are no much performance gain for hard disk
workload, the system time is dropped heavily, up to 97%. And as expected,
the performance increased a lot, up to 260%, for fast device(ram disk).
v2: use bits instead of array to note down wait queue need to wake up.
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
drivers/md/raid5.c | 27 +++++++++++++++++++--------
drivers/md/raid5.h | 2 +-
2 files changed, 20 insertions(+), 9 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 64d5bea..697d77a 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -344,7 +344,8 @@ static void release_inactive_stripe_list(struct r5conf *conf,
int hash)
{
int size;
- bool do_wakeup = false;
+ unsigned long do_wakeup = 0;
+ int i = 0;
unsigned long flags;
if (hash == NR_STRIPE_HASH_LOCKS) {
@@ -365,15 +366,19 @@ static void release_inactive_stripe_list(struct r5conf *conf,
!list_empty(list))
atomic_dec(&conf->empty_inactive_list_nr);
list_splice_tail_init(list, conf->inactive_list + hash);
- do_wakeup = true;
+ do_wakeup |= 1 << (size - 1);
spin_unlock_irqrestore(conf->hash_locks + hash, flags);
}
size--;
hash--;
}
+ for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++) {
+ if (do_wakeup & (1 << i))
+ wake_up(&conf->wait_for_stripe[i]);
+ }
+
if (do_wakeup) {
- wake_up(&conf->wait_for_stripe);
if (atomic_read(&conf->active_stripes) == 0)
wake_up(&conf->wait_for_quiescent);
if (conf->retry_read_aligned)
@@ -686,14 +691,15 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
if (!sh) {
set_bit(R5_INACTIVE_BLOCKED,
&conf->cache_state);
- wait_event_lock_irq(
- conf->wait_for_stripe,
+ wait_event_exclusive_cmd(
+ conf->wait_for_stripe[hash],
!list_empty(conf->inactive_list + hash) &&
(atomic_read(&conf->active_stripes)
< (conf->max_nr_stripes * 3 / 4)
|| !test_bit(R5_INACTIVE_BLOCKED,
&conf->cache_state)),
- *(conf->hash_locks + hash));
+ spin_unlock_irq(conf->hash_locks + hash),
+ spin_lock_irq(conf->hash_locks + hash));
clear_bit(R5_INACTIVE_BLOCKED,
&conf->cache_state);
} else {
@@ -718,6 +724,9 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
}
} while (sh == NULL);
+ if (!list_empty(conf->inactive_list + hash))
+ wake_up(&conf->wait_for_stripe[hash]);
+
spin_unlock_irq(conf->hash_locks + hash);
return sh;
}
@@ -2138,7 +2147,7 @@ static int resize_stripes(struct r5conf *conf, int newsize)
cnt = 0;
list_for_each_entry(nsh, &newstripes, lru) {
lock_device_hash_lock(conf, hash);
- wait_event_cmd(conf->wait_for_stripe,
+ wait_event_exclusive_cmd(conf->wait_for_stripe[hash],
!list_empty(conf->inactive_list + hash),
unlock_device_hash_lock(conf, hash),
lock_device_hash_lock(conf, hash));
@@ -6402,7 +6411,9 @@ static struct r5conf *setup_conf(struct mddev *mddev)
spin_lock_init(&conf->device_lock);
seqcount_init(&conf->gen_lock);
init_waitqueue_head(&conf->wait_for_quiescent);
- init_waitqueue_head(&conf->wait_for_stripe);
+ for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++) {
+ init_waitqueue_head(&conf->wait_for_stripe[i]);
+ }
init_waitqueue_head(&conf->wait_for_overlap);
INIT_LIST_HEAD(&conf->handle_list);
INIT_LIST_HEAD(&conf->hold_list);
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 4cc05ec..6307b90 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -509,7 +509,7 @@ struct r5conf {
atomic_t empty_inactive_list_nr;
struct llist_head released_stripes;
wait_queue_head_t wait_for_quiescent;
- wait_queue_head_t wait_for_stripe;
+ wait_queue_head_t wait_for_stripe[NR_STRIPE_HASH_LOCKS];
wait_queue_head_t wait_for_overlap;
unsigned long cache_state;
#define R5_INACTIVE_BLOCKED 1 /* release of inactive stripes blocked,
--
1.9.0
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH 07/10] block: kill merge_bvec_fn() completely
From: Ming Lin @ 2015-04-29 6:32 UTC (permalink / raw)
To: lkml, Christoph Hellwig, Jens Axboe, Kent Overstreet, Dongsu Park,
Lars Ellenberg, drbd-user, Jiri Kosina, Yehuda Sadeh, Sage Weil,
Alex Elder, ceph-devel, Alasdair Kergon, Mike Snitzer, dm-devel,
Neil Brown, linux-raid, Christoph Hellwig, Martin K. Petersen,
Dave Chinner, Wu Fengguang
In-Reply-To: <20150428173137.GC12975@agk-dp.fab.redhat.com>
On Tue, Apr 28, 2015 at 10:31 AM, Alasdair G Kergon <agk@redhat.com> wrote:
> On Mon, Apr 27, 2015 at 11:48:34PM -0700, Ming Lin wrote:
>> As generic_make_request() is now able to handle arbitrarily sized bios,
>> it's no longer necessary for each individual block driver to define its
>> own ->merge_bvec_fn() callback. Remove every invocation completely.
>
> merge_bvec_fn is also about telling the code building up bios what optimum
> maximum size to use.
>
> Have you got some test data to demonstrate that we won't find some situations
> where performance is now lost by code building up bios that are far too big
> and are always going to have to be split up later?
Not yet.
Fengguang will help to run some tests.
Thanks.
>
> Alasdair
>
^ permalink raw reply
* Re: [PATCH 1/3 v2] wait: introduce wait_event_exclusive_cmd
From: Peter Zijlstra @ 2015-04-29 9:31 UTC (permalink / raw)
To: Yuanhan Liu; +Cc: neilb, linux-raid, linux-kernel, Ingo Molnar
In-Reply-To: <1430275735-20290-1-git-send-email-yuanhan.liu@linux.intel.com>
On Wed, Apr 29, 2015 at 10:48:53AM +0800, Yuanhan Liu wrote:
> It's just a variant of wait_event_cmd(), with exclusive flag being set.
>
> For cases like RAID5, which puts many processes to sleep until 1/4
> resources are free, a wake_up wakes up all processes to run, but
> there is one process being able to get the resource as it's protected
> by a spin lock. That ends up introducing heavy lock contentions, and
> hurts performance badly.
>
> Here introduce wait_event_exclusive_cmd to relieve the lock contention
> naturally by letting wake_up just wake up one process.
>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Thanks!
Assuming you want to route this through Neil's MD tree:
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
> include/linux/wait.h | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
> diff --git a/include/linux/wait.h b/include/linux/wait.h
> index 2db8334..db78c72 100644
> --- a/include/linux/wait.h
> +++ b/include/linux/wait.h
> @@ -358,6 +358,19 @@ do { \
> __ret; \
> })
>
> +#define __wait_event_exclusive_cmd(wq, condition, cmd1, cmd2) \
> + (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 1, 0, \
> + cmd1; schedule(); cmd2)
> +/*
> + * Just like wait_event_cmd(), except it sets exclusive flag
> + */
> +#define wait_event_exclusive_cmd(wq, condition, cmd1, cmd2) \
> +do { \
> + if (condition) \
> + break; \
> + __wait_event_exclusive_cmd(wq, condition, cmd1, cmd2); \
> +} while (0)
> +
> #define __wait_event_cmd(wq, condition, cmd1, cmd2) \
> (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 0, 0, \
> cmd1; schedule(); cmd2)
> --
> 1.9.0
>
^ permalink raw reply
* [PATCH v2 0/4] dm-raid: Add support for the MD RAID0 personality
From: heinzm @ 2015-04-29 12:03 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This v2 patchset of 4 adds access to the MD RAID0 personality to dm-raid
(in addition to the already supported raid1/10/4/5/6 personalities)
to enable single zone striping.
The added functionality is the first step to allow full support
of MD kernel functionality via dm-raid and to eventually replace
the dm "striped" target altogether. Adding shrink/takeover and
reshape functionality to dm-raid is subject to future patches.
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
Heinz Mauelshagen (4):
dm-raid: Add support for the MD RAID0 personality
md-raid0: conditional mddev->queue access to suit dm-raid
Documentation: add version history entries for raid0 support _and_
forgotten devices_handle_discards_safely module paramater
dm-raid: remove bogus parse_raid_params() comment about
devices_handle_discards_safely
Documentation/device-mapper/dm-raid.txt | 3 +
drivers/md/dm-raid.c | 206 +++++++++++++++++---------------
drivers/md/raid0.c | 5 +-
3 files changed, 117 insertions(+), 97 deletions(-)
--
2.1.0
^ permalink raw reply
* [PATCH v2 1/4] dm-raid: Add support for the MD RAID0 personality
From: heinzm @ 2015-04-29 12:03 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch adds access to the MD RAID0 personality to dm-raid
(in addition to the already supported raid1/10/4/5/6 personalities)
to enable single zone striping.
It avoids setting rs->md.gendisk by introducing a conditional
to MD raid0, because device stack and discard settings are done
by dm core
The following changes enable that access:
- add type definition to raid_types array
- make bitmap creation conditonal in super_validate(), because
bitmaps are not allowed in raid0
- set rdev->sectors to the data image size in super_validate()
to allow the raid0 personality to calculate the MD array
size properly
- use mdddev(un)lock() functions instead of direct mutex_(un)lock()
(wrapped in here because it's a trivial change)
- enhance raid_status() to always report full sync for raid0
so that userspace checks for 100% sync will succeed and allow
for resize (and takeover/reshape once added in future paches)
- enhance raid_resume() to not load bitmap in case of raid0
- raise version to 1.7.0 because of the raid0 API change
As side changes I applied:
- ensure maximum device limit in superblock
- rename DMPF_* (print flags) to CTR_FLAG_* (constructor flags)
and their respective struct raid_set member
- use strcasecmp() in raid10_format_to_md_layout() as in the constructor
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/dm-raid.c | 204 ++++++++++++++++++++++++++++-----------------------
1 file changed, 111 insertions(+), 93 deletions(-)
diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
index 88e4c7f..c541d5a 100644
--- a/drivers/md/dm-raid.c
+++ b/drivers/md/dm-raid.c
@@ -1,6 +1,6 @@
/*
* Copyright (C) 2010-2011 Neil Brown
- * Copyright (C) 2010-2014 Red Hat, Inc. All rights reserved.
+ * Copyright (C) 2010-2015 Red Hat, Inc. All rights reserved.
*
* This file is released under the GPL.
*/
@@ -17,6 +17,7 @@
#include <linux/device-mapper.h>
#define DM_MSG_PREFIX "raid"
+#define MAX_RAID_DEVICES 253 /* raid4/5/6 limit */
static bool devices_handle_discard_safely = false;
@@ -45,25 +46,25 @@ struct raid_dev {
};
/*
- * Flags for rs->print_flags field.
+ * Flags for rs->ctr_flags field.
*/
-#define DMPF_SYNC 0x1
-#define DMPF_NOSYNC 0x2
-#define DMPF_REBUILD 0x4
-#define DMPF_DAEMON_SLEEP 0x8
-#define DMPF_MIN_RECOVERY_RATE 0x10
-#define DMPF_MAX_RECOVERY_RATE 0x20
-#define DMPF_MAX_WRITE_BEHIND 0x40
-#define DMPF_STRIPE_CACHE 0x80
-#define DMPF_REGION_SIZE 0x100
-#define DMPF_RAID10_COPIES 0x200
-#define DMPF_RAID10_FORMAT 0x400
+#define CTR_FLAG_SYNC 0x1
+#define CTR_FLAG_NOSYNC 0x2
+#define CTR_FLAG_REBUILD 0x4
+#define CTR_FLAG_DAEMON_SLEEP 0x8
+#define CTR_FLAG_MIN_RECOVERY_RATE 0x10
+#define CTR_FLAG_MAX_RECOVERY_RATE 0x20
+#define CTR_FLAG_MAX_WRITE_BEHIND 0x40
+#define CTR_FLAG_STRIPE_CACHE 0x80
+#define CTR_FLAG_REGION_SIZE 0x100
+#define CTR_FLAG_RAID10_COPIES 0x200
+#define CTR_FLAG_RAID10_FORMAT 0x400
struct raid_set {
struct dm_target *ti;
uint32_t bitmap_loaded;
- uint32_t print_flags;
+ uint32_t ctr_flags;
struct mddev md;
struct raid_type *raid_type;
@@ -81,6 +82,7 @@ static struct raid_type {
const unsigned level; /* RAID level. */
const unsigned algorithm; /* RAID algorithm. */
} raid_types[] = {
+ {"raid0", "RAID0 (striping)", 0, 2, 0, 0 /* NONE */},
{"raid1", "RAID1 (mirroring)", 0, 2, 1, 0 /* NONE */},
{"raid10", "RAID10 (striped mirrors)", 0, 2, 10, UINT_MAX /* Varies */},
{"raid4", "RAID4 (dedicated parity disk)", 1, 2, 5, ALGORITHM_PARITY_0},
@@ -119,15 +121,15 @@ static int raid10_format_to_md_layout(char *format, unsigned copies)
{
unsigned n = 1, f = 1;
- if (!strcmp("near", format))
+ if (!strcasecmp("near", format))
n = copies;
else
f = copies;
- if (!strcmp("offset", format))
+ if (!strcasecmp("offset", format))
return 0x30000 | (f << 8) | n;
- if (!strcmp("far", format))
+ if (!strcasecmp("far", format))
return 0x20000 | (f << 8) | n;
return (f << 8) | n;
@@ -555,12 +557,12 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
for (i = 0; i < num_raid_params; i++) {
if (!strcasecmp(argv[i], "nosync")) {
rs->md.recovery_cp = MaxSector;
- rs->print_flags |= DMPF_NOSYNC;
+ rs->ctr_flags |= CTR_FLAG_NOSYNC;
continue;
}
if (!strcasecmp(argv[i], "sync")) {
rs->md.recovery_cp = 0;
- rs->print_flags |= DMPF_SYNC;
+ rs->ctr_flags |= CTR_FLAG_SYNC;
continue;
}
@@ -585,7 +587,7 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
return -EINVAL;
}
raid10_format = argv[i];
- rs->print_flags |= DMPF_RAID10_FORMAT;
+ rs->ctr_flags |= CTR_FLAG_RAID10_FORMAT;
continue;
}
@@ -602,7 +604,7 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
}
clear_bit(In_sync, &rs->dev[value].rdev.flags);
rs->dev[value].rdev.recovery_offset = 0;
- rs->print_flags |= DMPF_REBUILD;
+ rs->ctr_flags |= CTR_FLAG_REBUILD;
} else if (!strcasecmp(key, "write_mostly")) {
if (rs->raid_type->level != 1) {
rs->ti->error = "write_mostly option is only valid for RAID1";
@@ -618,7 +620,7 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
rs->ti->error = "max_write_behind option is only valid for RAID1";
return -EINVAL;
}
- rs->print_flags |= DMPF_MAX_WRITE_BEHIND;
+ rs->ctr_flags |= CTR_FLAG_MAX_WRITE_BEHIND;
/*
* In device-mapper, we specify things in sectors, but
@@ -631,14 +633,14 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
}
rs->md.bitmap_info.max_write_behind = value;
} else if (!strcasecmp(key, "daemon_sleep")) {
- rs->print_flags |= DMPF_DAEMON_SLEEP;
+ rs->ctr_flags |= CTR_FLAG_DAEMON_SLEEP;
if (!value || (value > MAX_SCHEDULE_TIMEOUT)) {
rs->ti->error = "daemon sleep period out of range";
return -EINVAL;
}
rs->md.bitmap_info.daemon_sleep = value;
} else if (!strcasecmp(key, "stripe_cache")) {
- rs->print_flags |= DMPF_STRIPE_CACHE;
+ rs->ctr_flags |= CTR_FLAG_STRIPE_CACHE;
/*
* In device-mapper, we specify things in sectors, but
@@ -656,21 +658,21 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
return -EINVAL;
}
} else if (!strcasecmp(key, "min_recovery_rate")) {
- rs->print_flags |= DMPF_MIN_RECOVERY_RATE;
+ rs->ctr_flags |= CTR_FLAG_MIN_RECOVERY_RATE;
if (value > INT_MAX) {
rs->ti->error = "min_recovery_rate out of range";
return -EINVAL;
}
rs->md.sync_speed_min = (int)value;
} else if (!strcasecmp(key, "max_recovery_rate")) {
- rs->print_flags |= DMPF_MAX_RECOVERY_RATE;
+ rs->ctr_flags |= CTR_FLAG_MAX_RECOVERY_RATE;
if (value > INT_MAX) {
rs->ti->error = "max_recovery_rate out of range";
return -EINVAL;
}
rs->md.sync_speed_max = (int)value;
} else if (!strcasecmp(key, "region_size")) {
- rs->print_flags |= DMPF_REGION_SIZE;
+ rs->ctr_flags |= CTR_FLAG_REGION_SIZE;
region_size = value;
} else if (!strcasecmp(key, "raid10_copies") &&
(rs->raid_type->level == 10)) {
@@ -678,7 +680,7 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
rs->ti->error = "Bad value for 'raid10_copies'";
return -EINVAL;
}
- rs->print_flags |= DMPF_RAID10_COPIES;
+ rs->ctr_flags |= CTR_FLAG_RAID10_COPIES;
raid10_copies = value;
} else {
DMERR("Unable to parse RAID parameter: %s", key);
@@ -720,7 +722,7 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
rs->md.layout = raid10_format_to_md_layout(raid10_format,
raid10_copies);
rs->md.new_layout = rs->md.layout;
- } else if ((rs->raid_type->level > 1) &&
+ } else if ((!rs->raid_type->level || rs->raid_type->level > 1) &&
sector_div(sectors_per_dev,
(rs->md.raid_disks - rs->raid_type->parity_devs))) {
rs->ti->error = "Target length not divisible by number of data devices";
@@ -947,7 +949,7 @@ static int super_init_validation(struct mddev *mddev, struct md_rdev *rdev)
return -EINVAL;
}
- if (!(rs->print_flags & (DMPF_SYNC | DMPF_NOSYNC)))
+ if (!(rs->ctr_flags & (CTR_FLAG_SYNC | CTR_FLAG_NOSYNC)))
mddev->recovery_cp = le64_to_cpu(sb->array_resync_offset);
/*
@@ -1026,8 +1028,9 @@ static int super_init_validation(struct mddev *mddev, struct md_rdev *rdev)
return 0;
}
-static int super_validate(struct mddev *mddev, struct md_rdev *rdev)
+static int super_validate(struct raid_set *rs, struct md_rdev *rdev)
{
+ struct mddev *mddev = &rs->md;
struct dm_raid_superblock *sb = page_address(rdev->sb_page);
/*
@@ -1037,8 +1040,10 @@ static int super_validate(struct mddev *mddev, struct md_rdev *rdev)
if (!mddev->events && super_init_validation(mddev, rdev))
return -EINVAL;
- mddev->bitmap_info.offset = 4096 >> 9; /* Enable bitmap creation */
- rdev->mddev->bitmap_info.default_offset = 4096 >> 9;
+ /* Enable bitmap creation for RAID levels != 0 */
+ mddev->bitmap_info.offset = (rs->raid_type->level) ? to_sector(4096) : 0;
+ rdev->mddev->bitmap_info.default_offset = mddev->bitmap_info.offset;
+
if (!test_bit(FirstUse, &rdev->flags)) {
rdev->recovery_offset = le64_to_cpu(sb->disk_recovery_offset);
if (rdev->recovery_offset != MaxSector)
@@ -1073,7 +1078,7 @@ static int analyse_superblocks(struct dm_target *ti, struct raid_set *rs)
freshest = NULL;
rdev_for_each_safe(rdev, tmp, mddev) {
/*
- * Skipping super_load due to DMPF_SYNC will cause
+ * Skipping super_load due to CTR_FLAG_SYNC will cause
* the array to undergo initialization again as
* though it were new. This is the intended effect
* of the "sync" directive.
@@ -1082,7 +1087,9 @@ static int analyse_superblocks(struct dm_target *ti, struct raid_set *rs)
* that the "sync" directive is disallowed during the
* reshape.
*/
- if (rs->print_flags & DMPF_SYNC)
+ rdev->sectors = to_sector(i_size_read(rdev->bdev->bd_inode));
+
+ if (rs->ctr_flags & CTR_FLAG_SYNC)
continue;
if (!rdev->meta_bdev)
@@ -1140,11 +1147,11 @@ static int analyse_superblocks(struct dm_target *ti, struct raid_set *rs)
* validation for the remaining devices.
*/
ti->error = "Unable to assemble array: Invalid superblocks";
- if (super_validate(mddev, freshest))
+ if (super_validate(rs, freshest))
return -EINVAL;
rdev_for_each(rdev, mddev)
- if ((rdev != freshest) && super_validate(mddev, rdev))
+ if ((rdev != freshest) && super_validate(rs, rdev))
return -EINVAL;
return 0;
@@ -1243,7 +1250,7 @@ static int raid_ctr(struct dm_target *ti, unsigned argc, char **argv)
}
if ((kstrtoul(argv[num_raid_params], 10, &num_raid_devs) < 0) ||
- (num_raid_devs >= INT_MAX)) {
+ (num_raid_devs > MAX_RAID_DEVICES)) {
ti->error = "Cannot understand number of raid devices";
return -EINVAL;
}
@@ -1282,10 +1289,11 @@ static int raid_ctr(struct dm_target *ti, unsigned argc, char **argv)
*/
configure_discard_support(ti, rs);
- mutex_lock(&rs->md.reconfig_mutex);
+ /* Has to be held on running the array */
+ mddev_lock_nointr(&rs->md);
ret = md_run(&rs->md);
rs->md.in_sync = 0; /* Assume already marked dirty */
- mutex_unlock(&rs->md.reconfig_mutex);
+ mddev_unlock(&rs->md);
if (ret) {
ti->error = "Fail to run raid array";
@@ -1368,34 +1376,40 @@ static void raid_status(struct dm_target *ti, status_type_t type,
case STATUSTYPE_INFO:
DMEMIT("%s %d ", rs->raid_type->name, rs->md.raid_disks);
- if (test_bit(MD_RECOVERY_RUNNING, &rs->md.recovery))
- sync = rs->md.curr_resync_completed;
- else
- sync = rs->md.recovery_cp;
-
- if (sync >= rs->md.resync_max_sectors) {
- /*
- * Sync complete.
- */
+ if (rs->raid_type->level) {
+ if (test_bit(MD_RECOVERY_RUNNING, &rs->md.recovery))
+ sync = rs->md.curr_resync_completed;
+ else
+ sync = rs->md.recovery_cp;
+
+ if (sync >= rs->md.resync_max_sectors) {
+ /*
+ * Sync complete.
+ */
+ array_in_sync = 1;
+ sync = rs->md.resync_max_sectors;
+ } else if (test_bit(MD_RECOVERY_REQUESTED, &rs->md.recovery)) {
+ /*
+ * If "check" or "repair" is occurring, the array has
+ * undergone and initial sync and the health characters
+ * should not be 'a' anymore.
+ */
+ array_in_sync = 1;
+ } else {
+ /*
+ * The array may be doing an initial sync, or it may
+ * be rebuilding individual components. If all the
+ * devices are In_sync, then it is the array that is
+ * being initialized.
+ */
+ for (i = 0; i < rs->md.raid_disks; i++)
+ if (!test_bit(In_sync, &rs->dev[i].rdev.flags))
+ array_in_sync = 1;
+ }
+ } else {
+ /* RAID0 */
array_in_sync = 1;
sync = rs->md.resync_max_sectors;
- } else if (test_bit(MD_RECOVERY_REQUESTED, &rs->md.recovery)) {
- /*
- * If "check" or "repair" is occurring, the array has
- * undergone and initial sync and the health characters
- * should not be 'a' anymore.
- */
- array_in_sync = 1;
- } else {
- /*
- * The array may be doing an initial sync, or it may
- * be rebuilding individual components. If all the
- * devices are In_sync, then it is the array that is
- * being initialized.
- */
- for (i = 0; i < rs->md.raid_disks; i++)
- if (!test_bit(In_sync, &rs->dev[i].rdev.flags))
- array_in_sync = 1;
}
/*
@@ -1446,7 +1460,7 @@ static void raid_status(struct dm_target *ti, status_type_t type,
case STATUSTYPE_TABLE:
/* The string you would use to construct this array */
for (i = 0; i < rs->md.raid_disks; i++) {
- if ((rs->print_flags & DMPF_REBUILD) &&
+ if ((rs->ctr_flags & CTR_FLAG_REBUILD) &&
rs->dev[i].data_dev &&
!test_bit(In_sync, &rs->dev[i].rdev.flags))
raid_param_cnt += 2; /* for rebuilds */
@@ -1455,33 +1469,33 @@ static void raid_status(struct dm_target *ti, status_type_t type,
raid_param_cnt += 2;
}
- raid_param_cnt += (hweight32(rs->print_flags & ~DMPF_REBUILD) * 2);
- if (rs->print_flags & (DMPF_SYNC | DMPF_NOSYNC))
+ raid_param_cnt += (hweight32(rs->ctr_flags & ~CTR_FLAG_REBUILD) * 2);
+ if (rs->ctr_flags & (CTR_FLAG_SYNC | CTR_FLAG_NOSYNC))
raid_param_cnt--;
DMEMIT("%s %u %u", rs->raid_type->name,
raid_param_cnt, rs->md.chunk_sectors);
- if ((rs->print_flags & DMPF_SYNC) &&
+ if ((rs->ctr_flags & CTR_FLAG_SYNC) &&
(rs->md.recovery_cp == MaxSector))
DMEMIT(" sync");
- if (rs->print_flags & DMPF_NOSYNC)
+ if (rs->ctr_flags & CTR_FLAG_NOSYNC)
DMEMIT(" nosync");
for (i = 0; i < rs->md.raid_disks; i++)
- if ((rs->print_flags & DMPF_REBUILD) &&
+ if ((rs->ctr_flags & CTR_FLAG_REBUILD) &&
rs->dev[i].data_dev &&
!test_bit(In_sync, &rs->dev[i].rdev.flags))
DMEMIT(" rebuild %u", i);
- if (rs->print_flags & DMPF_DAEMON_SLEEP)
+ if (rs->ctr_flags & CTR_FLAG_DAEMON_SLEEP)
DMEMIT(" daemon_sleep %lu",
rs->md.bitmap_info.daemon_sleep);
- if (rs->print_flags & DMPF_MIN_RECOVERY_RATE)
+ if (rs->ctr_flags & CTR_FLAG_MIN_RECOVERY_RATE)
DMEMIT(" min_recovery_rate %d", rs->md.sync_speed_min);
- if (rs->print_flags & DMPF_MAX_RECOVERY_RATE)
+ if (rs->ctr_flags & CTR_FLAG_MAX_RECOVERY_RATE)
DMEMIT(" max_recovery_rate %d", rs->md.sync_speed_max);
for (i = 0; i < rs->md.raid_disks; i++)
@@ -1489,11 +1503,11 @@ static void raid_status(struct dm_target *ti, status_type_t type,
test_bit(WriteMostly, &rs->dev[i].rdev.flags))
DMEMIT(" write_mostly %u", i);
- if (rs->print_flags & DMPF_MAX_WRITE_BEHIND)
+ if (rs->ctr_flags & CTR_FLAG_MAX_WRITE_BEHIND)
DMEMIT(" max_write_behind %lu",
rs->md.bitmap_info.max_write_behind);
- if (rs->print_flags & DMPF_STRIPE_CACHE) {
+ if (rs->ctr_flags & CTR_FLAG_STRIPE_CACHE) {
struct r5conf *conf = rs->md.private;
/* convert from kiB to sectors */
@@ -1501,15 +1515,15 @@ static void raid_status(struct dm_target *ti, status_type_t type,
conf ? conf->max_nr_stripes * 2 : 0);
}
- if (rs->print_flags & DMPF_REGION_SIZE)
+ if (rs->ctr_flags & CTR_FLAG_REGION_SIZE)
DMEMIT(" region_size %lu",
rs->md.bitmap_info.chunksize >> 9);
- if (rs->print_flags & DMPF_RAID10_COPIES)
+ if (rs->ctr_flags & CTR_FLAG_RAID10_COPIES)
DMEMIT(" raid10_copies %u",
raid10_md_layout_to_copies(rs->md.layout));
- if (rs->print_flags & DMPF_RAID10_FORMAT)
+ if (rs->ctr_flags & CTR_FLAG_RAID10_FORMAT)
DMEMIT(" raid10_format %s",
raid10_md_layout_to_format(rs->md.layout));
@@ -1684,26 +1698,30 @@ static void raid_resume(struct dm_target *ti)
{
struct raid_set *rs = ti->private;
- set_bit(MD_CHANGE_DEVS, &rs->md.flags);
- if (!rs->bitmap_loaded) {
- bitmap_load(&rs->md);
- rs->bitmap_loaded = 1;
- } else {
- /*
- * A secondary resume while the device is active.
- * Take this opportunity to check whether any failed
- * devices are reachable again.
- */
- attempt_restore_of_faulty_devices(rs);
+ if (rs->raid_type->level) {
+ set_bit(MD_CHANGE_DEVS, &rs->md.flags);
+
+ if (!rs->bitmap_loaded) {
+ bitmap_load(&rs->md);
+ rs->bitmap_loaded = 1;
+ } else {
+ /*
+ * A secondary resume while the device is active.
+ * Take this opportunity to check whether any failed
+ * devices are reachable again.
+ */
+ attempt_restore_of_faulty_devices(rs);
+ }
+
+ clear_bit(MD_RECOVERY_FROZEN, &rs->md.recovery);
}
- clear_bit(MD_RECOVERY_FROZEN, &rs->md.recovery);
mddev_resume(&rs->md);
}
static struct target_type raid_target = {
.name = "raid",
- .version = {1, 6, 0},
+ .version = {1, 7, 0},
.module = THIS_MODULE,
.ctr = raid_ctr,
.dtr = raid_dtr,
--
2.1.0
^ permalink raw reply related
* [PATCH v2 2/4] md-raid0: conditional mddev->queue access to suit dm-raid
From: heinzm @ 2015-04-29 12:03 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch is a prerequisite for dm-raid "raid0" support to allow
dm-raid to access the MD RAID0 personality doing unconditional
accesses to mddev->queue, which is NULL in case of dm-raid stacked
on top of MD.
Most of the conditional mddev->queue accesses made it to upstream
but this missing one, which prohibits md raid0 to set disk stack limits
(being done in dm core in case of md underneath dm).
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/raid0.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index 2cb59a6..6a68ef5 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -188,8 +188,9 @@ static int create_strip_zones(struct mddev *mddev, struct r0conf **private_conf)
}
dev[j] = rdev1;
- disk_stack_limits(mddev->gendisk, rdev1->bdev,
- rdev1->data_offset << 9);
+ if (mddev->queue)
+ disk_stack_limits(mddev->gendisk, rdev1->bdev,
+ rdev1->data_offset << 9);
if (rdev1->bdev->bd_disk->queue->merge_bvec_fn)
conf->has_merge_bvec = 1;
--
2.1.0
^ permalink raw reply related
* [PATCH v2 3/4] Documentation: add version history entries for raid0 support _and_ forgotten devices_handle_discards_safely module paramater
From: heinzm @ 2015-04-29 12:03 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch adds version history for md raid0 to dm-raid.txt
_and_
adds forgotten doc for the 1.6.0 version change as well
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
---
Documentation/device-mapper/dm-raid.txt | 3 +++
1 file changed, 3 insertions(+)
diff --git a/Documentation/device-mapper/dm-raid.txt b/Documentation/device-mapper/dm-raid.txt
index ef8ba9f..bceb865 100644
--- a/Documentation/device-mapper/dm-raid.txt
+++ b/Documentation/device-mapper/dm-raid.txt
@@ -224,3 +224,6 @@ Version History
New status (STATUSTYPE_INFO) fields: sync_action and mismatch_cnt.
1.5.1 Add ability to restore transiently failed devices on resume.
1.5.2 'mismatch_cnt' is zero unless [last_]sync_action is "check".
+1.6.0 Add devices_handle_discards_safely module parameter to
+ allow discards on disks announcing uncertain discard_zeroes_data
+1.7.0 Add support for MD RAID0 mappings
--
2.1.0
^ permalink raw reply related
* [PATCH v2 4/4] dm-raid: remove bogus parse_raid_params() comment about devices_handle_discards_safely
From: heinzm @ 2015-04-29 12:03 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch removes a bogus comment on parse_raid_params() claiming
"devices_handle_discard_safely" being a table line argument when
it actually is a module parameter.
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/dm-raid.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
index c541d5a..97e1651 100644
--- a/drivers/md/dm-raid.c
+++ b/drivers/md/dm-raid.c
@@ -479,8 +479,6 @@ too_many:
* will form the "stripe"
* [[no]sync] Force or prevent recovery of the
* entire array
- * [devices_handle_discard_safely] Allow discards on RAID4/5/6; useful if RAID
- * member device(s) properly support TRIM/UNMAP
* [rebuild <idx>] Rebuild the drive indicated by the index
* [daemon_sleep <ms>] Time between bitmap daemon work to
* clear bits
--
2.1.0
^ permalink raw reply related
* Re: [PATCH -stable] block: destroy bdi before blockdev is unregistered.
From: Christoph Hellwig @ 2015-04-29 13:35 UTC (permalink / raw)
To: NeilBrown
Cc: Mike Snitzer, Jens Axboe, Azat Khuzhin, Christoph Hellwig,
Kernel.org-Linux-RAID, Guoqing Jiang, Tejun Heo, Jan Kara, lkml,
device-mapper development, Peter Zijlstra
In-Reply-To: <20150429072530.39d38b00@notabene.brown>
On Wed, Apr 29, 2015 at 07:25:30AM +1000, NeilBrown wrote:
> As bdi_set_min_ratio doesn't touch bdi->dev, there seems to be no need for
> the test, or the warning.
>
> I wonder if it would make sense to move the bdi_set_min_ratio() call to
> bdi_destroy, and discard bdi_unregister??
> There is a comment which suggests bdi_unregister might be of use later, but
> it might be best to have a clean slate in which to add whatever might be
> needed??
This seems fine to me from the block dev point of view. I don't really
understand the bdi_min_ratio logic, but Peter might have a better idea.
^ permalink raw reply
* Re: [PATCH -stable] block: destroy bdi before blockdev is unregistered.
From: Peter Zijlstra @ 2015-04-29 16:02 UTC (permalink / raw)
To: Christoph Hellwig
Cc: NeilBrown, Mike Snitzer, Jens Axboe, Azat Khuzhin,
Kernel.org-Linux-RAID, Guoqing Jiang, Tejun Heo, Jan Kara, lkml,
device-mapper development
In-Reply-To: <20150429133512.GA4436@lst.de>
On Wed, Apr 29, 2015 at 03:35:12PM +0200, Christoph Hellwig wrote:
> On Wed, Apr 29, 2015 at 07:25:30AM +1000, NeilBrown wrote:
> > As bdi_set_min_ratio doesn't touch bdi->dev, there seems to be no need for
> > the test, or the warning.
> >
> > I wonder if it would make sense to move the bdi_set_min_ratio() call to
> > bdi_destroy, and discard bdi_unregister??
> > There is a comment which suggests bdi_unregister might be of use later, but
> > it might be best to have a clean slate in which to add whatever might be
> > needed??
>
> This seems fine to me from the block dev point of view. I don't really
> understand the bdi_min_ratio logic, but Peter might have a better idea.
Ah, that was a bit of digging, I've not looked at that in ages :-)
So if you look at bdi_dirty_limit()'s comment:
* The bdi's share of dirty limit will be adapting to its throughput and
* bounded by the bdi->min_ratio and/or bdi->max_ratio parameters, if set.
So the min_ratio is a minimum guaranteed fraction of the total
throughput.
Now the problem before commit ccb6108f5b0b ("mm/backing-dev.c: reset bdi
min_ratio in bdi_unregister()") was that since bdi_set_min_ratio()
keeps a global sum of bdi->min_ratio, you need to subtract from said
global sum when taking the BDI away. Otherwise we loose/leak a fraction
of the total throughput available (to the other BDIs).
Which is what that bdi_set_min_ratio(bdi, 0) in unregister does. It
resets the min_ratio for the bdi being taken out and frees up the min
allocated bandwidth for the others.
So I think moving that do destroy would be fine; assuming the delay
between unregister and destroy is typically 'short'. Because without
that you can 'leak' this min ratio for extended periods which means the
bandwidth is unavailable for other BDIs.
Does that make sense?
^ permalink raw reply
* Re: Help needed recovering from raid failure
From: Peter van Es @ 2015-04-29 18:17 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
Dear Neil,
first of all, I really appreciate you trying to help me. This is the first time I’m deploying software raid, so really appreciate the guidance.
> On 29 Apr 2015, at 00:26, NeilBrown <neilb@suse.de> wrote:
>
> This isn't really reporting anything new.
> There is probably a daily cron job which reports all degraded arrays. This
> message is reported by that job.
I understand...
>
>
> Why do you think the array is off-line? The above message doesn't suggest
> that.
>
My Ubuntu server was accessible through ssh but did not serve webpages, files etc. When I went to the console,
it told me it had taken the array offline because of degraded /dev/sdd2 and /dev/sdc2
Those two drives were out of the array.
>
>>
>> Needless to say, I can't boot the system anymore as the boot drive is /dev/md0, and GRUB can't
>> get at it. I do need to recover data (I know, but there's stuf on there I have no backup for--yet).
>
> You boot off a RAID5? Does grub support that? I didn't know.
> But md0 hasn't failed, has it?
>
> Confused.
Well, it took a little time but yes, I managed to define a raid 5 array that the system was able to boot from.
> There is something VERY sick here. I suggest that you tread very carefully.
>
> All your '1' partitions should be about 2GB and the '2' parititions about 2TB
>
> But the --examine output suggests sda2 and sdb2 are 2TB, while sdd2 and sde2
> are 2GB.
>
> That really really shouldn't happen. Maybe check your partition table
> (fdisk).
> I really cannot see how this would happen.
But this question, and the previous question you asked, tell me a little of what I may have done…
I think confused /dev/md0 and /dev/md1 (now called /dev/md126 and /dev/md127 when running of the USB stick).
/dev/md0 is a swap array (around 6GB, comprised of 4 x 2 GB in raid 5)
/dev/md1 is the boot and data array (around 5 TB, comprised of 4 x ~2 TB in raid 5)
I must have confused them and tried to add the /dev/sdc2 and /dev/sdd2 drive to the /dev/md0 array (mdadm —add /dev/md0 /dev/sdc2)
instead of to the /dev/md1 array. They were then added as spare drives, their superblocks were overwritten, but since
a) no swap space was used, and
b) they were added as spares
The data should not have been overwritten.
>
> Can you
> mdadm -Ss
>
> to stop all the arrays, then
>
> fdisk -l /dev/sd?
>
> then
>
> mdadm -Esvv
>
Neil, here they are: again, I appreciate you taking the time and guiding me through this!
Is there any way to resurrect the super blocks and try to force assemble the array, skipping the failing drive /dev/sdd2 (the /dev/sdd2 drive created some errors I observed in the log, /dev/sdc2 must have had a one off issue to be taken out….). I have two new drives (arrived today), and a new SSD drive. I would want to get the new array assembled using /dev/sdc2 perhaps forcing it back to the array geometry and “hoping for the best” and then install a new /dev/sdd2 to be recovered. Then I’ll create a boot and swap drive off the SSD which means that any array failures should not prevent the system from booting…
Requested outputs are below
Thanks,
Peter
fdisk output: (USB devices deleted)
Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000f24ee
Device Boot Start End Blocks Id System
/dev/sda1 2048 3905535 1951744 fd Linux raid autodetect
/dev/sda2 * 3905536 3907028991 1951561728 fd Linux raid autodetect
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00029d5c
Device Boot Start End Blocks Id System
/dev/sdb1 2048 3905535 1951744 fd Linux raid autodetect
/dev/sdb2 * 3905536 3907028991 1951561728 fd Linux raid autodetect
Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000727bf
Device Boot Start End Blocks Id System
/dev/sdd1 2048 3905535 1951744 fd Linux raid autodetect
/dev/sdd2 * 3905536 3907028991 1951561728 fd Linux raid autodetect
Disk /dev/sde: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009fe7f
Device Boot Start End Blocks Id System
/dev/sde1 2048 3905535 1951744 fd Linux raid autodetect
/dev/sde2 * 3905536 3907028991 1951561728 fd Linux raid autodetect
mdadm -Esvv output (USB devices deleted)
/dev/sde2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : dbe238a3:c7a528c1:a1b78589:276ecfcf
Name : ubuntu:0 (local to host ubuntu)
Creation Time : Wed Apr 1 22:27:42 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3903121408 (1861.15 GiB 1998.40 GB)
Array Size : 5850624 (5.58 GiB 5.99 GB)
Used Dev Size : 3900416 (1904.82 MiB 1997.01 MB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : cdae3287:91168194:942ba99d:1a85c466
Update Time : Wed Apr 29 17:46:25 2015
Checksum : b8b84dad - correct
Events : 30
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAA ('A' == active, '.' == missing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : dbe238a3:c7a528c1:a1b78589:276ecfcf
Name : ubuntu:0 (local to host ubuntu)
Creation Time : Wed Apr 1 22:27:42 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3901440 (1905.32 MiB 1997.54 MB)
Array Size : 5850624 (5.58 GiB 5.99 GB)
Used Dev Size : 3900416 (1904.82 MiB 1997.01 MB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b051f523:4887e729:cd63bed1:8c2a7575
Update Time : Wed Apr 29 17:46:25 2015
Checksum : 453ddeef - correct
Events : 30
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing)
/dev/sde:
MBR Magic : aa55
Partition[0] : 3903488 sectors at 2048 (type fd)
Partition[1] : 3903123456 sectors at 3905536 (type fd)
/dev/sdd2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : dbe238a3:c7a528c1:a1b78589:276ecfcf
Name : ubuntu:0 (local to host ubuntu)
Creation Time : Wed Apr 1 22:27:42 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3903121408 (1861.15 GiB 1998.40 GB)
Array Size : 5850624 (5.58 GiB 5.99 GB)
Used Dev Size : 3900416 (1904.82 MiB 1997.01 MB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 0f3f2b91:09cbb344:e52c4c4b:722d65c4
Update Time : Wed Apr 29 17:46:25 2015
Checksum : 7e273c0f - correct
Events : 30
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : dbe238a3:c7a528c1:a1b78589:276ecfcf
Name : ubuntu:0 (local to host ubuntu)
Creation Time : Wed Apr 1 22:27:42 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3901440 (1905.32 MiB 1997.54 MB)
Array Size : 5850624 (5.58 GiB 5.99 GB)
Used Dev Size : 3900416 (1904.82 MiB 1997.01 MB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b6668730:3b1380bf:556700d9:30df829c
Update Time : Wed Apr 29 17:46:25 2015
Checksum : 15b83814 - correct
Events : 30
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 3903488 sectors at 2048 (type fd)
Partition[1] : 3903123456 sectors at 3905536 (type fd)
/dev/sdb2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 1f28f7bb:7b3ecd41:ca0fa5d1:ccd008df
Name : ubuntu:1 (local to host ubuntu)
Creation Time : Wed Apr 1 22:27:58 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3902861312 (1861.03 GiB 1998.26 GB)
Array Size : 5854290432 (5583.09 GiB 5994.79 GB)
Used Dev Size : 3902860288 (1861.03 GiB 1998.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f1e79609:79b7ac23:55197f70:e8fbfd58
Update Time : Sun Apr 26 05:59:13 2015
Checksum : 696f4e76 - correct
Events : 18014
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AA.. ('A' == active, '.' == missing)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : dbe238a3:c7a528c1:a1b78589:276ecfcf
Name : ubuntu:0 (local to host ubuntu)
Creation Time : Wed Apr 1 22:27:42 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3901440 (1905.32 MiB 1997.54 MB)
Array Size : 5850624 (5.58 GiB 5.99 GB)
Used Dev Size : 3900416 (1904.82 MiB 1997.01 MB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : f52239b1:0fb87e7e:71e29ea4:bf67184a
Update Time : Wed Apr 29 17:46:25 2015
Checksum : ce9c9cd0 - correct
Events : 30
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3903488 sectors at 2048 (type fd)
Partition[1] : 3903123456 sectors at 3905536 (type fd)
/dev/sda2:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 1f28f7bb:7b3ecd41:ca0fa5d1:ccd008df
Name : ubuntu:1 (local to host ubuntu)
Creation Time : Wed Apr 1 22:27:58 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3902861312 (1861.03 GiB 1998.26 GB)
Array Size : 5854290432 (5583.09 GiB 5994.79 GB)
Used Dev Size : 3902860288 (1861.03 GiB 1998.26 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 713e556d:ca104217:785db68a:d820a57b
Update Time : Sun Apr 26 05:59:13 2015
Checksum : fda151f9 - correct
Events : 18014
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AA.. ('A' == active, '.' == missing)
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : dbe238a3:c7a528c1:a1b78589:276ecfcf
Name : ubuntu:0 (local to host ubuntu)
Creation Time : Wed Apr 1 22:27:42 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 3901440 (1905.32 MiB 1997.54 MB)
Array Size : 5850624 (5.58 GiB 5.99 GB)
Used Dev Size : 3900416 (1904.82 MiB 1997.01 MB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c483532d:06f93351:cfdf5a92:e83855b5
Update Time : Wed Apr 29 17:46:25 2015
Checksum : 76650d1c - correct
Events : 30
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sda:
MBR Magic : aa55
Partition[0] : 3903488 sectors at 2048 (type fd)
Partition[1] : 3903123456 sectors at 3905536 (type fd)--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox