* Re: [PATCH] md/raid10: make sync_request_write() call bio_copy_data()
From: Christoph Hellwig @ 2015-04-25 9:47 UTC (permalink / raw)
To: Ming Lin; +Cc: Neil Brown, Jens Axboe, linux-raid, Kent Overstreet, Dongsu Park
In-Reply-To: <1429915930-22449-1-git-send-email-mlin@kernel.org>
On Fri, Apr 24, 2015 at 03:52:10PM -0700, Ming Lin wrote:
> From: Kent Overstreet <kent.overstreet@gmail.com>
>
> Refactor sync_request_write() of md/raid10 to use bio_copy_data()
> instead of open coding bio_vec iterations.
Do you plan to submit an updated version of the immutable biovecs set
soon? In that case it would be good to merge this patch through
the block tree, as it's a requirement for it AFAICS. Might be worth to
repost them together.
^ permalink raw reply
* Re: [PATCH] md/raid10: make sync_request_write() call bio_copy_data()
From: Dongsu Park @ 2015-04-25 7:51 UTC (permalink / raw)
To: Ming Lin
Cc: Neil Brown, Christoph Hellwig, Jens Axboe, linux-raid,
Kent Overstreet
In-Reply-To: <1429915930-22449-1-git-send-email-mlin@kernel.org>
On 24.04.2015 15:52, Ming Lin wrote:
> From: Kent Overstreet <kent.overstreet@gmail.com>
>
> Refactor sync_request_write() of md/raid10 to use bio_copy_data()
> instead of open coding bio_vec iterations.
Looks good. Keep up the good work! ;-)
Thanks,
Dongsu
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Kent Overstreet <kent.overstreet@gmail.com>
> [dpark: add more description in commit message]
> Signed-off-by: Dongsu Park <dongsu.park@profitbricks.com>
> Signed-off-by: Ming Lin <mlin@kernel.org>
> ---
> drivers/md/raid10.c | 20 +++++---------------
> 1 file changed, 5 insertions(+), 15 deletions(-)
>
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index a7196c4..02e33f1 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -2097,18 +2097,11 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
> tbio->bi_vcnt = vcnt;
> tbio->bi_iter.bi_size = r10_bio->sectors << 9;
> tbio->bi_rw = WRITE;
> - tbio->bi_private = r10_bio;
> tbio->bi_iter.bi_sector = r10_bio->devs[i].addr;
> -
> - for (j=0; j < vcnt ; j++) {
> - tbio->bi_io_vec[j].bv_offset = 0;
> - tbio->bi_io_vec[j].bv_len = PAGE_SIZE;
> -
> - memcpy(page_address(tbio->bi_io_vec[j].bv_page),
> - page_address(fbio->bi_io_vec[j].bv_page),
> - PAGE_SIZE);
> - }
> tbio->bi_end_io = end_sync_write;
> + tbio->bi_private = r10_bio;
> +
> + bio_copy_data(tbio, fbio);
>
> d = r10_bio->devs[i].devnum;
> atomic_inc(&conf->mirrors[d].rdev->nr_pending);
> @@ -2124,17 +2117,14 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
> * that are active
> */
> for (i = 0; i < conf->copies; i++) {
> - int j, d;
> + int d;
>
> tbio = r10_bio->devs[i].repl_bio;
> if (!tbio || !tbio->bi_end_io)
> continue;
> if (r10_bio->devs[i].bio->bi_end_io != end_sync_write
> && r10_bio->devs[i].bio != fbio)
> - for (j = 0; j < vcnt; j++)
> - memcpy(page_address(tbio->bi_io_vec[j].bv_page),
> - page_address(fbio->bi_io_vec[j].bv_page),
> - PAGE_SIZE);
> + bio_copy_data(tbio, fbio);
> d = r10_bio->devs[i].devnum;
> atomic_inc(&r10_bio->remaining);
> md_sync_acct(conf->mirrors[d].replacement->bdev,
> --
> 1.9.1
>
^ permalink raw reply
* Re: Massive RAID-1 desync
From: NeilBrown @ 2015-04-25 7:25 UTC (permalink / raw)
To: cau2jeaf1honoq; +Cc: linux-raid
In-Reply-To: <1810942606.18204832.1429908469113.JavaMail.zimbra@laposte.net>
[-- Attachment #1: Type: text/plain, Size: 2216 bytes --]
On Fri, 24 Apr 2015 22:47:49 +0200 (CEST) cau2jeaf1honoq@laposte.net wrote:
> Something is happening here. I don't know what, but I'm having
> fun trying to guess.
>
> The root file system (ext3) is on a 4 x 30 GB RAID-1 array. A
> couple hours after boot, the kernel detected something wrong in
> the file system and decided to remount it read-only.
>
> Comparing the component partitions finds many differences with a
> very uneven distribution :
>
> - sda1 and sdb1 are identical except for 4 bytes in the last
> 70 kB,
Perfectly normal. Metadata is at the end, at least 64K from the end and 64K
aligned.
>
> - sdd1 is identical to sda1 and sdb1 except for about 67,000
> differences in the last 70 kB.
Following the metadata is between 60K and 124K of nothing. It could easily
be completely different on different devices.
>
> - sdc1 is grossly out of sync with about 300 million differences
> with the others, all of them in the first 450 MB or so.
sdc1 is sick.
Maybe it has hardware problems. Maybe some hacker broke into your machine
and wrote garbage to it. Or maybe you triggered a bug that no one else has
ever come across (unlikely, but possible).
>
> I'm not sure what to make of this. The knee-jerk thought would
> be "/dev/sdc1 is the odd man out so sdc must be faulty". But
> that disk participates in other arrays without problems, I don't
> see anything obviously bad in its SMART data and the kernel
> messages just before the remount were actually about sda.
And what were those messages about sda?
>
> To be honest, I don't have a clear idea of how things got where
> they are. Since writing to a RAID-1 array writes the same data
> to all devices, how can you have so many differences ?
Cosmic rays? EMP?
I actually think that the most likely explanation is that someone was
careless and wrote something to sdc that they didn't mean to. But I'm
probably wrong. I like guessing too.
NeilBrown
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: Massive RAID-1 desync
From: Jean-Baptiste Thomas @ 2015-04-25 6:18 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1504250601200.16871@uplift.swm.pp.se>
On 2015-04-25 06:02 +0200, Mikael Abrahamsson wrote:
> What kernel version are you running?
Linux 3.0.14.
mdadm 3.1.4 (2010-08-31)
> Some other information would be
> interesting as well, such as what /proc/mdstat is saying,
Good point. Curiously, nothing :
md1 : active raid1 sdd1[3] sdb1[0] sdc1[2] sda1[1]
31463232 blocks [4/4] [UUUU]
> and anything from dmesg or similar logs leading up to this...
Too late, /var/log was on the root file system and lockd has
since helpfully flooded the kernel ring buffer.
^ permalink raw reply
* Re: Massive RAID-1 desync
From: Mikael Abrahamsson @ 2015-04-25 4:02 UTC (permalink / raw)
To: cau2jeaf1honoq; +Cc: linux-raid
In-Reply-To: <1810942606.18204832.1429908469113.JavaMail.zimbra@laposte.net>
On Fri, 24 Apr 2015, cau2jeaf1honoq@laposte.net wrote:
> Something is happening here. I don't know what, but I'm having
> fun trying to guess.
What kernel version are you running? Some other information would be
interesting as well, such as what /proc/mdstat is saying, and anything
from dmesg or similar logs leading up to this...
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* [PATCH] md/raid10: make sync_request_write() call bio_copy_data()
From: Ming Lin @ 2015-04-24 22:52 UTC (permalink / raw)
To: Neil Brown
Cc: Christoph Hellwig, Jens Axboe, linux-raid, Kent Overstreet,
Dongsu Park, Ming Lin
From: Kent Overstreet <kent.overstreet@gmail.com>
Refactor sync_request_write() of md/raid10 to use bio_copy_data()
instead of open coding bio_vec iterations.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Kent Overstreet <kent.overstreet@gmail.com>
[dpark: add more description in commit message]
Signed-off-by: Dongsu Park <dongsu.park@profitbricks.com>
Signed-off-by: Ming Lin <mlin@kernel.org>
---
drivers/md/raid10.c | 20 +++++---------------
1 file changed, 5 insertions(+), 15 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index a7196c4..02e33f1 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -2097,18 +2097,11 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
tbio->bi_vcnt = vcnt;
tbio->bi_iter.bi_size = r10_bio->sectors << 9;
tbio->bi_rw = WRITE;
- tbio->bi_private = r10_bio;
tbio->bi_iter.bi_sector = r10_bio->devs[i].addr;
-
- for (j=0; j < vcnt ; j++) {
- tbio->bi_io_vec[j].bv_offset = 0;
- tbio->bi_io_vec[j].bv_len = PAGE_SIZE;
-
- memcpy(page_address(tbio->bi_io_vec[j].bv_page),
- page_address(fbio->bi_io_vec[j].bv_page),
- PAGE_SIZE);
- }
tbio->bi_end_io = end_sync_write;
+ tbio->bi_private = r10_bio;
+
+ bio_copy_data(tbio, fbio);
d = r10_bio->devs[i].devnum;
atomic_inc(&conf->mirrors[d].rdev->nr_pending);
@@ -2124,17 +2117,14 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)
* that are active
*/
for (i = 0; i < conf->copies; i++) {
- int j, d;
+ int d;
tbio = r10_bio->devs[i].repl_bio;
if (!tbio || !tbio->bi_end_io)
continue;
if (r10_bio->devs[i].bio->bi_end_io != end_sync_write
&& r10_bio->devs[i].bio != fbio)
- for (j = 0; j < vcnt; j++)
- memcpy(page_address(tbio->bi_io_vec[j].bv_page),
- page_address(fbio->bi_io_vec[j].bv_page),
- PAGE_SIZE);
+ bio_copy_data(tbio, fbio);
d = r10_bio->devs[i].devnum;
atomic_inc(&r10_bio->remaining);
md_sync_acct(conf->mirrors[d].replacement->bdev,
--
1.9.1
^ permalink raw reply related
* Massive RAID-1 desync
From: cau2jeaf1honoq @ 2015-04-24 20:47 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1919189912.18202330.1429908372364.JavaMail.zimbra@laposte.net>
Something is happening here. I don't know what, but I'm having
fun trying to guess.
The root file system (ext3) is on a 4 x 30 GB RAID-1 array. A
couple hours after boot, the kernel detected something wrong in
the file system and decided to remount it read-only.
Comparing the component partitions finds many differences with a
very uneven distribution :
- sda1 and sdb1 are identical except for 4 bytes in the last
70 kB,
- sdd1 is identical to sda1 and sdb1 except for about 67,000
differences in the last 70 kB.
- sdc1 is grossly out of sync with about 300 million differences
with the others, all of them in the first 450 MB or so.
I'm not sure what to make of this. The knee-jerk thought would
be "/dev/sdc1 is the odd man out so sdc must be faulty". But
that disk participates in other arrays without problems, I don't
see anything obviously bad in its SMART data and the kernel
messages just before the remount were actually about sda.
To be honest, I don't have a clear idea of how things got where
they are. Since writing to a RAID-1 array writes the same data
to all devices, how can you have so many differences ?
^ permalink raw reply
* Re: PROBLEM: write to jbod with 3TB and 160GB drives hits BUG/oops
From: Charles Bertsch @ 2015-04-24 20:11 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, BertschC@acm.org
In-Reply-To: <KdvS1q00X1hjKLY01dvTMG>
On 04/23/2015 06:55 PM, NeilBrown wrote:
>
> By "jbod" I assume you mean "linear array".
>
> You say this happens without any filesystem on the array, yet the stack
> traces clearly show ext2 in use.
> Maybe some weird interaction is happening between the the filesystem and the
> linear array.
> But please confirm that the stack trace happened when there was no filesystem
> on the array you were testing, and report what filesystems you do have which
> use ext2.
>
Neil --
Yes, I do mean linear array.
At the point of the stack trace, there was no file-system on the linear
2-drive array. The test-jbod-2 script would create the array and then
write directly to /dev/md0. Any evidence of previous existence of a
file-system would have been obliterated by earlier runs copying
/dev/zero everywhere.
The file-systems in use --
-- The rootfs is an initrd file, squashfs, and mounted read-only.
-- An ext3 for configuration and logs is mounted RW on /flash
-- An ext2 using 8MB of RAM is mounted RW on /var
-- The file-server is derived from a much earlier design that required
some RW directories within the root. These entries appear in the mount
command as ext2, but are part of /var (and not separate file systems) --
-- mount --bind /var/hd /hd
-- mount --bind /var/home /home
-- A devtmpfs mounted on /dev, tmpfs on /dev/shm, proc on /proc, sysfs
on /sys, and another mount --bind from within /flash for nfs.
# mount
/dev/root on / type squashfs (ro,relatime)
devtmpfs on /dev type devtmpfs
(rw,relatime,size=1002600k,nr_inodes=250650,mode=755)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
/dev/ram1 on /var type ext2 (rw,relatime,errors=continue)
/dev/ram1 on /hd type ext2 (rw,relatime,errors=continue)
/dev/ram1 on /home type ext2 (rw,relatime,errors=continue)
tmpfs on /dev/shm type tmpfs (rw,relatime)
/dev/sdb1 on /flash type ext3
(rw,noatime,errors=continue,commit=60,barrier=1,data=ordered)
/dev/sdb1 on /var/lib/nfs type ext3
(rw,noatime,errors=continue,commit=60,barrier=1,data=ordered)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
#
> Is there any chance you could use "git bisect" to find out exactly which
> commit introduced the problem? That is the mostly likely path to a
solution.
>
I am not familiar with "git bisect". Would this be similar to
downloading a series of kernel releases from linux-3.3.5 up to 3.18.5
using a binary search to find which release (rather than which commit)
has the problem ?
Thanks
Charles Bertsch
^ permalink raw reply
* [PATCH 2/2] md/raid5: exclusive wait_for_stripe
From: Yuanhan Liu @ 2015-04-24 13:39 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, linux-kernel, Yuanhan Liu
In-Reply-To: <1429882744-22655-1-git-send-email-yuanhan.liu@linux.intel.com>
I noticed heavy spin lock contention at get_active_stripe() with fsmark
multiple thread write workloads.
Here is how this hot contention comes from. We have limited stripes, and
it's a multiple thread write workload. Hence, those stripes will be taken
soon, which puts later processes to sleep for waiting free stripes. When
enough stripes(> 1/4 total stripes) are released, all process are woken,
trying to get the lock. But there is one only being able to get this lock
for each hash lock, making other processes spinning out there for acquiring
the lock.
Thus, it's effectiveless to wakeup all processes and let them battle for
a lock that permits one to access only each time. Instead, we could make
it be a exclusive wake up: wake up one process only. That avoids the heavy
spin lock contention naturally.
Here are some test results I have got with this patch applied(all test run
3 times):
`fsmark.files_per_sec'
=====================
next-20150317 this patch
------------------------- -------------------------
metric_value ±stddev metric_value ±stddev change testbox/benchmark/testcase-params
------------------------- ------------------------- -------- ------------------------------
25.600 ±0.0 92.700 ±2.5 262.1% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-btrfs-4M-30G-fsyncBeforeClose
25.600 ±0.0 77.800 ±0.6 203.9% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-btrfs-4M-30G-fsyncBeforeClose
32.000 ±0.0 93.800 ±1.7 193.1% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-ext4-4M-30G-fsyncBeforeClose
32.000 ±0.0 81.233 ±1.7 153.9% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-ext4-4M-30G-fsyncBeforeClose
48.800 ±14.5 99.667 ±2.0 104.2% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-xfs-4M-30G-fsyncBeforeClose
6.400 ±0.0 12.800 ±0.0 100.0% ivb44/fsmark/1x-64t-3HDD-RAID5-btrfs-4M-40G-fsyncBeforeClose
63.133 ±8.2 82.800 ±0.7 31.2% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-xfs-4M-30G-fsyncBeforeClose
245.067 ±0.7 306.567 ±7.9 25.1% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-f2fs-4M-30G-fsyncBeforeClose
17.533 ±0.3 21.000 ±0.8 19.8% ivb44/fsmark/1x-1t-3HDD-RAID5-xfs-4M-40G-fsyncBeforeClose
188.167 ±1.9 215.033 ±3.1 14.3% ivb44/fsmark/1x-1t-4BRD_12G-RAID5-btrfs-4M-30G-NoSync
254.500 ±1.8 290.733 ±2.4 14.2% ivb44/fsmark/1x-1t-9BRD_6G-RAID5-btrfs-4M-30G-NoSync
`time.system_time'
=====================
next-20150317 this patch
------------------------- -------------------------
metric_value ±stddev metric_value ±stddev change testbox/benchmark/testcase-params
------------------------- ------------------------- -------- ------------------------------
7235.603 ±1.2 185.163 ±1.9 -97.4% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-btrfs-4M-30G-fsyncBeforeClose
7666.883 ±2.9 202.750 ±1.0 -97.4% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-btrfs-4M-30G-fsyncBeforeClose
14567.893 ±0.7 421.230 ±0.4 -97.1% ivb44/fsmark/1x-64t-3HDD-RAID5-btrfs-4M-40G-fsyncBeforeClose
3697.667 ±14.0 148.190 ±1.7 -96.0% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-xfs-4M-30G-fsyncBeforeClose
5572.867 ±3.8 310.717 ±1.4 -94.4% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-ext4-4M-30G-fsyncBeforeClose
5565.050 ±0.5 313.277 ±1.5 -94.4% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-ext4-4M-30G-fsyncBeforeClose
2420.707 ±17.1 171.043 ±2.7 -92.9% ivb44/fsmark/1x-64t-9BRD_6G-RAID5-xfs-4M-30G-fsyncBeforeClose
3743.300 ±4.6 379.827 ±3.5 -89.9% ivb44/fsmark/1x-64t-3HDD-RAID5-ext4-4M-40G-fsyncBeforeClose
3308.687 ±6.3 363.050 ±2.0 -89.0% ivb44/fsmark/1x-64t-3HDD-RAID5-xfs-4M-40G-fsyncBeforeClose
Where,
1x: where 'x' means iterations or loop, corresponding to the 'L' option of fsmark
1t, 64t: where 't' means thread
4M: means the single file size, corresponding to the '-s' option of fsmark
40G, 30G, 120G: means the total test size
4BRD_12G: BRD is the ramdisk, where '4' means 4 ramdisk, and where '12G' means
the size of one ramdisk. So, it would be 48G in total. And we made a
raid on those ramdisk
As you can see, though there are no much performance gain for hard disk
workload, the system time is dropped heavily, up to 97%. And as expected,
the performance increased a lot, up to 260%, for fast device(ram disk).
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
drivers/md/raid5.c | 46 +++++++++++++++++++++++++++++++++++-----------
drivers/md/raid5.h | 2 +-
2 files changed, 36 insertions(+), 12 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index b7e385f..2d8fcc1 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -344,7 +344,8 @@ static void release_inactive_stripe_list(struct r5conf *conf,
int hash)
{
int size;
- bool do_wakeup = false;
+ bool do_wakeup[NR_STRIPE_HASH_LOCKS] = { false, };
+ int i = 0;
unsigned long flags;
if (hash == NR_STRIPE_HASH_LOCKS) {
@@ -365,17 +366,22 @@ static void release_inactive_stripe_list(struct r5conf *conf,
!list_empty(list))
atomic_dec(&conf->empty_inactive_list_nr);
list_splice_tail_init(list, conf->inactive_list + hash);
- do_wakeup = true;
+ do_wakeup[size - 1] = true;
spin_unlock_irqrestore(conf->hash_locks + hash, flags);
}
size--;
hash--;
}
- if (do_wakeup) {
- wake_up(&conf->wait_for_stripe);
- if (conf->retry_read_aligned)
- md_wakeup_thread(conf->mddev->thread);
+ for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++) {
+ bool waked_thread = false;
+ if (do_wakeup[i]) {
+ wake_up(&conf->wait_for_stripe[i]);
+ if (!waked_thread) {
+ waked_thread = true;
+ md_wakeup_thread(conf->mddev->thread);
+ }
+ }
}
}
@@ -655,6 +661,18 @@ static int has_failed(struct r5conf *conf)
return 0;
}
+/* XXX: might put it to linux/wait.h to be a public API? */
+#define raid_wait_event_exclusive_cmd(wq, condition, cmd1, cmd2) \
+do { \
+ if (condition) \
+ break; \
+ (void)___wait_event(wq, condition, TASK_UNINTERRUPTIBLE, 1, 0, \
+ cmd1; \
+ schedule(); \
+ cmd2); \
+} while (0)
+
+
static struct stripe_head *
get_active_stripe(struct r5conf *conf, sector_t sector,
int previous, int noblock, int noquiesce)
@@ -684,14 +702,15 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
if (!sh) {
set_bit(R5_INACTIVE_BLOCKED,
&conf->cache_state);
- wait_event_lock_irq(
- conf->wait_for_stripe,
+ raid_wait_event_exclusive_cmd(
+ conf->wait_for_stripe[hash],
!list_empty(conf->inactive_list + hash) &&
(atomic_read(&conf->active_stripes)
< (conf->max_nr_stripes * 3 / 4)
|| !test_bit(R5_INACTIVE_BLOCKED,
&conf->cache_state)),
- *(conf->hash_locks + hash));
+ spin_unlock_irq(conf->hash_locks + hash),
+ spin_lock_irq(conf->hash_locks + hash));
clear_bit(R5_INACTIVE_BLOCKED,
&conf->cache_state);
} else {
@@ -716,6 +735,9 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
}
} while (sh == NULL);
+ if (!list_empty(conf->inactive_list + hash))
+ wake_up(&conf->wait_for_stripe[hash]);
+
spin_unlock_irq(conf->hash_locks + hash);
return sh;
}
@@ -2136,7 +2158,7 @@ static int resize_stripes(struct r5conf *conf, int newsize)
cnt = 0;
list_for_each_entry(nsh, &newstripes, lru) {
lock_device_hash_lock(conf, hash);
- wait_event_cmd(conf->wait_for_stripe,
+ raid_wait_event_exclusive_cmd(conf->wait_for_stripe[hash],
!list_empty(conf->inactive_list + hash),
unlock_device_hash_lock(conf, hash),
lock_device_hash_lock(conf, hash));
@@ -6391,7 +6413,9 @@ static struct r5conf *setup_conf(struct mddev *mddev)
spin_lock_init(&conf->device_lock);
seqcount_init(&conf->gen_lock);
init_waitqueue_head(&conf->wait_for_quiesce);
- init_waitqueue_head(&conf->wait_for_stripe);
+ for (i = 0; i < NR_STRIPE_HASH_LOCKS; i++) {
+ init_waitqueue_head(&conf->wait_for_stripe[i]);
+ }
init_waitqueue_head(&conf->wait_for_overlap);
INIT_LIST_HEAD(&conf->handle_list);
INIT_LIST_HEAD(&conf->hold_list);
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index fab53a3..cdad2d2 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -509,7 +509,7 @@ struct r5conf {
atomic_t empty_inactive_list_nr;
struct llist_head released_stripes;
wait_queue_head_t wait_for_quiesce;
- wait_queue_head_t wait_for_stripe;
+ wait_queue_head_t wait_for_stripe[NR_STRIPE_HASH_LOCKS];
wait_queue_head_t wait_for_overlap;
unsigned long cache_state;
#define R5_INACTIVE_BLOCKED 1 /* release of inactive stripes blocked,
--
1.9.0
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 1/2] md/raid5: split wait_for_stripe and introduce wait_for_quiesce
From: Yuanhan Liu @ 2015-04-24 13:39 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, linux-kernel, Yuanhan Liu
If I read code correctly, current wait_for_stripe actually has 2 usage:
- wait for there is enough free stripe cache, triggered when
get_free_stripe() failed. This is what wait_for_stripe intend
for literally.
- wait for quiesce == 0 or
active_aligned_reads == 0 && active_stripes == 0
It has nothing to do with wait_for_stripe literally, and releasing
an active stripe won't actually wake them up. On the contrary, wake_up
from under this case won't actually wake up the process waiting for
an free stripe being available.
Hence, we'd better split wait_for_stripe, and here I introduce
wait_for_quiesce for the second usage. The name may not well taken, or
even taken wrongly. Feel free to correct me then.
This is also a prepare patch for next patch: make wait_for_stripe
exclusive.
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
drivers/md/raid5.c | 13 +++++++------
drivers/md/raid5.h | 1 +
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 9716319..b7e385f 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -667,7 +667,7 @@ get_active_stripe(struct r5conf *conf, sector_t sector,
spin_lock_irq(conf->hash_locks + hash);
do {
- wait_event_lock_irq(conf->wait_for_stripe,
+ wait_event_lock_irq(conf->wait_for_quiesce,
conf->quiesce == 0 || noquiesce,
*(conf->hash_locks + hash));
sh = __find_stripe(conf, sector, conf->generation - previous);
@@ -4725,7 +4725,7 @@ static void raid5_align_endio(struct bio *bi, int error)
raid_bi, 0);
bio_endio(raid_bi, 0);
if (atomic_dec_and_test(&conf->active_aligned_reads))
- wake_up(&conf->wait_for_stripe);
+ wake_up(&conf->wait_for_quiesce);
return;
}
@@ -4820,7 +4820,7 @@ static int chunk_aligned_read(struct mddev *mddev, struct bio * raid_bio)
align_bi->bi_iter.bi_sector += rdev->data_offset;
spin_lock_irq(&conf->device_lock);
- wait_event_lock_irq(conf->wait_for_stripe,
+ wait_event_lock_irq(conf->wait_for_quiesce,
conf->quiesce == 0,
conf->device_lock);
atomic_inc(&conf->active_aligned_reads);
@@ -5659,7 +5659,7 @@ static int retry_aligned_read(struct r5conf *conf, struct bio *raid_bio)
bio_endio(raid_bio, 0);
}
if (atomic_dec_and_test(&conf->active_aligned_reads))
- wake_up(&conf->wait_for_stripe);
+ wake_up(&conf->wait_for_quiesce);
return handled;
}
@@ -6390,6 +6390,7 @@ static struct r5conf *setup_conf(struct mddev *mddev)
goto abort;
spin_lock_init(&conf->device_lock);
seqcount_init(&conf->gen_lock);
+ init_waitqueue_head(&conf->wait_for_quiesce);
init_waitqueue_head(&conf->wait_for_stripe);
init_waitqueue_head(&conf->wait_for_overlap);
INIT_LIST_HEAD(&conf->handle_list);
@@ -7413,7 +7414,7 @@ static void raid5_quiesce(struct mddev *mddev, int state)
* active stripes can drain
*/
conf->quiesce = 2;
- wait_event_cmd(conf->wait_for_stripe,
+ wait_event_cmd(conf->wait_for_quiesce,
atomic_read(&conf->active_stripes) == 0 &&
atomic_read(&conf->active_aligned_reads) == 0,
unlock_all_device_hash_locks_irq(conf),
@@ -7427,7 +7428,7 @@ static void raid5_quiesce(struct mddev *mddev, int state)
case 0: /* re-enable writes */
lock_all_device_hash_locks_irq(conf);
conf->quiesce = 0;
- wake_up(&conf->wait_for_stripe);
+ wake_up(&conf->wait_for_quiesce);
wake_up(&conf->wait_for_overlap);
unlock_all_device_hash_locks_irq(conf);
break;
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 7dc0dd8..fab53a3 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -508,6 +508,7 @@ struct r5conf {
struct list_head inactive_list[NR_STRIPE_HASH_LOCKS];
atomic_t empty_inactive_list_nr;
struct llist_head released_stripes;
+ wait_queue_head_t wait_for_quiesce;
wait_queue_head_t wait_for_stripe;
wait_queue_head_t wait_for_overlap;
unsigned long cache_state;
--
1.9.0
^ permalink raw reply related
* Re: Bisected, with rfc/patch - was Re: BUG: unable to handle kernel NULL pointer dereference at sysfs_do_create_link_sd (after mdadm)
From: Christoph Hellwig @ 2015-04-24 8:27 UTC (permalink / raw)
To: NeilBrown
Cc: Christoph Hellwig, Azat Khuzhin, Kernel.org-Linux-RAID,
Guoqing Jiang, Tejun Heo, Jan Kara, Jens Axboe, dm-devel
In-Reply-To: <20150424120932.3d554638@notabene.brown>
On Fri, Apr 24, 2015 at 12:09:32PM +1000, NeilBrown wrote:
> I'm pretty sure that the md code is already as close to the "dm model" as it
> meaningfully can be.
>
> If I move bdi_destroy out of blk_release_queue (which really think is too
> later) and place it in blk_cleanup_queue (which seems a credible place for
> it), and then move the blk_cleanup_queue call in md_free up before the
> del_gendisk() call (which is probably the right thing to do anyway, though dm
> has the same order that md currently has) then I don't get any crashes and
> I'm almost convince it is correct...
This looks reasonable to me, thanks.
^ permalink raw reply
* [PATCH 10/10] Reuse the write_bitmap for update uuid
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
Since write_bitmap is extended for handle different sistuations,
then it also could possible to support change the uuid of bitmap,
and remove bitmap_update_uuid accordingly.
Q: is the write_bitmap0 also impacted?
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Assemble.c | 5 ++---
bitmap.c | 20 --------------------
mdadm.h | 2 +-
super1.c | 4 ++++
4 files changed, 7 insertions(+), 24 deletions(-)
diff --git a/Assemble.c b/Assemble.c
index 22042a9..314e4b4 100644
--- a/Assemble.c
+++ b/Assemble.c
@@ -664,9 +664,8 @@ static int load_devices(struct devs *devices, char *devmap,
if (strcmp(c->update, "uuid")==0 &&
ident->bitmap_fd >= 0 && !bitmap_done) {
- if (bitmap_update_uuid(ident->bitmap_fd,
- content->uuid,
- tst->ss->swapuuid) != 0)
+ copy_uuid(tst->devs->uuid, content->uuid, tst->ss->swapuuid);
+ if (tst->ss->write_bitmap(tst, dfd, UUIDUpdate))
pr_err("Could not update uuid on external bitmap.\n");
else
bitmap_done = 1;
diff --git a/bitmap.c b/bitmap.c
index ab83f4e..cd65050 100644
--- a/bitmap.c
+++ b/bitmap.c
@@ -441,23 +441,3 @@ out:
unlink(filename); /* possibly corrupted, better get rid of it */
return rv;
}
-
-int bitmap_update_uuid(int fd, int *uuid, int swap)
-{
- struct bitmap_super_s bm;
- if (lseek(fd, 0, 0) != 0)
- return 1;
- if (read(fd, &bm, sizeof(bm)) != sizeof(bm))
- return 1;
- if (bm.magic != __cpu_to_le32(BITMAP_MAGIC))
- return 1;
- copy_uuid(bm.uuid, uuid, swap);
- if (lseek(fd, 0, 0) != 0)
- return 2;
- if (write(fd, &bm, sizeof(bm)) != sizeof(bm)) {
- lseek(fd, 0, 0);
- return 2;
- }
- lseek(fd, 0, 0);
- return 0;
-}
diff --git a/mdadm.h b/mdadm.h
index 97892e6..7b9bb28 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -358,6 +358,7 @@ enum bitmap_update {
NoUpdate,
NameUpdate,
NodeNumUpdate,
+ UUIDUpdate,
};
/* structures read from config file */
@@ -1273,7 +1274,6 @@ extern int CreateBitmap(char *filename, int force, char uuid[16],
int major);
extern int ExamineBitmap(char *filename, int brief, struct supertype *st);
extern int Write_rules(char *rule_name);
-extern int bitmap_update_uuid(int fd, int *uuid, int swap);
extern unsigned long bitmap_sectors(struct bitmap_super_s *bsb);
extern int Dump_metadata(char *dev, char *dir, struct context *c,
struct supertype *st);
diff --git a/super1.c b/super1.c
index 047e799..b2fc6d7 100644
--- a/super1.c
+++ b/super1.c
@@ -2228,6 +2228,10 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
case NodeNumUpdate:
bms->nodes = __cpu_to_le32(st->nodes);
break;
+ case UUIDUpdate:
+ memset((char *)bms->uuid, 0, sizeof(bms->uuid));
+ strncpy((char *)bms->uuid, (char *)st->devs->uuid, sizeof(bms->uuid));
+ break;
case NoUpdate:
default:
break;
--
1.7.12.4
^ permalink raw reply related
* [PATCH 09/10] mdadm: change the num of cluster node
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
This extends nodes option for assemble mode, make the num of
cluster node could be change by user.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Assemble.c | 4 ++++
ReadMe.c | 2 +-
mdadm.c | 3 +++
mdadm.h | 1 +
super1.c | 6 ++++++
5 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/Assemble.c b/Assemble.c
index e1b846c..22042a9 100644
--- a/Assemble.c
+++ b/Assemble.c
@@ -649,6 +649,10 @@ static int load_devices(struct devs *devices, char *devmap,
devname, 0, 0, c->homecluster);
tst->ss->write_bitmap(tst, dfd, NameUpdate);
}
+ if (strcmp(c->update, "nodes") == 0) {
+ tst->nodes = c->nodes;
+ tst->ss->write_bitmap(tst, dfd, NodeNumUpdate);
+ }
if (strcmp(c->update, "uuid")==0 &&
!ident->uuid_set) {
ident->uuid_set = 1;
diff --git a/ReadMe.c b/ReadMe.c
index c854cd5..d1830e1 100644
--- a/ReadMe.c
+++ b/ReadMe.c
@@ -140,7 +140,7 @@ struct option long_options[] = {
{"homehost", 1, 0, HomeHost},
{"symlinks", 1, 0, Symlinks},
{"data-offset",1, 0, DataOffset},
- {"nodes",1, 0, Nodes},
+ {"nodes",1, 0, Nodes}, /* also for --assemble */
{"home-cluster",1, 0, ClusterName},
/* For assemble */
diff --git a/mdadm.c b/mdadm.c
index 20f195d..344bde2 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -589,6 +589,7 @@ int main(int argc, char *argv[])
}
ident.raid_disks = s.raiddisks;
continue;
+ case O(ASSEMBLE, Nodes):
case O(CREATE, Nodes):
c.nodes = parse_num(optarg);
if (c.nodes <= 0) {
@@ -744,6 +745,8 @@ int main(int argc, char *argv[])
continue;
if (strcmp(c.update, "home-cluster")==0)
continue;
+ if (strcmp(c.update, "nodes")==0)
+ continue;
if (strcmp(c.update, "devicesize")==0)
continue;
if (strcmp(c.update, "no-bitmap")==0)
diff --git a/mdadm.h b/mdadm.h
index d8b0749..97892e6 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -357,6 +357,7 @@ enum prefix_standard {
enum bitmap_update {
NoUpdate,
NameUpdate,
+ NodeNumUpdate,
};
/* structures read from config file */
diff --git a/super1.c b/super1.c
index e43bef1..047e799 100644
--- a/super1.c
+++ b/super1.c
@@ -1329,6 +1329,9 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
sb->devflags |= WriteMostly1;
else if (strcmp(update, "readwrite")==0)
sb->devflags &= ~WriteMostly1;
+ else if (strcmp(update, "nodes")==0)
+ /* Just a placeholder since no related member in mdp_superblock_1 */
+ ;
else
rv = -1;
@@ -2222,6 +2225,9 @@ static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update
free(new_name);
break;
}
+ case NodeNumUpdate:
+ bms->nodes = __cpu_to_le32(st->nodes);
+ break;
case NoUpdate:
default:
break;
--
1.7.12.4
^ permalink raw reply related
* [PATCH 08/10] mdadm: add the ability to change cluster name
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
To support change the cluster name, the commit do the followings:
1. extend original write_bitmap function for new scenario.
2. add the scenarion to handle the modification of cluster's name
in write_bitmap1.
3. make update_super1 can change the name in mdp_superblock_1.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Assemble.c | 5 +++++
Grow.c | 2 +-
mdadm.c | 3 +++
mdadm.h | 7 ++++++-
super0.c | 4 ++--
super1.c | 46 ++++++++++++++++++++++++++++++++++++++++++++--
6 files changed, 61 insertions(+), 6 deletions(-)
diff --git a/Assemble.c b/Assemble.c
index 25a103d..e1b846c 100644
--- a/Assemble.c
+++ b/Assemble.c
@@ -644,6 +644,11 @@ static int load_devices(struct devs *devices, char *devmap,
*stp = st;
return -1;
}
+ if (strcmp(c->update, "home-cluster") == 0) {
+ err = tst->ss->update_super(tst, content, c->update,
+ devname, 0, 0, c->homecluster);
+ tst->ss->write_bitmap(tst, dfd, NameUpdate);
+ }
if (strcmp(c->update, "uuid")==0 &&
!ident->uuid_set) {
ident->uuid_set = 1;
diff --git a/Grow.c b/Grow.c
index 1122cec..bf44e66 100644
--- a/Grow.c
+++ b/Grow.c
@@ -420,7 +420,7 @@ int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
bitmapsize, offset_setable,
major)
)
- st->ss->write_bitmap(st, fd2);
+ st->ss->write_bitmap(st, fd2, NoUpdate);
else {
pr_err("failed to create internal bitmap - chunksize problem.\n");
close(fd2);
diff --git a/mdadm.c b/mdadm.c
index 5b4b3ef..20f195d 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -598,6 +598,7 @@ int main(int argc, char *argv[])
}
continue;
case O(CREATE, ClusterName):
+ case O(ASSEMBLE, ClusterName):
c.homecluster = optarg;
if (strlen(c.homecluster) > 64) {
pr_err("Cluster name too big.\n");
@@ -741,6 +742,8 @@ int main(int argc, char *argv[])
continue;
if (strcmp(c.update, "homehost")==0)
continue;
+ if (strcmp(c.update, "home-cluster")==0)
+ continue;
if (strcmp(c.update, "devicesize")==0)
continue;
if (strcmp(c.update, "no-bitmap")==0)
diff --git a/mdadm.h b/mdadm.h
index 00c726e..d8b0749 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -354,6 +354,11 @@ enum prefix_standard {
IEC
};
+enum bitmap_update {
+ NoUpdate,
+ NameUpdate,
+};
+
/* structures read from config file */
/* List of mddevice names and identifiers
* Identifiers can be:
@@ -850,7 +855,7 @@ extern struct superswitch {
/* if add_internal_bitmap succeeded for existing array, this
* writes it out.
*/
- int (*write_bitmap)(struct supertype *st, int fd);
+ int (*write_bitmap)(struct supertype *st, int fd, enum bitmap_update update);
/* Free the superblock and any other allocated data */
void (*free_super)(struct supertype *st);
diff --git a/super0.c b/super0.c
index deb5999..6ad9d39 100644
--- a/super0.c
+++ b/super0.c
@@ -900,7 +900,7 @@ static int write_init_super0(struct supertype *st)
rv = store_super0(st, di->fd);
if (rv == 0 && (sb->state & (1<<MD_SB_BITMAP_PRESENT)))
- rv = st->ss->write_bitmap(st, di->fd);
+ rv = st->ss->write_bitmap(st, di->fd, NoUpdate);
if (rv)
pr_err("failed to write superblock to %s\n",
@@ -1175,7 +1175,7 @@ static void locate_bitmap0(struct supertype *st, int fd)
lseek64(fd, offset, 0);
}
-static int write_bitmap0(struct supertype *st, int fd)
+static int write_bitmap0(struct supertype *st, int fd, enum bitmap_update update)
{
unsigned long long dsize;
unsigned long long offset;
diff --git a/super1.c b/super1.c
index 3c2fce2..e43bef1 100644
--- a/super1.c
+++ b/super1.c
@@ -1073,6 +1073,25 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
info->name[32] = 0;
}
+ if (strcmp(update, "home-cluster") == 0 &&
+ homehost) {
+ /* Note that 'home-cluster' is to change the name of cluster,
+ * it is another "name" update.
+ */
+ char *new_name = xmalloc(sizeof(sb->set_name));
+ if (strrchr(sb->set_name, ':')) {
+ strcpy(new_name, strchr(sb->set_name, ':'));
+ }
+
+ memset(sb->set_name, 0, sizeof(sb->set_name));
+ strcpy(sb->set_name, homehost);
+ if (new_name)
+ strcat(sb->set_name, new_name);
+
+ free(new_name);
+ goto out;
+ }
+
if (strcmp(update, "force-one")==0) {
/* Not enough devices for a working array,
* so bring this one up-to-date
@@ -1313,6 +1332,7 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
else
rv = -1;
+out:
sb->sb_csum = calc_sb_1_csum(sb);
return rv;
}
@@ -1691,7 +1711,7 @@ static int write_init_super1(struct supertype *st)
sb->sb_csum = calc_sb_1_csum(sb);
rv = store_super1(st, di->fd);
if (rv == 0 && (__le32_to_cpu(sb->feature_map) & 1))
- rv = st->ss->write_bitmap(st, di->fd);
+ rv = st->ss->write_bitmap(st, di->fd, NoUpdate);
close(di->fd);
di->fd = -1;
if (rv)
@@ -2175,7 +2195,7 @@ static void locate_bitmap1(struct supertype *st, int fd)
lseek64(fd, offset<<9, 0);
}
-static int write_bitmap1(struct supertype *st, int fd)
+static int write_bitmap1(struct supertype *st, int fd, enum bitmap_update update)
{
struct mdp_superblock_1 *sb = st->sb;
bitmap_super_t *bms = (bitmap_super_t*)(((char*)sb)+MAX_SB_SIZE);
@@ -2185,6 +2205,28 @@ static int write_bitmap1(struct supertype *st, int fd)
struct align_fd afd;
unsigned int i;
+ switch (update) {
+ case NameUpdate:
+ {
+ char *new_name = xmalloc(sizeof(sb->set_name));
+
+ strncpy(new_name, sb->set_name, sizeof(sb->set_name));
+ memset((char *)bms->cluster_name, 0, sizeof(bms->cluster_name));
+
+ if (strtok(new_name, ":"))
+ strncpy((char *)bms->cluster_name, new_name, strlen(sb->set_name));
+ else
+ /* In case the original set_name doesn't like aaa:md* */
+ strncpy((char *)bms->cluster_name, sb->set_name, strlen(sb->set_name));
+
+ free(new_name);
+ break;
+ }
+ case NoUpdate:
+ default:
+ break;
+ }
+
init_afd(&afd, fd);
locate_bitmap1(st, fd);
--
1.7.12.4
^ permalink raw reply related
* [PATCH 07/10] Skip clustered devices in incremental
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
We want the clustered devices to be started exclusively by a cluster
resource-agent. So, avoid starting using the incremental option.
This also skips a clustered md from starting during boot in inactive mode.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Incremental.c | 5 +++++
super1.c | 2 ++
2 files changed, 7 insertions(+)
diff --git a/Incremental.c b/Incremental.c
index 0c9a9a4..5450a5c 100644
--- a/Incremental.c
+++ b/Incremental.c
@@ -232,6 +232,11 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
devname);
goto out;
}
+ /* Skip the clustered ones. This should be started by
+ * clustering resource agents
+ */
+ if (info.array.state & (1 << MD_SB_CLUSTERED))
+ goto out;
/* 3a/ if not, check for homehost match. If no match, continue
* but don't trust the 'name' in the array. Thus a 'random' minor
diff --git a/super1.c b/super1.c
index ac1b011..3c2fce2 100644
--- a/super1.c
+++ b/super1.c
@@ -891,6 +891,8 @@ static void getinfo_super1(struct supertype *st, struct mdinfo *info, char *map)
info->array.state =
(__le64_to_cpu(sb->resync_offset) == MaxSector)
? 1 : 0;
+ if (__le32_to_cpu(bsb->nodes) > 1)
+ info->array.state |= (1 << MD_SB_CLUSTERED);
info->data_offset = __le64_to_cpu(sb->data_offset);
info->component_size = __le64_to_cpu(sb->size);
--
1.7.12.4
^ permalink raw reply related
* [PATCH 06/10] Convert a bitmap=none device to clustered
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
This adds the ability to convert a regular md without bitmap
(--bitmap=none) to a clustered device (--bitmap=clustered).
To convert a device with --bitmap=internal or --bitmap=external,
you have to convert to --bitmap=none and then re-execute the
command with --bitmap=clustered.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Grow.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/Grow.c b/Grow.c
index 9a573fd..1122cec 100644
--- a/Grow.c
+++ b/Grow.c
@@ -330,9 +330,16 @@ int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
}
return 0;
}
- pr_err("Internal bitmap already present on %s\n",
- devname);
- return 1;
+ if ((strcmp(s->bitmap_file, "clustered")==0) && (array.state & (1<<MD_SB_CLUSTERED))) {
+ pr_err("Clustered bitmap already present on %s\n",
+ devname);
+ return 1;
+ }
+ if ((strcmp(s->bitmap_file, "internal")==0) && (!(array.state & (1<<MD_SB_CLUSTERED)))) {
+ pr_err("Internal bitmap already present on %s\n",
+ devname);
+ return 1;
+ }
}
if (strcmp(s->bitmap_file, "none") == 0) {
@@ -375,7 +382,8 @@ int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
free(st);
return 1;
}
- if (strcmp(s->bitmap_file, "internal") == 0) {
+ if ((strcmp(s->bitmap_file, "internal") == 0) ||
+ (strcmp(s->bitmap_file, "clustered") == 0)) {
int rv;
int d;
int offset_setable = 0;
@@ -384,6 +392,8 @@ int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
pr_err("Internal bitmaps not supported with %s metadata\n", st->ss->name);
return 1;
}
+ st->nodes = c->nodes;
+ st->cluster_name = c->homecluster;
mdi = sysfs_read(fd, NULL, GET_BITMAP_LOCATION);
if (mdi)
offset_setable = 1;
@@ -426,6 +436,8 @@ int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
rv = sysfs_set_num_signed(mdi, NULL, "bitmap/location",
mdi->bitmap_offset);
} else {
+ if (strcmp(s->bitmap_file, "clustered") == 0)
+ array.state |= (1<<MD_SB_CLUSTERED);
array.state |= (1<<MD_SB_BITMAP_PRESENT);
rv = ioctl(fd, SET_ARRAY_INFO, &array);
}
--
1.7.12.4
^ permalink raw reply related
* [PATCH 05/10] Add a new clustered disk
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
A clustered disk is added by the traditional --add sequence.
However, other nodes need to acknowledge that they can "see"
the device. This is done by --cluster-confirm:
--cluster-confirm Y:/dev/whatever (if disk is found)
or
--cluster-confirm Y:missing (if disk is not found)
The node initiating the --add, has the disk state tagged with
MD_DISK_CLUSTER_ADD and the one confirming tag the disk with
MD_DISK_CANDIDATE.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Manage.c | 33 +++++++++++++++++++++++++++++----
ReadMe.c | 1 +
md_p.h | 7 +++++++
md_u.h | 1 +
mdadm.8.in | 9 +++++++++
mdadm.c | 4 ++++
mdadm.h | 2 ++
util.c | 11 +++++++++++
8 files changed, 64 insertions(+), 4 deletions(-)
diff --git a/Manage.c b/Manage.c
index d3cfb55..4c3d451 100644
--- a/Manage.c
+++ b/Manage.c
@@ -690,7 +690,8 @@ skip_re_add:
int Manage_add(int fd, int tfd, struct mddev_dev *dv,
struct supertype *tst, mdu_array_info_t *array,
int force, int verbose, char *devname,
- char *update, unsigned long rdev, unsigned long long array_size)
+ char *update, unsigned long rdev, unsigned long long array_size,
+ int raid_slot)
{
unsigned long long ldsize;
struct supertype *dev_st = NULL;
@@ -879,7 +880,10 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
}
disc.major = major(rdev);
disc.minor = minor(rdev);
- disc.number =j;
+ if (raid_slot < 0)
+ disc.number = j;
+ else
+ disc.number = raid_slot;
disc.state = 0;
if (array->not_persistent==0) {
int dfd;
@@ -920,6 +924,14 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
}
free(used);
}
+
+ if (array->state & (1 << MD_SB_CLUSTERED)) {
+ if (dv->disposition == 'c')
+ disc.state |= (1 << MD_DISK_CANDIDATE);
+ else
+ disc.state |= (1 << MD_DISK_CLUSTER_ADD);
+ }
+
if (dv->writemostly == 1)
disc.state |= (1 << MD_DISK_WRITEMOSTLY);
if (tst->ss->external) {
@@ -1239,6 +1251,7 @@ int Manage_subdevs(char *devname, int fd,
* variant on 'A'
* 'F' - Another variant of 'A', where the device was faulty
* so must be removed from the array first.
+ * 'c' - confirm the device as found (for clustered environments)
*
* For 'f' and 'r', the device can also be a kernel-internal
* name such as 'sdb'.
@@ -1254,6 +1267,7 @@ int Manage_subdevs(char *devname, int fd,
struct mdinfo info;
int frozen = 0;
int busy = 0;
+ int raid_slot = -1;
if (ioctl(fd, GET_ARRAY_INFO, &array)) {
pr_err("Cannot get array info for %s\n",
@@ -1282,6 +1296,11 @@ int Manage_subdevs(char *devname, int fd,
int rv;
int mj,mn;
+ raid_slot = -1;
+ if (dv->disposition == 'c')
+ parse_cluster_confirm_arg(dv->devname, &dv->devname,
+ &raid_slot);
+
if (strcmp(dv->devname, "failed") == 0 ||
strcmp(dv->devname, "faulty") == 0) {
if (dv->disposition != 'A'
@@ -1307,6 +1326,11 @@ int Manage_subdevs(char *devname, int fd,
if (strcmp(dv->devname, "missing") == 0) {
struct mddev_dev *add_devlist = NULL;
struct mddev_dev **dp;
+ if (dv->disposition == 'c') {
+ rv = ioctl(fd, CLUSTERED_DISK_NACK, NULL);
+ break;
+ }
+
if (dv->disposition != 'A') {
pr_err("'missing' only meaningful with --re-add\n");
goto abort;
@@ -1399,7 +1423,7 @@ int Manage_subdevs(char *devname, int fd,
else {
int open_err = errno;
if (stat(dv->devname, &stb) != 0) {
- pr_err("Cannot find %s: %s\n",
+ pr_err("%s: %d Cannot find %s: %s\n", __func__, __LINE__,
dv->devname, strerror(errno));
goto abort;
}
@@ -1437,6 +1461,7 @@ int Manage_subdevs(char *devname, int fd,
case 'A':
case 'M': /* --re-add missing */
case 'F': /* --re-add faulty */
+ case 'c': /* --cluster-confirm */
/* add the device */
if (subarray) {
pr_err("Cannot add disks to a \'member\' array, perform this operation on the parent container\n");
@@ -1470,7 +1495,7 @@ int Manage_subdevs(char *devname, int fd,
}
rv = Manage_add(fd, tfd, dv, tst, &array,
force, verbose, devname, update,
- rdev, array_size);
+ rdev, array_size, raid_slot);
close(tfd);
tfd = -1;
if (rv < 0)
diff --git a/ReadMe.c b/ReadMe.c
index c6286ae..c854cd5 100644
--- a/ReadMe.c
+++ b/ReadMe.c
@@ -169,6 +169,7 @@ struct option long_options[] = {
{"wait", 0, 0, WaitOpt},
{"wait-clean", 0, 0, Waitclean },
{"action", 1, 0, Action },
+ {"cluster-confirm", 0, 0, ClusterConfirm},
/* For Detail/Examine */
{"brief", 0, 0, Brief},
diff --git a/md_p.h b/md_p.h
index c4846ba..e59504f 100644
--- a/md_p.h
+++ b/md_p.h
@@ -78,6 +78,12 @@
#define MD_DISK_ACTIVE 1 /* disk is running but may not be in sync */
#define MD_DISK_SYNC 2 /* disk is in sync with the raid set */
#define MD_DISK_REMOVED 3 /* disk is in sync with the raid set */
+#define MD_DISK_CLUSTER_ADD 4 /* Initiate a disk add across the cluster
+ * For clustered enviroments only.
+ */
+#define MD_DISK_CANDIDATE 5 /* disk is added as spare (local) until confirmed
+ * For clustered enviroments only.
+ */
#define MD_DISK_WRITEMOSTLY 9 /* disk is "write-mostly" is RAID1 config.
* read requests will only be sent here in
@@ -106,6 +112,7 @@ typedef struct mdp_device_descriptor_s {
#define MD_SB_BLOCK_CONTAINER_RESHAPE 3 /* block container wide reshapes */
#define MD_SB_BLOCK_VOLUME 4 /* block activation of array, other arrays
* in container can be activated */
+#define MD_SB_CLUSTERED 5 /* MD is clustered */
#define MD_SB_BITMAP_PRESENT 8 /* bitmap may be present nearby */
typedef struct mdp_superblock_s {
diff --git a/md_u.h b/md_u.h
index be9868a..76068d6 100644
--- a/md_u.h
+++ b/md_u.h
@@ -44,6 +44,7 @@
#define STOP_ARRAY _IO (MD_MAJOR, 0x32)
#define STOP_ARRAY_RO _IO (MD_MAJOR, 0x33)
#define RESTART_ARRAY_RW _IO (MD_MAJOR, 0x34)
+#define CLUSTERED_DISK_NACK _IO (MD_MAJOR, 0x35)
typedef struct mdu_version_s {
int major;
diff --git a/mdadm.8.in b/mdadm.8.in
index c015cbf..6873cc7 100644
--- a/mdadm.8.in
+++ b/mdadm.8.in
@@ -1405,6 +1405,15 @@ will avoid reading from these devices if possible.
.BR \-\-readwrite
Subsequent devices that are added or re\-added will have the 'write-mostly'
flag cleared.
+.TP
+.BR \-\-cluster\-confirm
+Confirm the existence of the device. This is issued in response to an \-\-add
+request by a node in a cluster. When a node adds a device it sends a message
+to all nodes in the cluster to look for a device with a UUID. This translates
+to a udev notification with the UUID of the device to be added and the slot
+number. The receiving node must acknowledge this message
+with \-\-cluster\-confirm. Valid arguments are <slot>:<devicename> in case
+the device is found or <slot>:missing in case the device is not found.
.P
Each of these options requires that the first device listed is the array
diff --git a/mdadm.c b/mdadm.c
index 6963a09..5b4b3ef 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -196,6 +196,7 @@ int main(int argc, char *argv[])
case 'f':
case Fail:
case ReAdd: /* re-add */
+ case ClusterConfirm:
if (!mode) {
newmode = MANAGE;
shortopt = short_bitmap_options;
@@ -933,6 +934,9 @@ int main(int argc, char *argv[])
* remove the device */
devmode = 'f';
continue;
+ case O(MANAGE, ClusterConfirm):
+ devmode = 'c';
+ continue;
case O(MANAGE,Replace):
/* Mark these devices for replacement */
devmode = 'R';
diff --git a/mdadm.h b/mdadm.h
index f56d9d6..00c726e 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -346,6 +346,7 @@ enum special_options {
Action,
Nodes,
ClusterName,
+ ClusterConfirm,
};
enum prefix_standard {
@@ -1281,6 +1282,7 @@ extern int parse_uuid(char *str, int uuid[4]);
extern int parse_layout_10(char *layout);
extern int parse_layout_faulty(char *layout);
extern long parse_num(char *num);
+extern int parse_cluster_confirm_arg(char *inp, char **devname, int *slot);
extern int check_ext2(int fd, char *name);
extern int check_reiser(int fd, char *name);
extern int check_raid(int fd, char *name);
diff --git a/util.c b/util.c
index ed9a745..1d82fc7 100644
--- a/util.c
+++ b/util.c
@@ -273,6 +273,17 @@ long parse_num(char *num)
}
#endif
+int parse_cluster_confirm_arg(char *input, char **devname, int *slot)
+{
+ char *dev;
+ *slot = strtoul(input, &dev, 10);
+ if (dev[0] == ':')
+ *devname = dev+1;
+ else
+ return -1;
+ return 0;
+}
+
void remove_partitions(int fd)
{
/* remove partitions from this block devices.
--
1.7.12.4
^ permalink raw reply related
* [PATCH 04/10] Show all bitmaps while examining bitmap
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
This adds capability of exmining bitmaps corresponding to all
nodes/slots on the device.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
bitmap.c | 35 ++++++++++++++++++++++++++---------
1 file changed, 26 insertions(+), 9 deletions(-)
diff --git a/bitmap.c b/bitmap.c
index b1d54a6..ab83f4e 100644
--- a/bitmap.c
+++ b/bitmap.c
@@ -258,7 +258,7 @@ int ExamineBitmap(char *filename, int brief, struct supertype *st)
int rv = 1;
char buf[64];
int swap;
- int fd;
+ int fd, i;
__u32 uuid32[4];
fd = bitmap_file_open(filename, &st);
@@ -315,9 +315,6 @@ int ExamineBitmap(char *filename, int brief, struct supertype *st)
uuid32[2],
uuid32[3]);
- printf(" Events : %llu\n", (unsigned long long)sb->events);
- printf(" Events Cleared : %llu\n", (unsigned long long)sb->events_cleared);
- printf(" State : %s\n", bitmap_state(sb->state));
printf(" Chunksize : %s\n", human_chunksize(sb->chunksize));
printf(" Daemon : %ds flush period\n", sb->daemon_sleep);
if (sb->write_behind)
@@ -327,11 +324,31 @@ int ExamineBitmap(char *filename, int brief, struct supertype *st)
printf(" Write Mode : %s\n", buf);
printf(" Sync Size : %llu%s\n", (unsigned long long)sb->sync_size/2,
human_size(sb->sync_size * 512));
- if (brief)
- goto free_info;
- printf(" Bitmap : %llu bits (chunks), %llu dirty (%2.1f%%)\n",
- info->total_bits, info->dirty_bits,
- 100.0 * info->dirty_bits / (info->total_bits?:1));
+ if (sb->nodes > 0) {
+ printf(" Cluster nodes : %d\n", sb->nodes);
+ printf(" Cluster name : %s\n", sb->cluster_name);
+ }
+ i = 0;
+ do {
+ if (i) {
+ free(info);
+ info = bitmap_fd_read(fd, brief);
+ sb = &info->sb;
+ }
+ if (sb->magic != BITMAP_MAGIC) {
+ pr_err("invalid bitmap magic 0x%x, the bitmap file appears to be corrupted\n", sb->magic);
+ }
+ printf(" Node Slot : %d\n", i);
+ printf(" Events : %llu\n", (unsigned long long)sb->events);
+ printf(" Events Cleared : %llu\n", (unsigned long long)sb->events_cleared);
+ printf(" State : %s\n", bitmap_state(sb->state));
+ if (brief)
+ continue;
+ printf(" Bitmap : %llu bits (chunks), %llu dirty (%2.1f%%)\n",
+ info->total_bits, info->dirty_bits,
+ 100.0 * info->dirty_bits / (info->total_bits?:1));
+ } while (++i < (int)sb->nodes);
+
free_info:
free(info);
return rv;
--
1.7.12.4
^ permalink raw reply related
* [PATCH 03/10] Create n bitmaps for clustered mode
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
For a clustered MD, create bitmaps equal to number of nodes so
each node has an independent bitmap.
Only the first bitmap is has the bits set so that the first node
that assembles the device also performs the sync.
The bitmaps are aligned to 4k boundaries.
On-disk format:
0 4k 8k 12k
-------------------------------------------------------------------
| idle | md super | bm super [0] + bits |
| bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
| bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
| bm bits [3, contd] | | |
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Create.c | 3 ++-
bitmap.h | 7 +++++--
mdadm.8.in | 7 ++++++-
mdadm.c | 17 ++++++++++++++++-
super1.c | 59 +++++++++++++++++++++++++++++++++++++++++------------------
5 files changed, 70 insertions(+), 23 deletions(-)
diff --git a/Create.c b/Create.c
index cd5485b..9663dc4 100644
--- a/Create.c
+++ b/Create.c
@@ -752,7 +752,8 @@ int Create(struct supertype *st, char *mddev,
#endif
}
- if (s->bitmap_file && strcmp(s->bitmap_file, "internal")==0) {
+ if (s->bitmap_file && (strcmp(s->bitmap_file, "internal")==0
+ || strcmp(s->bitmap_file, "clustered")==0)) {
if ((vers%100) < 2) {
pr_err("internal bitmaps not supported by this kernel.\n");
goto abort_locked;
diff --git a/bitmap.h b/bitmap.h
index c8725a3..adbf0b4 100644
--- a/bitmap.h
+++ b/bitmap.h
@@ -154,8 +154,11 @@ typedef struct bitmap_super_s {
__u32 chunksize; /* 52 the bitmap chunk size in bytes */
__u32 daemon_sleep; /* 56 seconds between disk flushes */
__u32 write_behind; /* 60 number of outstanding write-behind writes */
-
- __u8 pad[256 - 64]; /* set to zero */
+ __u32 sectors_reserved; /* 64 number of 512-byte sectors that are
+ * reserved for the bitmap. */
+ __u32 nodes; /* 68 the maximum number of nodes in cluster. */
+ __u8 cluster_name[64]; /* 72 cluster name to which this md belongs */
+ __u8 pad[256 - 136]; /* set to zero */
} bitmap_super_t;
/* notes:
diff --git a/mdadm.8.in b/mdadm.8.in
index a0e8288..c015cbf 100644
--- a/mdadm.8.in
+++ b/mdadm.8.in
@@ -700,7 +700,12 @@ and so is replicated on all devices. If the word
.B "none"
is given with
.B \-\-grow
-mode, then any bitmap that is present is removed.
+mode, then any bitmap that is present is removed. If the word
+.B "clustered"
+is given, the array is created for a clustered environment. One bitmap
+is created for each node as defined by the
+.B \-\-nodes
+parameter and are stored internally.
To help catch typing errors, the filename must contain at least one
slash ('/') if it is a real file (not 'internal' or 'none').
diff --git a/mdadm.c b/mdadm.c
index e4f8568..6963a09 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -1111,6 +1111,15 @@ int main(int argc, char *argv[])
s.bitmap_file = optarg;
continue;
}
+ if (strcmp(optarg, "clustered")== 0) {
+ s.bitmap_file = optarg;
+ /* Set the default number of cluster nodes
+ * to 4 if not already set by user
+ */
+ if (c.nodes < 1)
+ c.nodes = 4;
+ continue;
+ }
/* probable typo */
pr_err("bitmap file must contain a '/', or be 'internal', or 'none'\n"
" not '%s'\n", optarg);
@@ -1404,7 +1413,13 @@ int main(int argc, char *argv[])
if (c.delay == 0)
c.delay = DEFAULT_BITMAP_DELAY;
- if (!strncmp(s.bitmap_file, "internal", 9) ||
+ if (!strncmp(s.bitmap_file, "clustered", 9)) {
+ if (s.level != 1) {
+ pr_err("--bitmap=clustered is currently supported with RAID mirror only\n");
+ rv = 1;
+ break;
+ }
+ } else if (!strncmp(s.bitmap_file, "internal", 9) ||
!strncmp(s.bitmap_file,"none", 4)) {
if (c.nodes) {
pr_err("--nodes argument is incompatible with --bitmap=%s.\n",
diff --git a/super1.c b/super1.c
index f0508fe..ac1b011 100644
--- a/super1.c
+++ b/super1.c
@@ -2144,6 +2144,10 @@ add_internal_bitmap1(struct supertype *st,
bms->daemon_sleep = __cpu_to_le32(delay);
bms->sync_size = __cpu_to_le64(size);
bms->write_behind = __cpu_to_le32(write_behind);
+ bms->nodes = __cpu_to_le32(st->nodes);
+ if (st->cluster_name)
+ strncpy((char *)bms->cluster_name,
+ st->cluster_name, strlen(st->cluster_name));
*chunkp = chunk;
return 1;
@@ -2177,6 +2181,7 @@ static int write_bitmap1(struct supertype *st, int fd)
void *buf;
int towrite, n;
struct align_fd afd;
+ unsigned int i;
init_afd(&afd, fd);
@@ -2185,27 +2190,45 @@ static int write_bitmap1(struct supertype *st, int fd)
if (posix_memalign(&buf, 4096, 4096))
return -ENOMEM;
- memset(buf, 0xff, 4096);
- memcpy(buf, (char *)bms, sizeof(bitmap_super_t));
-
- towrite = __le64_to_cpu(bms->sync_size) / (__le32_to_cpu(bms->chunksize)>>9);
- towrite = (towrite+7) >> 3; /* bits to bytes */
- towrite += sizeof(bitmap_super_t);
- towrite = ROUND_UP(towrite, 512);
- while (towrite > 0) {
- n = towrite;
- if (n > 4096)
- n = 4096;
- n = awrite(&afd, buf, n);
- if (n > 0)
- towrite -= n;
+ /* We use bms->nodes as opposed to st->nodes to
+ * be compatible with write-after-reads such as
+ * the GROW operation.
+ */
+ for (i = 0; i < __le32_to_cpu(bms->nodes); i++) {
+ /* Only the first bitmap should resync
+ * the whole device
+ */
+ if (i)
+ memset(buf, 0x00, 4096);
else
+ memset(buf, 0xff, 4096);
+ memcpy(buf, (char *)bms, sizeof(bitmap_super_t));
+
+ towrite = __le64_to_cpu(bms->sync_size) / (__le32_to_cpu(bms->chunksize)>>9);
+ towrite = (towrite+7) >> 3; /* bits to bytes */
+ towrite += sizeof(bitmap_super_t);
+ /* we need the bitmaps to be at 4k boundary */
+ towrite = ROUND_UP(towrite, 4096);
+ while (towrite > 0) {
+ n = towrite;
+ if (n > 4096)
+ n = 4096;
+ n = awrite(&afd, buf, n);
+ if (n > 0)
+ towrite -= n;
+ else
+ break;
+ if (i)
+ memset(buf, 0x00, 4096);
+ else
+ memset(buf, 0xff, 4096);
+ }
+ fsync(fd);
+ if (towrite) {
+ rv = -2;
break;
- memset(buf, 0xff, 4096);
+ }
}
- fsync(fd);
- if (towrite)
- rv = -2;
free(buf);
return rv;
--
1.7.12.4
^ permalink raw reply related
* [PATCH 02/10] home-cluster while creating an array
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
The home-cluster is stored in the bitmap super block of the
array. The device can be assembled on a cluster with the
cluster name same as the one recorded in the bitmap.
If home-cluster is not specified, this is auto-detected using
dlopen corosync cmap library.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Create.c | 1 +
Makefile | 1 +
ReadMe.c | 1 +
config.c | 27 ++++++++++++++++++++++++++-
mdadm.8.in | 6 ++++++
mdadm.c | 25 +++++++++++++++++++++++++
mdadm.h | 5 +++++
util.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
8 files changed, 115 insertions(+), 1 deletion(-)
diff --git a/Create.c b/Create.c
index b73f6cb..cd5485b 100644
--- a/Create.c
+++ b/Create.c
@@ -532,6 +532,7 @@ int Create(struct supertype *st, char *mddev,
warn = 1;
}
st->nodes = c->nodes;
+ st->cluster_name = c->homecluster;
if (warn) {
if (c->runstop!= 1) {
diff --git a/Makefile b/Makefile
index a7d8c5c..431f08b 100644
--- a/Makefile
+++ b/Makefile
@@ -101,6 +101,7 @@ endif
# If you want a static binary, you might uncomment these
# LDFLAGS = -static
# STRIP = -s
+LDLIBS=-ldl
INSTALL = /usr/bin/install
DESTDIR =
diff --git a/ReadMe.c b/ReadMe.c
index 30c569d..c6286ae 100644
--- a/ReadMe.c
+++ b/ReadMe.c
@@ -141,6 +141,7 @@ struct option long_options[] = {
{"symlinks", 1, 0, Symlinks},
{"data-offset",1, 0, DataOffset},
{"nodes",1, 0, Nodes},
+ {"home-cluster",1, 0, ClusterName},
/* For assemble */
{"uuid", 1, 0, 'u'},
diff --git a/config.c b/config.c
index 7342c42..21b6afd 100644
--- a/config.c
+++ b/config.c
@@ -77,7 +77,7 @@ char DefaultAltConfFile[] = CONFFILE2;
char DefaultAltConfDir[] = CONFFILE2 ".d";
enum linetype { Devices, Array, Mailaddr, Mailfrom, Program, CreateDev,
- Homehost, AutoMode, Policy, PartPolicy, LTEnd };
+ Homehost, HomeCluster, AutoMode, Policy, PartPolicy, LTEnd };
char *keywords[] = {
[Devices] = "devices",
[Array] = "array",
@@ -86,6 +86,7 @@ char *keywords[] = {
[Program] = "program",
[CreateDev]= "create",
[Homehost] = "homehost",
+ [HomeCluster] = "homecluster",
[AutoMode] = "auto",
[Policy] = "policy",
[PartPolicy]="part-policy",
@@ -562,6 +563,21 @@ void homehostline(char *line)
}
}
+static char *home_cluster = NULL;
+void homeclusterline(char *line)
+{
+ char *w;
+
+ for (w=dl_next(line); w != line ; w=dl_next(w)) {
+ if (home_cluster == NULL) {
+ if (strcasecmp(w, "<none>")==0)
+ home_cluster = xstrdup("");
+ else
+ home_cluster = xstrdup(w);
+ }
+ }
+}
+
char auto_yes[] = "yes";
char auto_no[] = "no";
char auto_homehost[] = "homehost";
@@ -724,6 +740,9 @@ void conf_file(FILE *f)
case Homehost:
homehostline(line);
break;
+ case HomeCluster:
+ homeclusterline(line);
+ break;
case AutoMode:
autoline(line);
break;
@@ -884,6 +903,12 @@ char *conf_get_homehost(int *require_homehostp)
return home_host;
}
+char *conf_get_homecluster(void)
+{
+ load_conffile();
+ return home_cluster;
+}
+
struct createinfo *conf_get_create_info(void)
{
load_conffile();
diff --git a/mdadm.8.in b/mdadm.8.in
index bd8d59e..a0e8288 100644
--- a/mdadm.8.in
+++ b/mdadm.8.in
@@ -415,6 +415,12 @@ This functionality is currently only provided by
and
.BR \-\-monitor .
+.TP
+.B \-\-home\-cluster=
+specifies the cluster name for the md device. The md device can be assembled
+only on the cluster which matches the name specified. If this option is not
+provided, mdadm tried to detect the cluster name automatically.
+
.SH For create, build, or grow:
.TP
diff --git a/mdadm.c b/mdadm.c
index bce6a76..e4f8568 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -596,6 +596,13 @@ int main(int argc, char *argv[])
exit(2);
}
continue;
+ case O(CREATE, ClusterName):
+ c.homecluster = optarg;
+ if (strlen(c.homecluster) > 64) {
+ pr_err("Cluster name too big.\n");
+ exit(ERANGE);
+ }
+ continue;
case O(CREATE,'x'): /* number of spare (eXtra) disks */
if (s.sparedisks) {
pr_err("spare-devices set twice: %d and %s\n",
@@ -1267,6 +1274,18 @@ int main(int argc, char *argv[])
c.require_homehost = 0;
}
+ if (c.homecluster == NULL && (c.nodes > 0)) {
+ c.homecluster = conf_get_homecluster();
+ if (c.homecluster == NULL)
+ rv = get_cluster_name(&c.homecluster);
+ if (rv == 0) {
+ c.homehost = xstrdup(c.homecluster);
+ /* Add a : to differentiate between a host
+ * and a cluster */
+ strcat(c.homehost, ":");
+ }
+ }
+
if (c.backup_file && data_offset != INVALID_SECTORS) {
pr_err("--backup-file and --data-offset are incompatible\n");
exit(2);
@@ -1393,6 +1412,12 @@ int main(int argc, char *argv[])
rv = 1;
break;
}
+ if (c.homecluster) {
+ pr_err("--home-cluster argument is incompatible with --bitmap=%s.\n",
+ s.bitmap_file);
+ rv = 1;
+ break;
+ }
}
if (s.write_behind && !s.bitmap_file) {
diff --git a/mdadm.h b/mdadm.h
index 9d55801..f56d9d6 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -345,6 +345,7 @@ enum special_options {
Restore,
Action,
Nodes,
+ ClusterName,
};
enum prefix_standard {
@@ -420,6 +421,7 @@ struct context {
int invalid_backup;
char *action;
int nodes;
+ char *homecluster;
};
struct shape {
@@ -1032,6 +1034,7 @@ struct supertype {
int devcnt;
int retry_soon;
int nodes;
+ char *cluster_name;
struct mdinfo *devs;
@@ -1308,6 +1311,7 @@ extern char *conf_get_mailaddr(void);
extern char *conf_get_mailfrom(void);
extern char *conf_get_program(void);
extern char *conf_get_homehost(int *require_homehostp);
+extern char *conf_get_homecluster(void);
extern char *conf_line(FILE *file);
extern char *conf_word(FILE *file, int allow_key);
extern void print_quoted(char *str);
@@ -1416,6 +1420,7 @@ extern char *stat2devnm(struct stat *st);
extern char *fd2devnm(int fd);
extern int in_initrd(void);
+extern int get_cluster_name(char **name);
#define _ROUND_UP(val, base) (((val) + (base) - 1) & ~(base - 1))
#define ROUND_UP(val, base) _ROUND_UP(val, (typeof(val))(base))
diff --git a/util.c b/util.c
index cc98d3b..ed9a745 100644
--- a/util.c
+++ b/util.c
@@ -34,6 +34,8 @@
#include <ctype.h>
#include <dirent.h>
#include <signal.h>
+#include <dlfcn.h>
+#include <corosync/cmap.h>
/*
* following taken from linux/blkpg.h because they aren't
@@ -1976,3 +1978,51 @@ void reopen_mddev(int mdfd)
if (fd >= 0 && fd != mdfd)
dup2(fd, mdfd);
}
+
+int get_cluster_name(char **cluster_name)
+{
+ void *lib_handle = NULL;
+ int rv = -1;
+
+ cmap_handle_t handle;
+ static int (*initialize)(cmap_handle_t *handle);
+ static int (*get_string)(cmap_handle_t handle,
+ const char *string,
+ char **name);
+ static int (*finalize)(cmap_handle_t handle);
+
+
+ lib_handle = dlopen("libcmap.so.4", RTLD_NOW | RTLD_LOCAL);
+ if (!lib_handle)
+ return rv;
+
+ initialize = dlsym(lib_handle, "cmap_initialize");
+ if (!initialize)
+ goto out;
+
+ get_string = dlsym(lib_handle, "cmap_get_string");
+ if (!get_string)
+ goto out;
+
+ finalize = dlsym(lib_handle, "cmap_finalize");
+ if (!finalize)
+ goto out;
+
+ rv = initialize(&handle);
+ if (rv != CS_OK)
+ goto out;
+
+ rv = get_string(handle, "totem.cluster_name", cluster_name);
+ if (rv != CS_OK) {
+ free(*cluster_name);
+ rv = -1;
+ goto name_err;
+ }
+
+ rv = 0;
+name_err:
+ finalize(handle);
+out:
+ dlclose(lib_handle);
+ return rv;
+}
--
1.7.12.4
^ permalink raw reply related
* [PATCH 01/10] Add nodes option while creating md
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
In-Reply-To: <1429860641-5839-1-git-send-email-gqjiang@suse.com>
From: Guoqing Jiang <gqjiang@suse.com>
Specifies the maximum number of nodes in the cluster that may use
this device simultaneously. This is equivalent to the number of
bitmaps created in the internal superblock (patches to follow).
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Create.c | 1 +
ReadMe.c | 1 +
mdadm.8.in | 5 +++++
mdadm.c | 20 +++++++++++++++++++-
mdadm.h | 3 +++
5 files changed, 29 insertions(+), 1 deletion(-)
diff --git a/Create.c b/Create.c
index ef28da0..b73f6cb 100644
--- a/Create.c
+++ b/Create.c
@@ -531,6 +531,7 @@ int Create(struct supertype *st, char *mddev,
st->ss->name);
warn = 1;
}
+ st->nodes = c->nodes;
if (warn) {
if (c->runstop!= 1) {
diff --git a/ReadMe.c b/ReadMe.c
index 87a4916..30c569d 100644
--- a/ReadMe.c
+++ b/ReadMe.c
@@ -140,6 +140,7 @@ struct option long_options[] = {
{"homehost", 1, 0, HomeHost},
{"symlinks", 1, 0, Symlinks},
{"data-offset",1, 0, DataOffset},
+ {"nodes",1, 0, Nodes},
/* For assemble */
{"uuid", 1, 0, 'u'},
diff --git a/mdadm.8.in b/mdadm.8.in
index a630310..bd8d59e 100644
--- a/mdadm.8.in
+++ b/mdadm.8.in
@@ -966,6 +966,11 @@ However for RAID0, it is not possible to add spares. So to increase
the number of devices in a RAID0, it is necessary to set the new
number of devices, and to add the new devices, in the same command.
+.TP
+.BR \-\-nodes
+Specify the maximum number of nodes in the cluster that will use this
+device simultaneously. If not specified, this defaults to 4.
+
.SH For assemble:
.TP
diff --git a/mdadm.c b/mdadm.c
index 3e8c49b..bce6a76 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -588,7 +588,14 @@ int main(int argc, char *argv[])
}
ident.raid_disks = s.raiddisks;
continue;
-
+ case O(CREATE, Nodes):
+ c.nodes = parse_num(optarg);
+ if (c.nodes <= 0) {
+ pr_err("invalid number for the number of "
+ "cluster nodes: %s\n", optarg);
+ exit(2);
+ }
+ continue;
case O(CREATE,'x'): /* number of spare (eXtra) disks */
if (s.sparedisks) {
pr_err("spare-devices set twice: %d and %s\n",
@@ -1377,6 +1384,17 @@ int main(int argc, char *argv[])
case CREATE:
if (c.delay == 0)
c.delay = DEFAULT_BITMAP_DELAY;
+
+ if (!strncmp(s.bitmap_file, "internal", 9) ||
+ !strncmp(s.bitmap_file,"none", 4)) {
+ if (c.nodes) {
+ pr_err("--nodes argument is incompatible with --bitmap=%s.\n",
+ s.bitmap_file);
+ rv = 1;
+ break;
+ }
+ }
+
if (s.write_behind && !s.bitmap_file) {
pr_err("write-behind mode requires a bitmap.\n");
rv = 1;
diff --git a/mdadm.h b/mdadm.h
index 141f963..9d55801 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -344,6 +344,7 @@ enum special_options {
Dump,
Restore,
Action,
+ Nodes,
};
enum prefix_standard {
@@ -418,6 +419,7 @@ struct context {
char *backup_file;
int invalid_backup;
char *action;
+ int nodes;
};
struct shape {
@@ -1029,6 +1031,7 @@ struct supertype {
*/
int devcnt;
int retry_soon;
+ int nodes;
struct mdinfo *devs;
--
1.7.12.4
^ permalink raw reply related
* [PATCH 00/10] mdadm tool: add the support for cluster-md
From: gqjiang @ 2015-04-24 7:30 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, rgoldwyn
From: Guoqing Jiang <gqjiang@suse.com>
Basic background for Cluster MD: Cluster MD is a shared-device RAID for a
cluster, currently, the implementation is limited to RAID1 but with further
work (and some positive feedback), it could be extend to other RAID levels.
The kernel part code of cluster-md has been sent to maillist several month
ago by Goldywyn, and to make cluster-md works, the mdadm tools also need to
do some changes accordingly.
This patch set extends mdadm tool to aware cluster MD scenario, and handle
related cluster-md scenario.
1. the first part (0001-0007) comes from Goldwyn, which add initial
support for cluster-md, those changes included make mdadm awares nodes,
home-cluster and n bitmaps for clustered mode, also let mdadm can
confirm disk which is added by another node.
2. the second part is for support change cluster-name and node nums under
assemble mode. Which extend write-bitmap to handle above cases, and also
use the extended write_bitmap for update uuid. [PATCH 10/10] is just compiled
test only.
BTW: this series is based on commit "72a457 IMSM: Count arrays per orom".
Some reltated links:
[1] http://marc.info/?l=linux-raid&m=141891941330336&w=2
[2] http://marc.info/?l=linux-raid&m=141935561418770&w=2
Guoqing Jiang (10):
Add nodes option while creating md
home-cluster while creating an array
Create n bitmaps for clustered mode
Show all bitmaps while examining bitmap
Add a new clustered disk
Convert a bitmap=none device to clustered
Skip clustered devices in incremental
mdadm: add the ability to change cluster name
mdadm: change the num of cluster node
Reuse the write_bitmap for update uuid
Assemble.c | 14 +++++--
Create.c | 5 ++-
Grow.c | 22 ++++++++---
Incremental.c | 5 +++
Makefile | 1 +
Manage.c | 33 +++++++++++++++--
ReadMe.c | 3 ++
bitmap.c | 55 +++++++++++++--------------
bitmap.h | 7 +++-
config.c | 27 +++++++++++++-
md_p.h | 7 ++++
md_u.h | 1 +
mdadm.8.in | 27 +++++++++++++-
mdadm.c | 70 ++++++++++++++++++++++++++++++++++-
mdadm.h | 20 +++++++++-
super0.c | 4 +-
super1.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++----------
util.c | 61 ++++++++++++++++++++++++++++++
18 files changed, 408 insertions(+), 71 deletions(-)
--
1.7.12.4
^ permalink raw reply
* Re: [PATCH v3 2/4] md/raid10: make sync_request_write() call bio_copy_data()
From: Christoph Hellwig @ 2015-04-24 7:08 UTC (permalink / raw)
To: Ming Lin
Cc: linux-kernel, Christoph Hellwig, Kent Overstreet, Jens Axboe,
Dongsu Park, Neil Brown, linux-raid
In-Reply-To: <1429830275-6792-3-git-send-email-mlin@kernel.org>
On Thu, Apr 23, 2015 at 04:04:33PM -0700, Ming Lin wrote:
> From: Kent Overstreet <kent.overstreet@gmail.com>
>
> Refactor sync_request_write() of md/raid10 to use bio_copy_data()
> instead of open coding bio_vec iterations.
Looks good,
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: Bisected, with rfc/patch - was Re: BUG: unable to handle kernel NULL pointer dereference at sysfs_do_create_link_sd (after mdadm)
From: NeilBrown @ 2015-04-24 2:09 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Azat Khuzhin, Kernel.org-Linux-RAID, Guoqing Jiang, Tejun Heo,
Jan Kara, Jens Axboe, dm-devel
In-Reply-To: <20150423161051.GA18971@lst.de>
[-- Attachment #1: Type: text/plain, Size: 3201 bytes --]
On Thu, 23 Apr 2015 18:10:51 +0200 Christoph Hellwig <hch@lst.de> wrote:
> On Thu, Apr 23, 2015 at 06:03:14PM +1000, NeilBrown wrote:
> > On Thu, 23 Apr 2015 09:37:24 +0200 Christoph Hellwig <hch@lst.de> wrote:
> >
> > > Plase fix your device name lifetimes.
> >
> > Any chance you could be more explicit?
> >
> > The commit you identified doesn't seem to help much - md and dm are quite
> > different in this area.
> >
> > It seems that it is no longer safe to call 'add_disk' between calling
> > 'del_gendisk' and bdi_destroy being called. How can I find out if I am in
> > that window, or wait for bdi_destroy to be called?
>
> The bdi is only around if the device is open, either through a device
> node, or through a blkdev_get from a file system. If you get duplicate
> names that means you're trying to allocate a new gendisk while the old
> one is still around.
>
> In theory you're fine once the device gets ->release called.
In practice .... the put_disk() shortly after the ->release call in
__blkdev_put() is what ultimately releases the name of the bdi - at least in
the cases where I get a crash.
>
> Except that we can hold sysfs reference to the qeue, eww. So for now
> try to follow the dm model, but I'll need to add a callback to the
> queue called once the request_queue actually is released for this.
I'm pretty sure that the md code is already as close to the "dm model" as it
meaningfully can be.
If I move bdi_destroy out of blk_release_queue (which really think is too
later) and place it in blk_cleanup_queue (which seems a credible place for
it), and then move the blk_cleanup_queue call in md_free up before the
del_gendisk() call (which is probably the right thing to do anyway, though dm
has the same order that md currently has) then I don't get any crashes and
I'm almost convince it is correct...
Thoughts?
Thanks,
NeilBrown
diff --git a/block/blk-core.c b/block/blk-core.c
index 794c3e7f01cf..66406474f0c4 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -552,6 +552,8 @@ void blk_cleanup_queue(struct request_queue *q)
q->queue_lock = &q->__queue_lock;
spin_unlock_irq(lock);
+ bdi_destroy(&q->backing_dev_info);
+
/* @q is and will stay empty, shutdown and put */
blk_put_queue(q);
}
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index faaf36ade7eb..2b8fd302f677 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -522,8 +522,6 @@ static void blk_release_queue(struct kobject *kobj)
blk_trace_shutdown(q);
- bdi_destroy(&q->backing_dev_info);
-
ida_simple_remove(&blk_queue_ida, q->id);
call_rcu(&q->rcu_head, blk_free_queue_rcu);
}
diff --git a/drivers/md/md.c b/drivers/md/md.c
index d4f31e195e26..593a02476c78 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -4818,12 +4818,12 @@ static void md_free(struct kobject *ko)
if (mddev->sysfs_state)
sysfs_put(mddev->sysfs_state);
+ if (mddev->queue)
+ blk_cleanup_queue(mddev->queue);
if (mddev->gendisk) {
del_gendisk(mddev->gendisk);
put_disk(mddev->gendisk);
}
- if (mddev->queue)
- blk_cleanup_queue(mddev->queue);
kfree(mddev);
}
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: PROBLEM: write to jbod with 3TB and 160GB drives hits BUG/oops
From: NeilBrown @ 2015-04-24 1:55 UTC (permalink / raw)
To: Charles Bertsch; +Cc: linux-raid, BertschC@acm.org
In-Reply-To: <55398462.1000202@cox.net>
[-- Attachment #1: Type: text/plain, Size: 971 bytes --]
On Thu, 23 Apr 2015 16:46:42 -0700 Charles Bertsch <cbertsch@cox.net> wrote:
> 1. PROBLEM: write to jbod with 3TB and 160GB drives hits BUG/oops
>
> 2. Description. Using linux-3.18.5, linux-3.18.7, and linux-4.0 a
> two-drive jbod created using one 3TB drive and one 160GB drive will
> SOMETIMES hit a BUG call at linux-4.0/fs/buffer.c:3220
By "jbod" I assume you mean "linear array".
You say this happens without any filesystem on the array, yet the stack
traces clearly show ext2 in use.
Maybe some weird interaction is happening between the the filesystem and the
linear array.
But please confirm that the stack trace happened when there was no filesystem
on the array you were testing, and report what filesystems you do have which
use ext2.
Is there any chance you could use "git bisect" to find out exactly which
commit introduced the problem? That is the mostly likely path to a solution.
Thanks for the report,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox