[PATCH md 002 of 6] Don't allow new md/bitmap file to be set if one already exists

[PATCH md 002 of 6] Don't allow new md/bitmap file to be set if one already exists

[NeilBrown]

... otherwise we loose a reference and can never free the file.

Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>

### Diffstat output
 ./drivers/md/md.c |    2 +-
 1 files changed, 1 insertion(+), 1 deletion(-)

diff ./drivers/md/md.c~current~ ./drivers/md/md.c
--- ./drivers/md/md.c~current~	2005-08-09 10:22:01.000000000 +1000
+++ ./drivers/md/md.c	2005-08-09 10:25:22.000000000 +1000
@@ -2430,7 +2430,7 @@ static int set_bitmap_file(mddev_t *mdde
 {
 	int err;
 
-	if (mddev->pers)
+	if (mddev->pers || mddev->bitmap_file)
 		return -EBUSY;
 
 	mddev->bitmap_file = fget(fd);

[PATCH md 001 of 6] Make sure mddev->bitmap_offset gets cleared between array instantiations.

[NeilBrown]

... otherwise we might try to load a bitmap from an array which hasn't one.

Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>

### Diffstat output
 ./drivers/md/md.c |    3 +++
 1 files changed, 3 insertions(+)

diff ./drivers/md/md.c~current~ ./drivers/md/md.c
--- ./drivers/md/md.c~current~	2005-08-09 10:22:01.000000000 +1000
+++ ./drivers/md/md.c	2005-08-09 10:22:01.000000000 +1000
@@ -623,6 +623,7 @@ static int super_90_validate(mddev_t *md
 		mddev->raid_disks = sb->raid_disks;
 		mddev->size = sb->size;
 		mddev->events = md_event(sb);
+		mddev->bitmap_offset = 0;
 
 		if (sb->state & (1<<MD_SB_CLEAN))
 			mddev->recovery_cp = MaxSector;
@@ -938,6 +939,7 @@ static int super_1_validate(mddev_t *mdd
 		mddev->raid_disks = le32_to_cpu(sb->raid_disks);
 		mddev->size = le64_to_cpu(sb->size)/2;
 		mddev->events = le64_to_cpu(sb->events);
+		mddev->bitmap_offset = 0;
 		
 		mddev->recovery_cp = le64_to_cpu(sb->resync_offset);
 		memcpy(mddev->uuid, sb->set_uuid, 16);
@@ -1824,6 +1826,7 @@ static int do_md_stop(mddev_t * mddev, i
 		fput(mddev->bitmap_file);
 		mddev->bitmap_file = NULL;
 	}
+	mddev->bitmap_offset = 0;
 
 	/*
 	 * Free resources if final stop

[PATCH md 003 of 6] Improve handling of bitmap initialisation.

[NeilBrown]

When we find a 'stale' bitmap, possibly because it is new,
we should just assume every bit needs to be set, but rather
base the setting of bits on the current state of the array
(degraded and recovery_cp).

Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>

### Diffstat output
 ./drivers/md/bitmap.c |   28 +++++++++++++++++++++-------
 1 files changed, 21 insertions(+), 7 deletions(-)

diff ./drivers/md/bitmap.c~current~ ./drivers/md/bitmap.c
--- ./drivers/md/bitmap.c~current~	2005-08-11 15:12:54.000000000 +1000
+++ ./drivers/md/bitmap.c	2005-08-11 15:12:54.000000000 +1000
@@ -520,6 +520,8 @@ success:
 	bitmap->daemon_sleep = daemon_sleep;
 	bitmap->flags |= sb->state;
 	bitmap->events_cleared = le64_to_cpu(sb->events_cleared);
+	if (sb->state & BITMAP_STALE)
+		bitmap->events_cleared = bitmap->mddev->events;
 	err = 0;
 out:
 	kunmap(bitmap->sb_page);
@@ -818,7 +820,7 @@ int bitmap_unplug(struct bitmap *bitmap)
 	return 0;
 }
 
-static void bitmap_set_memory_bits(struct bitmap *bitmap, sector_t offset);
+static void bitmap_set_memory_bits(struct bitmap *bitmap, sector_t offset, int needed);
 /* * bitmap_init_from_disk -- called at bitmap_create time to initialize
  * the in-memory bitmap from the on-disk bitmap -- also, sets up the
  * memory mapping of the bitmap file
@@ -826,8 +828,11 @@ static void bitmap_set_memory_bits(struc
  *   if there's no bitmap file, or if the bitmap file had been
  *   previously kicked from the array, we mark all the bits as
  *   1's in order to cause a full resync.
+ *
+ * We ignore all bits for sectors that end earlier than 'start'.
+ * This is used when reading an out-of-date bitmap...
  */
-static int bitmap_init_from_disk(struct bitmap *bitmap)
+static int bitmap_init_from_disk(struct bitmap *bitmap, sector_t start)
 {
 	unsigned long i, chunks, index, oldindex, bit;
 	struct page *page = NULL, *oldpage = NULL;
@@ -914,7 +919,7 @@ static int bitmap_init_from_disk(struct 
 			 	 * whole page and write it out
 				 */
 				memset(page_address(page) + offset, 0xff,
-					PAGE_SIZE - offset);
+				       PAGE_SIZE - offset);
 				ret = write_page(bitmap, page, 1);
 				if (ret) {
 					kunmap(page);
@@ -928,8 +933,11 @@ static int bitmap_init_from_disk(struct 
 		}
 		if (test_bit(bit, page_address(page))) {
 			/* if the disk bit is set, set the memory bit */
-			bitmap_set_memory_bits(bitmap, i << CHUNK_BLOCK_SHIFT(bitmap));
+			bitmap_set_memory_bits(bitmap, i << CHUNK_BLOCK_SHIFT(bitmap),
+					       ((i+1) << (CHUNK_BLOCK_SHIFT(bitmap)) >= start)
+				);
 			bit_cnt++;
+			set_page_attr(bitmap, page, BITMAP_PAGE_CLEAN);
 		}
 	}
 
@@ -1424,7 +1432,7 @@ void bitmap_close_sync(struct bitmap *bi
 	}
 }
 
-static void bitmap_set_memory_bits(struct bitmap *bitmap, sector_t offset)
+static void bitmap_set_memory_bits(struct bitmap *bitmap, sector_t offset, int needed)
 {
 	/* For each chunk covered by any of these sectors, set the
 	 * counter to 1 and set resync_needed.  They should all
@@ -1441,7 +1449,7 @@ static void bitmap_set_memory_bits(struc
 	}
 	if (! *bmc) {
 		struct page *page;
-		*bmc = 1 | NEEDED_MASK;
+		*bmc = 1 | (needed?NEEDED_MASK:0);
 		bitmap_count_page(bitmap, offset, 1);
 		page = filemap_get_page(bitmap, offset >> CHUNK_BLOCK_SHIFT(bitmap));
 		set_page_attr(bitmap, page, BITMAP_PAGE_CLEAN);
@@ -1517,6 +1525,7 @@ int bitmap_create(mddev_t *mddev)
 	unsigned long pages;
 	struct file *file = mddev->bitmap_file;
 	int err;
+	sector_t start;
 
 	BUG_ON(sizeof(bitmap_super_t) != 256);
 
@@ -1581,7 +1590,12 @@ int bitmap_create(mddev_t *mddev)
 
 	/* now that we have some pages available, initialize the in-memory
 	 * bitmap from the on-disk bitmap */
-	err = bitmap_init_from_disk(bitmap);
+	start = 0;
+	if (mddev->degraded == 0
+	    || bitmap->events_cleared == mddev->events)
+		/* no need to keep dirty bits to optimise a re-add of a missing device */
+		start = mddev->recovery_cp;
+	err = bitmap_init_from_disk(bitmap, start);
 
 	if (err)
 		return err;

[PATCH md 000 of 6] Introduction

[NeilBrown]

The following 6 patches for md mostly add some new functionality (hot-adding 
write-intent bitmaps and write-behind support for raid1).
They can, and should, wait for after 2.6.13 is released.

The first patch fixes a bug which could be annoying, and is probably
suitable for 2.6.13, however I suspect it is too late for 2.6.13, and
it is ok to be left.

The bug is that if you create an array with an internal bitmap, shut
it down, and then create an array with the same md device, the md
drive will assume it should have a bitmap too.  As the array can be
created with a different md device, it is mostly an inconvenience.
I'm pretty sure there is no risk of data corruption.

NeilBrown


 [PATCH md 001 of 6] Make sure mddev->bitmap_offset gets cleared between array instantiations.
 [PATCH md 002 of 6] Don't allow new md/bitmap file to be set if one already exists
 [PATCH md 003 of 6] Improve handling of bitmap initialisation.
 [PATCH md 004 of 6] All hot-add and hot-remove of md intent logging bitmaps
 [PATCH md 005 of 6] Support write-mostly device in raid1
 [PATCH md 006 of 6] Add write-behind support for md/raid1

[PATCH md 005 of 6] Support write-mostly device in raid1

[NeilBrown]

This allows a device in a raid1 to be marked as "write mostly".
Read requests will only be sent if there is no other option.

Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>

### Diffstat output
 ./drivers/md/md.c           |   18 ++++++++++
 ./drivers/md/raid1.c        |   76 ++++++++++++++++++++++++++++++--------------
 ./include/linux/raid/md_k.h |    3 +
 ./include/linux/raid/md_p.h |   11 +++++-
 4 files changed, 82 insertions(+), 26 deletions(-)

diff ./drivers/md/md.c~current~ ./drivers/md/md.c
--- ./drivers/md/md.c~current~	2005-08-11 15:35:33.000000000 +1000
+++ ./drivers/md/md.c	2005-08-11 16:03:03.000000000 +1000
@@ -671,6 +671,7 @@ static int super_90_validate(mddev_t *md
 
 	if (mddev->level != LEVEL_MULTIPATH) {
 		rdev->faulty = 0;
+		rdev->flags = 0;
 		desc = sb->disks + rdev->desc_nr;
 
 		if (desc->state & (1<<MD_DISK_FAULTY))
@@ -680,6 +681,8 @@ static int super_90_validate(mddev_t *md
 			rdev->in_sync = 1;
 			rdev->raid_disk = desc->raid_disk;
 		}
+		if (desc->state & (1<<MD_DISK_WRITEMOSTLY))
+			set_bit(WriteMostly, &rdev->flags);
 	} else /* MULTIPATH are always insync */
 		rdev->in_sync = 1;
 	return 0;
@@ -778,6 +781,8 @@ static void super_90_sync(mddev_t *mddev
 			spare++;
 			working++;
 		}
+		if (test_bit(WriteMostly, &rdev2->flags))
+			d->state |= (1<<MD_DISK_WRITEMOSTLY);
 	}
 	
 	/* now set the "removed" and "faulty" bits on any missing devices */
@@ -991,6 +996,9 @@ static int super_1_validate(mddev_t *mdd
 			rdev->raid_disk = role;
 			break;
 		}
+		rdev->flags = 0;
+		if (sb->devflags & WriteMostly1)
+			set_bit(WriteMostly, &rdev->flags);
 	} else /* MULTIPATH are always insync */
 		rdev->in_sync = 1;
 
@@ -2151,6 +2159,8 @@ static int get_disk_info(mddev_t * mddev
 			info.state |= (1<<MD_DISK_ACTIVE);
 			info.state |= (1<<MD_DISK_SYNC);
 		}
+		if (test_bit(WriteMostly, &rdev->flags))
+			info.state |= (1<<MD_DISK_WRITEMOSTLY);
 	} else {
 		info.major = info.minor = 0;
 		info.raid_disk = -1;
@@ -2236,6 +2246,9 @@ static int add_new_disk(mddev_t * mddev,
 		rdev->saved_raid_disk = rdev->raid_disk;
 
 		rdev->in_sync = 0; /* just to be sure */
+		if (info->state & (1<<MD_DISK_WRITEMOSTLY))
+			set_bit(WriteMostly, &rdev->flags);
+
 		rdev->raid_disk = -1;
 		err = bind_rdev_to_array(rdev, mddev);
 		if (err)
@@ -2277,6 +2290,9 @@ static int add_new_disk(mddev_t * mddev,
 		else
 			rdev->in_sync = 0;
 
+		if (info->state & (1<<MD_DISK_WRITEMOSTLY))
+			set_bit(WriteMostly, &rdev->flags);
+
 		err = bind_rdev_to_array(rdev, mddev);
 		if (err) {
 			export_rdev(rdev);
@@ -3329,6 +3345,8 @@ static int md_seq_show(struct seq_file *
 			char b[BDEVNAME_SIZE];
 			seq_printf(seq, " %s[%d]",
 				bdevname(rdev->bdev,b), rdev->desc_nr);
+			if (test_bit(WriteMostly, &rdev->flags))
+				seq_printf(seq, "(W)");
 			if (rdev->faulty) {
 				seq_printf(seq, "(F)");
 				continue;

diff ./drivers/md/raid1.c~current~ ./drivers/md/raid1.c
--- ./drivers/md/raid1.c~current~	2005-08-11 15:14:05.000000000 +1000
+++ ./drivers/md/raid1.c	2005-08-11 16:03:03.000000000 +1000
@@ -360,13 +360,14 @@ static int read_balance(conf_t *conf, r1
 {
 	const unsigned long this_sector = r1_bio->sector;
 	int new_disk = conf->last_used, disk = new_disk;
+	int wonly_disk = -1;
 	const int sectors = r1_bio->sectors;
 	sector_t new_distance, current_distance;
-	mdk_rdev_t *new_rdev, *rdev;
+	mdk_rdev_t *rdev;
 
 	rcu_read_lock();
 	/*
-	 * Check if it if we can balance. We can balance on the whole
+	 * Check if we can balance. We can balance on the whole
 	 * device if no resync is going on, or below the resync window.
 	 * We take the first readable disk when above the resync window.
 	 */
@@ -376,11 +377,16 @@ static int read_balance(conf_t *conf, r1
 		/* Choose the first operation device, for consistancy */
 		new_disk = 0;
 
-		while ((new_rdev=conf->mirrors[new_disk].rdev) == NULL ||
-		       !new_rdev->in_sync) {
-			new_disk++;
-			if (new_disk == conf->raid_disks) {
-				new_disk = -1;
+		for (rdev = conf->mirrors[new_disk].rdev;
+		     !rdev || !rdev->in_sync
+			     || test_bit(WriteMostly, &rdev->flags);
+		     rdev = conf->mirrors[++new_disk].rdev) {
+		     
+			if (rdev && rdev->in_sync)
+				wonly_disk = new_disk;
+
+			if (new_disk == conf->raid_disks - 1) {
+				new_disk = wonly_disk;
 				break;
 			}
 		}
@@ -389,16 +395,26 @@ static int read_balance(conf_t *conf, r1
 
 
 	/* make sure the disk is operational */
-	while ((new_rdev=conf->mirrors[new_disk].rdev) == NULL ||
-	       !new_rdev->in_sync) {
+	for (rdev = conf->mirrors[new_disk].rdev;
+	     !rdev || !rdev->in_sync ||
+		     test_bit(WriteMostly, &rdev->flags);
+	     rdev = conf->mirrors[new_disk].rdev) {
+
+		if (rdev && rdev->in_sync)
+			wonly_disk = new_disk;
+
 		if (new_disk <= 0)
 			new_disk = conf->raid_disks;
 		new_disk--;
 		if (new_disk == disk) {
-			new_disk = -1;
-			goto rb_out;
+			new_disk = wonly_disk;
+			break;
 		}
 	}
+
+	if (new_disk < 0) 
+		goto rb_out;
+
 	disk = new_disk;
 	/* now disk == new_disk == starting point for search */
 
@@ -419,37 +435,41 @@ static int read_balance(conf_t *conf, r1
 			disk = conf->raid_disks;
 		disk--;
 
-		if ((rdev=conf->mirrors[disk].rdev) == NULL ||
-		    !rdev->in_sync)
+		rdev = conf->mirrors[disk].rdev;
+
+		if (!rdev ||
+		    !rdev->in_sync ||
+		    test_bit(WriteMostly, &rdev->flags))
 			continue;
 
 		if (!atomic_read(&rdev->nr_pending)) {
 			new_disk = disk;
-			new_rdev = rdev;
 			break;
 		}
 		new_distance = abs(this_sector - conf->mirrors[disk].head_position);
 		if (new_distance < current_distance) {
 			current_distance = new_distance;
 			new_disk = disk;
-			new_rdev = rdev;
 		}
 	} while (disk != conf->last_used);
 
-rb_out:
+ rb_out:
 
 
 	if (new_disk >= 0) {
-		conf->next_seq_sect = this_sector + sectors;
-		conf->last_used = new_disk;
-		atomic_inc(&new_rdev->nr_pending);
-		if (!new_rdev->in_sync) {
+		rdev = conf->mirrors[new_disk].rdev;
+		if (!rdev)
+			goto retry;
+		atomic_inc(&rdev->nr_pending);
+		if (!rdev->in_sync) {
 			/* cannot risk returning a device that failed
 			 * before we inc'ed nr_pending
 			 */
-			atomic_dec(&new_rdev->nr_pending);
+			atomic_dec(&rdev->nr_pending);
 			goto retry;
 		}
+		conf->next_seq_sect = this_sector + sectors;
+		conf->last_used = new_disk;
 	}
 	rcu_read_unlock();
 
@@ -1109,6 +1129,7 @@ static sector_t sync_request(mddev_t *md
 	sector_t max_sector, nr_sectors;
 	int disk;
 	int i;
+	int wonly;
 	int write_targets = 0;
 	int sync_blocks;
 	int still_degraded = 0;
@@ -1164,14 +1185,21 @@ static sector_t sync_request(mddev_t *md
 	 */
 	disk = conf->last_used;
 	/* make sure disk is operational */
-
+	wonly = disk;
 	while (conf->mirrors[disk].rdev == NULL ||
-	       !conf->mirrors[disk].rdev->in_sync) {
+	       !conf->mirrors[disk].rdev->in_sync ||
+	       test_bit(WriteMostly, &conf->mirrors[disk].rdev->flags)
+		) {
+		if (conf->mirrors[disk].rdev  &&
+		    conf->mirrors[disk].rdev->in_sync)
+			wonly = disk;
 		if (disk <= 0)
 			disk = conf->raid_disks;
 		disk--;
-		if (disk == conf->last_used)
+		if (disk == conf->last_used) {
+			disk = wonly;
 			break;
+		}
 	}
 	conf->last_used = disk;
 	atomic_inc(&conf->mirrors[disk].rdev->nr_pending);

diff ./include/linux/raid/md_k.h~current~ ./include/linux/raid/md_k.h
--- ./include/linux/raid/md_k.h~current~	2005-08-11 15:28:56.000000000 +1000
+++ ./include/linux/raid/md_k.h	2005-08-11 16:03:03.000000000 +1000
@@ -181,6 +181,9 @@ struct mdk_rdev_s
 	int faulty;			/* if faulty do not issue IO requests */
 	int in_sync;			/* device is a full member of the array */
 
+	unsigned long	flags;		/* Should include faulty and in_sync here. */
+#define	WriteMostly	4		/* Avoid reading if at all possible */
+
 	int desc_nr;			/* descriptor index in the superblock */
 	int raid_disk;			/* role of device in array */
 	int saved_raid_disk;		/* role that device used to have in the

diff ./include/linux/raid/md_p.h~current~ ./include/linux/raid/md_p.h
--- ./include/linux/raid/md_p.h~current~	2005-08-11 16:03:03.000000000 +1000
+++ ./include/linux/raid/md_p.h	2005-08-11 16:03:03.000000000 +1000
@@ -79,6 +79,11 @@
 #define MD_DISK_SYNC		2 /* disk is in sync with the raid set */
 #define MD_DISK_REMOVED		3 /* disk is in sync with the raid set */
 
+#define	MD_DISK_WRITEMOSTLY	9 /* disk is "write-mostly" is RAID1 config.
+				   * read requests will only be sent here in 
+				   * dire need
+				   */
+
 typedef struct mdp_device_descriptor_s {
 	__u32 number;		/* 0 Device number in the entire set	      */
 	__u32 major;		/* 1 Device major number		      */
@@ -193,7 +198,7 @@ struct mdp_superblock_1 {
 
 	__u64	ctime;		/* lo 40 bits are seconds, top 24 are microseconds or 0*/
 	__u32	level;		/* -4 (multipath), -1 (linear), 0,1,4,5 */
-	__u32	layout;		/* only for raid5 currently */
+	__u32	layout;		/* only for raid5 and raid10 currently */
 	__u64	size;		/* used size of component devices, in 512byte sectors */
 
 	__u32	chunksize;	/* in 512byte sectors */
@@ -212,7 +217,9 @@ struct mdp_superblock_1 {
 	__u32	dev_number;	/* permanent identifier of this  device - not role in raid */
 	__u32	cnt_corrected_read; /* number of read errors that were corrected by re-writing */
 	__u8	device_uuid[16]; /* user-space setable, ignored by kernel */
-	__u8	pad2[64-56];	/* set to 0 when writing */
+	__u8	devflags;	/* per-device flags.  Only one defined...*/
+#define	WriteMostly1	1	/* mask for writemostly flag in above */
+	__u8	pad2[64-57];	/* set to 0 when writing */
 
 	/* array state information - 64 bytes */
 	__u64	utime;		/* 40 bits second, 24 btes microseconds */

[PATCH md 006 of 6] Add write-behind support for md/raid1

[NeilBrown]

If a device is flagged 'WriteMostly' and the array has a bitmap,
and the bitmap superblock indicates that write_behind is allowed,
then write_behind is enabled for WriteMostly devices.

Write requests will be acknowledges as complete to the caller
(via b_end_io) when all non-WriteMostly devices have completed the write,
but will not be cleared from the bitmap until all devices complete.

This requires memory allocation to make a local copy of the data
being written.  If there is insufficient memory, then we fall-back
on normal write semantics.

Signed-Off-By: Paul Clements <paul.clements@steeleye.com>
Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>

### Diffstat output
 ./drivers/md/bitmap.c         |   26 +++++++-
 ./drivers/md/raid1.c          |  124 +++++++++++++++++++++++++++++++++++++++---
 ./include/linux/raid/bitmap.h |   15 +++--
 ./include/linux/raid/md_k.h   |    3 +
 ./include/linux/raid/raid1.h  |   13 ++++
 5 files changed, 165 insertions(+), 16 deletions(-)

diff ./drivers/md/bitmap.c~current~ ./drivers/md/bitmap.c
--- ./drivers/md/bitmap.c~current~	2005-08-11 16:17:06.000000000 +1000
+++ ./drivers/md/bitmap.c	2005-08-11 16:04:11.000000000 +1000
@@ -437,6 +437,7 @@ void bitmap_print_sb(struct bitmap *bitm
 	printk(KERN_DEBUG "  daemon sleep: %ds\n", le32_to_cpu(sb->daemon_sleep));
 	printk(KERN_DEBUG "     sync size: %llu KB\n",
 			(unsigned long long)le64_to_cpu(sb->sync_size)/2);
+	printk(KERN_DEBUG "max write behind: %d\n", le32_to_cpu(sb->write_behind));
 	kunmap(bitmap->sb_page);
 }
 
@@ -445,7 +446,7 @@ static int bitmap_read_sb(struct bitmap 
 {
 	char *reason = NULL;
 	bitmap_super_t *sb;
-	unsigned long chunksize, daemon_sleep;
+	unsigned long chunksize, daemon_sleep, write_behind;
 	unsigned long bytes_read;
 	unsigned long long events;
 	int err = -EINVAL;
@@ -474,6 +475,7 @@ static int bitmap_read_sb(struct bitmap 
 
 	chunksize = le32_to_cpu(sb->chunksize);
 	daemon_sleep = le32_to_cpu(sb->daemon_sleep);
+	write_behind = le32_to_cpu(sb->write_behind);
 
 	/* verify that the bitmap-specific fields are valid */
 	if (sb->magic != cpu_to_le32(BITMAP_MAGIC))
@@ -485,7 +487,9 @@ static int bitmap_read_sb(struct bitmap 
 	else if ((1 << ffz(~chunksize)) != chunksize)
 		reason = "bitmap chunksize not a power of 2";
 	else if (daemon_sleep < 1 || daemon_sleep > 15)
-		reason = "daemon sleep period out of range";
+		reason = "daemon sleep period out of range (1-15s)";
+	else if (write_behind > COUNTER_MAX)
+		reason = "write-behind limit out of range (0 - 16383)";
 	if (reason) {
 		printk(KERN_INFO "%s: invalid bitmap file superblock: %s\n",
 			bmname(bitmap), reason);
@@ -518,6 +522,7 @@ success:
 	/* assign fields using values from superblock */
 	bitmap->chunksize = chunksize;
 	bitmap->daemon_sleep = daemon_sleep;
+	bitmap->max_write_behind = write_behind;
 	bitmap->flags |= sb->state;
 	bitmap->events_cleared = le64_to_cpu(sb->events_cleared);
 	if (sb->state & BITMAP_STALE)
@@ -1282,9 +1287,16 @@ static bitmap_counter_t *bitmap_get_coun
 	}
 }
 
-int bitmap_startwrite(struct bitmap *bitmap, sector_t offset, unsigned long sectors)
+int bitmap_startwrite(struct bitmap *bitmap, sector_t offset, unsigned long sectors, int behind)
 {
 	if (!bitmap) return 0;
+
+	if (behind) {
+		atomic_inc(&bitmap->behind_writes);
+		PRINTK(KERN_DEBUG "inc write-behind count %d/%d\n",
+		  atomic_read(&bitmap->behind_writes), bitmap->max_write_behind);
+	}
+
 	while (sectors) {
 		int blocks;
 		bitmap_counter_t *bmc;
@@ -1319,9 +1331,15 @@ int bitmap_startwrite(struct bitmap *bit
 }
 
 void bitmap_endwrite(struct bitmap *bitmap, sector_t offset, unsigned long sectors,
-		     int success)
+		     int success, int behind)
 {
 	if (!bitmap) return;
+	if (behind) {
+		atomic_dec(&bitmap->behind_writes);
+		PRINTK(KERN_DEBUG "dec write-behind count %d/%d\n",
+		  atomic_read(&bitmap->behind_writes), bitmap->max_write_behind);
+	}
+
 	while (sectors) {
 		int blocks;
 		unsigned long flags;

diff ./drivers/md/raid1.c~current~ ./drivers/md/raid1.c
--- ./drivers/md/raid1.c~current~	2005-08-11 16:17:06.000000000 +1000
+++ ./drivers/md/raid1.c	2005-08-11 16:22:45.000000000 +1000
@@ -222,8 +222,17 @@ static void raid_end_bio_io(r1bio_t *r1_
 {
 	struct bio *bio = r1_bio->master_bio;
 
-	bio_endio(bio, bio->bi_size,
-		test_bit(R1BIO_Uptodate, &r1_bio->state) ? 0 : -EIO);
+	/* if nobody has done the final endio yet, do it now */
+	if (!test_and_set_bit(R1BIO_Returned, &r1_bio->state)) {
+		PRINTK(KERN_DEBUG "raid1: sync end %s on sectors %llu-%llu\n",
+			(bio_data_dir(bio) == WRITE) ? "write" : "read",
+			(unsigned long long) bio->bi_sector,
+			(unsigned long long) bio->bi_sector +
+				(bio->bi_size >> 9) - 1);
+	
+		bio_endio(bio, bio->bi_size,
+			test_bit(R1BIO_Uptodate, &r1_bio->state) ? 0 : -EIO);
+	}
 	free_r1bio(r1_bio);
 }
 
@@ -292,7 +301,7 @@ static int raid1_end_write_request(struc
 {
 	int uptodate = test_bit(BIO_UPTODATE, &bio->bi_flags);
 	r1bio_t * r1_bio = (r1bio_t *)(bio->bi_private);
-	int mirror;
+	int mirror, behind;
 	conf_t *conf = mddev_to_conf(r1_bio->mddev);
 
 	if (bio->bi_size)
@@ -323,16 +332,46 @@ static int raid1_end_write_request(struc
 
 	update_head_pos(mirror, r1_bio);
 
+	behind = test_bit(R1BIO_BehindIO, &r1_bio->state);
+	if (behind) {
+		if (test_bit(WriteMostly, &conf->mirrors[mirror].rdev->flags))
+			atomic_dec(&r1_bio->behind_remaining);
+
+		/* In behind mode, we ACK the master bio once the I/O has safely
+		 * reached all non-writemostly disks. Setting the Returned bit
+		 * ensures that this gets done only once -- we don't ever want to
+		 * return -EIO here, instead we'll wait */
+
+		if (atomic_read(&r1_bio->behind_remaining) >= (atomic_read(&r1_bio->remaining)-1) &&
+		    test_bit(R1BIO_Uptodate, &r1_bio->state)) {
+			/* Maybe we can return now */
+			if (!test_and_set_bit(R1BIO_Returned, &r1_bio->state)) {
+				struct bio *mbio = r1_bio->master_bio;
+				PRINTK(KERN_DEBUG "raid1: behind end write sectors %llu-%llu\n",
+				       (unsigned long long) mbio->bi_sector,
+				       (unsigned long long) mbio->bi_sector +
+				       (mbio->bi_size >> 9) - 1);
+				bio_endio(mbio, mbio->bi_size, 0);
+			}
+		}
+	}
 	/*
 	 *
 	 * Let's see if all mirrored write operations have finished
 	 * already.
 	 */
 	if (atomic_dec_and_test(&r1_bio->remaining)) {
+		if (test_bit(R1BIO_BehindIO, &r1_bio->state)) {
+			/* free extra copy of the data pages */
+			int i = bio->bi_vcnt;
+			while (i--)
+				__free_page(bio->bi_io_vec[i].bv_page);
+		}
 		/* clear the bitmap if all writes complete successfully */
 		bitmap_endwrite(r1_bio->mddev->bitmap, r1_bio->sector,
 				r1_bio->sectors,
-				!test_bit(R1BIO_Degraded, &r1_bio->state));
+				!test_bit(R1BIO_Degraded, &r1_bio->state),
+				behind);
 		md_write_end(r1_bio->mddev);
 		raid_end_bio_io(r1_bio);
 	}
@@ -562,6 +601,39 @@ static void device_barrier(conf_t *conf,
 	spin_unlock_irq(&conf->resync_lock);
 }
 
+/* duplicate the data pages for behind I/O */
+static struct page **alloc_behind_pages(struct bio *bio)
+{
+	int i;
+	struct bio_vec *bvec;
+	struct page **pages = kmalloc(bio->bi_vcnt * sizeof(struct page *),
+					GFP_NOIO);
+	if (unlikely(!pages))
+		goto do_sync_io;
+
+	memset(pages, 0, bio->bi_vcnt * sizeof(struct page *));
+
+	bio_for_each_segment(bvec, bio, i) {
+		pages[i] = alloc_page(GFP_NOIO);
+		if (unlikely(!pages[i]))
+			goto do_sync_io;
+		memcpy(kmap(pages[i]) + bvec->bv_offset,
+			kmap(bvec->bv_page) + bvec->bv_offset, bvec->bv_len);
+		kunmap(pages[i]);
+		kunmap(bvec->bv_page);
+	}
+
+	return pages;
+
+do_sync_io:
+	if (pages)
+		for (i = 0; i < bio->bi_vcnt && pages[i]; i++)
+			__free_page(pages[i]);
+	kfree(pages);
+	PRINTK("%dB behind alloc failed, doing sync I/O\n", bio->bi_size);
+	return NULL;
+}
+
 static int make_request(request_queue_t *q, struct bio * bio)
 {
 	mddev_t *mddev = q->queuedata;
@@ -574,6 +646,7 @@ static int make_request(request_queue_t 
 	struct bitmap *bitmap = mddev->bitmap;
 	unsigned long flags;
 	struct bio_list bl;
+	struct page **behind_pages = NULL;
 
 	if (unlikely(bio_barrier(bio))) {
 		bio_endio(bio, bio->bi_size, -EOPNOTSUPP);
@@ -613,8 +686,6 @@ static int make_request(request_queue_t 
 	r1_bio->mddev = mddev;
 	r1_bio->sector = bio->bi_sector;
 
-	r1_bio->state = 0;
-
 	if (bio_data_dir(bio) == READ) {
 		/*
 		 * read balancing logic:
@@ -675,13 +746,22 @@ static int make_request(request_queue_t 
 	}
 	rcu_read_unlock();
 
+	BUG_ON(targets == 0); /* we never fail the last device */
+
 	if (targets < conf->raid_disks) {
 		/* array is degraded, we will not clear the bitmap
 		 * on I/O completion (see raid1_end_write_request) */
 		set_bit(R1BIO_Degraded, &r1_bio->state);
 	}
 
+	/* do behind I/O ? */
+	if (bitmap &&
+	    atomic_read(&bitmap->behind_writes) < bitmap->max_write_behind &&
+	    (behind_pages = alloc_behind_pages(bio)) != NULL)
+		set_bit(R1BIO_BehindIO, &r1_bio->state);
+
 	atomic_set(&r1_bio->remaining, 0);
+	atomic_set(&r1_bio->behind_remaining, 0);
 
 	bio_list_init(&bl);
 	for (i = 0; i < disks; i++) {
@@ -698,12 +778,31 @@ static int make_request(request_queue_t 
 		mbio->bi_rw = WRITE;
 		mbio->bi_private = r1_bio;
 
+		if (behind_pages) {
+			struct bio_vec *bvec;
+			int j;
+
+			/* Yes, I really want the '__' version so that
+			 * we clear any unused pointer in the io_vec, rather
+			 * than leave them unchanged.  This is important
+			 * because when we come to free the pages, we won't
+			 * know the originial bi_idx, so we just free 
+			 * them all
+			 */
+			__bio_for_each_segment(bvec, mbio, j, 0)
+				bvec->bv_page = behind_pages[j];
+			if (test_bit(WriteMostly, &conf->mirrors[i].rdev->flags))
+				atomic_inc(&r1_bio->behind_remaining);
+		}
+
 		atomic_inc(&r1_bio->remaining);
 
 		bio_list_add(&bl, mbio);
 	}
+	kfree(behind_pages); /* the behind pages are attached to the bios now */
 
-	bitmap_startwrite(bitmap, bio->bi_sector, r1_bio->sectors);
+	bitmap_startwrite(bitmap, bio->bi_sector, r1_bio->sectors,
+				test_bit(R1BIO_BehindIO, &r1_bio->state));
 	spin_lock_irqsave(&conf->device_lock, flags);
 	bio_list_merge(&conf->pending_bio_list, &bl);
 	bio_list_init(&bl);
@@ -1471,6 +1570,17 @@ out:
 static int stop(mddev_t *mddev)
 {
 	conf_t *conf = mddev_to_conf(mddev);
+	struct bitmap *bitmap = mddev->bitmap;
+	int behind_wait = 0;
+
+	/* wait for behind writes to complete */
+	while (bitmap && atomic_read(&bitmap->behind_writes) > 0) {
+		behind_wait++;
+		printk(KERN_INFO "raid1: behind writes in progress on device %s, waiting to stop (%d)\n", mdname(mddev), behind_wait);
+		set_current_state(TASK_UNINTERRUPTIBLE);
+		schedule_timeout(HZ); /* wait a second */
+		/* need to kick something here to make sure I/O goes? */
+	}
 
 	md_unregister_thread(mddev->thread);
 	mddev->thread = NULL;

diff ./include/linux/raid/bitmap.h~current~ ./include/linux/raid/bitmap.h
--- ./include/linux/raid/bitmap.h~current~	2005-08-11 16:17:06.000000000 +1000
+++ ./include/linux/raid/bitmap.h	2005-08-11 16:04:11.000000000 +1000
@@ -7,7 +7,7 @@
 #define BITMAP_H 1
 
 #define BITMAP_MAJOR 3
-#define BITMAP_MINOR 38
+#define BITMAP_MINOR 39
 
 /*
  * in-memory bitmap:
@@ -147,8 +147,9 @@ typedef struct bitmap_super_s {
 	__u32 state;        /* 48  bitmap state information */
 	__u32 chunksize;    /* 52  the bitmap chunk size in bytes */
 	__u32 daemon_sleep; /* 56  seconds between disk flushes */
+	__u32 write_behind; /* 60  number of outstanding write-behind writes */
 
-	__u8  pad[256 - 60]; /* set to zero */
+	__u8  pad[256 - 64]; /* set to zero */
 } bitmap_super_t;
 
 /* notes:
@@ -226,6 +227,9 @@ struct bitmap {
 
 	unsigned long flags;
 
+	unsigned long max_write_behind; /* write-behind mode */
+	atomic_t behind_writes;
+
 	/*
 	 * the bitmap daemon - periodically wakes up and sweeps the bitmap
 	 * file, cleaning up bits and flushing out pages to disk as necessary
@@ -260,9 +264,10 @@ int  bitmap_setallbits(struct bitmap *bi
 void bitmap_write_all(struct bitmap *bitmap);
 
 /* these are exported */
-int bitmap_startwrite(struct bitmap *bitmap, sector_t offset, unsigned long sectors);
-void bitmap_endwrite(struct bitmap *bitmap, sector_t offset, unsigned long sectors,
-		     int success);
+int bitmap_startwrite(struct bitmap *bitmap, sector_t offset,
+			unsigned long sectors, int behind);
+void bitmap_endwrite(struct bitmap *bitmap, sector_t offset,
+			unsigned long sectors, int success, int behind);
 int bitmap_start_sync(struct bitmap *bitmap, sector_t offset, int *blocks, int degraded);
 void bitmap_end_sync(struct bitmap *bitmap, sector_t offset, int *blocks, int aborted);
 void bitmap_close_sync(struct bitmap *bitmap);

diff ./include/linux/raid/md_k.h~current~ ./include/linux/raid/md_k.h
--- ./include/linux/raid/md_k.h~current~	2005-08-11 16:17:06.000000000 +1000
+++ ./include/linux/raid/md_k.h	2005-08-11 16:04:11.000000000 +1000
@@ -275,6 +275,9 @@ struct mddev_s
 	atomic_t			writes_pending; 
 	request_queue_t			*queue;	/* for plugging ... */
 
+	atomic_t                        write_behind; /* outstanding async IO */
+	unsigned int                    max_write_behind; /* 0 = sync */
+
 	struct bitmap                   *bitmap; /* the bitmap for the device */
 	struct file			*bitmap_file; /* the bitmap file */
 	long				bitmap_offset; /* offset from superblock of

diff ./include/linux/raid/raid1.h~current~ ./include/linux/raid/raid1.h
--- ./include/linux/raid/raid1.h~current~	2005-08-11 16:17:06.000000000 +1000
+++ ./include/linux/raid/raid1.h	2005-08-11 16:04:11.000000000 +1000
@@ -80,6 +80,9 @@ struct r1bio_s {
 	atomic_t		remaining; /* 'have we finished' count,
 					    * used from IRQ handlers
 					    */
+	atomic_t		behind_remaining; /* number of write-behind ios remaining
+						 * in this BehindIO request
+						 */
 	sector_t		sector;
 	int			sectors;
 	unsigned long		state;
@@ -107,4 +110,14 @@ struct r1bio_s {
 #define	R1BIO_Uptodate	0
 #define	R1BIO_IsSync	1
 #define	R1BIO_Degraded	2
+#define	R1BIO_BehindIO   3
+/* For write-behind requests, we call bi_end_io when 
+ * the last non-write-behind device completes, providing 
+ * any write was successful.  Otherwise we call when 
+ * any write-behind write succeeds, otherwise we call 
+ * with failure when last write completes (and all failed).
+ * Record that bi_end_io was called with this flag...
+ */
+#define	R1BIO_Returned 4 
+				 
 #endif

[PATCH md 004 of 6] All hot-add and hot-remove of md intent logging bitmaps

[NeilBrown]

Both file-bitmaps and superblock bitmaps are supported.

If you add a bitmap file on the array device, you lose.

This introduces a 'default_bitmap_offset' field in mddev,
as the ioctl used for adding a superblock bitmap doesn't have
room for giving an offset.  Later, this value will be setable
via sysfs.

Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>

### Diffstat output
 ./drivers/md/md.c           |  104 ++++++++++++++++++++++++++++++++++++--------
 ./drivers/md/raid1.c        |   30 ++++++++++++
 ./include/linux/raid/md_k.h |   10 ++++
 3 files changed, 127 insertions(+), 17 deletions(-)

diff ./drivers/md/md.c~current~ ./drivers/md/md.c
--- ./drivers/md/md.c~current~	2005-08-11 15:12:51.000000000 +1000
+++ ./drivers/md/md.c	2005-08-11 15:35:33.000000000 +1000
@@ -624,6 +624,7 @@ static int super_90_validate(mddev_t *md
 		mddev->size = sb->size;
 		mddev->events = md_event(sb);
 		mddev->bitmap_offset = 0;
+		mddev->default_bitmap_offset = MD_SB_BYTES >> 9;
 
 		if (sb->state & (1<<MD_SB_CLEAN))
 			mddev->recovery_cp = MaxSector;
@@ -649,7 +650,7 @@ static int super_90_validate(mddev_t *md
 				printk(KERN_WARNING "md: bitmaps only support for raid1\n");
 				return -EINVAL;
 			}
-			mddev->bitmap_offset = (MD_SB_BYTES >> 9);
+			mddev->bitmap_offset = mddev->default_bitmap_offset;
 		}
 
 	} else if (mddev->pers == NULL) {
@@ -940,6 +941,9 @@ static int super_1_validate(mddev_t *mdd
 		mddev->size = le64_to_cpu(sb->size)/2;
 		mddev->events = le64_to_cpu(sb->events);
 		mddev->bitmap_offset = 0;
+		mddev->default_bitmap_offset = 0;
+		if (mddev->minor_version == 0)
+			mddev->default_bitmap_offset = -(64*1024)/512;
 		
 		mddev->recovery_cp = le64_to_cpu(sb->resync_offset);
 		memcpy(mddev->uuid, sb->set_uuid, 16);
@@ -2072,6 +2076,8 @@ static int get_array_info(mddev_t * mdde
 	info.state         = 0;
 	if (mddev->in_sync)
 		info.state = (1<<MD_SB_CLEAN);
+	if (mddev->bitmap && mddev->bitmap_offset)
+		info.state = (1<<MD_SB_BITMAP_PRESENT);
 	info.active_disks  = active;
 	info.working_disks = working;
 	info.failed_disks  = failed;
@@ -2430,25 +2436,51 @@ static int set_bitmap_file(mddev_t *mdde
 {
 	int err;
 
-	if (mddev->pers || mddev->bitmap_file)
-		return -EBUSY;
+	if (mddev->pers) {
+		if (!mddev->pers->quiesce)
+			return -EBUSY;
+		if (mddev->recovery || mddev->sync_thread)
+			return -EBUSY;
+		/* we should be able to change the bitmap.. */
+	}
 
-	mddev->bitmap_file = fget(fd);
 
-	if (mddev->bitmap_file == NULL) {
-		printk(KERN_ERR "%s: error: failed to get bitmap file\n",
-			mdname(mddev));
-		return -EBADF;
-	}
+	if (fd >= 0) {
+		if (mddev->bitmap)
+			return -EEXIST; /* cannot add when bitmap is present */
+		mddev->bitmap_file = fget(fd);
 
-	err = deny_bitmap_write_access(mddev->bitmap_file);
-	if (err) {
-		printk(KERN_ERR "%s: error: bitmap file is already in use\n",
-			mdname(mddev));
-		fput(mddev->bitmap_file);
-		mddev->bitmap_file = NULL;
-	} else
+		if (mddev->bitmap_file == NULL) {
+			printk(KERN_ERR "%s: error: failed to get bitmap file\n",
+			       mdname(mddev));
+			return -EBADF;
+		}
+
+		err = deny_bitmap_write_access(mddev->bitmap_file);
+		if (err) {
+			printk(KERN_ERR "%s: error: bitmap file is already in use\n",
+			       mdname(mddev));
+			fput(mddev->bitmap_file);
+			mddev->bitmap_file = NULL;
+			return err;
+		}
 		mddev->bitmap_offset = 0; /* file overrides offset */
+	} else if (mddev->bitmap == NULL)
+		return -ENOENT; /* cannot remove what isn't there */
+	err = 0;
+	if (mddev->pers) {
+		mddev->pers->quiesce(mddev, 1);
+		if (fd >= 0)
+			err = bitmap_create(mddev);
+		if (fd < 0 || err)
+			bitmap_destroy(mddev);
+		mddev->pers->quiesce(mddev, 0);
+	} else if (fd < 0) {
+		if (mddev->bitmap_file)
+			fput(mddev->bitmap_file);
+		mddev->bitmap_file = NULL;
+	}
+
 	return err;
 }
 
@@ -2528,6 +2560,11 @@ static int update_array_info(mddev_t *md
 {
 	int rv = 0;
 	int cnt = 0;
+	int state = 0;
+
+	/* calculate expected state,ignoring low bits */
+	if (mddev->bitmap && mddev->bitmap_offset)
+		state |= (1 << MD_SB_BITMAP_PRESENT);
 
 	if (mddev->major_version != info->major_version ||
 	    mddev->minor_version != info->minor_version ||
@@ -2536,12 +2573,16 @@ static int update_array_info(mddev_t *md
 	    mddev->level         != info->level         ||
 /*	    mddev->layout        != info->layout        || */
 	    !mddev->persistent	 != info->not_persistent||
-	    mddev->chunk_size    != info->chunk_size    )
+	    mddev->chunk_size    != info->chunk_size    ||
+	    /* ignore bottom 8 bits of state, and allow SB_BITMAP_PRESENT to change */
+	    ((state^info->state) & 0xfffffe00)
+		)
 		return -EINVAL;
 	/* Check there is only one change */
 	if (mddev->size != info->size) cnt++;
 	if (mddev->raid_disks != info->raid_disks) cnt++;
 	if (mddev->layout != info->layout) cnt++;
+	if ((state ^ info->state) & (1<<MD_SB_BITMAP_PRESENT)) cnt++;
 	if (cnt == 0) return 0;
 	if (cnt > 1) return -EINVAL;
 
@@ -2620,6 +2661,35 @@ static int update_array_info(mddev_t *md
 			}
 		}
 	}
+	if ((state ^ info->state) & (1<<MD_SB_BITMAP_PRESENT)) {
+		if (mddev->pers->quiesce == NULL)
+			return -EINVAL;
+		if (mddev->recovery || mddev->sync_thread)
+			return -EBUSY;
+		if (info->state & (1<<MD_SB_BITMAP_PRESENT)) {
+			/* add the bitmap */
+			if (mddev->bitmap)
+				return -EEXIST;
+			if (mddev->default_bitmap_offset == 0)
+				return -EINVAL;
+			mddev->bitmap_offset = mddev->default_bitmap_offset;
+			mddev->pers->quiesce(mddev, 1);
+			rv = bitmap_create(mddev);
+			if (rv)
+				bitmap_destroy(mddev);
+			mddev->pers->quiesce(mddev, 0);
+		} else {
+			/* remove the bitmap */
+			if (!mddev->bitmap)
+				return -ENOENT;
+			if (mddev->bitmap->file)
+				return -EINVAL;
+			mddev->pers->quiesce(mddev, 1);
+			bitmap_destroy(mddev);
+			mddev->pers->quiesce(mddev, 0);
+			mddev->bitmap_offset = 0;
+		}
+	}
 	md_update_sb(mddev);
 	return rv;
 }

diff ./drivers/md/raid1.c~current~ ./drivers/md/raid1.c
--- ./drivers/md/raid1.c~current~	2005-08-11 15:14:01.000000000 +1000
+++ ./drivers/md/raid1.c	2005-08-11 15:14:05.000000000 +1000
@@ -1565,6 +1565,35 @@ static int raid1_reshape(mddev_t *mddev,
 	return 0;
 }
 
+void raid1_quiesce(mddev_t *mddev, int state)
+{
+	conf_t *conf = mddev_to_conf(mddev);
+
+	switch(state) {
+	case 0:
+		spin_lock_irq(&conf->resync_lock);
+		conf->barrier++;
+		wait_event_lock_irq(conf->wait_idle, !conf->nr_pending,
+				    conf->resync_lock, raid1_unplug(mddev->queue));
+		spin_unlock_irq(&conf->resync_lock);
+		break;
+	case 1:
+		spin_lock_irq(&conf->resync_lock);
+		conf->barrier--;
+		spin_unlock_irq(&conf->resync_lock);
+		wake_up(&conf->wait_resume);
+		wake_up(&conf->wait_idle);
+		break;
+	}
+	if (mddev->thread) {
+		if (mddev->bitmap)
+			mddev->thread->timeout = mddev->bitmap->daemon_sleep * HZ;
+		else
+			mddev->thread->timeout = MAX_SCHEDULE_TIMEOUT;
+		md_wakeup_thread(mddev->thread);
+	}
+}
+
 
 static mdk_personality_t raid1_personality =
 {
@@ -1581,6 +1610,7 @@ static mdk_personality_t raid1_personali
 	.sync_request	= sync_request,
 	.resize		= raid1_resize,
 	.reshape	= raid1_reshape,
+	.quiesce	= raid1_quiesce,
 };
 
 static int __init raid_init(void)

diff ./include/linux/raid/md_k.h~current~ ./include/linux/raid/md_k.h
--- ./include/linux/raid/md_k.h~current~	2005-08-11 15:14:02.000000000 +1000
+++ ./include/linux/raid/md_k.h	2005-08-11 15:28:56.000000000 +1000
@@ -278,6 +278,10 @@ struct mddev_s
 							* start of bitmap. May be
 							* negative, but not '0'
 							*/
+	long				default_bitmap_offset; /* this is the offset to use when
+								* hot-adding a bitmap.  It should
+								* eventually be settable by sysfs.
+								*/
 
 	struct list_head		all_mddevs;
 };
@@ -314,6 +318,12 @@ struct mdk_personality_s
 	int (*resize) (mddev_t *mddev, sector_t sectors);
 	int (*reshape) (mddev_t *mddev, int raid_disks);
 	int (*reconfig) (mddev_t *mddev, int layout, int chunk_size);
+	/* quiesce moves between quiescence states
+	 * 0 - fully active
+	 * 1 - no new requests allowed
+	 * others - reserved
+	 */
+	void (*quiesce) (mddev_t *mddev, int state);
 };
 
 

RAID5 spare change

[djani22]

Hello list,

It is possible, to chage (swap) the spare drive and another one drive on the
working raid5 array without failing any device?
I have an good, but slow drive in the array, and I want to swap it with the
spare, because it is faster.
I know, when I mark the slow drive to failed-device, the system does it
automatically, but when the resync is happening, the whole system slows down
too much.

It is possible to mirror only one drive, and after it is synced, remove the
old, or similar?

Any idea?

Thanks!

Janos


----- Original Message -----
From: "NeilBrown" <neilb@cse.unsw.edu.au>
To: "Andrew Morton" <akpm@osdl.org>
Cc: <linux-raid@vger.kernel.org>
Sent: Thursday, August 11, 2005 9:22 AM
Subject: [PATCH md 005 of 6] Support write-mostly device in raid1


>
> This allows a device in a raid1 to be marked as "write mostly".
> Read requests will only be sent if there is no other option.
>
> Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au>
>
> ### Diffstat output
>  ./drivers/md/md.c           |   18 ++++++++++
>  ./drivers/md/raid1.c        |   76
++++++++++++++++++++++++++++++--------------
>  ./include/linux/raid/md_k.h |    3 +
>  ./include/linux/raid/md_p.h |   11 +++++-
>  4 files changed, 82 insertions(+), 26 deletions(-)
>
> diff ./drivers/md/md.c~current~ ./drivers/md/md.c
> --- ./drivers/md/md.c~current~ 2005-08-11 15:35:33.000000000 +1000
> +++ ./drivers/md/md.c 2005-08-11 16:03:03.000000000 +1000
> @@ -671,6 +671,7 @@ static int super_90_validate(mddev_t *md
>
>   if (mddev->level != LEVEL_MULTIPATH) {
>   rdev->faulty = 0;
> + rdev->flags = 0;
>   desc = sb->disks + rdev->desc_nr;
>
>   if (desc->state & (1<<MD_DISK_FAULTY))
> @@ -680,6 +681,8 @@ static int super_90_validate(mddev_t *md
>   rdev->in_sync = 1;
>   rdev->raid_disk = desc->raid_disk;
>   }
> + if (desc->state & (1<<MD_DISK_WRITEMOSTLY))
> + set_bit(WriteMostly, &rdev->flags);
>   } else /* MULTIPATH are always insync */
>   rdev->in_sync = 1;
>   return 0;
> @@ -778,6 +781,8 @@ static void super_90_sync(mddev_t *mddev
>   spare++;
>   working++;
>   }
> + if (test_bit(WriteMostly, &rdev2->flags))
> + d->state |= (1<<MD_DISK_WRITEMOSTLY);
>   }
>
>   /* now set the "removed" and "faulty" bits on any missing devices */
> @@ -991,6 +996,9 @@ static int super_1_validate(mddev_t *mdd
>   rdev->raid_disk = role;
>   break;
>   }
> + rdev->flags = 0;
> + if (sb->devflags & WriteMostly1)
> + set_bit(WriteMostly, &rdev->flags);
>   } else /* MULTIPATH are always insync */
>   rdev->in_sync = 1;
>
> @@ -2151,6 +2159,8 @@ static int get_disk_info(mddev_t * mddev
>   info.state |= (1<<MD_DISK_ACTIVE);
>   info.state |= (1<<MD_DISK_SYNC);
>   }
> + if (test_bit(WriteMostly, &rdev->flags))
> + info.state |= (1<<MD_DISK_WRITEMOSTLY);
>   } else {
>   info.major = info.minor = 0;
>   info.raid_disk = -1;
> @@ -2236,6 +2246,9 @@ static int add_new_disk(mddev_t * mddev,
>   rdev->saved_raid_disk = rdev->raid_disk;
>
>   rdev->in_sync = 0; /* just to be sure */
> + if (info->state & (1<<MD_DISK_WRITEMOSTLY))
> + set_bit(WriteMostly, &rdev->flags);
> +
>   rdev->raid_disk = -1;
>   err = bind_rdev_to_array(rdev, mddev);
>   if (err)
> @@ -2277,6 +2290,9 @@ static int add_new_disk(mddev_t * mddev,
>   else
>   rdev->in_sync = 0;
>
> + if (info->state & (1<<MD_DISK_WRITEMOSTLY))
> + set_bit(WriteMostly, &rdev->flags);
> +
>   err = bind_rdev_to_array(rdev, mddev);
>   if (err) {
>   export_rdev(rdev);
> @@ -3329,6 +3345,8 @@ static int md_seq_show(struct seq_file *
>   char b[BDEVNAME_SIZE];
>   seq_printf(seq, " %s[%d]",
>   bdevname(rdev->bdev,b), rdev->desc_nr);
> + if (test_bit(WriteMostly, &rdev->flags))
> + seq_printf(seq, "(W)");
>   if (rdev->faulty) {
>   seq_printf(seq, "(F)");
>   continue;
>
> diff ./drivers/md/raid1.c~current~ ./drivers/md/raid1.c
> --- ./drivers/md/raid1.c~current~ 2005-08-11 15:14:05.000000000 +1000
> +++ ./drivers/md/raid1.c 2005-08-11 16:03:03.000000000 +1000
> @@ -360,13 +360,14 @@ static int read_balance(conf_t *conf, r1
>  {
>   const unsigned long this_sector = r1_bio->sector;
>   int new_disk = conf->last_used, disk = new_disk;
> + int wonly_disk = -1;
>   const int sectors = r1_bio->sectors;
>   sector_t new_distance, current_distance;
> - mdk_rdev_t *new_rdev, *rdev;
> + mdk_rdev_t *rdev;
>
>   rcu_read_lock();
>   /*
> - * Check if it if we can balance. We can balance on the whole
> + * Check if we can balance. We can balance on the whole
>   * device if no resync is going on, or below the resync window.
>   * We take the first readable disk when above the resync window.
>   */
> @@ -376,11 +377,16 @@ static int read_balance(conf_t *conf, r1
>   /* Choose the first operation device, for consistancy */
>   new_disk = 0;
>
> - while ((new_rdev=conf->mirrors[new_disk].rdev) == NULL ||
> -        !new_rdev->in_sync) {
> - new_disk++;
> - if (new_disk == conf->raid_disks) {
> - new_disk = -1;
> + for (rdev = conf->mirrors[new_disk].rdev;
> +      !rdev || !rdev->in_sync
> +      || test_bit(WriteMostly, &rdev->flags);
> +      rdev = conf->mirrors[++new_disk].rdev) {
> +
> + if (rdev && rdev->in_sync)
> + wonly_disk = new_disk;
> +
> + if (new_disk == conf->raid_disks - 1) {
> + new_disk = wonly_disk;
>   break;
>   }
>   }
> @@ -389,16 +395,26 @@ static int read_balance(conf_t *conf, r1
>
>
>   /* make sure the disk is operational */
> - while ((new_rdev=conf->mirrors[new_disk].rdev) == NULL ||
> -        !new_rdev->in_sync) {
> + for (rdev = conf->mirrors[new_disk].rdev;
> +      !rdev || !rdev->in_sync ||
> +      test_bit(WriteMostly, &rdev->flags);
> +      rdev = conf->mirrors[new_disk].rdev) {
> +
> + if (rdev && rdev->in_sync)
> + wonly_disk = new_disk;
> +
>   if (new_disk <= 0)
>   new_disk = conf->raid_disks;
>   new_disk--;
>   if (new_disk == disk) {
> - new_disk = -1;
> - goto rb_out;
> + new_disk = wonly_disk;
> + break;
>   }
>   }
> +
> + if (new_disk < 0)
> + goto rb_out;
> +
>   disk = new_disk;
>   /* now disk == new_disk == starting point for search */
>
> @@ -419,37 +435,41 @@ static int read_balance(conf_t *conf, r1
>   disk = conf->raid_disks;
>   disk--;
>
> - if ((rdev=conf->mirrors[disk].rdev) == NULL ||
> -     !rdev->in_sync)
> + rdev = conf->mirrors[disk].rdev;
> +
> + if (!rdev ||
> +     !rdev->in_sync ||
> +     test_bit(WriteMostly, &rdev->flags))
>   continue;
>
>   if (!atomic_read(&rdev->nr_pending)) {
>   new_disk = disk;
> - new_rdev = rdev;
>   break;
>   }
>   new_distance = abs(this_sector - conf->mirrors[disk].head_position);
>   if (new_distance < current_distance) {
>   current_distance = new_distance;
>   new_disk = disk;
> - new_rdev = rdev;
>   }
>   } while (disk != conf->last_used);
>
> -rb_out:
> + rb_out:
>
>
>   if (new_disk >= 0) {
> - conf->next_seq_sect = this_sector + sectors;
> - conf->last_used = new_disk;
> - atomic_inc(&new_rdev->nr_pending);
> - if (!new_rdev->in_sync) {
> + rdev = conf->mirrors[new_disk].rdev;
> + if (!rdev)
> + goto retry;
> + atomic_inc(&rdev->nr_pending);
> + if (!rdev->in_sync) {
>   /* cannot risk returning a device that failed
>   * before we inc'ed nr_pending
>   */
> - atomic_dec(&new_rdev->nr_pending);
> + atomic_dec(&rdev->nr_pending);
>   goto retry;
>   }
> + conf->next_seq_sect = this_sector + sectors;
> + conf->last_used = new_disk;
>   }
>   rcu_read_unlock();
>
> @@ -1109,6 +1129,7 @@ static sector_t sync_request(mddev_t *md
>   sector_t max_sector, nr_sectors;
>   int disk;
>   int i;
> + int wonly;
>   int write_targets = 0;
>   int sync_blocks;
>   int still_degraded = 0;
> @@ -1164,14 +1185,21 @@ static sector_t sync_request(mddev_t *md
>   */
>   disk = conf->last_used;
>   /* make sure disk is operational */
> -
> + wonly = disk;
>   while (conf->mirrors[disk].rdev == NULL ||
> -        !conf->mirrors[disk].rdev->in_sync) {
> +        !conf->mirrors[disk].rdev->in_sync ||
> +        test_bit(WriteMostly, &conf->mirrors[disk].rdev->flags)
> + ) {
> + if (conf->mirrors[disk].rdev  &&
> +     conf->mirrors[disk].rdev->in_sync)
> + wonly = disk;
>   if (disk <= 0)
>   disk = conf->raid_disks;
>   disk--;
> - if (disk == conf->last_used)
> + if (disk == conf->last_used) {
> + disk = wonly;
>   break;
> + }
>   }
>   conf->last_used = disk;
>   atomic_inc(&conf->mirrors[disk].rdev->nr_pending);
>
> diff ./include/linux/raid/md_k.h~current~ ./include/linux/raid/md_k.h
> --- ./include/linux/raid/md_k.h~current~ 2005-08-11 15:28:56.000000000
+1000
> +++ ./include/linux/raid/md_k.h 2005-08-11 16:03:03.000000000 +1000
> @@ -181,6 +181,9 @@ struct mdk_rdev_s
>   int faulty; /* if faulty do not issue IO requests */
>   int in_sync; /* device is a full member of the array */
>
> + unsigned long flags; /* Should include faulty and in_sync here. */
> +#define WriteMostly 4 /* Avoid reading if at all possible */
> +
>   int desc_nr; /* descriptor index in the superblock */
>   int raid_disk; /* role of device in array */
>   int saved_raid_disk; /* role that device used to have in the
>
> diff ./include/linux/raid/md_p.h~current~ ./include/linux/raid/md_p.h
> --- ./include/linux/raid/md_p.h~current~ 2005-08-11 16:03:03.000000000
+1000
> +++ ./include/linux/raid/md_p.h 2005-08-11 16:03:03.000000000 +1000
> @@ -79,6 +79,11 @@
>  #define MD_DISK_SYNC 2 /* disk is in sync with the raid set */
>  #define MD_DISK_REMOVED 3 /* disk is in sync with the raid set */
>
> +#define MD_DISK_WRITEMOSTLY 9 /* disk is "write-mostly" is RAID1 config.
> +    * read requests will only be sent here in
> +    * dire need
> +    */
> +
>  typedef struct mdp_device_descriptor_s {
>   __u32 number; /* 0 Device number in the entire set       */
>   __u32 major; /* 1 Device major number       */
> @@ -193,7 +198,7 @@ struct mdp_superblock_1 {
>
>   __u64 ctime; /* lo 40 bits are seconds, top 24 are microseconds or 0*/
>   __u32 level; /* -4 (multipath), -1 (linear), 0,1,4,5 */
> - __u32 layout; /* only for raid5 currently */
> + __u32 layout; /* only for raid5 and raid10 currently */
>   __u64 size; /* used size of component devices, in 512byte sectors */
>
>   __u32 chunksize; /* in 512byte sectors */
> @@ -212,7 +217,9 @@ struct mdp_superblock_1 {
>   __u32 dev_number; /* permanent identifier of this  device - not role in
raid */
>   __u32 cnt_corrected_read; /* number of read errors that were corrected
by re-writing */
>   __u8 device_uuid[16]; /* user-space setable, ignored by kernel */
> - __u8 pad2[64-56]; /* set to 0 when writing */
> + __u8 devflags; /* per-device flags.  Only one defined...*/
> +#define WriteMostly1 1 /* mask for writemostly flag in above */
> + __u8 pad2[64-57]; /* set to 0 when writing */
>
>   /* array state information - 64 bytes */
>   __u64 utime; /* 40 bits second, 24 btes microseconds */
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5 spare change

[Neil Brown]

On Thursday August 11, djani22@dynamicweb.hu wrote:
> 
> It is possible to mirror only one drive, and after it is synced, remove the
> old, or similar?
> 

That would be nice, wouldn't it.
It isn't possible yet.
When we get the bitmap intent logging working for raid5, it will be
possible to do it with just 2 small windows while the array is
degraded. 
Whether it will ever be possible to do it with no window, I don't
know.

NeilBrown

Re: [PATCH md 006 of 6] Add write-behind support for md/raid1

[Al Boldi]

NeilBrown wrote:
> If a device is flagged 'WriteMostly' and the array has a bitmap,
> and the bitmap superblock indicates that write_behind is allowed,
> then write_behind is enabled for WriteMostly devices.

Nice, but why is it dependent on WriteMostly?

> This requires memory allocation to make a local copy of the data
> being written.  If there is insufficient memory, then we fall-back
> on normal write semantics.

Good!

Also, it would be nice to have something like:
echo 1 > /proc/md../../WriteBehindEnabled
echo 10 > /proc/md../../WriteBehindMaxDelayBeforeSync

Thanks!

--
Al

Re: [PATCH md 006 of 6] Add write-behind support for md/raid1

[Paul Clements]

Al Boldi wrote:
> NeilBrown wrote:
> 
>>If a device is flagged 'WriteMostly' and the array has a bitmap,
>>and the bitmap superblock indicates that write_behind is allowed,
>>then write_behind is enabled for WriteMostly devices.
> 
> 
> Nice, but why is it dependent on WriteMostly?

WriteMostly is just a flag that tells us which devices will get the 
write-behinds, and which will not. You'll be able to mix any combination 
of WriteMostly devices and normal devices in a raid1.

--
Paul

Re: [PATCH md 006 of 6] Add write-behind support for md/raid1

[Al Boldi]

Paul Clements wrote:
> Al Boldi wrote:
> > NeilBrown wrote:
> >>If a device is flagged 'WriteMostly' and the array has a bitmap,
> >>and the bitmap superblock indicates that write_behind is allowed,
> >>then write_behind is enabled for WriteMostly devices.
> >
> > Nice, but why is it dependent on WriteMostly?
>
> WriteMostly is just a flag that tells us which devices will get the
> write-behinds, and which will not. You'll be able to mix any
> combination of WriteMostly devices and normal devices in a raid1.
>

Yes, but doesn't WriteMostly imply ReadDelay?
If so, doesn't that mean that WriteBehind is dependent on ReadDelay?

--
Al

Re: [PATCH md 006 of 6] Add write-behind support for md/raid1

[Neil Brown]

On Friday August 12, a1426z@gawab.com wrote:
> 
> Also, it would be nice to have something like:
> echo 1 > /proc/md../../WriteBehindEnabled
> echo 10 > /proc/md../../WriteBehindMaxDelayBeforeSync

Something like
   echo 1 > /sys/block/md0/md/d4/write_behind
or
   echo 10 > /sys/block/md0/md/max_write_behind

are real possibilities.
However "WriteBehindMaxDelayBeforeSync" doesn't make sense I don't
think.  When doing write behind, we don't delay the write at all.  We
just assume that it will take longer because it is on a slow device.
The only limit is the number of outstanding write requests.  Once this
crosses a maximum, all further requests a fully synchronous until the
number of outstanding writes drops below the maximum.

NeilBrown

Re: [PATCH md 006 of 6] Add write-behind support for md/raid1

[Neil Brown]

On Saturday August 13, a1426z@gawab.com wrote:
> >
> > WriteMostly is just a flag that tells us which devices will get the
> > write-behinds, and which will not. You'll be able to mix any
> > combination of WriteMostly devices and normal devices in a raid1.
> >
> 
> Yes, but doesn't WriteMostly imply ReadDelay?
> If so, doesn't that mean that WriteBehind is dependent on ReadDelay?

I'm not sure what you mean by "ReadDelay".
We never deliberately delay reads.  However some devices (e.g. network
connection to remote mirror) may have a higher read latency, and these
would be appropriate for WriteMostly.

As read-latency and write-latency are likely to go together, it seems
reasonable for WriteMostly and WriteBehind to go together.  Do you
disagree?

NeilBrown

Re: [PATCH md 006 of 6] Add write-behind support for md/raid1

[Al Boldi]

Neil Brown wrote:
> As read-latency and write-latency are likely to go together, it
> seems reasonable for WriteMostly and WriteBehind to go together. 
> Do you disagree?

In OOP terms, always try to remove dependencies.
Try to make your implementation as general as possible to facilitate 
code reuse thus increasing scalability!

--
Al