multi-hosting support for carrier grade Linux

multi-hosting support for carrier grade Linux

[Dave Jiang]

I'm attempting to implement multihost support of the MD for environments
such as carrier grade Linux. Multihost support allows the RAID array to
be claimed by a particular host via a unique ID (unique SCSI host ID,
FibreChannel WWN, or geographical address of a chassis blade. That way
another host that can access the disks do not claim the same disks that
are used by the RAID array. 

I would like to store a 64bit unique ID on the
superblock of the device. The least intrusive way IMHO to do this is
implementing the feature via the management app such as mdadm in
userland. However, it seems that after I instruct the kernel to create the MD array via mdadm, the kernel starts out with a blank superblock and clobbers the
data I have stored on the superblock via mdadm. Would it be acceptable
to modify the kernel such that the unique ID info is preserved during
the creation of the superblock by the kernel? Example patch follows:

Index: linux-2.6.10/drivers/md/md.c
===================================================================
--- linux-2.6.10.orig/drivers/md/md.c
+++ linux-2.6.10/drivers/md/md.c
@@ -640,6 +640,10 @@
 	struct list_head *tmp;
 	mdk_rdev_t *rdev2;
 	int next_spare = mddev->raid_disks;
+#ifdef CONFIG_MD_MULTIHOST	
+	u32 host_id_type;
+	u64 host_identifier;
+#endif	
 
 	/* make rdev->sb match mddev data..
 	 *
@@ -656,6 +660,13 @@
 
 	sb = (mdp_super_t*)page_address(rdev->sb_page);
 
+#ifdef CONFIG_MD_MULTIHOST
+	/* saving MULTIHOST entries */
+	read_disk_sb(rdev);
+	host_id_type = sb->this_disk.host_id_type;
+	host_identifier = sb->this_disk.host_identifier;
+#endif	
+
 	memset(sb, 0, sizeof(*sb));
 
 	sb->md_magic = MD_SB_MAGIC;
@@ -741,6 +752,13 @@
 	sb->spare_disks = spare;
 
 	sb->this_disk = sb->disks[rdev->desc_nr];
+
+#ifdef CONFIG_MD_MULTIHOST
+	/* MULTIHOST */
+	sb->this_disk.host_id_type = host_id_type;
+	sb->this_disk.host_identifier = host_identifier;
+#endif	
+
 	sb->sb_csum = calc_sb_csum(sb);
 }
 
@@ -3103,7 +3121,23 @@
 				seq_printf(seq, "	resync=DELAYED");
 		}
 
+#ifdef CONFIG_MD_MULTIHOST
+		ITERATE_RDEV(mddev,rdev,tmp2) {
+			char b[BDEVNAME_SIZE];
+			mdp_super_t *sb = (mdp_super_t*)page_address(rdev->sb_page);
+			if(sb->this_disk.host_id_type == 0)
+				break;
+			seq_printf(seq, "\n      %s[%d] : ", 
+					bdevname(rdev->bdev,b), rdev->desc_nr);
+			seq_printf(seq, "\n          MultiHost ID type : %d",
+					sb->this_disk.host_id_type);
+			seq_printf(seq, "\n          MultiHost ID : %llu",
+					sb->this_disk.host_identifier);
+		}
+		seq_printf(seq, "\n\n");
+#else		
 		seq_printf(seq, "\n");
+#endif /* CONFIG_MD_MULTIHOST */
 	}
 	mddev_unlock(mddev);
 	
Index: linux-2.6.10/include/linux/raid/md_p.h
===================================================================
--- linux-2.6.10.orig/include/linux/raid/md_p.h
+++ linux-2.6.10/include/linux/raid/md_p.h
@@ -85,7 +85,13 @@
 	__u32 minor;		/* 2 Device minor number		      */
 	__u32 raid_disk;	/* 3 The role of the device in the raid set   */
 	__u32 state;		/* 4 Operational state			      */
+#if CONFIG_MD_MULTIHOST
+	__u32 reserved[MD_SB_DESCRIPTOR_WORDS - 8];
+	__u32 host_id_type;
+	__u64 host_identifier;
+#else
 	__u32 reserved[MD_SB_DESCRIPTOR_WORDS - 5];
+#endif	/* CONFIG_MD_MULTIHOST */
 } mdp_disk_t;
 
 #define MD_SB_MAGIC		0xa92b4efc
Index: linux-2.6.10/drivers/md/Kconfig
===================================================================
--- linux-2.6.10.orig/drivers/md/Kconfig
+++ linux-2.6.10/drivers/md/Kconfig
@@ -164,6 +164,17 @@
 
 	  If unsure, say N.
 
+config MD_MULTIHOST
+	bool "Multi-Host support"
+	depends on BLK_DEV_MD
+	help
+	  Multi-Host is the ability to bind an MD volume to a particular host via
+	  a unique identifier. The choices of identifiers can be used are: unique
+	  SCSI host ID, Fibre Channel IEEE unique identifier (WWN), or telecom
+	  blade geographical address (IPMB address). 
+
+	  If unsure, say N
+	  
 config MD_FAULTY
 	tristate "Faulty test module for MD"
 	depends on BLK_DEV_MD


------------------------------------------------------
Dave Jiang
Software Engineer                               
MontaVista Software, Inc.                       
mailto:djiang@mvista.com
------------------------------------------------------

Re: multi-hosting support for carrier grade Linux

[Paul Clements]

Dave Jiang wrote:
> I'm attempting to implement multihost support of the MD for environments
> such as carrier grade Linux. Multihost support allows the RAID array to
> be claimed by a particular host via a unique ID (unique SCSI host ID,
> FibreChannel WWN, or geographical address of a chassis blade. That way
> another host that can access the disks do not claim the same disks that
> are used by the RAID array. 

Why not just use SCSI reservations?

> I would like to store a 64bit unique ID on the
> superblock of the device. The least intrusive way IMHO to do this is
> implementing the feature via the management app such as mdadm in
> userland. However, it seems that after I instruct the kernel to create the MD array via mdadm, the kernel starts out with a blank superblock and clobbers the

If you write a valid superblock to the disk and then assemble the array, 
the superblock doesn't get clobbered.

> data I have stored on the superblock via mdadm. Would it be acceptable
> to modify the kernel such that the unique ID info is preserved during
> the creation of the superblock by the kernel? Example patch follows:

As for adding additional fields to the various superblock formats, you'd 
have to ask Neil if he's open to that.

--
Paul

Re: multi-hosting support for carrier grade Linux

[Dave Jiang]

Paul Clements wrote:
> Dave Jiang wrote:
> 
>> I'm attempting to implement multihost support of the MD for environments
>> such as carrier grade Linux. Multihost support allows the RAID array to
>> be claimed by a particular host via a unique ID (unique SCSI host ID,
>> FibreChannel WWN, or geographical address of a chassis blade. That way
>> another host that can access the disks do not claim the same disks that
>> are used by the RAID array. 
> 
> 
> Why not just use SCSI reservations?

 From my understanding of SCSI reservations ownership of the entire disk 
claimed. If we have multiple host blades, it is possible that multiple 
hosts to claim separate partitions on a disk, each for their own RAIDs - 
not optimal for performance but useful in many embedded environments.

Even if reservations targeted specific portions of the disk the 
partitioning tools (e.g. fdisk) would have to be involved to keep things 
straight and deal with the multiple initiator situations. Tagging the 
superblocks IMHO seems much simpler.

Another issue comes up due to hotswapping. In some environments, such as 
FibreChannel, SCSI addresses may be dynamic making releasing a 
reservation impractical - the replacement host may not have the same 
SCSI addresses as its predecessor.

> 
>> I would like to store a 64bit unique ID on the
>> superblock of the device. The least intrusive way IMHO to do this is
>> implementing the feature via the management app such as mdadm in
>> userland. However, it seems that after I instruct the kernel to create 
>> the MD array via mdadm, the kernel starts out with a blank superblock 
>> and clobbers the
> 
> 
> If you write a valid superblock to the disk and then assemble the array, 
> the superblock doesn't get clobbered.

In order to target a particular partition to a specific host the 
superblock is the obvious place to place the discriminator. The assembly 
process simply bypasses the partitions that are not specifically 
targeted to it. This way a group of hosts on a shared peripheral 
connection (e.g. FibreChannel) would only claim what they should and 
ignore what they shouldn't.

Perhaps the UUID within the superblock could be built via a function 
that could be abstracted or overloaded. The data within only needs to be 
unique within its environment so it could contain geographical 
addressing, target information or whatever fits. Perhaps this sounds a 
bit less intrusive and requires little or perhaps no kernel changes?

>> data I have stored on the superblock via mdadm. Would it be acceptable
>> to modify the kernel such that the unique ID info is preserved during
>> the creation of the superblock by the kernel? Example patch follows:
> 
> 
> As for adding additional fields to the various superblock formats, you'd 
> have to ask Neil if he's open to that.
> 
> -- 
> Paul

------------------------------------------------------
Dave Jiang
Software Engineer
MontaVista Software, Inc.
mailto:djiang@mvista.com
------------------------------------------------------

Re: multi-hosting support for carrier grade Linux

[Paul Clements]

Dave Jiang wrote:
> Paul Clements wrote:
>> Dave Jiang wrote:

>  From my understanding of SCSI reservations ownership of the entire disk 
> claimed. If we have multiple host blades, it is possible that multiple 

SCSI 3 reservations can be used to lock an extent (e.g., a partition) of 
a disk (you can even do read only reservations, which lock out only 
writes). But yes, given your requirements, that may not be the easiest 
solution.


> Perhaps the UUID within the superblock could be built via a function 
> that could be abstracted or overloaded. The data within only needs to be 
> unique within its environment so it could contain geographical 
> addressing, target information or whatever fits. Perhaps this sounds a 
> bit less intrusive and requires little or perhaps no kernel changes?

Yes, the md driver doesn't care what the UUID is, so you can make it 
anything you want and then assemble the array. When you change 
initiators, you rewrite the UUID (you just have to make sure that the 
UUID is unique and that all UUIDs for disks in a given array are the 
same). The UUID is 128 bits long, so it should suffice for your purposes.

--
Paul