Re: Greater than 16 xvd devices for blkfront

["Daniel P. Berrange" <berrange@redhat.com>]

On Wed, May 07, 2008 at 02:55:02AM +0100, Daniel P. Berrange wrote:
> On Tue, May 06, 2008 at 01:36:05PM -0400, Chris Lalancette wrote:
> > All,
> >      We've had a number of requests to increase the number of xvd devices that a
> > PV guest can have.  Currently, if you try to connect > 16 disks, you get an
> > error from xend.  The problem ends up being that both xend and blkfront assume
> > that for dev_t, major/minor is 8 bits each, where in fact there are actually 10
> > bits for major and 22 bits for minor.
> >      Therefore, it shouldn't really be a problem giving lots of disks to guests.
> >  The problem is in backwards compatibility, and the details.  What I am
> > initially proposing to do is to leave things where they are for /dev/xvd[a-p];
> > that is, still put the xenstore entries in the same place, and use 8 bits for
> > the major and 8 bits for the minor.  For anything above that, we would end up
> > putting the xenstore entry in a different place, and pushing the major into the
> > top 10 bits (leaving the bottom 22 bits for the minor); that way old guests
> > won't fire when the entry is added, and we will add code to newer guests
> > blkfront so that they will fire when they see that entry.  Does anyone see any
> > problems with this setup, or have any ideas how to do it better?
> 
> Looking at the blkfront code I think we can increase the minor numbers
> available for xvdX devices without requiring changes to the where stuff
> is stored.

Have a go with this proof of concept patch to blkfront. I built pv-on-hvm drivers
with this and successfully booted my guest with 25 disks (xvdb -> xvdz) and saw
them registered in /dev as can be seen from /proc/partitions:

major minor  #blocks  name

   3     0    5242880 hda
   3     1     104391 hda1
   3     2    5132767 hda2
 253     0    4096000 dm-0
 253     1    1015808 dm-1
 202    16     102400 xvdb
 202    32     102400 xvdc
 202    48     102400 xvdd
 202    49      48163 xvdd1
 202    50      48195 xvdd2
 202    64     102400 xvde
 202    80     102400 xvdf
 202    96     102400 xvdg
 202   112     102400 xvdh
 202   128     102400 xvdi
 202   144     102400 xvdj
 202   160     102400 xvdk
 202   176     102400 xvdl
 202   192     102400 xvdm
 202   208     102400 xvdn
 202   224     102400 xvdo
 202   240     102400 xvdp
 202   256     102400 xvdq
 202   272     102400 xvdr
 202   288     102400 xvds
 202   304     102400 xvdt
 202   320     102400 xvdu
 202   336     102400 xvdv
 202   352     102400 xvdw
 202   368     102400 xvdx
 202   384     102400 xvdy
 202   400     102400 xvdz
 202   401      96358 xvdz1

NB, requires the regex tweak to blkif.py in XenD to allow xvd[a-z] naming.

Regards,
Daniel.

diff -r 57ab8dd47580 drivers/xen/blkfront/vbd.c
--- a/drivers/xen/blkfront/vbd.c	Sun Jul 01 22:07:32 2007 +0100
+++ b/drivers/xen/blkfront/vbd.c	Tue May 06 23:38:20 2008 -0400
@@ -166,7 +166,14 @@ xlbd_get_major_info(int vdevice)
                 index = 18 + major - SCSI_DISK8_MAJOR;
                 break;
         case SCSI_CDROM_MAJOR: index = 26; break;
-        default: index = 27; break;
+        default:
+		index = 27;
+		if (major >  XLBD_MAJOR_VBD_START) {
+			printk("xen-vbd: fixup major/minor %d -> %d,%d\n", vdevice, major, minor);
+			minor += (16 * 16 * (major - 202));
+			major = 202;
+		}
+		printk("xen-vbd: process major/minor %d -> %d,%d\n", vdevice, major, minor);
 	}
 
 	mi = ((major_info[index] != NULL) ? major_info[index] :
@@ -315,14 +322,42 @@ xlvbd_add(blkif_sector_t capacity, int v
 {
 	struct block_device *bd;
 	int err = 0;
+	int major, minor;
 
-	info->dev = MKDEV(BLKIF_MAJOR(vdevice), BLKIF_MINOR(vdevice));
+	major = BLKIF_MAJOR(vdevice);
+	minor = BLKIF_MINOR(vdevice);
+
+	switch (major) {
+	case IDE0_MAJOR:
+	case IDE1_MAJOR:
+	case IDE2_MAJOR:
+	case IDE3_MAJOR:
+	case IDE4_MAJOR:
+	case IDE5_MAJOR:
+	case IDE6_MAJOR:
+	case IDE7_MAJOR:
+	case IDE8_MAJOR:
+	case IDE9_MAJOR:
+	case SCSI_DISK0_MAJOR:
+	case SCSI_DISK1_MAJOR ... SCSI_DISK7_MAJOR:
+	case SCSI_DISK8_MAJOR ... SCSI_DISK15_MAJOR:
+	case SCSI_CDROM_MAJOR:
+		break;
+
+	default:
+		if (major > 202) {
+			minor += (16 * 16 * (major - 202));
+			major = 202;
+		}
+	}
+
+	info->dev = MKDEV(major, minor);
 
 	bd = bdget(info->dev);
 	if (bd == NULL)
 		return -ENODEV;
 
-	err = xlvbd_alloc_gendisk(BLKIF_MINOR(vdevice), capacity, vdevice,
+	err = xlvbd_alloc_gendisk(minor, capacity, vdevice,
 				  vdisk_info, sector_size, info);
 
 	bdput(bd);


-- 
|: Red Hat, Engineering, Boston   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|