Xen guest disk online resize, xenstore/blkback/blkfront questions

Xen guest disk online resize, xenstore/blkback/blkfront questions

[Pasi Kärkkäinen]

Hello,

I started implementing support for on-the-fly/online resizing of Xen PV
guest disks.

I added 'block-resize' command to xm, and added device_resize() function
to xend.

I'm able to do 'xm block-resize <guest> <guest_disk>" and my new
device_resize() function in xend gets called. All fine so far.

Now I'm trying to understand the internals of how kernel
blkback/blkfront and xenstore interact with each other, so I can
implement the actual resizing.

This is how I've understood the flow of adding/creating a block device
in Xen:

- blkback driver in dom0 kernel runs xenbus_register_backend(), which
  makes xenstore to notify blkback about new block devices via xenbus.

- When xend DevController.createDevice() for new block device gets called, 
  it writes the initial block device configuration to xenstore. 
  
- Xenstore then notifies blkback via xenbus, and the blkback_probe() function 
  of blkback gets called.

- blkback_probe() creates the backend device in dom0 kernel, and
  sets up a xenbus_watch to get notified about 'physical-device' changes
  in xenstore block device backend information.

- udev hotplug scripts get executed when the new backend device is
  created in the dom0 kernel, and when the udev hotplug scripts are done 
  they update the xenstore block device backend 'physical-device' information 
  to trigger blkback notification.
  (what do the udev hotplug scripts actually do here?)

- Xenstore block device backend 'physical-device' updates cause xenbus_watch to call 
  blkback backend_changed(). Physical device major/minor, mode and type
  are then fetched from xenstore. Then vbd is created in the kernel, 
  and update_blkif_status() is called.

- update_blkif_status() calls connect() which tries to connect the
  backend with the blkfront frontend.

- connect() writes block device size in sectors and sector-size
  to xenstore, and changes the state to 'connected' so frontend can
  fetch the information from xenstore and bring itself online.


Is this more or less correct? Please correct if I've missed something or
understood something wrong.

So, what I need to do next:

- LVM online-resize the guest disk LV in dom0.

- write something to xenstore block device backend structures, 
  to get the blkback driver notified about the 'block-resize'.

  Should I add a new xenbus_watch for some, say, 'resize' field, so I could get
  callback to blkback device_resize() easily when xenstore is updated? 

- blkback driver then needs to update/fetch the new size of the vbd, 
  and update the xenstore /local/domain/0/backend/vbd/X/sectors field.

  Any problems getting the new size/sectors on-the-fly in the kernel? 

- blkback then needs to write something to xenstore block device frontend
  /local/domain/X/device/vbd/Y/ to notify the blkfront driver in the guest.

  Same thing here, should blkfront have a watch for some 'resize' field
  or so?


How does that sound like? All comments and help appreciated!

-- Pasi

Re: Xen guest disk online resize, xenstore/blkback/blkfront questions

[Ian Campbell]

On Wed, 2010-01-06 at 20:26 +0000, Pasi Kärkkäinen wrote:
[...] 
> Is this more or less correct? Please correct if I've missed something or
> understood something wrong.

All sounds vaguely familiar, although it's been a while since I last had
to follow those twisty paths around...

> So, what I need to do next:
> 
> - LVM online-resize the guest disk LV in dom0.
> 
> - write something to xenstore block device backend structures, 
>   to get the blkback driver notified about the 'block-resize'.
> 
>   Should I add a new xenbus_watch for some, say, 'resize' field, so I could get
>   callback to blkback device_resize() easily when xenstore is updated? 
> 
> - blkback driver then needs to update/fetch the new size of the vbd, 
>   and update the xenstore /local/domain/0/backend/vbd/X/sectors field.
> 
>   Any problems getting the new size/sectors on-the-fly in the kernel? 

You might need to close and reopen the block device? 

Idle speculation: perhaps instead of an explicit "resize" field in
xenstore you could just have the backend continue watching the
'physical-device' node even after everything is connected the first time
and if it is rewritten (including to the same value) reopening the
physical device and setting things up again (picking up a size change as
a side effect).

There's no particular reason why the physical device couldn't change
over this operation either, you could maybe imagine changing to a
different device mapper node (e.g. perhaps some sort of wierd snapshot
mechanism?) or perhaps implementing some sort of PV-CDROM media change
in the same way.

> - blkback then needs to write something to xenstore block device frontend
>   /local/domain/X/device/vbd/Y/ to notify the blkfront driver in the guest.
> 
>   Same thing here, should blkfront have a watch for some 'resize' field
>   or so?

blkfront can probably just watch /local/domain/0/backend/vbd/X/sectors

Ian.

Re: Xen guest disk online resize, xenstore/blkback/blkfront questions

[Pasi Kärkkäinen]

On Wed, Jan 06, 2010 at 08:44:25PM +0000, Ian Campbell wrote:
> On Wed, 2010-01-06 at 20:26 +0000, Pasi Kärkkäinen wrote:
> [...] 
> > Is this more or less correct? Please correct if I've missed something or
> > understood something wrong.
> 
> All sounds vaguely familiar, although it's been a while since I last had
> to follow those twisty paths around...
> 
> > So, what I need to do next:
> > 
> > - LVM online-resize the guest disk LV in dom0.
> > 
> > - write something to xenstore block device backend structures, 
> >   to get the blkback driver notified about the 'block-resize'.
> > 
> >   Should I add a new xenbus_watch for some, say, 'resize' field, so I could get
> >   callback to blkback device_resize() easily when xenstore is updated? 
> > 
> > - blkback driver then needs to update/fetch the new size of the vbd, 
> >   and update the xenstore /local/domain/0/backend/vbd/X/sectors field.
> > 
> >   Any problems getting the new size/sectors on-the-fly in the kernel? 
> 
> You might need to close and reopen the block device? 
> 
> Idle speculation: perhaps instead of an explicit "resize" field in
> xenstore you could just have the backend continue watching the
> 'physical-device' node even after everything is connected the first time
> and if it is rewritten (including to the same value) reopening the
> physical device and setting things up again (picking up a size change as
> a side effect).
> 

Good idea. I'll experiment with this.

> There's no particular reason why the physical device couldn't change
> over this operation either, you could maybe imagine changing to a
> different device mapper node (e.g. perhaps some sort of wierd snapshot
> mechanism?) or perhaps implementing some sort of PV-CDROM media change
> in the same way.
> 
> > - blkback then needs to write something to xenstore block device frontend
> >   /local/domain/X/device/vbd/Y/ to notify the blkfront driver in the guest.
> > 
> >   Same thing here, should blkfront have a watch for some 'resize' field
> >   or so?
> 
> blkfront can probably just watch /local/domain/0/backend/vbd/X/sectors
> 

Sounds good. 
Thanks for the reply!

-- Pasi

Re: Xen guest disk online resize, xenstore/blkback/blkfront questions

[Daniel Stodden]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On Wed, 2010-01-06 at 16:57 -0500, Pasi Kärkkäinen wrote:

> > >   Any problems getting the new size/sectors on-the-fly in the kernel? 
> > 
> > You might need to close and reopen the block device? 
> > 
> > Idle speculation: perhaps instead of an explicit "resize" field in
> > xenstore you could just have the backend continue watching the
> > 'physical-device' node even after everything is connected the first time
> > and if it is rewritten (including to the same value) reopening the
> > physical device and setting things up again (picking up a size change as
> > a side effect).
> > 
> 
> Good idea. I'll experiment with this.

The XCP kernel has support for transparent pause/unpause. Which takes of
the queue flush in order to get the bdev closed.

Below is that patch which enables switching the physical node,
safety-checking if the VBD is paused etc. Matters not so much for
snapshotting, rather for cross-vg stuff.

It happens to work otherwise as IanC suggested.

It may be more than you're up to in order to get a resize done, e.g.
depends on whether you want to piggyback close/open into pause/unpause
or not (as a rule of thumb, it's safe to say that pause applications
want a close anyway :]). 

Didn't check for bitrot, sorry in advance.

HTH. Let me know if you're missing something or how it's going.

Cheers,
Daniel

[-- Attachment #2: blkback-device-switch.diff --]
[-- Type: text/x-patch, Size: 4501 bytes --]

diff -r e069e08b56f0 drivers/xen/blkback/xenbus.c
--- a/drivers/xen/blkback/xenbus.c	Tue Apr 14 21:22:05 2009 -0700
+++ b/drivers/xen/blkback/xenbus.c	Tue Apr 14 21:23:38 2009 -0700
@@ -37,6 +37,7 @@
 	unsigned major;
 	unsigned minor;
 	char *mode;
+	int cdrom;
 	int group_added;
 	char *nodename;
 	atomic_t refcnt;
@@ -50,6 +51,8 @@
 static int connect_ring(struct backend_info *);
 static void backend_changed(struct xenbus_watch *, const char **,
 			    unsigned int);
+static int blkback_open_bdev(struct backend_info *be);
+static void blkback_close_bdev(struct backend_info *be);
 
 static int blkback_name(blkif_t *blkif, char *buf)
 {
@@ -349,9 +352,7 @@
 
 	down(&blkback_dev_sem);
 
-	vbd_free(&blkif->vbd);
-	blkif->be->major = 0;
-	blkif->be->minor = 0;
+	blkback_close_bdev(blkif->be);
 	blkif->remove_requested = 0;
 	if (blkif->be->dev)
 		xenvbd_sysfs_delif(blkif->be->dev);
@@ -425,6 +426,10 @@
 	struct backend_info *be = blkif->be;
 
 	down(&blkback_dev_sem);
+
+	if (blkif->vbd.bdev)
+		vbd_free(&blkif->vbd);
+
 	if (be->dev) {
 		err = xenbus_write(XBT_NIL,
 				   be->dev->nodename, "pause-done", "");
@@ -471,7 +476,8 @@
 
 	} else if (xenbus_exists(XBT_NIL, dev->nodename, "pause-done")) {
 		WPRINTK("resuming %s\n", dev->nodename);
-		blkback_resume(be->blkif);
+		if (!blkback_open_bdev(be))
+			blkback_resume(be->blkif);
 	}
 }
 
@@ -555,12 +561,25 @@
 	return err;
 }
 
+static void blkback_close_bdev(struct backend_info *be)
+{
+	if (be->major || be->minor) {
+		vbd_free(&be->blkif->vbd);
+		be->major = 0;
+		be->minor = 0;
+	}
+
+	if (be->mode) {
+		kfree(be->mode);
+		be->mode = NULL;
+	}
+}
+
 static int blkback_open_bdev(struct backend_info *be)
 {
 	int err;
 	char *p;
 	long handle;
-	int cdrom = 0;
 	unsigned major;
 	unsigned minor;
 	char *device_type;
@@ -580,15 +599,16 @@
 
 	if ((be->major || be->minor) &&
 	    ((be->major != major) || (be->minor != minor))) {
-		printk(KERN_WARNING
-		       "blkback: changing physical device (from %x:%x to "
-		       "%x:%x) not supported.\n", be->major, be->minor,
-		       major, minor);
-		xenbus_dev_fatal(dev, err, "invalid physical-device change");
-		return -EINVAL;
+		printk(KERN_INFO
+		       "blkback: changing physical device (from %x:%x to %x:%x)\n",
+		       be->major, be->minor, major, minor);
 	}
 
 	if (!be->mode) {
+		/* Reopen on resume ignores the vbd info, as we're not
+		   reissuing the frontend information
+		   either. Reconnect will. */
+
 		be->mode = xenbus_read(XBT_NIL, dev->nodename, "mode", NULL);
 		if (IS_ERR(be->mode)) {
 			err = PTR_ERR(be->mode);
@@ -596,12 +616,12 @@
 			xenbus_dev_fatal(dev, err, "reading mode");
 			return err;
 		}
-	}
 
-	device_type = xenbus_read(XBT_NIL, dev->otherend, "device-type", NULL);
-	if (!IS_ERR(device_type)) {
-		cdrom = strcmp(device_type, "cdrom") == 0;
-		kfree(device_type);
+		device_type = xenbus_read(XBT_NIL, dev->otherend, "device-type", NULL);
+		if (!IS_ERR(device_type)) {
+			be->cdrom = strcmp(device_type, "cdrom") == 0;
+			kfree(device_type);
+		}
 	}
 
 	/* Front end dir is a number, which is used as the handle. */
@@ -609,9 +629,10 @@
 	handle = simple_strtoul(p, NULL, 0);
 
 	err = vbd_create(be->blkif, handle, major, minor,
-			 (NULL == strchr(be->mode, 'w')), cdrom);
+			 (NULL == strchr(be->mode, 'w')), be->cdrom);
 	if (err) {
 		xenbus_dev_fatal(dev, err, "creating vbd structure");
+		blkback_close_bdev(be);
 		return err;
 	}
 
@@ -652,24 +673,12 @@
 	}
 
 	if ((be->major || be->minor) &&
-	    ((be->major != major) || (be->minor != minor))) {
+	    ((be->major != major) || (be->minor != minor)) &&
+	    !xenbus_exists(XBT_NIL, dev->nodename, "pause-done")) {
 		printk(KERN_WARNING
-		       "blkback: changing physical device (from %x:%x to "
-		       "%x:%x) not supported.\n", be->major, be->minor,
+		       "blkback: attempt to change physical device (from %x:%x to "
+		       "%x:%x) on running blkif.\n", be->major, be->minor,
 		       major, minor);
-		return;
-	}
-
-	if (be->mode) {
-		kfree(be->mode);
-		be->mode = NULL;
-	}
-
-	be->mode = xenbus_read(XBT_NIL, dev->nodename, "mode", NULL);
-	if (IS_ERR(be->mode)) {
-		err = PTR_ERR(be->mode);
-		be->mode = NULL;
-		xenbus_dev_fatal(dev, err, "reading mode");
 		return;
 	}
 
@@ -680,8 +689,7 @@
 
 		err = xenvbd_sysfs_addif(dev);
 		if (err) {
-			vbd_free(&be->blkif->vbd);
-			be->major = be->minor = 0;
+			blkback_close_bdev(be);
 			xenbus_dev_fatal(dev, err, "creating sysfs entries");
 			return;
 		}

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen guest disk online resize, xenstore/blkback/blkfront questions

[Pasi Kärkkäinen]

On Thu, Jan 07, 2010 at 11:54:19PM -0800, Daniel Stodden wrote:
> On Wed, 2010-01-06 at 16:57 -0500, Pasi Kärkkäinen wrote:
> 
> > > >   Any problems getting the new size/sectors on-the-fly in the kernel? 
> > > 
> > > You might need to close and reopen the block device? 
> > > 
> > > Idle speculation: perhaps instead of an explicit "resize" field in
> > > xenstore you could just have the backend continue watching the
> > > 'physical-device' node even after everything is connected the first time
> > > and if it is rewritten (including to the same value) reopening the
> > > physical device and setting things up again (picking up a size change as
> > > a side effect).
> > > 
> > 
> > Good idea. I'll experiment with this.
> 
> The XCP kernel has support for transparent pause/unpause. Which takes of
> the queue flush in order to get the bdev closed.
> 
> Below is that patch which enables switching the physical node,
> safety-checking if the VBD is paused etc. Matters not so much for
> snapshotting, rather for cross-vg stuff.
> 
> It happens to work otherwise as IanC suggested.
> 
> It may be more than you're up to in order to get a resize done, e.g.
> depends on whether you want to piggyback close/open into pause/unpause
> or not (as a rule of thumb, it's safe to say that pause applications
> want a close anyway :]). 
> 
> Didn't check for bitrot, sorry in advance.
>

Interesting. I'll take a look at this aswell. Thanks!

> HTH. Let me know if you're missing something or how it's going.
> 

Yep. I'll get back to you. 

-- Pasi

Re: Xen guest disk online resize, xenstore/blkback/blkfront questions

[feisky]

Has anyone tried is this method ok?

--
View this message in context: http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5447334.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

Re: Xen guest disk online resize, xenstore/blkback/blkfront questions

[Pasi Kärkkäinen]

On Wed, Feb 01, 2012 at 04:29:58AM -0800, feisky wrote:
> Has anyone tried is this method ok?
> 

Yep, I've used it, and verified it works.

-- Pasi

Re: Xen guest disk online resize, xenstore/blkback/blkfront questions

[feisky]

[-- Attachment #1.1: Type: text/plain, Size: 1647 bytes --]

Could you send your patch to me?

thanks.

On Wed, Feb 1, 2012 at 11:52 PM, Pasi Kärkkäinen [via Xen] <
ml-node+s1045712n5447888h47@n5.nabble.com> wrote:

> On Wed, Feb 01, 2012 at 04:29:58AM -0800, feisky wrote:
> > Has anyone tried is this method ok?
> >
>
> Yep, I've used it, and verified it works.
>
> -- Pasi
>
>
> _______________________________________________
> Xen-devel mailing list
> [hidden email] <http://user/SendEmail.jtp?type=node&node=5447888&i=0>
> http://lists.xensource.com/xen-devel
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5447888.html
>  To unsubscribe from Xen guest disk online resize,
> xenstore/blkback/blkfront questions, click here<http://xen.1045712.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2547847&code=ZmVpc2t5ZXJAZ21haWwuY29tfDI1NDc4NDd8MTg5Nzg0NTI4Nw==>
> .
> NAML<http://xen.1045712.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>


--
View this message in context: http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5449661.html
Sent from the Xen - Dev mailing list archive at Nabble.com.

[-- Attachment #1.2: Type: text/html, Size: 2822 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen guest disk online resize, xenstore/blkback/blkfront questions

[Pasi Kärkkäinen]

On Wed, Feb 01, 2012 at 09:31:54PM -0800, feisky wrote:
>    Could you send your patch to me?Â
>    thanks.
> 

It doesn't require patches. Support for online domU disk resize
is in upstream kernel.org kernels (and in rhel/sles vendor kernels).

It only requires support from xen-blkback and xen-blkfront,
and LVM volumes in dom0.

-- Pasi

>    On Wed, Feb 1, 2012 at 11:52 PM, Pasi KÀrkkÀinen [via Xen] <[1][hidden
>    email]> wrote:
> 
>      On Wed, Feb 01, 2012 at 04:29:58AM -0800, feisky wrote:
>      > Has anyone tried is this method ok?
>      >
> 
>      Yep, I've used it, and verified it works.
> 
>      -- Pasi
> 
>      _______________________________________________
>      Xen-devel mailing list
>      [2][hidden email]
>      [3]http://lists.xensource.com/xen-devel
> 
>    --------------------------------------------------------------------------
> 
>      If you reply to this email, your message will be added to the discussion
>      below:
>      [4]http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5447888.html
>      To unsubscribe from Xen guest disk online resize,
>      xenstore/blkback/blkfront questions, [5]click here.
>      [6]NAML
> 
>    -------------------------------------------
> 
>    View this message in context: [7]Re: Xen guest disk online resize,
>    xenstore/blkback/blkfront questions
>    Sent from the [8]Xen - Dev mailing list archive at Nabble.com.
> 
> References
> 
>    Visible links
>    1. file:///user/SendEmail.jtp?type=node&node=5449661&i=0
>    2. http://user/SendEmail.jtp?type=node&node=5447888&i=0
>    3. http://lists.xensource.com/xen-devel
>    4. http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5447888.html
>    5. file:///tmp/
>    6. http://xen.1045712.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>    7. http://xen.1045712.n5.nabble.com/Xen-guest-disk-online-resize-xenstore-blkback-blkfront-questions-tp2547847p5449661.html
>    8. http://xen.1045712.n5.nabble.com/Xen-Dev-f2473738.html

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel