Race condition in /etc/xen/scripts/block

Race condition in /etc/xen/scripts/block

[Bastian de Groot]

Hello,

I am trying to figure out the behavior of the hook-script for block devices 
(/etc/xen/scripts/block).

Here is the problem:
I want to start a VM. It shall boot from a dm-device, which is setup by a  
script executed in the script /etc/xen/scripts/block. The dm-device doesn't  
exist when "xm start " is executed . 

When I place the setup script at the top of the block-script, everything  
works fine (http://pastebin.com/UYYbGupC). But when the script is placed in  
the phy case branch of the block-script, the VM won't start 
(http://pastebin.com/iPdJ65AH).

Whether the VM starts or not, also depends on the runtime of the setup script. 
When the script is executed fast, it is much more likely that the VM starts.

So, is there a way to setup a device in the hook-script?

My configuration: Xen 3.4 on a 2.6.32 Kernel

I would appreciate your help on that very much,
Bastian

RE: Race condition in /etc/xen/scripts/block

[James Harper]

> Hello,
> 
> I am trying to figure out the behavior of the hook-script for block
devices
> (/etc/xen/scripts/block).
> 
> Here is the problem:
> I want to start a VM. It shall boot from a dm-device, which is setup
by a
> script executed in the script /etc/xen/scripts/block. The dm-device
doesn't
> exist when "xm start " is executed .
> 
> When I place the setup script at the top of the block-script,
everything
> works fine (http://pastebin.com/UYYbGupC). But when the script is
placed in
> the phy case branch of the block-script, the VM won't start
> (http://pastebin.com/iPdJ65AH).
> 
> Whether the VM starts or not, also depends on the runtime of the setup
script.
> When the script is executed fast, it is much more likely that the VM
starts.
> 
> So, is there a way to setup a device in the hook-script?
> 
> My configuration: Xen 3.4 on a 2.6.32 Kernel
> 
> I would appreciate your help on that very much,
> Bastian
> 

I've seen a similar thing when using drbd. When you run 'xm start' the
drbd device is 'secondary' and one of the things the block setup script
does is make it 'primary' which means it can be opened/accessed.
Unfortunately most of the time qemu-dm has already tried to open it and
failed by then.

In that case I solved it by having qemu-dm try to open in a loop with a
sleep each time. Not the most elegant solution but it works.

I don't think the same problem exists with PV domains so if you are
using a PV DomU then I've probably not been much help...

James

Re: Race condition in /etc/xen/scripts/block

[Bastian de Groot]

Hi James, 

thank you for your fast reply. The VMs are not PV domains, so the problem 
seems to be pretty much the same as yours. I would be very glad if you could 
publish the patch, so that I can test if it works for me.

Thanks again,
Bastian

Am Freitag 16 Juli 2010 12:07:29 schrieben Sie:
> > Hello,
> >
> > I am trying to figure out the behavior of the hook-script for block
>
> devices
>
> > (/etc/xen/scripts/block).
> >
> > Here is the problem:
> > I want to start a VM. It shall boot from a dm-device, which is setup
>
> by a
>
> > script executed in the script /etc/xen/scripts/block. The dm-device
>
> doesn't
>
> > exist when "xm start " is executed .
> >
> > When I place the setup script at the top of the block-script,
>
> everything
>
> > works fine (http://pastebin.com/UYYbGupC). But when the script is
>
> placed in
>
> > the phy case branch of the block-script, the VM won't start
> > (http://pastebin.com/iPdJ65AH).
> >
> > Whether the VM starts or not, also depends on the runtime of the setup
>
> script.
>
> > When the script is executed fast, it is much more likely that the VM
>
> starts.
>
> > So, is there a way to setup a device in the hook-script?
> >
> > My configuration: Xen 3.4 on a 2.6.32 Kernel
> >
> > I would appreciate your help on that very much,
> > Bastian
>
> I've seen a similar thing when using drbd. When you run 'xm start' the
> drbd device is 'secondary' and one of the things the block setup script
> does is make it 'primary' which means it can be opened/accessed.
> Unfortunately most of the time qemu-dm has already tried to open it and
> failed by then.
>
> In that case I solved it by having qemu-dm try to open in a loop with a
> sleep each time. Not the most elegant solution but it works.
>
> I don't think the same problem exists with PV domains so if you are
> using a PV DomU then I've probably not been much help...
>
> James

RE: Race condition in /etc/xen/scripts/block

[James Harper]

> 
> Hi James,
> 
> thank you for your fast reply. The VMs are not PV domains, so the
problem
> seems to be pretty much the same as yours. I would be very glad if you
could
> publish the patch, so that I can test if it works for me.
> 

This is the patch I'm using for drbd. You can probably remove the drbd
bit if it doesn't interest you. If line wrapping makes the patch
unusable let me know and I'll attach it as a file. It's against 3.4.x
but is simple enough that you should be able to re-invent it for a later
or earlier version. Obviously if qemu isn't the thing that's having
problems then this patch won't help.

James


diff --git a/xenstore.c b/xenstore.c
index 9360771..bd0b5e0 100644
--- a/xenstore.c
+++ b/xenstore.c
@@ -317,6 +317,7 @@ void xenstore_parse_domain_config(int hvm_domid)
     unsigned int len, num, hd_index, pci_devid = 0;
     BlockDriverState *bs;
     BlockDriver *format;
+    int retry_count;

     /* paths controlled by untrustworthy guest, and values read from
them */
     char *danger_path;
@@ -428,6 +429,14 @@ void xenstore_parse_domain_config(int hvm_domid)
             params = newparams;
            format = &bdrv_raw;
         }
+        /* handle drbd mapping */
+        if (!strcmp(drv, "drbd")) {
+            char *newparams = malloc(17 + strlen(params) + 1);
+            sprintf(newparams, "/dev/drbd/by-res/%s", params);
+            free(params);
+            params = newparams;
+           format = &bdrv_raw;
+        }

 #if 0
        /* Phantom VBDs are disabled because the use of paths
@@ -500,8 +509,15 @@ void xenstore_parse_domain_config(int hvm_domid)
                }
            }
             pstrcpy(bs->filename, sizeof(bs->filename), params);
-            if (bdrv_open2(bs, params, BDRV_O_CACHE_WB /* snapshot and
write-back */, format) < 0)
-                fprintf(stderr, "qemu: could not open vbd '%s' or hard
disk image '%s' (drv '%s' format '%s')\n", buf, params, drv ? drv : "?",
format ? format->format_name : "0");
+            retry_count = 0;
+            while ((ret = bdrv_open2(bs, params, BDRV_O_CACHE_WB /*
snapshot and write-back */, format)) < 0)
+            {
+               if (retry_count++ > 10)
+                  break;
+               sleep(1);
+            }
+            if (ret < 0)
+               fprintf(stderr, "qemu: could not open vbd '%s' or hard
disk image '%s' (drv '%s' format '%s')\n", buf, params, drv ? drv : "?",
format ? format->format_name : "0");
         }

        drives_table[nb_drives].bdrv = bs;

Re: Race condition in /etc/xen/scripts/block

[Bastian de Groot]

Hi James,

thanks again for the fast reply. I've tested the patch over the last few days 
and it seems to work just fine. 

As you said, it's not the most elegant way, but it looks like there is no 
other simple way around this problem.

Does anyone know whether xen 4.x also has this bug?

Bastian

Am Freitag 16 Juli 2010 13:35:03 schrieben Sie:
> This is the patch I'm using for drbd. You can probably remove the drbd
> bit if it doesn't interest you. If line wrapping makes the patch
> unusable let me know and I'll attach it as a file. It's against 3.4.x
> but is simple enough that you should be able to re-invent it for a later
> or earlier version. Obviously if qemu isn't the thing that's having
> problems then this patch won't help.
>
> James
>
>
> diff --git a/xenstore.c b/xenstore.c
> index 9360771..bd0b5e0 100644
> --- a/xenstore.c
> +++ b/xenstore.c
> @@ -317,6 +317,7 @@ void xenstore_parse_domain_config(int hvm_domid)
>      unsigned int len, num, hd_index, pci_devid = 0;
>      BlockDriverState *bs;
>      BlockDriver *format;
> +    int retry_count;
>
>      /* paths controlled by untrustworthy guest, and values read from
> them */
>      char *danger_path;
> @@ -428,6 +429,14 @@ void xenstore_parse_domain_config(int hvm_domid)
>              params = newparams;
>             format = &bdrv_raw;
>          }
> +        /* handle drbd mapping */
> +        if (!strcmp(drv, "drbd")) {
> +            char *newparams = malloc(17 + strlen(params) + 1);
> +            sprintf(newparams, "/dev/drbd/by-res/%s", params);
> +            free(params);
> +            params = newparams;
> +           format = &bdrv_raw;
> +        }
>
>  #if 0
>         /* Phantom VBDs are disabled because the use of paths
> @@ -500,8 +509,15 @@ void xenstore_parse_domain_config(int hvm_domid)
>                 }
>             }
>              pstrcpy(bs->filename, sizeof(bs->filename), params);
> -            if (bdrv_open2(bs, params, BDRV_O_CACHE_WB /* snapshot and
> write-back */, format) < 0)
> -                fprintf(stderr, "qemu: could not open vbd '%s' or hard
> disk image '%s' (drv '%s' format '%s')\n", buf, params, drv ? drv : "?",
> format ? format->format_name : "0");
> +            retry_count = 0;
> +            while ((ret = bdrv_open2(bs, params, BDRV_O_CACHE_WB /*
> snapshot and write-back */, format)) < 0)
> +            {
> +               if (retry_count++ > 10)
> +                  break;
> +               sleep(1);
> +            }
> +            if (ret < 0)
> +               fprintf(stderr, "qemu: could not open vbd '%s' or hard
> disk image '%s' (drv '%s' format '%s')\n", buf, params, drv ? drv : "?",
> format ? format->format_name : "0");
>          }
>
>         drives_table[nb_drives].bdrv = bs;