tap:qcow causes dom-U to hang in 3.0.3

tap:qcow causes dom-U to hang in 3.0.3

[Roland Paterson-Jones]

Hi all

I have had some success with the new blktap support in Xen 3.0.3. In 
particular, the loopback drop-in, tap:aio: works somewhat reliably.

However, I am really interested in copy-on-write support, and hence have 
been trying tap:qcow:. This has had limited success, in that dom-U's 
appear to launch. However, as soon as I try to console or ssh into the 
dom-U's they apparently freeze, and I can get no further with them.

I have managed to 'xm terminate' the frozen dom-U's, but this leaves 
xend in some inconsistent state. All further 'xm xxx' commands barf with 
the following error:

[root@dom0]# xm list
Error: Device 2050 not connected
Usage: xm list [options] [Domain, ...]

Curiously, it seems that device 2050 was a secondary block device 
exposed using phy:.

So, any advice on how to debug this?

Thanks and kind regards
Roland

Re: [Xen-devel] tap:qcow causes dom-U to hang in 3.0.3

[Ewan Mellor]

On Wed, Nov 08, 2006 at 04:50:43PM +0200, Roland Paterson-Jones wrote:

> Hi all
> 
> I have had some success with the new blktap support in Xen 3.0.3. In 
> particular, the loopback drop-in, tap:aio: works somewhat reliably.
> 
> However, I am really interested in copy-on-write support, and hence have 
> been trying tap:qcow:. This has had limited success, in that dom-U's 
> appear to launch. However, as soon as I try to console or ssh into the 
> dom-U's they apparently freeze, and I can get no further with them.
> 
> I have managed to 'xm terminate' the frozen dom-U's, but this leaves 
> xend in some inconsistent state. All further 'xm xxx' commands barf with 
> the following error:
> 
> [root@dom0]# xm list
> Error: Device 2050 not connected
> Usage: xm list [options] [Domain, ...]
> 
> Curiously, it seems that device 2050 was a secondary block device 
> exposed using phy:.
> 
> So, any advice on how to debug this?

Could we see your /var/log/xen/*  and the output of xenstore-ls?

Thanks,

Ewan.

Re: [Xen-users] Re: tap:qcow causes dom-U to hang in 3.0.3

[Roland Paterson-Jones]

[-- Attachment #1: Type: text/plain, Size: 8616 bytes --]

Ewan Mellor wrote:

>On Wed, Nov 08, 2006 at 04:50:43PM +0200, Roland Paterson-Jones wrote:
>  
>
>>
>>[root@dom0]# xm list
>>Error: Device 2050 not connected
>>Usage: xm list [options] [Domain, ...]
>>
>>Curiously, it seems that device 2050 was a secondary block device 
>>exposed using phy:.
>>
>>So, any advice on how to debug this?
>>    
>>
>
>Could we see your /var/log/xen/*  and the output of xenstore-ls?
>  
>
Logs attached. xenstore-ls below.

I had two domains running. One with root-fs using 
tap:aio:/mnt/instance_image_store_1/3811, and a second dom-U on 
tap:qcow:/mnt/instance_image_store_0/3811.qcow, which was set up as a 
COW overlay of image file /mnt/instance_image_store_0/3811 (not the same 
as the tap:aio: image for the first domU - the directories differ in 
their last digit). 

Then both dom-U's also had/have a swap partition (vbd phy:) on 
/dev/VolGroupDomU/instance_swap_store_1/0 and a second vbd phy: on 
/dev/VolGroupDomU/instance_ephemeral_store_1/0.

The 'xenstore-ls' also shows qcow:/mnt/instance_image_store_2/3811.qcow 
which I don't think should be there at all (maybe from a previous failed 
attempt).

Could this be related to tap:qcow: mounts not umounting? I previously 
set up a test tap:qcow: device in dom0 using 'block-attach 0 ...', then 
mounted it, which seemed to work OK. However 'umount ...' then hung 
(seemingly in the kernel - I had to reboot to get rid of it).

Thx Roland

-----------------------------------------
[root@dom0 ~]# xenstore-ls
tool = ""
 xenstored = ""
vm = ""
 00000000-0000-0000-0000-000000000000 = ""
  shadow_memory = "0"
  uuid = "00000000-0000-0000-0000-000000000000"
  on_reboot = "restart"
  on_poweroff = "destroy"
  name = "Domain-0"
  xend = ""
   restart_count = "0"
  vcpus = "2"
  vcpu_avail = "3"
  memory = "1023"
  on_crash = "restart"
  maxmem = "1023"
 00000000-0000-0000-0000-0000ec291715 = ""
  image = "(linux (kernel /boot/vmlinuz-2.6.16.29-xen) (ramdisk 
/boot/initrd-2.6.16.29-xen.img) (..."
   ostype = "linux"
   kernel = "/boot/vmlinuz-2.6.16.29-xen"
   cmdline = " root=/dev/sda1 ro 4"
   ramdisk = "/boot/initrd-2.6.16.29-xen.img"
  shadow_memory = "0"
  uuid = "00000000-0000-0000-0000-0000ec291715"
  on_reboot = "restart"
  start_time = "1162994682.56"
  on_poweroff = "destroy"
  name = "dom_91715"
  xend = ""
   restart_count = "0"
  vcpus = "1"
  vcpu_avail = "1"
  memory = "1700"
  on_crash = "restart"
  maxmem = "1700"
local = ""
 domain = ""
  0 = ""
   cpu = ""
    0 = ""
     availability = "online"
    1 = ""
     availability = "online"
   memory = ""
    target = "1047552"
   name = "Domain-0"
   console = ""
    limit = "1048576"
   vm = "/vm/00000000-0000-0000-0000-000000000000"
   domid = "0"
   backend = ""
    tap = ""
     1 = ""
      2049 = ""
       domain = "dom_91715"
       frontend = "/local/domain/1/device/vbd/2049"
       dev = "sda1"
       state = "4"
       params = "aio:/mnt/instance_image_store_1/3811"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "tap"
       sectors = "3123200"
       sector-size = "512"
       info = "0"
       hotplug-status = "connected"
     2 = ""
      2049 = ""
       domain = "dom_91721"
       frontend = "/local/domain/2/device/vbd/2049"
       dev = "sda1"
       state = "4"
       params = "qcow:/mnt/instance_image_store_0/3811.qcow"
       mode = "w"
       online = "1"
       frontend-id = "2"
       type = "tap"
       sectors = "3123200"
       sector-size = "512"
       info = "0"
       hotplug-status = "connected"
     3 = ""
      2049 = ""
       domain = "dom_91723"
       frontend = "/local/domain/3/device/vbd/2049"
       dev = "sda1"
       state = "1"
       params = "qcow:/mnt/instance_image_store_2/3811.qcow"
       mode = "w"
       online = "1"
       frontend-id = "3"
       type = "tap"
       sectors = "3123200"
       sector-size = "512"
       info = "0"
    vbd = ""
     1 = ""
      2050 = ""
       domain = "dom_91715"
       frontend = "/local/domain/1/device/vbd/2050"
       dev = "sda2"
       state = "4"
       params = "/dev/VolGroupDomU/instance_ephemeral_store_1"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "phy"
       physical-device = "fd:6"
       hotplug-status = "connected"
       sectors = "312737792"
       info = "0"
       sector-size = "512"
      2051 = ""
       domain = "dom_91715"
       frontend = "/local/domain/1/device/vbd/2051"
       dev = "sda3"
       state = "4"
       params = "/dev/VolGroupDomU/instance_swap_store_1"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "phy"
       physical-device = "fd:7"
       hotplug-status = "connected"
       sectors = "1835008"
       info = "0"
       sector-size = "512"
     2 = ""
      2050 = ""
       domain = "dom_91721"
       frontend = "/local/domain/2/device/vbd/2050"
       dev = "sda2"
       state = "4"
       params = "/dev/VolGroupDomU/instance_ephemeral_store_0"
       mode = "w"
       online = "1"
       frontend-id = "2"
       type = "phy"
       physical-device = "fd:3"
       hotplug-status = "connected"
       sectors = "312737792"
       info = "0"
       sector-size = "512"
      2051 = ""
       domain = "dom_91721"
       frontend = "/local/domain/2/device/vbd/2051"
       dev = "sda3"
       state = "4"
       params = "/dev/VolGroupDomU/instance_swap_store_0"
       mode = "w"
       online = "1"
       frontend-id = "2"
       type = "phy"
       physical-device = "fd:4"
       hotplug-status = "connected"
       sectors = "1835008"
       info = "0"
       sector-size = "512"
     3 = ""
      2050 = ""
       domain = "dom_91723"
       frontend = "/local/domain/3/device/vbd/2050"
       dev = "sda2"
       state = "1"
       params = "/dev/VolGroupDomU/instance_ephemeral_store_2"
       mode = "w"
       online = "1"
       frontend-id = "3"
       type = "phy"
      2051 = ""
       domain = "dom_91723"
       frontend = "/local/domain/3/device/vbd/2051"
       dev = "sda3"
       state = "1"
       params = "/dev/VolGroupDomU/instance_swap_store_2"
       mode = "w"
       online = "1"
       frontend-id = "3"
       type = "phy"
    vif = ""
     1 = ""
      0 = ""
       domain = "dom_91715"
       handle = "0"
       script = "/etc/xen/scripts/vif-aes"
       state = "4"
       frontend = "/local/domain/1/device/vif/0"
       mac = "12:31:34:00:03:8F"
       online = "1"
       frontend-id = "1"
       feature-sg = "1"
       feature-gso-tcpv4 = "1"
       feature-rx-copy = "1"
       hotplug-status = "connected"
     3 = ""
      0 = ""
       domain = "dom_91723"
       handle = "0"
       script = "/etc/xen/scripts/vif-aes"
       state = "5"
       frontend = "/local/domain/3/device/vif/0"
       mac = "12:31:34:00:03:8D"
       online = "0"
       frontend-id = "3"
   error = ""
    backend = ""
     tap = ""
      1 = ""
       2049 = ""
        error = "2 getting info"
      2 = ""
       2049 = ""
        error = "2 getting info"
  1 = ""
   device = ""
    vbd = ""
     2049 = ""
      backend-id = "0"
      virtual-device = "2049"
      device-type = "disk"
      state = "4"
      backend = "/local/domain/0/backend/tap/1/2049"
      ring-ref = "8"
      event-channel = "6"
     2050 = ""
      backend-id = "0"
      virtual-device = "2050"
      device-type = "disk"
      state = "4"
      backend = "/local/domain/0/backend/vbd/1/2050"
      ring-ref = "9"
      event-channel = "7"
     2051 = ""
      backend-id = "0"
      virtual-device = "2051"
      device-type = "disk"
      state = "4"
      backend = "/local/domain/0/backend/vbd/1/2051"
      ring-ref = "10"
      event-channel = "8"
    vif = ""
     0 = ""
      backend-id = "0"
      mac = "12:31:34:00:03:8F"
      handle = "0"
      state = "4"
      backend = "/local/domain/0/backend/vif/1/0"
      tx-ring-ref = "523"
      rx-ring-ref = "524"
      event-channel = "9"
      request-rx-copy = "0"
      feature-rx-notify = "1"
      feature-sg = "1"
      feature-gso-tcpv4 = "1"
   device-misc = ""
    vif = ""
     nextDeviceID = "1"
   console = ""
    ring-ref = "995310"
    port = "2"
    limit = "1048576"
    tty = "/dev/pts/2"
   name = "dom_91715"
   vm = "/vm/00000000-0000-0000-0000-0000ec291715"
   domid = "1"
   cpu = ""
    0 = ""
     availability = "online"
   memory = ""
    target = "1740800"
   store = ""
    ring-ref = "995311"
    port = "1"


>Thanks,
>
>Ewan.
>
>_______________________________________________
>Xen-users mailing list
>Xen-users@lists.xensource.com
>http://lists.xensource.com/xen-users
>
>
>  
>


[-- Attachment #2: xen-logs.tgz --]
[-- Type: application/x-compressed-tar, Size: 6908 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [Xen-users] Re: tap:qcow causes dom-U to hang in 3.0.3

[Julian Chesterfield]

Roland,

Can you also verify whether there's an active tapdisk process running 
in Dom0 for each tap:{aio,qcow} vbd. We are aware of a bug with the 
qcow implementation that we hope to submit a fix for very soon. It's 
likely that you are seeing the same issue.

- Julian

On 9 Nov 2006, at 07:58, Roland Paterson-Jones wrote:

> Ewan Mellor wrote:
>
>> On Wed, Nov 08, 2006 at 04:50:43PM +0200, Roland Paterson-Jones wrote:
>>
>>>
>>> [root@dom0]# xm list
>>> Error: Device 2050 not connected
>>> Usage: xm list [options] [Domain, ...]
>>>
>>> Curiously, it seems that device 2050 was a secondary block device 
>>> exposed using phy:.
>>>
>>> So, any advice on how to debug this?
>>>
>>
>> Could we see your /var/log/xen/*  and the output of xenstore-ls?
>>
> Logs attached. xenstore-ls below.
>
> I had two domains running. One with root-fs using 
> tap:aio:/mnt/instance_image_store_1/3811, and a second dom-U on 
> tap:qcow:/mnt/instance_image_store_0/3811.qcow, which was set up as a 
> COW overlay of image file /mnt/instance_image_store_0/3811 (not the 
> same as the tap:aio: image for the first domU - the directories differ 
> in their last digit).
> Then both dom-U's also had/have a swap partition (vbd phy:) on 
> /dev/VolGroupDomU/instance_swap_store_1/0 and a second vbd phy: on 
> /dev/VolGroupDomU/instance_ephemeral_store_1/0.
>
> The 'xenstore-ls' also shows 
> qcow:/mnt/instance_image_store_2/3811.qcow which I don't think should 
> be there at all (maybe from a previous failed attempt).
>
> Could this be related to tap:qcow: mounts not umounting? I previously 
> set up a test tap:qcow: device in dom0 using 'block-attach 0 ...', 
> then mounted it, which seemed to work OK. However 'umount ...' then 
> hung (seemingly in the kernel - I had to reboot to get rid of it).
>
> Thx Roland
>
> -----------------------------------------
> [root@dom0 ~]# xenstore-ls
> tool = ""
> xenstored = ""
> vm = ""
> 00000000-0000-0000-0000-000000000000 = ""
>  shadow_memory = "0"
>  uuid = "00000000-0000-0000-0000-000000000000"
>  on_reboot = "restart"
>  on_poweroff = "destroy"
>  name = "Domain-0"
>  xend = ""
>   restart_count = "0"
>  vcpus = "2"
>  vcpu_avail = "3"
>  memory = "1023"
>  on_crash = "restart"
>  maxmem = "1023"
> 00000000-0000-0000-0000-0000ec291715 = ""
>  image = "(linux (kernel /boot/vmlinuz-2.6.16.29-xen) (ramdisk 
> /boot/initrd-2.6.16.29-xen.img) (..."
>   ostype = "linux"
>   kernel = "/boot/vmlinuz-2.6.16.29-xen"
>   cmdline = " root=/dev/sda1 ro 4"
>   ramdisk = "/boot/initrd-2.6.16.29-xen.img"
>  shadow_memory = "0"
>  uuid = "00000000-0000-0000-0000-0000ec291715"
>  on_reboot = "restart"
>  start_time = "1162994682.56"
>  on_poweroff = "destroy"
>  name = "dom_91715"
>  xend = ""
>   restart_count = "0"
>  vcpus = "1"
>  vcpu_avail = "1"
>  memory = "1700"
>  on_crash = "restart"
>  maxmem = "1700"
> local = ""
> domain = ""
>  0 = ""
>   cpu = ""
>    0 = ""
>     availability = "online"
>    1 = ""
>     availability = "online"
>   memory = ""
>    target = "1047552"
>   name = "Domain-0"
>   console = ""
>    limit = "1048576"
>   vm = "/vm/00000000-0000-0000-0000-000000000000"
>   domid = "0"
>   backend = ""
>    tap = ""
>     1 = ""
>      2049 = ""
>       domain = "dom_91715"
>       frontend = "/local/domain/1/device/vbd/2049"
>       dev = "sda1"
>       state = "4"
>       params = "aio:/mnt/instance_image_store_1/3811"
>       mode = "w"
>       online = "1"
>       frontend-id = "1"
>       type = "tap"
>       sectors = "3123200"
>       sector-size = "512"
>       info = "0"
>       hotplug-status = "connected"
>     2 = ""
>      2049 = ""
>       domain = "dom_91721"
>       frontend = "/local/domain/2/device/vbd/2049"
>       dev = "sda1"
>       state = "4"
>       params = "qcow:/mnt/instance_image_store_0/3811.qcow"
>       mode = "w"
>       online = "1"
>       frontend-id = "2"
>       type = "tap"
>       sectors = "3123200"
>       sector-size = "512"
>       info = "0"
>       hotplug-status = "connected"
>     3 = ""
>      2049 = ""
>       domain = "dom_91723"
>       frontend = "/local/domain/3/device/vbd/2049"
>       dev = "sda1"
>       state = "1"
>       params = "qcow:/mnt/instance_image_store_2/3811.qcow"
>       mode = "w"
>       online = "1"
>       frontend-id = "3"
>       type = "tap"
>       sectors = "3123200"
>       sector-size = "512"
>       info = "0"
>    vbd = ""
>     1 = ""
>      2050 = ""
>       domain = "dom_91715"
>       frontend = "/local/domain/1/device/vbd/2050"
>       dev = "sda2"
>       state = "4"
>       params = "/dev/VolGroupDomU/instance_ephemeral_store_1"
>       mode = "w"
>       online = "1"
>       frontend-id = "1"
>       type = "phy"
>       physical-device = "fd:6"
>       hotplug-status = "connected"
>       sectors = "312737792"
>       info = "0"
>       sector-size = "512"
>      2051 = ""
>       domain = "dom_91715"
>       frontend = "/local/domain/1/device/vbd/2051"
>       dev = "sda3"
>       state = "4"
>       params = "/dev/VolGroupDomU/instance_swap_store_1"
>       mode = "w"
>       online = "1"
>       frontend-id = "1"
>       type = "phy"
>       physical-device = "fd:7"
>       hotplug-status = "connected"
>       sectors = "1835008"
>       info = "0"
>       sector-size = "512"
>     2 = ""
>      2050 = ""
>       domain = "dom_91721"
>       frontend = "/local/domain/2/device/vbd/2050"
>       dev = "sda2"
>       state = "4"
>       params = "/dev/VolGroupDomU/instance_ephemeral_store_0"
>       mode = "w"
>       online = "1"
>       frontend-id = "2"
>       type = "phy"
>       physical-device = "fd:3"
>       hotplug-status = "connected"
>       sectors = "312737792"
>       info = "0"
>       sector-size = "512"
>      2051 = ""
>       domain = "dom_91721"
>       frontend = "/local/domain/2/device/vbd/2051"
>       dev = "sda3"
>       state = "4"
>       params = "/dev/VolGroupDomU/instance_swap_store_0"
>       mode = "w"
>       online = "1"
>       frontend-id = "2"
>       type = "phy"
>       physical-device = "fd:4"
>       hotplug-status = "connected"
>       sectors = "1835008"
>       info = "0"
>       sector-size = "512"
>     3 = ""
>      2050 = ""
>       domain = "dom_91723"
>       frontend = "/local/domain/3/device/vbd/2050"
>       dev = "sda2"
>       state = "1"
>       params = "/dev/VolGroupDomU/instance_ephemeral_store_2"
>       mode = "w"
>       online = "1"
>       frontend-id = "3"
>       type = "phy"
>      2051 = ""
>       domain = "dom_91723"
>       frontend = "/local/domain/3/device/vbd/2051"
>       dev = "sda3"
>       state = "1"
>       params = "/dev/VolGroupDomU/instance_swap_store_2"
>       mode = "w"
>       online = "1"
>       frontend-id = "3"
>       type = "phy"
>    vif = ""
>     1 = ""
>      0 = ""
>       domain = "dom_91715"
>       handle = "0"
>       script = "/etc/xen/scripts/vif-aes"
>       state = "4"
>       frontend = "/local/domain/1/device/vif/0"
>       mac = "12:31:34:00:03:8F"
>       online = "1"
>       frontend-id = "1"
>       feature-sg = "1"
>       feature-gso-tcpv4 = "1"
>       feature-rx-copy = "1"
>       hotplug-status = "connected"
>     3 = ""
>      0 = ""
>       domain = "dom_91723"
>       handle = "0"
>       script = "/etc/xen/scripts/vif-aes"
>       state = "5"
>       frontend = "/local/domain/3/device/vif/0"
>       mac = "12:31:34:00:03:8D"
>       online = "0"
>       frontend-id = "3"
>   error = ""
>    backend = ""
>     tap = ""
>      1 = ""
>       2049 = ""
>        error = "2 getting info"
>      2 = ""
>       2049 = ""
>        error = "2 getting info"
>  1 = ""
>   device = ""
>    vbd = ""
>     2049 = ""
>      backend-id = "0"
>      virtual-device = "2049"
>      device-type = "disk"
>      state = "4"
>      backend = "/local/domain/0/backend/tap/1/2049"
>      ring-ref = "8"
>      event-channel = "6"
>     2050 = ""
>      backend-id = "0"
>      virtual-device = "2050"
>      device-type = "disk"
>      state = "4"
>      backend = "/local/domain/0/backend/vbd/1/2050"
>      ring-ref = "9"
>      event-channel = "7"
>     2051 = ""
>      backend-id = "0"
>      virtual-device = "2051"
>      device-type = "disk"
>      state = "4"
>      backend = "/local/domain/0/backend/vbd/1/2051"
>      ring-ref = "10"
>      event-channel = "8"
>    vif = ""
>     0 = ""
>      backend-id = "0"
>      mac = "12:31:34:00:03:8F"
>      handle = "0"
>      state = "4"
>      backend = "/local/domain/0/backend/vif/1/0"
>      tx-ring-ref = "523"
>      rx-ring-ref = "524"
>      event-channel = "9"
>      request-rx-copy = "0"
>      feature-rx-notify = "1"
>      feature-sg = "1"
>      feature-gso-tcpv4 = "1"
>   device-misc = ""
>    vif = ""
>     nextDeviceID = "1"
>   console = ""
>    ring-ref = "995310"
>    port = "2"
>    limit = "1048576"
>    tty = "/dev/pts/2"
>   name = "dom_91715"
>   vm = "/vm/00000000-0000-0000-0000-0000ec291715"
>   domid = "1"
>   cpu = ""
>    0 = ""
>     availability = "online"
>   memory = ""
>    target = "1740800"
>   store = ""
>    ring-ref = "995311"
>    port = "1"
>
>
>> Thanks,
>>
>> Ewan.
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>>
>>
>>
>
> <xen-logs.tgz>_______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: [Xen-devel] tap:qcow causes dom-U to hang in 3.0.3

[Roland Paterson-Jones]

Julian Chesterfield wrote:

> Can you also verify whether there's an active tapdisk process running 
> in Dom0 for each tap:{aio,qcow} vbd. We are aware of a bug with the 
> qcow implementation that we hope to submit a fix for very soon. It's 
> likely that you are seeing the same issue.

Sorry, this is not what you asked for, but this is what an 'ls' of the  
qcow file shows:

 [root@dom0-0-50-45-5d-6a-bc ~]# ls -als /mnt/instance_image_store_0/
total 1564108
      4 drwxr-xr-x  2 root root          4096 Nov 10 15:42 .
      8 drwxr-xr-x  8 root root          4096 Nov  7 17:56 ..
1563132 -rw-r--r--  1 root root    1599078400 Nov 10 15:42 2
    964 -rw-r--r--  1 root root 1099645846016 Nov 10 15:50 2.qcow

It looks like the qcow file has grown to > 1000GB in (apparent) size(!) 
Could this be the root of the problem?

At this stage the dom-U is still running, but the qcow file is clearly 
not right. The file called '2' is a loopback image backing the qcow file.

The qcow overlay was created using '/usr/sbin/qcow-create $((10*1024)) 
/mnt/instance_image_store_0/2.qcow /mnt/instance_image_store_0/2'. I'm 
guessing there's no danger in making the qcow overlay extent (10GB) 
larger than the underlying loopback file extend (1.5GB) but please 
correct me if I'm wrong.

Earlier, an ls of the same directory (soon after dom-U creation) looked 
like:

[root@dom0-0-50-45-5d-6a-bc ~]# ls -als /mnt/instance_image_store_0/
total 1564052
      4 drwxr-xr-x  2 root root       4096 Nov 10 15:42 .
      8 drwxr-xr-x  8 root root       4096 Nov  7 17:56 ..
1563132 -rw-r--r--  1 root root 1599078400 Nov 10 15:42 2
    908 -rw-r--r--  1 root root     925696 Nov 10 15:43 2.qcow

Could this be the root of the problem - i.e. something in the qcow 
driver is getting it's offsets in the qcow file mangled?

Can I try to compile the qcow (tapdisk) driver with debug enabled? Where 
would the output go?

Regards
Roland

Re: [Xen-users] Re: tap:qcow causes dom-U to hang in 3.0.3

[Julian Chesterfield]

On 10 Nov 2006, at 14:00, Roland Paterson-Jones wrote:

> Julian Chesterfield wrote:
>
>> Can you also verify whether there's an active tapdisk process running 
>> in Dom0 for each tap:{aio,qcow} vbd. We are aware of a bug with the 
>> qcow implementation that we hope to submit a fix for very soon. It's 
>> likely that you are seeing the same issue.
>
> Sorry, this is not what you asked for, but this is what an 'ls' of the 
>  qcow file shows:
>
> [root@dom0-0-50-45-5d-6a-bc ~]# ls -als /mnt/instance_image_store_0/
> total 1564108
>      4 drwxr-xr-x  2 root root          4096 Nov 10 15:42 .
>      8 drwxr-xr-x  8 root root          4096 Nov  7 17:56 ..
> 1563132 -rw-r--r--  1 root root    1599078400 Nov 10 15:42 2
>    964 -rw-r--r--  1 root root 1099645846016 Nov 10 15:50 2.qcow
>
> It looks like the qcow file has grown to > 1000GB in (apparent) 
> size(!) Could this be the root of the problem?

Hmm, this shoudn't happen. Looks like the block offset lookup is 
screwed up.

>
> At this stage the dom-U is still running, but the qcow file is clearly 
> not right. The file called '2' is a loopback image backing the qcow 
> file.
>
> The qcow overlay was created using '/usr/sbin/qcow-create $((10*1024)) 
> /mnt/instance_image_store_0/2.qcow /mnt/instance_image_store_0/2'. I'm 
> guessing there's no danger in making the qcow overlay extent (10GB) 
> larger than the underlying loopback file extend (1.5GB) but please 
> correct me if I'm wrong.

So the current overlay support actually ignores the size value and 
creates a sparse overlay file with the same virtual size as the 
original. Technically it would be possible to create an overlay file 
larger than the source and fault read/writes for larger block offsets 
to the overlay disk, however I'm not certain this is a good idea.

>
> Earlier, an ls of the same directory (soon after dom-U creation) 
> looked like:
>
> [root@dom0-0-50-45-5d-6a-bc ~]# ls -als /mnt/instance_image_store_0/
> total 1564052
>      4 drwxr-xr-x  2 root root       4096 Nov 10 15:42 .
>      8 drwxr-xr-x  8 root root       4096 Nov  7 17:56 ..
> 1563132 -rw-r--r--  1 root root 1599078400 Nov 10 15:42 2
>    908 -rw-r--r--  1 root root     925696 Nov 10 15:43 2.qcow
>
> Could this be the root of the problem - i.e. something in the qcow 
> driver is getting it's offsets in the qcow file mangled?
>
> Can I try to compile the qcow (tapdisk) driver with debug enabled? 
> Where would the output go?
There's a bunch of DPRINTFs in the code which send debug output to 
syslog. You can play with syslog settings to switch on the debug 
messages and direct them to a log file.

Thanks for the file size tip, I'll explore further.

Thanks,
Julian

>
> Regards
> Roland
>

Re: [Xen-users] Re: tap:qcow causes dom-U to hang in 3.0.3

[Roland Paterson-Jones]

Julian Chesterfield wrote:

>
> There's a bunch of DPRINTFs in the code which send debug output to 
> syslog. You can play with syslog settings to switch on the debug 
> messages and direct them to a log file.
>
Here is another clue from syslog. Below is the full log, but the hang 
seems to accur when the following log messages are triggered (there's a 
bunch of them at the end):

Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869040]

I'm a little frustrated with re-building the blktap user-space tools 
myself. When I do so, I get the following kernel messages:

Nov 13 13:44:08 dom0-0-50-45-5d-6a-bc kernel: blk_tap: invalid user 
buffer -- could not remap it
Nov 13 13:44:08 dom0-0-50-45-5d-6a-bc kernel: blk_tap: Reached Fail_flush
Nov 13 13:44:08 dom0-0-50-45-5d-6a-bc kernel: blk_tap: BLKTAP_INVALID_HANDLE

But I don't get these when I use the (PAE) binary distribution. I'm only 
rebuilding the blktap tools, not the rest, so perhaps my config is not 
right (incompatible with the PAE kernel somehow). If you know off-hand 
what the problem is then I can maybe chase down the offset issue more 
productively.

Regards
Roland

Full debug syslog of qcow creation and dom-U launch using tap:qcow root FS:

Nov 14 11:56:26 dom0-0-50-45-5d-6a-bc qcow-create: Qcow_create: size 
10737418240
Nov 14 11:56:26 dom0-0-50-45-5d-6a-bc qcow-create: Backing file size 
detected: 3123200 sectors(total 1599078400 [1525 MB])
Nov 14 11:56:26 dom0-0-50-45-5d-6a-bc qcow-create: L1 Table offset: 
4096, size 6104
Nov 14 11:56:26 dom0-0-50-45-5d-6a-bc qcow-create: Adjusted filelength 
to 12288 for 4 Kbyte alignment
Nov 14 11:56:26 dom0-0-50-45-5d-6a-bc qcow-create: QCOW file 
successfully created
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Received a poll for a 
new vbd
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: /dev/xen/blktap1 
device already exists
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Received device id 1 
and major 254, sent domid 3 and be_id 2049
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Detected handle: [qcow]
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Process does not exist:
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Launching process, 
CMDLINE [tapdisk /dev/xen/tapctrlwrite1 /dev/xen/tapctrlread1]
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Write_msg called: 
CTLMSG_PID
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc TAPDISK: Tapdisk: Received msg, 
len 8, type 9, UID 13691
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Received CTLMSG_PID_RSP
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL:       PID: [21862]
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Write_msg called: 
CTLMSG_PARAMS, sending [qcow:/mnt/instance_image_store_0/2.qcow, 
/mnt/instance_image_store_0/2.qcow]
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Generated cookie, 13691
Nov 14 11:56:27 dom0-0-50-45-5d-6a-bc TAPDISK: Tapdisk: Received msg, 
len 43, type 1, UID 13691
Nov 14 11:56:28 dom0-0-50-45-5d-6a-bc kernel: tap tap-3-2049: 2 getting info
Nov 14 11:56:28 dom0-0-50-45-5d-6a-bc TAPDISK: Received CTLMSG_PARAMS: 
[/mnt/instance_image_store_0/2.qcow]
Nov 14 11:56:28 dom0-0-50-45-5d-6a-bc TAPDISK: Loaded driver: name 
[tapdisk_qcow], type [4]
Nov 14 11:56:28 dom0-0-50-45-5d-6a-bc TAPDISK: QCOW: Opening 
/mnt/instance_image_store_0/2.qcow
Nov 14 11:56:29 dom0-0-50-45-5d-6a-bc TAPDISK: L1 Table offset detected: 
4096, size 6104 (8192)
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc TAPDISK: Reading backing file data
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc TAPDISK: AIO state initialised
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc TAPDISK: Adding fd_list_entry
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc TAPDISK: Entered cookie 13691
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Received CTLMSG_IMG: 
3123200, 512, 0
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Received a poll for a 
new devmap
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/block: 
add XENBUS_PATH=backend/vbd/3/2050
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/block: 
add XENBUS_PATH=backend/vbd/3/2051
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/blktap: 
add XENBUS_PATH=backend/tap/3/2049
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/vif-aes: 
online XENBUS_PATH=backend/vif/3/0
Nov 14 11:56:30 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Write_msg called: 
CTLMSG_NEWDEV
Nov 14 11:56:31 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/blktap: 
Writing backend/tap/3/2049/hotplug-status connected to xenstore.
Nov 14 11:56:31 dom0-0-50-45-5d-6a-bc TAPDISK: Tapdisk: Received msg, 
len 12, type 4, UID 13691
Nov 14 11:56:31 dom0-0-50-45-5d-6a-bc TAPDISK: Retrieving state, cookie 
13691.....[OK]
Nov 14 11:56:31 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Received CTLMSG_NEWDEV_RSP
Nov 14 11:56:31 dom0-0-50-45-5d-6a-bc BLKTAPCTRL: Exiting map_new_blktapctrl
Nov 14 11:56:31 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/block: 
Writing backend/vbd/3/2051/physical-device fd:4 to xenstore.
Nov 14 11:56:31 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/block: 
Writing backend/vbd/3/2051/hotplug-status connected to xenstore.
Nov 14 11:56:32 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/block: 
Writing backend/vbd/3/2050/physical-device fd:3 to xenstore.
Nov 14 11:56:32 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/block: 
Writing backend/vbd/3/2050/hotplug-status connected to xenstore.
Nov 14 11:56:32 dom0-0-50-45-5d-6a-bc kernel: ADDRCONF(NETDEV_UP): 
vifI93276.0: link is not ready
Nov 14 11:56:33 dom0-0-50-45-5d-6a-bc logger: /etc/xen/scripts/vif-aes: 
Writing backend/vif/3/0/hotplug-status connected to xenstore.
Nov 14 11:56:50 dom0-0-50-45-5d-6a-bc kernel: ADDRCONF(NETDEV_CHANGE): 
vifI93276.0: link becomes ready
Nov 14 11:57:01 dom0-0-50-45-5d-6a-bc kernel: vifI93276.0: no IPv6 
routers present
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869040]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869048]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869056]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869064]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869072]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869080]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869088]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869096]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869104]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869112]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869120]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869128]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869136]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869144]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869152]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869160]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869168]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869176]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869184]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869192]
Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
iocb_free_count (0) failed[869200]

Re: Re: [Xen-devel] tap:qcow causes dom-U to hang in 3.0.3

[Roland Paterson-Jones]

Roland Paterson-Jones wrote:

> Julian Chesterfield wrote:
>
>>
>> There's a bunch of DPRINTFs in the code which send debug output to 
>> syslog. You can play with syslog settings to switch on the debug 
>> messages and direct them to a log file.
>>
> Here is another clue from syslog. Below is the full log, but the hang 
> seems to accur when the following log messages are triggered (there's 
> a bunch of them at the end):
>
> Nov 14 11:59:47 dom0-0-50-45-5d-6a-bc TAPDISK: AIO_LOCK or 
> iocb_free_count (0) failed[869040]
>
And when I try to 'xm destroy' the instance after it hangs, I get a slew 
of the following :

Nov 14 12:41:35 dom0-0-50-45-5d-6a-bc kernel: blk_tap: invalid kernel 
buffer -- could not remap it
Nov 14 12:41:35 dom0-0-50-45-5d-6a-bc kernel: blk_tap: Reached Fail_flush
Nov 14 12:41:35 dom0-0-50-45-5d-6a-bc kernel: blk_tap: BLKTAP_INVALID_HANDLE

This is the inverse of the error messages I see when I build the 
user-land tools myself. In the above it's the kernel buffer that can't 
be re-mapped. With my user-land tools it's the user buffer that can't be 
remapped. I wonder (but have no idea) whether this is related to PAE at all.

Roland

tap:qcow causes dom-U to hang in 3.0.4-rc1

[Hiromichi Itou]

Hi,

This problem had been discussed on this ML before.
Please refer to following URL.
	http://lists.xensource.com/archives/html/xen-devel/2006-11/ 
msg00391.html

This problem still remains in Xen 3.0.4-rc1.　(I tested on  
x86-64.)

Steps to Reproduce:

1) Create bootable DomainU image.
	dd if=/dev/zero of=/var/images/storage.disk1.sys1 bs=1024k count=1  
seek=8191
	  etc.

2) Create qcow file that has backing file.
	qcow-create 100 /var/images/qcowbak.img /var/images/storage.disk1.sys1

3) Boot DomainU that has below config.
	disk = [ 'tap:qcow:/var/images/qcowbak.img.img,hda,w' ]


I plan to start the investigation of the cause of this problem.

Has anyone already solved this problem?


Hiromichi Ito
Director of Engineering Department
---
Begi.net (http://Begi.net)

Re: tap:qcow causes dom-U to hang in 3.0.4-rc1

[Julian Chesterfield]

On 15/12/06 12:24 am, "Hiromichi Itou" <ito@begi.net> wrote:

> 
> 2) Create qcow file that has backing file.
> qcow-create 100 /var/images/qcowbak.img /var/images/storage.disk1.sys1
> 
> 3) Boot DomainU that has below config.
> disk = [ 'tap:qcow:/var/images/qcowbak.img.img,hda,w' ]
> 
Is 'qcowbak.img.img' a typo? By the looks of step 2, it should read
'qcowbak.img'

- Julian

Re: tap:qcow causes dom-U to hang in 3.0.4-rc1

[Hiromichi Itou]

Hi,

On 2006/12/20, at 7:32, Julian Chesterfield wrote:
> Is 'qcowbak.img.img' a typo? By the looks of step 2, it should read
> 'qcowbak.img'

Yes, it is a typo.
a correct step 3 is as follows.

3) Boot DomainU that has below config.
  disk = [ 'tap:qcow:/var/images/qcowbak.img,hda,w' ]


Hiromichi Ito

Re: Re: [Xen-devel] tap:qcow causes dom-U to hang in 3.0.3

[Roland Paterson-Jones]

Julian Chesterfield wrote:

> Roland,
>
> Can you also verify whether there's an active tapdisk process running 
> in Dom0 for each tap:{aio,qcow} vbd. We are aware of a bug with the 
> qcow implementation that we hope to submit a fix for very soon. It's 
> likely that you are seeing the same issue.

To answer your question, yes, it does appear that a tapdisk process is 
still running (this is after the dom-U has hung):

[root@dom0-0-50-45-5d-6a-bc ~]# ps -aef | grep tapdisk
root      4135     1  0 15:42 ?        00:00:01 tapdisk 
/dev/xen/tapctrlwrite1 /dev/xen/tapctrlread1

There is only one tap device, and the pid is the same as the single 
candidate while the dom-U was still reachable.

The hand seems to occur on the first (significant?) disk write inside 
the dom-U. For example:

-bash-3.00# dd if=/dev/zero of=./test-10MB bs=1k count=$((10*1024))

Has hung the dom-U, and I can no longer console or ssh into the dom-U.

Interestingly, on the dom-U, the qcow file has shrunk from its pervious 
peak of > 1TB, and is now appearing modestly as:

[root@dom0-0-50-45-5d-6a-bc ~]# ls -als /mnt/instance_image_store_0/
total 1564432
      4 drwxr-xr-x  2 root root       4096 Nov 10 15:42 .
      8 drwxr-xr-x  8 root root       4096 Nov  7 17:56 ..
1563132 -rw-r--r--  1 root root 1599078400 Nov 10 15:42 2
   1288 -rw-r--r--  1 root root    2466816 Nov 10 16:09 2.qcow

It's all very confusing. I'd love it to work, of course. Let me know 
what I can do to help with a diagnosis.

I'm running on the (binary) PAE-enabled 3.0.3 release.

Thanks and kind regards
Roland

Re: [Xen-users] Re: tap:qcow causes dom-U to hang in 3.0.3

[Roland Paterson-Jones]

[-- Attachment #1: Type: text/plain, Size: 8701 bytes --]

Ewan Mellor wrote:

> On Wed, Nov 08, 2006 at 04:50:43PM +0200, Roland Paterson-Jones wrote:
>  
>
>>
>> [root@dom0]# xm list
>> Error: Device 2050 not connected
>> Usage: xm list [options] [Domain, ...]
>>
>> Curiously, it seems that device 2050 was a secondary block device 
>> exposed using phy:.
>>
>> So, any advice on how to debug this?
>>   
>
>
> Could we see your /var/log/xen/*  and the output of xenstore-ls?
>  
>
Not sure this got through first time round - I got 'quarantine' messages 
about the .tgz attachment. Attached here uncompressed (!)

Logs attached. xenstore-ls below.

I had two domains running. One with root-fs using 
tap:aio:/mnt/instance_image_store_1/3811, and a second dom-U on 
tap:qcow:/mnt/instance_image_store_0/3811.qcow, which was set up as a 
COW overlay of image file /mnt/instance_image_store_0/3811 (not the same 
as the tap:aio: image for the first domU - the directories differ in 
their last digit).
Then both dom-U's also had/have a swap partition (vbd phy:) on 
/dev/VolGroupDomU/instance_swap_store_1/0 and a second vbd phy: on 
/dev/VolGroupDomU/instance_ephemeral_store_1/0.

The 'xenstore-ls' also shows qcow:/mnt/instance_image_store_2/3811.qcow 
which I don't think should be there at all (maybe from a previous failed 
attempt).

Could this be related to tap:qcow: mounts not umounting? I previously 
set up a test tap:qcow: device in dom0 using 'block-attach 0 ...', then 
mounted it, which seemed to work OK. However 'umount ...' then hung 
(seemingly in the kernel - I had to reboot to get rid of it).

Thx Roland

-----------------------------------------
[root@dom0 ~]# xenstore-ls
tool = ""
xenstored = ""
vm = ""
00000000-0000-0000-0000-000000000000 = ""
 shadow_memory = "0"
 uuid = "00000000-0000-0000-0000-000000000000"
 on_reboot = "restart"
 on_poweroff = "destroy"
 name = "Domain-0"
 xend = ""
  restart_count = "0"
 vcpus = "2"
 vcpu_avail = "3"
 memory = "1023"
 on_crash = "restart"
 maxmem = "1023"
00000000-0000-0000-0000-0000ec291715 = ""
 image = "(linux (kernel /boot/vmlinuz-2.6.16.29-xen) (ramdisk 
/boot/initrd-2.6.16.29-xen.img) (..."
  ostype = "linux"
  kernel = "/boot/vmlinuz-2.6.16.29-xen"
  cmdline = " root=/dev/sda1 ro 4"
  ramdisk = "/boot/initrd-2.6.16.29-xen.img"
 shadow_memory = "0"
 uuid = "00000000-0000-0000-0000-0000ec291715"
 on_reboot = "restart"
 start_time = "1162994682.56"
 on_poweroff = "destroy"
 name = "dom_91715"
 xend = ""
  restart_count = "0"
 vcpus = "1"
 vcpu_avail = "1"
 memory = "1700"
 on_crash = "restart"
 maxmem = "1700"
local = ""
domain = ""
 0 = ""
  cpu = ""
   0 = ""
    availability = "online"
   1 = ""
    availability = "online"
  memory = ""
   target = "1047552"
  name = "Domain-0"
  console = ""
   limit = "1048576"
  vm = "/vm/00000000-0000-0000-0000-000000000000"
  domid = "0"
  backend = ""
   tap = ""
    1 = ""
     2049 = ""
      domain = "dom_91715"
      frontend = "/local/domain/1/device/vbd/2049"
      dev = "sda1"
      state = "4"
      params = "aio:/mnt/instance_image_store_1/3811"
      mode = "w"
      online = "1"
      frontend-id = "1"
      type = "tap"
      sectors = "3123200"
      sector-size = "512"
      info = "0"
      hotplug-status = "connected"
    2 = ""
     2049 = ""
      domain = "dom_91721"
      frontend = "/local/domain/2/device/vbd/2049"
      dev = "sda1"
      state = "4"
      params = "qcow:/mnt/instance_image_store_0/3811.qcow"
      mode = "w"
      online = "1"
      frontend-id = "2"
      type = "tap"
      sectors = "3123200"
      sector-size = "512"
      info = "0"
      hotplug-status = "connected"
    3 = ""
     2049 = ""
      domain = "dom_91723"
      frontend = "/local/domain/3/device/vbd/2049"
      dev = "sda1"
      state = "1"
      params = "qcow:/mnt/instance_image_store_2/3811.qcow"
      mode = "w"
      online = "1"
      frontend-id = "3"
      type = "tap"
      sectors = "3123200"
      sector-size = "512"
      info = "0"
   vbd = ""
    1 = ""
     2050 = ""
      domain = "dom_91715"
      frontend = "/local/domain/1/device/vbd/2050"
      dev = "sda2"
      state = "4"
      params = "/dev/VolGroupDomU/instance_ephemeral_store_1"
      mode = "w"
      online = "1"
      frontend-id = "1"
      type = "phy"
      physical-device = "fd:6"
      hotplug-status = "connected"
      sectors = "312737792"
      info = "0"
      sector-size = "512"
     2051 = ""
      domain = "dom_91715"
      frontend = "/local/domain/1/device/vbd/2051"
      dev = "sda3"
      state = "4"
      params = "/dev/VolGroupDomU/instance_swap_store_1"
      mode = "w"
      online = "1"
      frontend-id = "1"
      type = "phy"
      physical-device = "fd:7"
      hotplug-status = "connected"
      sectors = "1835008"
      info = "0"
      sector-size = "512"
    2 = ""
     2050 = ""
      domain = "dom_91721"
      frontend = "/local/domain/2/device/vbd/2050"
      dev = "sda2"
      state = "4"
      params = "/dev/VolGroupDomU/instance_ephemeral_store_0"
      mode = "w"
      online = "1"
      frontend-id = "2"
      type = "phy"
      physical-device = "fd:3"
      hotplug-status = "connected"
      sectors = "312737792"
      info = "0"
      sector-size = "512"
     2051 = ""
      domain = "dom_91721"
      frontend = "/local/domain/2/device/vbd/2051"
      dev = "sda3"
      state = "4"
      params = "/dev/VolGroupDomU/instance_swap_store_0"
      mode = "w"
      online = "1"
      frontend-id = "2"
      type = "phy"
      physical-device = "fd:4"
      hotplug-status = "connected"
      sectors = "1835008"
      info = "0"
      sector-size = "512"
    3 = ""
     2050 = ""
      domain = "dom_91723"
      frontend = "/local/domain/3/device/vbd/2050"
      dev = "sda2"
      state = "1"
      params = "/dev/VolGroupDomU/instance_ephemeral_store_2"
      mode = "w"
      online = "1"
      frontend-id = "3"
      type = "phy"
     2051 = ""
      domain = "dom_91723"
      frontend = "/local/domain/3/device/vbd/2051"
      dev = "sda3"
      state = "1"
      params = "/dev/VolGroupDomU/instance_swap_store_2"
      mode = "w"
      online = "1"
      frontend-id = "3"
      type = "phy"
   vif = ""
    1 = ""
     0 = ""
      domain = "dom_91715"
      handle = "0"
      script = "/etc/xen/scripts/vif-aes"
      state = "4"
      frontend = "/local/domain/1/device/vif/0"
      mac = "12:31:34:00:03:8F"
      online = "1"
      frontend-id = "1"
      feature-sg = "1"
      feature-gso-tcpv4 = "1"
      feature-rx-copy = "1"
      hotplug-status = "connected"
    3 = ""
     0 = ""
      domain = "dom_91723"
      handle = "0"
      script = "/etc/xen/scripts/vif-aes"
      state = "5"
      frontend = "/local/domain/3/device/vif/0"
      mac = "12:31:34:00:03:8D"
      online = "0"
      frontend-id = "3"
  error = ""
   backend = ""
    tap = ""
     1 = ""
      2049 = ""
       error = "2 getting info"
     2 = ""
      2049 = ""
       error = "2 getting info"
 1 = ""
  device = ""
   vbd = ""
    2049 = ""
     backend-id = "0"
     virtual-device = "2049"
     device-type = "disk"
     state = "4"
     backend = "/local/domain/0/backend/tap/1/2049"
     ring-ref = "8"
     event-channel = "6"
    2050 = ""
     backend-id = "0"
     virtual-device = "2050"
     device-type = "disk"
     state = "4"
     backend = "/local/domain/0/backend/vbd/1/2050"
     ring-ref = "9"
     event-channel = "7"
    2051 = ""
     backend-id = "0"
     virtual-device = "2051"
     device-type = "disk"
     state = "4"
     backend = "/local/domain/0/backend/vbd/1/2051"
     ring-ref = "10"
     event-channel = "8"
   vif = ""
    0 = ""
     backend-id = "0"
     mac = "12:31:34:00:03:8F"
     handle = "0"
     state = "4"
     backend = "/local/domain/0/backend/vif/1/0"
     tx-ring-ref = "523"
     rx-ring-ref = "524"
     event-channel = "9"
     request-rx-copy = "0"
     feature-rx-notify = "1"
     feature-sg = "1"
     feature-gso-tcpv4 = "1"
  device-misc = ""
   vif = ""
    nextDeviceID = "1"
  console = ""
   ring-ref = "995310"
   port = "2"
   limit = "1048576"
   tty = "/dev/pts/2"
  name = "dom_91715"
  vm = "/vm/00000000-0000-0000-0000-0000ec291715"
  domid = "1"
  cpu = ""
   0 = ""
    availability = "online"
  memory = ""
   target = "1740800"
  store = ""
   ring-ref = "995311"
   port = "1"


> Thanks,
>
> Ewan.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>
>  
>

------------------------------------------------------------------------

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel


[-- Attachment #2: xend-debug.log --]
[-- Type: text/x-log, Size: 36 bytes --]

Nothing to flush.
Nothing to flush.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: xend.log --]
[-- Type: text/x-log; name="xend.log", Size: 88578 bytes --]

[2006-11-07 12:03:33 xend 2677] INFO (SrvDaemon:283) Xend Daemon started
[2006-11-07 12:03:33 xend 2677] INFO (SrvDaemon:287) Xend changeset: Sun Oct 15 13:19:07 2006 +0100 11774:620cb0064d9b.
[2006-11-07 12:03:33 xend.XendDomainInfo 2677] DEBUG (XendDomainInfo:212) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 18030905254L, 'ssidref': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'mem_kb': 1047296, 'maxmem_kb': -4, 'max_vcpu_id': 1, 'crashed': 0, 'running': 1, 'shutdown': 0, 'online_vcpus': 2, 'blocked': 0})
[2006-11-07 12:03:33 xend.XendDomainInfo 2677] INFO (XendDomainInfo:224) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2006-11-07 12:03:33 xend.XendDomainInfo 2677] WARNING (XendDomainInfo:246) No vm path in store for existing domain 0
[2006-11-07 12:03:33 xend.XendDomainInfo 2677] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'on_poweroff': 'destroy', 'name': 'Domain-0', 'xend/restart_count': '0', 'vcpus': '2', 'vcpu_avail': '3', 'memory': '1023', 'on_crash': 'restart', 'maxmem': '1023'}
[2006-11-07 12:03:33 xend.XendDomainInfo 2677] DEBUG (XendDomainInfo:747) Storing domain details: {'cpu/0/availability': 'online', 'memory/target': '1047552', 'cpu/1/availability': 'online', 'name': 'Domain-0', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0'}
[2006-11-07 12:03:33 xend 2677] DEBUG (XendDomain:153) number of vcpus to use is 0
[2006-11-07 12:03:33 xend.XendDomainInfo 2677] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-07 12:14:49 xend 2676] INFO (SrvDaemon:190) Xend stopped due to signal 15.
[2006-11-07 12:24:48 xend 2682] INFO (SrvDaemon:283) Xend Daemon started
[2006-11-07 12:24:48 xend 2682] INFO (SrvDaemon:287) Xend changeset: Sun Oct 15 13:19:07 2006 +0100 11774:620cb0064d9b.
[2006-11-07 12:24:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:212) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 17739491734L, 'ssidref': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'mem_kb': 1047296, 'maxmem_kb': -4, 'max_vcpu_id': 1, 'crashed': 0, 'running': 1, 'shutdown': 0, 'online_vcpus': 2, 'blocked': 0})
[2006-11-07 12:24:49 xend.XendDomainInfo 2682] INFO (XendDomainInfo:224) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2006-11-07 12:24:49 xend.XendDomainInfo 2682] WARNING (XendDomainInfo:246) No vm path in store for existing domain 0
[2006-11-07 12:24:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'on_poweroff': 'destroy', 'name': 'Domain-0', 'xend/restart_count': '0', 'vcpus': '2', 'vcpu_avail': '3', 'memory': '1023', 'on_crash': 'restart', 'maxmem': '1023'}
[2006-11-07 12:24:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:747) Storing domain details: {'cpu/0/availability': 'online', 'memory/target': '1047552', 'cpu/1/availability': 'online', 'name': 'Domain-0', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0'}
[2006-11-07 12:24:49 xend 2682] DEBUG (XendDomain:153) number of vcpus to use is 0
[2006-11-07 12:24:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-07 13:27:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291284'], ['name', 'dom_91284'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['vbd', ['uname', 'file:/mnt/instance_image_store_0/2866'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-07 13:27:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291284'], ['name', 'dom_91284'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['vbd', ['uname', 'file:/mnt/instance_image_store_0/2866'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-07 13:27:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291284', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91284', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('vbd', ['vbd', ['uname', 'file:/mnt/instance_image_store_0/2866'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8E']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-07 13:27:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-07 13:27:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 1 1.0
[2006-11-07 13:27:45 xend 2682] DEBUG (balloon:127) Balloon: 7268224 KiB free; need 1740800; done.
[2006-11-07 13:27:45 xend 2682] INFO (image:136) buildDomain os=linux dom=1 vcpus=1
[2006-11-07 13:27:45 xend 2682] DEBUG (image:191) dom            = 1
[2006-11-07 13:27:45 xend 2682] DEBUG (image:192) image          = /boot/vmlinuz-2.6-xenU
[2006-11-07 13:27:45 xend 2682] DEBUG (image:193) store_evtchn   = 1
[2006-11-07 13:27:45 xend 2682] DEBUG (image:194) console_evtchn = 2
[2006-11-07 13:27:45 xend 2682] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-07 13:27:45 xend 2682] DEBUG (image:196) ramdisk        = 
[2006-11-07 13:27:45 xend 2682] DEBUG (image:197) vcpus          = 1
[2006-11-07 13:27:45 xend 2682] DEBUG (image:198) features       = 
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2049'} to /local/domain/1/device/vbd/2049.
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91284', 'frontend': '/local/domain/1/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': '/mnt/instance_image_store_0/2866', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'file'} to /local/domain/0/backend/vbd/1/2049.
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2050'} to /local/domain/1/device/vbd/2050.
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91284', 'frontend': '/local/domain/1/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'phy'} to /local/domain/0/backend/vbd/1/2050.
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2051'} to /local/domain/1/device/vbd/2051.
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91284', 'frontend': '/local/domain/1/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'phy'} to /local/domain/0/backend/vbd/1/2051.
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8E', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91284', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/1/device/vif/0', 'mac': '12:31:34:00:03:8E', 'online': '1', 'frontend-id': '1'} to /local/domain/0/backend/vif/1/0.
[2006-11-07 13:27:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291284', 'on_reboot': 'restart', 'start_time': '1162898865.35', 'on_poweroff': 'destroy', 'name': 'dom_91284', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6-xenU) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-07 13:27:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '870785', 'console/port': '2', 'name': 'dom_91284', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291284', 'domid': '1', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '870786', 'store/port': '1'}
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-07 13:27:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:149) Waiting for 0.
[2006-11-07 13:27:45 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:143) Waiting for devices usb.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:143) Waiting for devices vbd.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:149) Waiting for 2049.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/1/2049/hotplug-status.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:149) Waiting for 2050.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/1/2050/hotplug-status.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:149) Waiting for 2051.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/1/2051/hotplug-status.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:143) Waiting for devices irq.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:143) Waiting for devices pci.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:143) Waiting for devices ioports.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:143) Waiting for devices tap.
[2006-11-07 13:27:47 xend 2682] DEBUG (DevController:143) Waiting for devices vtpm.
[2006-11-07 13:27:47 xend 2682] INFO (XendDomain:370) Domain dom_91284 (1) unpaused.
[2006-11-07 16:46:26 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-07 16:46:26 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-07 16:46:41 xend.XendDomainInfo 2682] INFO (XendDomainInfo:892) Domain has shutdown: name=dom_91284 id=1 reason=poweroff.
[2006-11-07 16:46:41 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1449) XendDomainInfo.destroy: domid=1
[2006-11-07 16:46:41 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1457) XendDomainInfo.destroyDomain(1)
[2006-11-07 17:03:10 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291365'], ['name', 'dom_91365'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['vbd', ['uname', 'file:/mnt/instance_image_store_1/2866'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_1'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_1'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8F']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-07 17:03:10 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291365'], ['name', 'dom_91365'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['vbd', ['uname', 'file:/mnt/instance_image_store_1/2866'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_1'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_1'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8F']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-07 17:03:10 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291365', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91365', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('vbd', ['vbd', ['uname', 'file:/mnt/instance_image_store_1/2866'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_1'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_1'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8F']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-07 17:03:10 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-07 17:03:10 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 2 1.0
[2006-11-07 17:03:10 xend 2682] DEBUG (balloon:127) Balloon: 7268076 KiB free; need 1740800; done.
[2006-11-07 17:03:10 xend 2682] INFO (image:136) buildDomain os=linux dom=2 vcpus=1
[2006-11-07 17:03:10 xend 2682] DEBUG (image:191) dom            = 2
[2006-11-07 17:03:10 xend 2682] DEBUG (image:192) image          = /boot/vmlinuz-2.6-xenU
[2006-11-07 17:03:10 xend 2682] DEBUG (image:193) store_evtchn   = 1
[2006-11-07 17:03:10 xend 2682] DEBUG (image:194) console_evtchn = 2
[2006-11-07 17:03:10 xend 2682] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-07 17:03:10 xend 2682] DEBUG (image:196) ramdisk        = 
[2006-11-07 17:03:10 xend 2682] DEBUG (image:197) vcpus          = 1
[2006-11-07 17:03:10 xend 2682] DEBUG (image:198) features       = 
[2006-11-07 17:03:10 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/2049'} to /local/domain/2/device/vbd/2049.
[2006-11-07 17:03:10 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91365', 'frontend': '/local/domain/2/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': '/mnt/instance_image_store_1/2866', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'file'} to /local/domain/0/backend/vbd/2/2049.
[2006-11-07 17:03:10 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/2050'} to /local/domain/2/device/vbd/2050.
[2006-11-07 17:03:10 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91365', 'frontend': '/local/domain/2/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_1', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'phy'} to /local/domain/0/backend/vbd/2/2050.
[2006-11-07 17:03:10 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/2051'} to /local/domain/2/device/vbd/2051.
[2006-11-07 17:03:10 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91365', 'frontend': '/local/domain/2/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_1', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'phy'} to /local/domain/0/backend/vbd/2/2051.
[2006-11-07 17:03:10 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8F', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/2/0'} to /local/domain/2/device/vif/0.
[2006-11-07 17:03:10 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91365', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/2/device/vif/0', 'mac': '12:31:34:00:03:8F', 'online': '1', 'frontend-id': '2'} to /local/domain/0/backend/vif/2/0.
[2006-11-07 17:03:11 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291365', 'on_reboot': 'restart', 'start_time': '1162911791.0', 'on_poweroff': 'destroy', 'name': 'dom_91365', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6-xenU) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-07 17:03:11 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '618709', 'console/port': '2', 'name': 'dom_91365', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291365', 'domid': '2', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '618710', 'store/port': '1'}
[2006-11-07 17:03:11 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-07 17:03:11 xend 2682] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-07 17:03:11 xend 2682] DEBUG (DevController:149) Waiting for 0.
[2006-11-07 17:03:11 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:143) Waiting for devices usb.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:143) Waiting for devices vbd.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:149) Waiting for 2049.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/2/2049/hotplug-status.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:149) Waiting for 2050.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/2/2050/hotplug-status.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:149) Waiting for 2051.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/2/2051/hotplug-status.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:143) Waiting for devices irq.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:143) Waiting for devices pci.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:143) Waiting for devices ioports.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:143) Waiting for devices tap.
[2006-11-07 17:03:13 xend 2682] DEBUG (DevController:143) Waiting for devices vtpm.
[2006-11-07 17:03:13 xend 2682] INFO (XendDomain:370) Domain dom_91365 (2) unpaused.
[2006-11-07 17:18:08 xend 2682] INFO (XendDomain:388) Domain dom_91365 (2) paused.
[2006-11-07 17:56:08 xend 2682] INFO (XendDomain:370) Domain dom_91365 (2) unpaused.
[2006-11-07 18:10:46 xend 2682] DEBUG (blkif:24) exception looking up device number for /dev/xvda1: [Errno 2] No such file or directory: '/dev/xvda1'
[2006-11-07 18:10:46 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '51713', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/0/51713'} to /local/domain/0/device/vbd/51713.
[2006-11-07 18:10:46 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'Domain-0', 'frontend': '/local/domain/0/device/vbd/51713', 'dev': '/dev/xvda1', 'state': '1', 'params': 'qcow:/mnt/ami_cache/tmp/gsg-jamesg.qcow', 'mode': 'w', 'online': '1', 'frontend-id': '0', 'type': 'tap'} to /local/domain/0/backend/tap/0/51713.
[2006-11-07 18:10:46 xend 2682] DEBUG (DevController:149) Waiting for 51713.
[2006-11-07 18:10:46 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/tap/0/51713/hotplug-status.
[2006-11-07 18:10:47 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/tap/0/51713/hotplug-status.
[2006-11-07 18:10:47 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 10:53:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291641'], ['name', 'dom_91641'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:aio:/mnt/instance_image_store_0/2866'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-08 10:53:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291641'], ['name', 'dom_91641'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:aio:/mnt/instance_image_store_0/2866'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-08 10:53:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291641', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91641', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('tap', ['tap', ['uname', 'tap:aio:/mnt/instance_image_store_0/2866'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8E']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-08 10:53:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-08 10:53:44 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 3 1.0
[2006-11-08 10:53:44 xend 2682] DEBUG (balloon:127) Balloon: 5527420 KiB free; need 1740800; done.
[2006-11-08 10:53:44 xend 2682] INFO (image:136) buildDomain os=linux dom=3 vcpus=1
[2006-11-08 10:53:44 xend 2682] DEBUG (image:191) dom            = 3
[2006-11-08 10:53:44 xend 2682] DEBUG (image:192) image          = /boot/vmlinuz-2.6-xenU
[2006-11-08 10:53:44 xend 2682] DEBUG (image:193) store_evtchn   = 1
[2006-11-08 10:53:44 xend 2682] DEBUG (image:194) console_evtchn = 2
[2006-11-08 10:53:44 xend 2682] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-08 10:53:44 xend 2682] DEBUG (image:196) ramdisk        = 
[2006-11-08 10:53:44 xend 2682] DEBUG (image:197) vcpus          = 1
[2006-11-08 10:53:44 xend 2682] DEBUG (image:198) features       = 
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/3/2049'} to /local/domain/3/device/vbd/2049.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91641', 'frontend': '/local/domain/3/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': 'aio:/mnt/instance_image_store_0/2866', 'mode': 'w', 'online': '1', 'frontend-id': '3', 'type': 'tap'} to /local/domain/0/backend/tap/3/2049.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/3/2050'} to /local/domain/3/device/vbd/2050.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91641', 'frontend': '/local/domain/3/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '3', 'type': 'phy'} to /local/domain/0/backend/vbd/3/2050.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/3/2051'} to /local/domain/3/device/vbd/2051.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91641', 'frontend': '/local/domain/3/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '3', 'type': 'phy'} to /local/domain/0/backend/vbd/3/2051.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8E', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/3/0'} to /local/domain/3/device/vif/0.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91641', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/3/device/vif/0', 'mac': '12:31:34:00:03:8E', 'online': '1', 'frontend-id': '3'} to /local/domain/0/backend/vif/3/0.
[2006-11-08 10:53:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291641', 'on_reboot': 'restart', 'start_time': '1162976025.11', 'on_poweroff': 'destroy', 'name': 'dom_91641', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6-xenU) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-08 10:53:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '2071943', 'console/port': '2', 'name': 'dom_91641', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291641', 'domid': '3', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '2071944', 'store/port': '1'}
[2006-11-08 10:53:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:149) Waiting for 0.
[2006-11-08 10:53:45 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/3/0/hotplug-status.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/3/0/hotplug-status.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:143) Waiting for devices usb.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:143) Waiting for devices vbd.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:149) Waiting for 2050.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/3/2050/hotplug-status.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:149) Waiting for 2051.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/3/2051/hotplug-status.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:143) Waiting for devices irq.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:143) Waiting for devices pci.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:143) Waiting for devices ioports.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:143) Waiting for devices tap.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:149) Waiting for 2049.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/tap/3/2049/hotplug-status.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 10:53:48 xend 2682] DEBUG (DevController:143) Waiting for devices vtpm.
[2006-11-08 10:53:48 xend 2682] INFO (XendDomain:370) Domain dom_91641 (3) unpaused.
[2006-11-08 11:24:39 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291662'], ['name', 'dom_91662'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_2/2866.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_2'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_2'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8D']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-08 11:24:39 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291662'], ['name', 'dom_91662'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_2/2866.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_2'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_2'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8D']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-08 11:24:39 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291662', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91662', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('tap', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_2/2866.qcow'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_2'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_2'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8D']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-08 11:24:39 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-08 11:24:39 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 4 1.0
[2006-11-08 11:24:39 xend 2682] DEBUG (balloon:127) Balloon: 3786932 KiB free; need 1740800; done.
[2006-11-08 11:24:39 xend 2682] INFO (image:136) buildDomain os=linux dom=4 vcpus=1
[2006-11-08 11:24:39 xend 2682] DEBUG (image:191) dom            = 4
[2006-11-08 11:24:39 xend 2682] DEBUG (image:192) image          = /boot/vmlinuz-2.6-xenU
[2006-11-08 11:24:39 xend 2682] DEBUG (image:193) store_evtchn   = 1
[2006-11-08 11:24:39 xend 2682] DEBUG (image:194) console_evtchn = 2
[2006-11-08 11:24:39 xend 2682] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-08 11:24:39 xend 2682] DEBUG (image:196) ramdisk        = 
[2006-11-08 11:24:39 xend 2682] DEBUG (image:197) vcpus          = 1
[2006-11-08 11:24:39 xend 2682] DEBUG (image:198) features       = 
[2006-11-08 11:24:39 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/4/2049'} to /local/domain/4/device/vbd/2049.
[2006-11-08 11:24:39 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91662', 'frontend': '/local/domain/4/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': 'qcow:/mnt/instance_image_store_2/2866.qcow', 'mode': 'w', 'online': '1', 'frontend-id': '4', 'type': 'tap'} to /local/domain/0/backend/tap/4/2049.
[2006-11-08 11:24:39 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/4/2050'} to /local/domain/4/device/vbd/2050.
[2006-11-08 11:24:39 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91662', 'frontend': '/local/domain/4/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_2', 'mode': 'w', 'online': '1', 'frontend-id': '4', 'type': 'phy'} to /local/domain/0/backend/vbd/4/2050.
[2006-11-08 11:24:39 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/4/2051'} to /local/domain/4/device/vbd/2051.
[2006-11-08 11:24:39 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91662', 'frontend': '/local/domain/4/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_2', 'mode': 'w', 'online': '1', 'frontend-id': '4', 'type': 'phy'} to /local/domain/0/backend/vbd/4/2051.
[2006-11-08 11:24:39 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8D', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/4/0'} to /local/domain/4/device/vif/0.
[2006-11-08 11:24:39 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91662', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/4/device/vif/0', 'mac': '12:31:34:00:03:8D', 'online': '1', 'frontend-id': '4'} to /local/domain/0/backend/vif/4/0.
[2006-11-08 11:24:40 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291662', 'on_reboot': 'restart', 'start_time': '1162977879.99', 'on_poweroff': 'destroy', 'name': 'dom_91662', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6-xenU) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-08 11:24:40 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '1505780', 'console/port': '2', 'name': 'dom_91662', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291662', 'domid': '4', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '1505781', 'store/port': '1'}
[2006-11-08 11:24:40 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 11:24:40 xend 2682] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-08 11:24:40 xend 2682] DEBUG (DevController:149) Waiting for 0.
[2006-11-08 11:24:40 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/4/0/hotplug-status.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/4/0/hotplug-status.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:143) Waiting for devices usb.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:143) Waiting for devices vbd.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:149) Waiting for 2050.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/4/2050/hotplug-status.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:149) Waiting for 2051.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/4/2051/hotplug-status.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:143) Waiting for devices irq.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:143) Waiting for devices pci.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:143) Waiting for devices ioports.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:143) Waiting for devices tap.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:149) Waiting for 2049.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/tap/4/2049/hotplug-status.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 11:24:44 xend 2682] DEBUG (DevController:143) Waiting for devices vtpm.
[2006-11-08 11:24:44 xend 2682] INFO (XendDomain:370) Domain dom_91662 (4) unpaused.
[2006-11-08 11:37:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 11:37:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 11:37:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 11:37:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 11:37:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 11:37:45 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 11:38:09 xend.XendDomainInfo 2682] INFO (XendDomainInfo:892) Domain has shutdown: name=dom_91641 id=3 reason=poweroff.
[2006-11-08 11:38:09 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1449) XendDomainInfo.destroy: domid=3
[2006-11-08 11:38:09 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1457) XendDomainInfo.destroyDomain(3)
[2006-11-08 11:38:10 xend.XendDomainInfo 2682] INFO (XendDomainInfo:892) Domain has shutdown: name=dom_91365 id=2 reason=poweroff.
[2006-11-08 11:38:10 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1449) XendDomainInfo.destroy: domid=2
[2006-11-08 11:38:10 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1457) XendDomainInfo.destroyDomain(2)
[2006-11-08 11:38:15 xend.XendDomainInfo 2682] INFO (XendDomainInfo:923) Domain shutdown timeout expired: name=dom_91662 id=4
[2006-11-08 11:38:15 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1449) XendDomainInfo.destroy: domid=4
[2006-11-08 11:38:15 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1457) XendDomainInfo.destroyDomain(4)
[2006-11-08 11:43:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291667'], ['name', 'dom_91667'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-08 11:43:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291667'], ['name', 'dom_91667'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-08 11:43:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291667', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91667', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('tap', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8E']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-08 11:43:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-08 11:43:49 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 5 1.0
[2006-11-08 11:43:49 xend 2682] DEBUG (balloon:127) Balloon: 7268056 KiB free; need 1740800; done.
[2006-11-08 11:43:49 xend 2682] INFO (image:136) buildDomain os=linux dom=5 vcpus=1
[2006-11-08 11:43:49 xend 2682] DEBUG (image:191) dom            = 5
[2006-11-08 11:43:49 xend 2682] DEBUG (image:192) image          = /boot/vmlinuz-2.6-xenU
[2006-11-08 11:43:49 xend 2682] DEBUG (image:193) store_evtchn   = 1
[2006-11-08 11:43:49 xend 2682] DEBUG (image:194) console_evtchn = 2
[2006-11-08 11:43:49 xend 2682] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-08 11:43:49 xend 2682] DEBUG (image:196) ramdisk        = 
[2006-11-08 11:43:49 xend 2682] DEBUG (image:197) vcpus          = 1
[2006-11-08 11:43:49 xend 2682] DEBUG (image:198) features       = 
[2006-11-08 11:43:49 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/5/2049'} to /local/domain/5/device/vbd/2049.
[2006-11-08 11:43:49 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91667', 'frontend': '/local/domain/5/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': 'qcow:/mnt/instance_image_store_0/3811.qcow', 'mode': 'w', 'online': '1', 'frontend-id': '5', 'type': 'tap'} to /local/domain/0/backend/tap/5/2049.
[2006-11-08 11:43:49 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/5/2050'} to /local/domain/5/device/vbd/2050.
[2006-11-08 11:43:49 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91667', 'frontend': '/local/domain/5/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '5', 'type': 'phy'} to /local/domain/0/backend/vbd/5/2050.
[2006-11-08 11:43:49 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/5/2051'} to /local/domain/5/device/vbd/2051.
[2006-11-08 11:43:49 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91667', 'frontend': '/local/domain/5/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '5', 'type': 'phy'} to /local/domain/0/backend/vbd/5/2051.
[2006-11-08 11:43:50 xend 2682] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8E', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/5/0'} to /local/domain/5/device/vif/0.
[2006-11-08 11:43:50 xend 2682] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91667', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/5/device/vif/0', 'mac': '12:31:34:00:03:8E', 'online': '1', 'frontend-id': '5'} to /local/domain/0/backend/vif/5/0.
[2006-11-08 11:43:50 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291667', 'on_reboot': 'restart', 'start_time': '1162979030.04', 'on_poweroff': 'destroy', 'name': 'dom_91667', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6-xenU) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-08 11:43:50 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '2065335', 'console/port': '2', 'name': 'dom_91667', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291667', 'domid': '5', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '2065186', 'store/port': '1'}
[2006-11-08 11:43:50 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 11:43:50 xend 2682] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-08 11:43:50 xend 2682] DEBUG (DevController:149) Waiting for 0.
[2006-11-08 11:43:50 xend 2682] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/5/0/hotplug-status.
[2006-11-08 11:45:30 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1449) XendDomainInfo.destroy: domid=5
[2006-11-08 11:45:30 xend.XendDomainInfo 2682] DEBUG (XendDomainInfo:1457) XendDomainInfo.destroyDomain(5)
[2006-11-08 12:52:34 xend 2890] INFO (SrvDaemon:283) Xend Daemon started
[2006-11-08 12:52:34 xend 2890] INFO (SrvDaemon:287) Xend changeset: Sun Oct 15 13:19:07 2006 +0100 11774:620cb0064d9b.
[2006-11-08 12:52:34 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:212) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 18452402899L, 'ssidref': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'mem_kb': 1047296, 'maxmem_kb': -4, 'max_vcpu_id': 1, 'crashed': 0, 'running': 1, 'shutdown': 0, 'online_vcpus': 2, 'blocked': 0})
[2006-11-08 12:52:34 xend.XendDomainInfo 2890] INFO (XendDomainInfo:224) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2006-11-08 12:52:34 xend.XendDomainInfo 2890] WARNING (XendDomainInfo:246) No vm path in store for existing domain 0
[2006-11-08 12:52:34 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'on_poweroff': 'destroy', 'name': 'Domain-0', 'xend/restart_count': '0', 'vcpus': '2', 'vcpu_avail': '3', 'memory': '1023', 'on_crash': 'restart', 'maxmem': '1023'}
[2006-11-08 12:52:34 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:747) Storing domain details: {'cpu/0/availability': 'online', 'memory/target': '1047552', 'cpu/1/availability': 'online', 'name': 'Domain-0', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0'}
[2006-11-08 12:52:34 xend 2890] DEBUG (XendDomain:153) number of vcpus to use is 0
[2006-11-08 12:52:34 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 12:55:58 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291677'], ['name', 'dom_91677'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-08 12:55:58 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291677'], ['name', 'dom_91677'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-08 12:55:58 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291677', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6-xenU'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91677', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('tap', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8E']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-08 12:55:58 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-08 12:55:58 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 1 1.0
[2006-11-08 12:55:58 xend 2890] DEBUG (balloon:127) Balloon: 7268224 KiB free; need 1740800; done.
[2006-11-08 12:55:58 xend 2890] INFO (image:136) buildDomain os=linux dom=1 vcpus=1
[2006-11-08 12:55:58 xend 2890] DEBUG (image:191) dom            = 1
[2006-11-08 12:55:58 xend 2890] DEBUG (image:192) image          = /boot/vmlinuz-2.6-xenU
[2006-11-08 12:55:58 xend 2890] DEBUG (image:193) store_evtchn   = 1
[2006-11-08 12:55:58 xend 2890] DEBUG (image:194) console_evtchn = 2
[2006-11-08 12:55:58 xend 2890] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-08 12:55:58 xend 2890] DEBUG (image:196) ramdisk        = 
[2006-11-08 12:55:58 xend 2890] DEBUG (image:197) vcpus          = 1
[2006-11-08 12:55:58 xend 2890] DEBUG (image:198) features       = 
[2006-11-08 12:55:58 xend 2890] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/1/2049'} to /local/domain/1/device/vbd/2049.
[2006-11-08 12:55:58 xend 2890] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91677', 'frontend': '/local/domain/1/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': 'qcow:/mnt/instance_image_store_0/3811.qcow', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'tap'} to /local/domain/0/backend/tap/1/2049.
[2006-11-08 12:55:58 xend 2890] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2050'} to /local/domain/1/device/vbd/2050.
[2006-11-08 12:55:58 xend 2890] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91677', 'frontend': '/local/domain/1/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'phy'} to /local/domain/0/backend/vbd/1/2050.
[2006-11-08 12:55:59 xend 2890] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2051'} to /local/domain/1/device/vbd/2051.
[2006-11-08 12:55:59 xend 2890] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91677', 'frontend': '/local/domain/1/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'phy'} to /local/domain/0/backend/vbd/1/2051.
[2006-11-08 12:55:59 xend 2890] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8E', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.
[2006-11-08 12:55:59 xend 2890] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91677', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/1/device/vif/0', 'mac': '12:31:34:00:03:8E', 'online': '1', 'frontend-id': '1'} to /local/domain/0/backend/vif/1/0.
[2006-11-08 12:55:59 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291677', 'on_reboot': 'restart', 'start_time': '1162983359.04', 'on_poweroff': 'destroy', 'name': 'dom_91677', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6-xenU) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-08 12:55:59 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '1009738', 'console/port': '2', 'name': 'dom_91677', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291677', 'domid': '1', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '1009739', 'store/port': '1'}
[2006-11-08 12:55:59 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 12:55:59 xend 2890] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-08 12:55:59 xend 2890] DEBUG (DevController:149) Waiting for 0.
[2006-11-08 12:55:59 xend 2890] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:143) Waiting for devices usb.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:143) Waiting for devices vbd.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:149) Waiting for 2050.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/1/2050/hotplug-status.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/1/2050/hotplug-status.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:149) Waiting for 2051.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/1/2051/hotplug-status.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:143) Waiting for devices irq.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:143) Waiting for devices pci.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:143) Waiting for devices ioports.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:143) Waiting for devices tap.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:149) Waiting for 2049.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/tap/1/2049/hotplug-status.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 12:56:00 xend 2890] DEBUG (DevController:143) Waiting for devices vtpm.
[2006-11-08 12:56:00 xend 2890] INFO (XendDomain:370) Domain dom_91677 (1) unpaused.
[2006-11-08 13:27:29 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 13:27:29 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 13:27:59 xend.XendDomainInfo 2890] INFO (XendDomainInfo:923) Domain shutdown timeout expired: name=dom_91677 id=1
[2006-11-08 13:27:59 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:1449) XendDomainInfo.destroy: domid=1
[2006-11-08 13:27:59 xend.XendDomainInfo 2890] DEBUG (XendDomainInfo:1457) XendDomainInfo.destroyDomain(1)
[2006-11-08 15:57:17 xend 2893] INFO (SrvDaemon:283) Xend Daemon started
[2006-11-08 15:57:17 xend 2893] INFO (SrvDaemon:287) Xend changeset: Sun Oct 15 13:19:07 2006 +0100 11774:620cb0064d9b.
[2006-11-08 15:57:17 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:212) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 17879451979L, 'ssidref': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'mem_kb': 1047296, 'maxmem_kb': -4, 'max_vcpu_id': 1, 'crashed': 0, 'running': 1, 'shutdown': 0, 'online_vcpus': 2, 'blocked': 0})
[2006-11-08 15:57:17 xend.XendDomainInfo 2893] INFO (XendDomainInfo:224) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2006-11-08 15:57:17 xend.XendDomainInfo 2893] WARNING (XendDomainInfo:246) No vm path in store for existing domain 0
[2006-11-08 15:57:17 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'on_poweroff': 'destroy', 'name': 'Domain-0', 'xend/restart_count': '0', 'vcpus': '2', 'vcpu_avail': '3', 'memory': '1023', 'on_crash': 'restart', 'maxmem': '1023'}
[2006-11-08 15:57:17 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:747) Storing domain details: {'cpu/0/availability': 'online', 'memory/target': '1047552', 'cpu/1/availability': 'online', 'name': 'Domain-0', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0'}
[2006-11-08 15:57:17 xend 2893] DEBUG (XendDomain:153) number of vcpus to use is 0
[2006-11-08 15:57:17 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 16:04:39 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291715'], ['name', 'dom_91715'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:aio:/mnt/instance_image_store_1/3811'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_1'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_1'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8F']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-08 16:04:39 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291715'], ['name', 'dom_91715'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:aio:/mnt/instance_image_store_1/3811'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_1'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_1'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8F']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-08 16:04:39 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291715', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91715', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('tap', ['tap', ['uname', 'tap:aio:/mnt/instance_image_store_1/3811'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_1'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_1'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8F']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-08 16:04:39 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-08 16:04:39 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 1 1.0
[2006-11-08 16:04:39 xend 2893] DEBUG (balloon:127) Balloon: 7268224 KiB free; need 1740800; done.
[2006-11-08 16:04:39 xend 2893] INFO (image:136) buildDomain os=linux dom=1 vcpus=1
[2006-11-08 16:04:39 xend 2893] DEBUG (image:191) dom            = 1
[2006-11-08 16:04:39 xend 2893] DEBUG (image:192) image          = /boot/vmlinuz-2.6.16.29-xen
[2006-11-08 16:04:39 xend 2893] DEBUG (image:193) store_evtchn   = 1
[2006-11-08 16:04:39 xend 2893] DEBUG (image:194) console_evtchn = 2
[2006-11-08 16:04:39 xend 2893] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-08 16:04:39 xend 2893] DEBUG (image:196) ramdisk        = /boot/initrd-2.6.16.29-xen.img
[2006-11-08 16:04:39 xend 2893] DEBUG (image:197) vcpus          = 1
[2006-11-08 16:04:39 xend 2893] DEBUG (image:198) features       = 
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/1/2049'} to /local/domain/1/device/vbd/2049.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91715', 'frontend': '/local/domain/1/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': 'aio:/mnt/instance_image_store_1/3811', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'tap'} to /local/domain/0/backend/tap/1/2049.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2050'} to /local/domain/1/device/vbd/2050.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91715', 'frontend': '/local/domain/1/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_1', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'phy'} to /local/domain/0/backend/vbd/1/2050.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/2051'} to /local/domain/1/device/vbd/2051.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91715', 'frontend': '/local/domain/1/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_1', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'phy'} to /local/domain/0/backend/vbd/1/2051.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8F', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91715', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/1/device/vif/0', 'mac': '12:31:34:00:03:8F', 'online': '1', 'frontend-id': '1'} to /local/domain/0/backend/vif/1/0.
[2006-11-08 16:04:42 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291715', 'on_reboot': 'restart', 'start_time': '1162994682.56', 'on_poweroff': 'destroy', 'name': 'dom_91715', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6.16.29-xen) (ramdisk /boot/initrd-2.6.16.29-xen.img) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-08 16:04:42 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '995310', 'console/port': '2', 'name': 'dom_91715', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291715', 'domid': '1', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '995311', 'store/port': '1'}
[2006-11-08 16:04:42 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:149) Waiting for 0.
[2006-11-08 16:04:42 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:143) Waiting for devices usb.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:143) Waiting for devices vbd.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:149) Waiting for 2050.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/1/2050/hotplug-status.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:149) Waiting for 2051.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/1/2051/hotplug-status.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:143) Waiting for devices irq.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:143) Waiting for devices pci.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:143) Waiting for devices ioports.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:143) Waiting for devices tap.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:149) Waiting for 2049.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/tap/1/2049/hotplug-status.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 16:04:45 xend 2893] DEBUG (DevController:143) Waiting for devices vtpm.
[2006-11-08 16:04:45 xend 2893] INFO (XendDomain:370) Domain dom_91715 (1) unpaused.
[2006-11-08 16:16:08 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291721'], ['name', 'dom_91721'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-08 16:16:08 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291721'], ['name', 'dom_91721'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8E']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-08 16:16:08 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291721', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91721', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('tap', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_0/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_0'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_0'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8E']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-08 16:16:08 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-08 16:16:08 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 2 1.0
[2006-11-08 16:16:08 xend 2893] DEBUG (balloon:127) Balloon: 5528620 KiB free; need 1740800; done.
[2006-11-08 16:16:08 xend 2893] INFO (image:136) buildDomain os=linux dom=2 vcpus=1
[2006-11-08 16:16:08 xend 2893] DEBUG (image:191) dom            = 2
[2006-11-08 16:16:08 xend 2893] DEBUG (image:192) image          = /boot/vmlinuz-2.6.16.29-xen
[2006-11-08 16:16:08 xend 2893] DEBUG (image:193) store_evtchn   = 1
[2006-11-08 16:16:08 xend 2893] DEBUG (image:194) console_evtchn = 2
[2006-11-08 16:16:08 xend 2893] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-08 16:16:08 xend 2893] DEBUG (image:196) ramdisk        = /boot/initrd-2.6.16.29-xen.img
[2006-11-08 16:16:08 xend 2893] DEBUG (image:197) vcpus          = 1
[2006-11-08 16:16:08 xend 2893] DEBUG (image:198) features       = 
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/2/2049'} to /local/domain/2/device/vbd/2049.
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91721', 'frontend': '/local/domain/2/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': 'qcow:/mnt/instance_image_store_0/3811.qcow', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'tap'} to /local/domain/0/backend/tap/2/2049.
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/2050'} to /local/domain/2/device/vbd/2050.
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91721', 'frontend': '/local/domain/2/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'phy'} to /local/domain/0/backend/vbd/2/2050.
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/2051'} to /local/domain/2/device/vbd/2051.
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91721', 'frontend': '/local/domain/2/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_0', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'phy'} to /local/domain/0/backend/vbd/2/2051.
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8E', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/2/0'} to /local/domain/2/device/vif/0.
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91721', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/2/device/vif/0', 'mac': '12:31:34:00:03:8E', 'online': '1', 'frontend-id': '2'} to /local/domain/0/backend/vif/2/0.
[2006-11-08 16:16:09 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291721', 'on_reboot': 'restart', 'start_time': '1162995369.36', 'on_poweroff': 'destroy', 'name': 'dom_91721', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6.16.29-xen) (ramdisk /boot/initrd-2.6.16.29-xen.img) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-08 16:16:09 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '1939705', 'console/port': '2', 'name': 'dom_91721', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291721', 'domid': '2', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '1939706', 'store/port': '1'}
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-08 16:16:09 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:149) Waiting for 0.
[2006-11-08 16:16:09 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2006-11-08 16:16:10 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2006-11-08 16:16:10 xend 2893] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:143) Waiting for devices usb.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:143) Waiting for devices vbd.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:149) Waiting for 2050.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/2/2050/hotplug-status.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:149) Waiting for 2051.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/2/2051/hotplug-status.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vbd/2/2051/hotplug-status.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:143) Waiting for devices irq.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:143) Waiting for devices pci.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:143) Waiting for devices ioports.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:143) Waiting for devices tap.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:149) Waiting for 2049.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/tap/2/2049/hotplug-status.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:478) hotplugStatusCallback 1.
[2006-11-08 16:16:11 xend 2893] DEBUG (DevController:143) Waiting for devices vtpm.
[2006-11-08 16:16:11 xend 2893] INFO (XendDomain:370) Domain dom_91721 (2) unpaused.
[2006-11-08 16:29:49 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 16:29:49 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 16:32:44 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1449) XendDomainInfo.destroy: domid=2
[2006-11-08 16:32:44 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1457) XendDomainInfo.destroyDomain(2)
[2006-11-08 16:44:55 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:190) XendDomainInfo.create(['vm', ['uuid', '00000000-0000-0000-0000-0000ec291723'], ['name', 'dom_91723'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_2/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_2'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_2'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8D']]], ['memmap'], ['device_model'], ['device_config']])
[2006-11-08 16:44:55 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:296) parseConfig: config is ['vm', ['uuid', '00000000-0000-0000-0000-0000ec291723'], ['name', 'dom_91723'], ['memory', '1700'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']]], ['device', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_2/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_2'], ['dev', 'sda2'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_2'], ['dev', 'sda3'], ['mode', 'w']]], ['device', ['vif', ['mac', '12:31:34:00:03:8D']]], ['memmap'], ['device_model'], ['device_config']]
[2006-11-08 16:44:55 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:395) parseConfig: result is {'shadow_memory': None, 'uuid': '00000000-0000-0000-0000-0000ec291723', 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6.16.29-xen'], ['ramdisk', '/boot/initrd-2.6.16.29-xen.img'], ['root', '/dev/sda1 ro'], ['args', '4'], ['vcpus', '1']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'dom_91723', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 1700, 'device': [('tap', ['tap', ['uname', 'tap:qcow:/mnt/instance_image_store_2/3811.qcow'], ['dev', 'sda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_ephemeral_store_2'], ['dev', 'sda2'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'phy:/dev/VolGroupDomU/instance_swap_store_2'], ['dev', 'sda3'], ['mode', 'w']]), ('vif', ['vif', ['mac', '12:31:34:00:03:8D']])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-08 16:44:55 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1253) XendDomainInfo.construct: None
[2006-11-08 16:44:55 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1285) XendDomainInfo.initDomain: 3 1.0
[2006-11-08 16:44:55 xend 2893] DEBUG (balloon:127) Balloon: 5528628 KiB free; need 1740800; done.
[2006-11-08 16:44:55 xend 2893] INFO (image:136) buildDomain os=linux dom=3 vcpus=1
[2006-11-08 16:44:55 xend 2893] DEBUG (image:191) dom            = 3
[2006-11-08 16:44:55 xend 2893] DEBUG (image:192) image          = /boot/vmlinuz-2.6.16.29-xen
[2006-11-08 16:44:55 xend 2893] DEBUG (image:193) store_evtchn   = 1
[2006-11-08 16:44:55 xend 2893] DEBUG (image:194) console_evtchn = 2
[2006-11-08 16:44:55 xend 2893] DEBUG (image:195) cmdline        =  root=/dev/sda1 ro 4
[2006-11-08 16:44:55 xend 2893] DEBUG (image:196) ramdisk        = /boot/initrd-2.6.16.29-xen.img
[2006-11-08 16:44:55 xend 2893] DEBUG (image:197) vcpus          = 1
[2006-11-08 16:44:55 xend 2893] DEBUG (image:198) features       = 
[2006-11-08 16:44:55 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2049', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/3/2049'} to /local/domain/3/device/vbd/2049.
[2006-11-08 16:44:55 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91723', 'frontend': '/local/domain/3/device/vbd/2049', 'dev': 'sda1', 'state': '1', 'params': 'qcow:/mnt/instance_image_store_2/3811.qcow', 'mode': 'w', 'online': '1', 'frontend-id': '3', 'type': 'tap'} to /local/domain/0/backend/tap/3/2049.
[2006-11-08 16:44:55 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2050', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/3/2050'} to /local/domain/3/device/vbd/2050.
[2006-11-08 16:44:55 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91723', 'frontend': '/local/domain/3/device/vbd/2050', 'dev': 'sda2', 'state': '1', 'params': '/dev/VolGroupDomU/instance_ephemeral_store_2', 'mode': 'w', 'online': '1', 'frontend-id': '3', 'type': 'phy'} to /local/domain/0/backend/vbd/3/2050.
[2006-11-08 16:44:56 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '2051', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/3/2051'} to /local/domain/3/device/vbd/2051.
[2006-11-08 16:44:56 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91723', 'frontend': '/local/domain/3/device/vbd/2051', 'dev': 'sda3', 'state': '1', 'params': '/dev/VolGroupDomU/instance_swap_store_2', 'mode': 'w', 'online': '1', 'frontend-id': '3', 'type': 'phy'} to /local/domain/0/backend/vbd/3/2051.
[2006-11-08 16:44:56 xend 2893] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '12:31:34:00:03:8D', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/3/0'} to /local/domain/3/device/vif/0.
[2006-11-08 16:44:56 xend 2893] DEBUG (DevController:112) DevController: writing {'domain': 'dom_91723', 'handle': '0', 'script': '/etc/xen/scripts/vif-aes', 'state': '1', 'frontend': '/local/domain/3/device/vif/0', 'mac': '12:31:34:00:03:8D', 'online': '1', 'frontend-id': '3'} to /local/domain/0/backend/vif/3/0.
[2006-11-08 16:44:56 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:712) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-0000ec291723', 'on_reboot': 'restart', 'start_time': '1162997096.03', 'on_poweroff': 'destroy', 'name': 'dom_91723', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1700', 'on_crash': 'restart', 'image': "(linux (kernel /boot/vmlinuz-2.6.16.29-xen) (ramdisk /boot/initrd-2.6.16.29-xen.img) (root '/dev/sda1 ro') (args 4) (vcpus 1))", 'maxmem': '1700'}
[2006-11-08 16:44:56 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:747) Storing domain details: {'console/ring-ref': '1760378', 'console/port': '2', 'name': 'dom_91723', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-0000ec291723', 'domid': '3', 'cpu/0/availability': 'online', 'memory/target': '1740800', 'store/ring-ref': '1760379', 'store/port': '1'}
[2006-11-08 16:44:56 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:933) XendDomainInfo.handleShutdownWatch
[2006-11-08 16:44:56 xend 2893] DEBUG (DevController:143) Waiting for devices vif.
[2006-11-08 16:44:56 xend 2893] DEBUG (DevController:149) Waiting for 0.
[2006-11-08 16:44:56 xend 2893] DEBUG (DevController:464) hotplugStatusCallback /local/domain/0/backend/vif/3/0/hotplug-status.
[2006-11-08 16:46:36 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1449) XendDomainInfo.destroy: domid=3
[2006-11-08 16:46:36 xend.XendDomainInfo 2893] DEBUG (XendDomainInfo:1457) XendDomainInfo.destroyDomain(3)

[-- Attachment #4: xen-hotplug.log --]
[-- Type: text/x-log, Size: 292 bytes --]

usage: ifdown <device name>
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
usage: ifdown <device name>
usage: ifdown <device name>
usage: ifdown <device name>
usage: ifdown <device name>
RTNETLINK answers: File exists
usage: ifdown <device name>

[-- Attachment #5: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [Xen-users] Re: tap:qcow causes dom-U to hang in 3.0.3

[Roland Paterson-Jones]

Ewan Mellor wrote:

>Could we see your /var/log/xen/*  and the output of xenstore-ls?
>  
>
I suspect this part of the xenstore-ls may be significant:

   error = ""
    backend = ""
     tap = ""
      1 = ""
       2049 = ""
        error = "2 getting info"

Is there somewhere I can get more info on the cause of the error? Also, 
given that the error seems 'sticky', in that 'xm shutdown' and 'xm 
destroy' don't appear to be able to get rid of the tap device, how do I 
identify where the tapdisk (or whatever) process is stuck?

Thanks for any advice
Regards
Roland

Here's the full xenstore-ls after the dom-U hangs and before I've done 
anything to try to shut it down:

[root@dom0 ~]# xenstore-ls
tool = ""
 xenstored = ""
vm = ""
 00000000-0000-0000-0000-000000000000 = ""
  shadow_memory = "0"
  uuid = "00000000-0000-0000-0000-000000000000"
  on_reboot = "restart"
  on_poweroff = "destroy"
  name = "Domain-0"
  xend = ""
   restart_count = "0"
  vcpus = "2"
  vcpu_avail = "3"
  memory = "1023"
  on_crash = "restart"
  maxmem = "1023"
 00000000-0000-0000-0000-0000ec292018 = ""
  image = "(linux (kernel /boot/vmlinuz-2.6.16.29-xen) (ramdisk 
/boot/initrd-2.6.16.29-xen.img) (..."
   ostype = "linux"
   kernel = "/boot/vmlinuz-2.6.16.29-xen"
   cmdline = " root=/dev/sda1 ro 4"
   ramdisk = "/boot/initrd-2.6.16.29-xen.img"
  shadow_memory = "0"
  uuid = "00000000-0000-0000-0000-0000ec292018"
  on_reboot = "restart"
  start_time = "1163078366.42"
  on_poweroff = "destroy"
  name = "dom_92018"
  xend = ""
   restart_count = "0"
  vcpus = "1"
  vcpu_avail = "1"
  memory = "1700"
  on_crash = "restart"
  maxmem = "1700"
local = ""
 domain = ""
  0 = ""
   cpu = ""
    0 = ""
     availability = "online"
    1 = ""
     availability = "online"
   memory = ""
    target = "1047552"
   name = "Domain-0"
   console = ""
    limit = "1048576"
   vm = "/vm/00000000-0000-0000-0000-000000000000"
   domid = "0"
   backend = ""
    tap = ""
     1 = ""
      2049 = ""
       domain = "dom_92018"
       frontend = "/local/domain/1/device/vbd/2049"
       dev = "sda1"
       state = "4"
       params = "qcow:/mnt/instance_image_store_1/3811.qcow"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "tap"
       sectors = "3123200"
       sector-size = "512"
       info = "0"
       hotplug-status = "connected"
    vbd = ""
     1 = ""
      2050 = ""
       domain = "dom_92018"
       frontend = "/local/domain/1/device/vbd/2050"
       dev = "sda2"
       state = "4"
       params = "/dev/VolGroupDomU/instance_ephemeral_store_1"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "phy"
       physical-device = "fd:6"
       hotplug-status = "connected"
       sectors = "312737792"
       info = "0"
       sector-size = "512"
      2051 = ""
       domain = "dom_92018"
       frontend = "/local/domain/1/device/vbd/2051"
       dev = "sda3"
       state = "4"
       params = "/dev/VolGroupDomU/instance_swap_store_1"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "phy"
       physical-device = "fd:7"
       hotplug-status = "connected"
       sectors = "1835008"
       info = "0"
       sector-size = "512"
    vif = ""
     1 = ""
      0 = ""
       domain = "dom_92018"
       handle = "0"
       script = "/etc/xen/scripts/vif-aes"
       state = "4"
       frontend = "/local/domain/1/device/vif/0"
       mac = "12:31:34:00:03:8F"
       online = "1"
       frontend-id = "1"
       feature-sg = "1"
       feature-gso-tcpv4 = "1"
       feature-rx-copy = "1"
       hotplug-status = "connected"
   error = ""
    backend = ""
     tap = ""
      1 = ""
       2049 = ""
        error = "2 getting info"
  1 = ""
   device = ""
    vbd = ""
     2049 = ""
      backend-id = "0"
      virtual-device = "2049"
      device-type = "disk"
      state = "4"
      backend = "/local/domain/0/backend/tap/1/2049"
      ring-ref = "8"
      event-channel = "6"
     2050 = ""
      backend-id = "0"
      virtual-device = "2050"
      device-type = "disk"
      state = "4"
      backend = "/local/domain/0/backend/vbd/1/2050"
      ring-ref = "9"
      event-channel = "7"
     2051 = ""
      backend-id = "0"
      virtual-device = "2051"
      device-type = "disk"
      state = "4"
      backend = "/local/domain/0/backend/vbd/1/2051"
      ring-ref = "10"
      event-channel = "8"
    vif = ""
     0 = ""
      backend-id = "0"
      mac = "12:31:34:00:03:8F"
      handle = "0"
      state = "4"
      backend = "/local/domain/0/backend/vif/1/0"
      tx-ring-ref = "523"
      rx-ring-ref = "524"
      event-channel = "9"
      request-rx-copy = "0"
      feature-rx-notify = "1"
      feature-sg = "1"
      feature-gso-tcpv4 = "1"
   device-misc = ""
    vif = ""
     nextDeviceID = "1"
   console = ""
    ring-ref = "786743"
    port = "2"
    limit = "1048576"
    tty = "/dev/pts/2"
   name = "dom_92018"
   vm = "/vm/00000000-0000-0000-0000-0000ec292018"
   domid = "1"
   cpu = ""
    0 = ""
     availability = "online"
   memory = ""
    target = "1740800"
   store = ""
    ring-ref = "786744"
    port = "1"

Re: Re: [Xen-devel] tap:qcow causes dom-U to hang in 3.0.3

[Ewan Mellor]

On Thu, Nov 09, 2006 at 03:34:16PM +0200, Roland Paterson-Jones wrote:

> Ewan Mellor wrote:
> 
> >Could we see your /var/log/xen/*  and the output of xenstore-ls?
> > 
> >
> I suspect this part of the xenstore-ls may be significant:
> 
>   error = ""
>    backend = ""
>     tap = ""
>      1 = ""
>       2049 = ""
>        error = "2 getting info"
> 
> Is there somewhere I can get more info on the cause of the error? Also, 
> given that the error seems 'sticky', in that 'xm shutdown' and 'xm 
> destroy' don't appear to be able to get rid of the tap device, how do I 
> identify where the tapdisk (or whatever) process is stuck?

I think that this might be misleading.  This error will be recorded when the
frontend comes up if the backend isn't yet ready, but once the backend becomes
ready, the frontend should retry.  It looks like that retry has been
successful (everything is in state 4 == Connected) so the error message is
bogus (we should actually delete that error when the connection completes
successfully).

I don't see any other smoking gun in your logs, I'm afraid, and I'm out of
ideas.  I'll see if we can repro this.

Ewan.

Re: tap:qcow causes dom-U to hang in 3.0.3

[Wolfgang Schnerring]

* Ewan Mellor <ewan@xensource.com>:
> On Wed, Nov 08, 2006 at 04:50:43PM +0200, Roland Paterson-Jones wrote:
>> However, I am really interested in copy-on-write support, and hence have 
>> been trying tap:qcow:. This has had limited success, in that dom-U's 
>> appear to launch. However, as soon as I try to console or ssh into the 
>> dom-U's they apparently freeze, and I can get no further with them.

I'm seeing (what I believe to be) the same thing, so here's some more data
points. This is Xen 3.0.3, compiled from the source.tgz, dom0 is Ubuntu 6.06,
domU is Debian Sarge. The error occurs when using a file-backed qcow-image,
using a standalone qcow image or tap:aio works just fine.

# qcow-create 1024 vm01.qcow vm01.img
# cat sample.xen
name="vm01"
kernel="/boot/vmlinuz-2.6-xen0"
ramdisk="/boot/initrd.img-2.6.16.29-xen0"
root="/dev/hda1"
extra="3"
memory=32
disk=["tap:qcow:/opt/wosc/vm01.qcow,hda1,w", "tap:aio:/opt/wosc/vm01-swap.img,hda2,w"]
vif=[""]
dhcp="off"
# xm create sample.xen -c
(boots up to login prompt, but freezes after entering the username)
# xm destroy vm01
# xm list
Error: Device 769 not connected

>From then on I haven't been able to un-wedge Xen other than by rebooting the
machine.

Please let me know if there's anything I can do to help debug this.

Thanks,
Wolfgang

> Could we see your /var/log/xen/*  and the output of xenstore-ls?

# xenstore-ls
tool = ""
 xenstored = ""
vm = ""
 00000000-0000-0000-0000-000000000000 = ""
  shadow_memory = "0"
  uuid = "00000000-0000-0000-0000-000000000000"
  on_reboot = "restart"
  on_poweroff = "destroy"
  name = "Domain-0"
  xend = ""
   restart_count = "0"
  vcpus = "1"
  vcpu_avail = "1"
  memory = "461"
  on_crash = "restart"
  maxmem = "461"
local = ""
 domain = ""
  0 = ""
   cpu = ""
    0 = ""
     availability = "online"
   memory = ""
    target = "472064"
   name = "Domain-0"
   console = ""
    limit = "1048576"
   vm = "/vm/00000000-0000-0000-0000-000000000000"
   domid = "0"
   backend = ""
    tap = ""
     1 = ""
      769 = ""
       domain = "vm01"
       frontend = "/local/domain/1/device/vbd/769"
       dev = "hda1"
       state = "4"
       params = "qcow:/opt/wosc/vm01.qcow"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "tap"
       sectors = "2097152"
       sector-size = "512"
       info = "0"
       hotplug-status = "connected"
      770 = ""
       domain = "vm01"
       frontend = "/local/domain/1/device/vbd/770"
       dev = "hda2"
       state = "4"
       params = "aio:/opt/wosc/vm01-swap.img"
       mode = "w"
       online = "1"
       frontend-id = "1"
       type = "tap"
       sectors = "262144"
       sector-size = "512"
       info = "0"
       hotplug-status = "connected"
   error = ""
    backend = ""
     tap = ""
      1 = ""
       769 = ""
        error = "2 getting info"
       770 = ""
        error = "2 getting info"

# cat /var/log/xend.log
[2006-11-10 09:07:17 xend 3432] INFO (__init__:1072) Xend stopped due to signal 15.
[2006-11-10 09:08:42 xend 3446] INFO (__init__:1072) Xend Daemon started
[2006-11-10 09:08:42 xend 3446] INFO (__init__:1072) Xend changeset: unavailable .
[2006-11-10 09:08:42 xend.XendDomainInfo 3446] DEBUG (__init__:1072) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 18163054582L, 'ssidref': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'shutdown_reason': 0, 'dying': 0, 'dom': 0, 'mem_kb': 471940, 'maxmem_kb': -4, 'max_vcpu_id': 0, 'crashed': 0, 'running': 1, 'shutdown': 0, 'online_vcpus': 1, 'blocked': 0})
[2006-11-10 09:08:42 xend.XendDomainInfo 3446] INFO (__init__:1072) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2006-11-10 09:08:42 xend.XendDomainInfo 3446] WARNING (__init__:1072) No vm path in store for existing domain 0
[2006-11-10 09:08:42 xend.XendDomainInfo 3446] DEBUG (__init__:1072) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'on_poweroff': 'destroy', 'name': 'Domain-0', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '461', 'on_crash': 'restart', 'maxmem': '461'}
[2006-11-10 09:08:42 xend.XendDomainInfo 3446] DEBUG (__init__:1072) Storing domain details: {'cpu/0/availability': 'online', 'memory/target': '472064', 'name': 'Domain-0', 'console/limit': '1048576', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0'}
[2006-11-10 09:08:42 xend 3446] DEBUG (__init__:1072) number of vcpus to use is 0
[2006-11-10 09:08:42 xend.XendDomainInfo 3446] DEBUG (__init__:1072) XendDomainInfo.handleShutdownWatch
[2006-11-10 09:35:36 xend.XendDomainInfo 3446] DEBUG (__init__:1072) XendDomainInfo.create(['vm', ['name', 'vm01'], ['memory', 32], ['vcpus', 1], ['image', ['linux', ['kernel', '
boot/vmlinuz-2.6-xen0'], ['ramdisk', '/boot/initrd.img-2.6.16.29-xenU'], ['root', '/dev/hda1'], ['args', '3']]], ['device', ['tap', ['uname', 'tap:qcow:/opt/wosc/vm01.qcow'], ['dev', 'hda1'], ['mode', 'w']]], ['device', ['tap', ['uname', 'tap:aio:/opt/wosc/vm01-swap.img'], ['dev', 'hda2'], ['mode', 'w']]], ['device', ['vif']]])
[2006-11-10 09:35:36 xend.XendDomainInfo 3446] DEBUG (__init__:1072) parseConfig: config is ['vm', ['name', 'vm01'], ['memory', 32], ['vcpus', 1], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6-xen0'], ['ramdisk', '/boot/initrd.img-2.6.16.29-xenU'], ['root', '/dev/hda1'], ['args', '3']]], ['device', ['tap', ['uname', 'tap:qcow:/opt/wosc/vm01.qcow'], ['dev', 'hda1'], ['mode', 'w']]], ['device', ['tap', ['uname', 'tap:aio:/opt/wosc/vm01-swap.img'], ['dev', 'hda2'], ['mode', 'w']]], ['device', ['vif']]]
[2006-11-10 09:35:36 xend.XendDomainInfo 3446] DEBUG (__init__:1072) parseConfig: result is {'shadow_memory': None, 'uuid': None, 'on_crash': None, 'on_reboot': None, 'localtime': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6-xen0'], ['ramdisk', '/boot/initrd.img-2.6.16.29-xenU'], ['root', '/dev/hda1'], ['args', '3']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'vm01', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 32, 'device': [('tap', ['tap', ['uname', 'tap:qcow:/opt/wosc/vm01.qcow'], ['dev', 'hda1'], ['mode', 'w']]), ('tap', ['tap', ['uname', 'tap:aio:/opt/wosc/vm01-swap.img'], ['dev', 'hda2'], ['mode', 'w']]), ('vif', ['vif'])], 'bootloader': None, 'cpu': None, 'maxmem': None}
[2006-11-10 09:35:36 xend.XendDomainInfo 3446] DEBUG (__init__:1072) XendDomainInfo.construct: None
[2006-11-10 09:35:36 xend.XendDomainInfo 3446] DEBUG (__init__:1072) XendDomainInfo.initDomain: 1 1.0
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) Balloon: 32828 KiB free; need 32768; done.
[2006-11-10 09:35:36 xend 3446] INFO (__init__:1072) buildDomain os=linux dom=1 vcpus=1
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) dom            = 1
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) image          = /boot/vmlinuz-2.6-xen0
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) store_evtchn   = 1
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) console_evtchn = 2
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) cmdline        =  root=/dev/hda1 3
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) ramdisk        = /boot/initrd.img-2.6.16.29-xenU
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) vcpus          = 1
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) features       =
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) DevController: writing {'backend-id': '0', 'virtual-device': '769', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/1/769'} to /local/domain/1/device/vbd/769.
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) DevController: writing {'domain': 'vm01', 'frontend': '/local/domain/1/device/vbd/769', 'dev': 'hda1', 'state': '1', 'params': 'qcow:/opt/wosc/vm01.qcow', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'tap'} to /local/domain/0/backend/tap/1/769.
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) DevController: writing {'backend-id': '0', 'virtual-device': '770', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/1/770'} to /local/domain/1/device/vbd/770.
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) DevController: writing {'domain': 'vm01', 'frontend': '/local/domain/1/device/vbd/770', 'dev': 'hda2', 'state': '1', 'params': 'aio:/opt/wosc/vm01-swap.img', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'tap'} to /local/domain/0/backend/tap/1/770.
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) DevController: writing {'backend-id': '0', 'mac': '00:16:3e:36:5b:cb', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.
[2006-11-10 09:35:36 xend 3446] DEBUG (__init__:1072) DevController: writing {'domain': 'vm01', 'handle': '0', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': '/local/domain/1/device/vif/0', 'mac': '00:16:3e:36:5b:cb', 'online': '1', 'frontend-id': '1'} to /local/domain/0/backend/vif/1/0.
[2006-11-10 09:35:37 xend.XendDomainInfo 3446] DEBUG (__init__:1072) Storing VM details: {'shadow_memory': '0', 'uuid': '071a2275-e0b2-4511-ecd8-f7510366f6a1', 'on_reboot': 'restart', 'start_time': '1163147737.0', 'on_poweroff': 'destroy', 'name': 'vm01', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '32', 'on_crash': 'restart', 'image': '(linux (kernel /boot/vmlinuz-2.6-xen0) (ramdisk /boot/initrd.img-2.6.16.29-xenU) (root /dev/hda1) (args 3))', 'maxmem': '32'}
[2006-11-10 09:35:37 xend.XendDomainInfo 3446] DEBUG (__init__:1072) Storing domain details: {'console/ring-ref': '71163', 'console/port': '2', 'name': 'vm01', 'console/limit': '1048576', 'vm': '/vm/071a2275-e0b2-4511-ecd8-f7510366f6a1', 'domid': '1', 'cpu/0/availability': 'online', 'memory/target': '32768', 'store/ring-ref': '71164', 'store/port': '1'}
[2006-11-10 09:35:37 xend.XendDomainInfo 3446] DEBUG (__init__:1072) XendDomainInfo.handleShutdownWatch
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for devices vif.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for 0.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) hotplugStatusCallback 1.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for devices usb.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for devices vbd.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for devices irq.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for devices pci.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for devices ioports.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for devices tap.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for 769.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/tap/1/769/hotplug-status.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) hotplugStatusCallback 1.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for 770.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/tap/1/770/hotplug-status.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) hotplugStatusCallback 1.
[2006-11-10 09:35:37 xend 3446] DEBUG (__init__:1072) Waiting for devices vtpm.
[2006-11-10 09:35:37 xend 3446] INFO (__init__:1072) Domain vm01 (1) unpaused.
[2006-11-10 09:36:20 xend.XendDomainInfo 3446] DEBUG (__init__:1072) XendDomainInfo.destroy: domid=1
[2006-11-10 09:36:20 xend.XendDomainInfo 3446] DEBUG (__init__:1072) XendDomainInfo.destroyDomain(1)

# cat /var/log/xen/xend-debug.log
Traceback (most recent call last):
  File "SocketServer.py", line 463, in process_request_thread
    self.finish_request(request, client_address)
  File "SocketServer.py", line 254, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "SocketServer.py", line 521, in __init__
    self.handle()
  File "BaseHTTPServer.py", line 316, in handle
    self.handle_one_request()
  File "BaseHTTPServer.py", line 310, in handle_one_request
    method()
  File "/usr/lib/python/xen/util/xmlrpclib2.py", line 66, in do_POST
    self.send_response(200)
  File "BaseHTTPServer.py", line 367, in send_response
    self.wfile.write("%s %d %s\r\n" %
  File "socket.py", line 248, in write
    self.flush()
  File "socket.py", line 235, in flush
    self._sock.sendall(buffer)
error: (32, 'Broken pipe')

# cat /var/log/xen/xen-hotplug.log
Nothing to flush.
Nothing to flush.
Nothing to flush.
Nothing to flush.