Unable to create VM with nic device on Arndale

Unable to create VM with nic device on Arndale

[Eugene Fedotov]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

Hello!
Could someone check VIF creation on ARM ?

My domU's config is following:
extra = "console=hvc0 debug ipv6.disable=1";
kernel = "/boot/zImageU"
memory = 128
name = "domU-1"
vcpus = 1
vif = [ 'script=/etc/xen/scripts/vif-route, ip=10.0.0.1' ];

"xl create domU.conf" causes following output:
Parsing config from domU.conf
libxl: error: libxl_create.c:1180:domcreate_attach_vtpms: unable to add 
nic devices
libxl: error: libxl_device.c:780:libxl__initiate_device_remove: unable 
to get my domid
libxl: error: libxl.c:1452:devices_destroy_cb: libxl__devices_destroy 
failed for 1

Libxl code assumes that domcreate_attach_vtpms handler called, that 
calls domcreate_attach_pci.
Why vif is logically depend on PCI and vtpm on ARM platform? (it is 
confusing).

Best regards,
Eugene

[-- Attachment #2: e_fedotov.vcf --]
[-- Type: text/x-vcard, Size: 192 bytes --]

begin:vcard
fn:Evgeny Fedotov  
n:Fedotov;Evgeny
org:Samsung;CSG
adr;dom:;;Dvintsev 12/1 Building C;Moscow
email;internet:e.fedotov@samsung.com
title:Engineer
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Unable to create VM with nic device on Arndale

[Ian Campbell]

On Wed, 2013-12-18 at 15:07 +0400, Eugene Fedotov wrote:
> Hello!
> Could someone check VIF creation on ARM ?
> 
> My domU's config is following:
> extra = "console=hvc0 debug ipv6.disable=1";
> kernel = "/boot/zImageU"
> memory = 128
> name = "domU-1"
> vcpus = 1
> vif = [ 'script=/etc/xen/scripts/vif-route, ip=10.0.0.1' ];

The parser doesn't handle spaces well.

> 
> "xl create domU.conf" causes following output:
> Parsing config from domU.conf
> libxl: error: libxl_create.c:1180:domcreate_attach_vtpms: unable to add 
> nic devices
> libxl: error: libxl_device.c:780:libxl__initiate_device_remove: unable 
> to get my domid
> libxl: error: libxl.c:1452:devices_destroy_cb: libxl__devices_destroy 
> failed for 1
> 
> Libxl code assumes that domcreate_attach_vtpms handler called, that 
> calls domcreate_attach_pci.
> Why vif is logically depend on PCI and vtpm on ARM platform? (it is 
> confusing).

It's a nasty side effect of the way libxl is structured internally as a
sequence of steps. The vtpm step follows the nic test and each step
checks for success failure in the previous step. This is very
confusing...

Ian.

Re: Unable to create VM with nic device on Arndale

[Stefano Stabellini]

On Wed, 18 Dec 2013, Eugene Fedotov wrote:
> Hello!
> Could someone check VIF creation on ARM ?
> 
> My domU's config is following:
> extra = "console=hvc0 debug ipv6.disable=1";
> kernel = "/boot/zImageU"
> memory = 128
> name = "domU-1"
> vcpus = 1
> vif = [ 'script=/etc/xen/scripts/vif-route, ip=10.0.0.1' ];
> 
> "xl create domU.conf" causes following output:
> Parsing config from domU.conf
> libxl: error: libxl_create.c:1180:domcreate_attach_vtpms: unable to add nic
> devices
> libxl: error: libxl_device.c:780:libxl__initiate_device_remove: unable to get
> my domid
> libxl: error: libxl.c:1452:devices_destroy_cb: libxl__devices_destroy failed
> for 1
> 
> Libxl code assumes that domcreate_attach_vtpms handler called, that calls
> domcreate_attach_pci.
> Why vif is logically depend on PCI and vtpm on ARM platform? (it is
> confusing).

domcreate_attach_vtpms shouldn't actually do anything unless you
requested a vtpm in the VM config, that you haven't done.

Is it possible that the error message is actually wrong?
I guess the error is just related to the creation of the vif somehow but
it is printed by domcreate_attach_vtpms because it is the first callback
being called.

Re: Unable to create VM with nic device on Arndale

[Ian Jackson]

Stefano Stabellini writes ("Re: [Xen-devel] Unable to create VM with nic device on Arndale"):
> On Wed, 18 Dec 2013, Eugene Fedotov wrote:
...
> > "xl create domU.conf" causes following output:
> > Parsing config from domU.conf
> > libxl: error: libxl_create.c:1180:domcreate_attach_vtpms: unable to add nic
> > devices
...
> > Libxl code assumes that domcreate_attach_vtpms handler called, that calls
> > domcreate_attach_pci.
> > Why vif is logically depend on PCI and vtpm on ARM platform? (it is
> > confusing).

This is just the asynchronous callback structure.  There's a function
which starts the addition of NICs and arranges when it's done for the
next function to be called, which is the one for adding vtpms.

> Is it possible that the error message is actually wrong?
> I guess the error is just related to the creation of the vif somehow but
> it is printed by domcreate_attach_vtpms because it is the first callback
> being called.

Exactly.

It's very unhelpful that this is the only message being printed.
There really ought to be a message about which NIC failed.

Can you try it again with "xl -vvvv" ?

Ian.

Re: Unable to create VM with nic device on Arndale

[Ian Campbell]

On Wed, 2013-12-18 at 12:46 +0000, Ian Jackson wrote:
> Stefano Stabellini writes ("Re: [Xen-devel] Unable to create VM with nic device on Arndale"):
> > On Wed, 18 Dec 2013, Eugene Fedotov wrote:
> ...
> > > "xl create domU.conf" causes following output:
> > > Parsing config from domU.conf
> > > libxl: error: libxl_create.c:1180:domcreate_attach_vtpms: unable to add nic
> > > devices
> ...
> > > Libxl code assumes that domcreate_attach_vtpms handler called, that calls
> > > domcreate_attach_pci.
> > > Why vif is logically depend on PCI and vtpm on ARM platform? (it is
> > > confusing).
> 
> This is just the asynchronous callback structure.  There's a function
> which starts the addition of NICs and arranges when it's done for the
> next function to be called, which is the one for adding vtpms.

The confusion arises because our LOG macros include the function name,
so checking for nic errors in a vtpm function is always going to produce
something confusing.

We should introduce a veneer here, e.g. nic_done(), which logs any nic
related errors and then calls the next step. Same for each step in the
async chain.

Ian.

Re: Unable to create VM with nic device on Arndale

[Eugene Fedotov]

> Stefano Stabellini writes ("Re: [Xen-devel] Unable to create VM with nic device on Arndale"):
>> On Wed, 18 Dec 2013, Eugene Fedotov wrote:
> ...
>>> "xl create domU.conf" causes following output:
>>> Parsing config from domU.conf
>>> libxl: error: libxl_create.c:1180:domcreate_attach_vtpms: unable to add nic
>>> devices
> ...
>>> Libxl code assumes that domcreate_attach_vtpms handler called, that calls
>>> domcreate_attach_pci.
>>> Why vif is logically depend on PCI and vtpm on ARM platform? (it is
>>> confusing).
> This is just the asynchronous callback structure.  There's a function
> which starts the addition of NICs and arranges when it's done for the
> next function to be called, which is the one for adding vtpms.
>
>> Is it possible that the error message is actually wrong?
>> I guess the error is just related to the creation of the vif somehow but
>> it is printed by domcreate_attach_vtpms because it is the first callback
>> being called.
> Exactly.
>
> It's very unhelpful that this is the only message being printed.
> There really ought to be a message about which NIC failed.
>
> Can you try it again with "xl -vvvv" ?

arndale# xl -vvvv create domU.conf
Parsing config from domU.conf
libxl: debug: libxl_create.c:1296:do_domain_create: ao 0x1b3af28: 
create: how=(nil) callback=(nil) poller=0x1b34e88
libxl: verbose: libxl_create.c:134:libxl__domain_build_info_setdefault: 
qemu-xen is unavailable, use qemu-xen-traditional instead: No such file 
or directory
libxl: debug: libxl_create.c:740:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no 
bootloader configured, using user supplied kernel
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch 
w=0x1b35188: deregister unregistered
libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best NUMA 
placement candidate found: nr_nodes=1, nr_cpus=1, nr_vcpus=2, 
free_memkb=1500
libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement 
candidate with 1 nodes, 1 cpus and 1500 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="console=hvc0 debug 
rdinit=/bin/sh", features="(null)"
libxl: debug: libxl_dom.c:353:libxl__build_pv: pv kernel mapped 0 path 
/boot/zImageU
domainbuilder: detail: xc_dom_kernel_file: filename="/boot/zImageU"
domainbuilder: detail: xc_dom_malloc_filemap    : 2816 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps xen-3.0-armv7l
domainbuilder: detail: xc_dom_rambase_init: RAM starts at 80000
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary 
loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64) 
loader ...
domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an 
arm64 Image
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32) 
loader ...
domainbuilder: detail: loader probe OK
domainbuilder: detail: xc_dom_parse_zimage32_kernel: called
domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l: 
0x80008000 -> 0x802c8158
libxl: debug: libxl_arm.c:433:libxl__arch_domain_configure: constructing 
DTB for Xen version 4.4 guest
libxl: debug: libxl_arm.c:497:libxl__arch_domain_configure: fdt total 
size 1174
domainbuilder: detail: xc_dom_devicetree_mem: called
domainbuilder: detail: xc_dom_mem_init: mem 128 MB, pages 0x8000 pages, 
4k each
domainbuilder: detail: xc_dom_mem_init: 0x8000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32
domainbuilder: detail: xc_dom_malloc            : 256 kB
domainbuilder: detail: arch_setup_meminit: devicetree: 0x87fff000 -> 
0x87fff496
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x80008000 
-> 0x802c9000  (pfn 0x80008 + 0x2c1 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x80008+0x2c1 at 0xb6771000
domainbuilder: detail: xc_dom_load_zimage_kernel: called
domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg 
0x80008000-0x802c9000
domainbuilder: detail: xc_dom_load_zimage_kernel: copy 2883928 bytes 
from blob 0xb6a73000 to dst 0xb6771000
domainbuilder: detail: xc_dom_alloc_segment:   devicetree   : 0x87fff000 
-> 0x88000000  (pfn 0x87fff + 0x1 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x87fff+0x1 at 0xb6770000
domainbuilder: detail: alloc_magic_pages: called
domainbuilder: detail: count_pgtables_arm: called
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x88000000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-armv7l <= matches
domainbuilder: detail: setup_pgtables_arm: called
domainbuilder: detail: clear_page: pfn 0x88000, mfn 0x88000
domainbuilder: detail: clear_page: pfn 0x88001, mfn 0x88001
domainbuilder: detail: start_info_arm: called
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 274 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 2816 kB
domainbuilder: detail:       domU mmap          : 2824 kB
domainbuilder: detail: vcpu_arm32: called
domainbuilder: detail: Initial state CPSR 0x1d3 PC 0x80008000
domainbuilder: detail: launch_vm: called, ctxt=0xb6ff6004
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch 
w=0x1b37078 wpath=/local/domain/0/backend/vif/12/0/state token=3/0: 
register slotnum=3
libxl: debug: libxl_create.c:1310:do_domain_create: ao 0x1b3af28: 
inprogress: poller=0x1b34e88, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1b37078 
wpath=/local/domain/0/backend/vif/12/0/state token=3/0: event 
epath=/local/domain/0/backend/vif/12/0/state
libxl: debug: libxl_event.c:646:devstate_watch_callback: backend 
/local/domain/0/backend/vif/12/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1b37078 
wpath=/local/domain/0/backend/vif/12/0/state token=3/0: event 
epath=/local/domain/0/backend/vif/12/0/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend 
/local/domain/0/backend/vif/12/0/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch 
w=0x1b37078 wpath=/local/domain/0/backend/vif/12/0/state token=3/0: 
deregister slotnum=3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch 
w=0x1b37078: deregister unregistered
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch 
w=0x1b370c0: deregister unregistered
libxl: error: libxl_create.c:1180:domcreate_attach_vtpms: unable to add 
nic devices
libxl: error: libxl_device.c:780:libxl__initiate_device_remove: unable 
to get my domid
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch 
w=0x1b37e88: deregister unregistered
libxl: error: libxl.c:1452:devices_destroy_cb: libxl__devices_destroy 
failed for 12
libxl: debug: libxl_event.c:1560:libxl__ao_complete: ao 0x1b3af28: 
complete, rc=-3
libxl: debug: libxl_event.c:1532:libxl__ao__destroy: ao 0x1b3af28: destroy
xc: debug: hypercall buffer: total allocations:216 total releases:216
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:208 misses:4 toobig:4

Best regards,
Evgeny

Re: Unable to create VM with nic device on Arndale

[Ian Campbell]

On Wed, 2013-12-18 at 17:23 +0400, Eugene Fedotov wrote:

> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1b37078 
> wpath=/local/domain/0/backend/vif/12/0/state token=3/0: event 
> epath=/local/domain/0/backend/vif/12/0/state
> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend 
> /local/domain/0/backend/vif/12/0/state wanted state 2 still waiting state 1

Do you have netback enabled in your dom0 kernel?

Re: Unable to create VM with nic device on Arndale

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] Unable to create VM with nic device on Arndale"):
> The confusion arises because our LOG macros include the function name,
> so checking for nic errors in a vtpm function is always going to produce
> something confusing.
>
> We should introduce a veneer here, e.g. nic_done(), which logs any nic
> related errors and then calls the next step. Same for each step in the
> async chain.

That would be tangling the code just to make the logging output
right.  Perhaps we could just fix the logging to DTRT.

How about a new logging macro which takes an argument for the context
to print instead of the function name ?

   static void domcreate_attach_vtpms(libxl__egc *egc,
                                      libxl__multidev *multidev,
                                      int ret)
   {
...
      libxl_domain_config* const d_config = dcs->guest_config;

      if (ret) {
-         LOG(ERROR, "unable to add nic devices");
+         CLOG("domcreate nics", ERROR, "unable to add nic devices");
          goto error_out;
      }

We'd probably only need to change about one call in each callback
function - the one which prints an error message for the previous
operation.

Ian.

Re: Unable to create VM with nic device on Arndale

[Eugene Fedotov]

18.12.2013 17:30, Ian Campbell пишет:
> On Wed, 2013-12-18 at 17:23 +0400, Eugene Fedotov wrote:
>
>> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1b37078
>> wpath=/local/domain/0/backend/vif/12/0/state token=3/0: event
>> epath=/local/domain/0/backend/vif/12/0/state
>> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend
>> /local/domain/0/backend/vif/12/0/state wanted state 2 still waiting state 1
> Do you have netback enabled in your dom0 kernel?

Looks like yes.

zcat /proc/config.gz | grep XEN
CONFIG_XEN_DOM0=y
CONFIG_XEN=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
# CONFIG_XEN_NETDEV_FRONTEND is not set
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_XEN_FBDEV_FRONTEND is not set
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_XEN_PRIVCMD=y

Best regards,
Eugene.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Unable to create VM with nic device on Arndale

[Eugene Fedotov]

> On Wed, 2013-12-18 at 17:23 +0400, Eugene Fedotov wrote:
>
>> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1b37078
>> wpath=/local/domain/0/backend/vif/12/0/state token=3/0: event
>> epath=/local/domain/0/backend/vif/12/0/state
>> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend
>> /local/domain/0/backend/vif/12/0/state wanted state 2 still waiting state 1
> Do you have netback enabled in your dom0 kernel?
>
>
Yes, 3.9.1 kernel.
Can it be outdated ?

Best regards,
Eugene.

Re: Unable to create VM with nic device on Arndale

[Ian Campbell]

On Wed, 2013-12-18 at 14:44 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] Unable to create VM with nic device on Arndale"):
> > The confusion arises because our LOG macros include the function name,
> > so checking for nic errors in a vtpm function is always going to produce
> > something confusing.
> >
> > We should introduce a veneer here, e.g. nic_done(), which logs any nic
> > related errors and then calls the next step. Same for each step in the
> > async chain.
> 
> That would be tangling the code just to make the logging output
> right.

I'm not sure of that, I think having paired start+done functions would
actually be clearer in its own right, but it would also make the error
paths less obtuse IMHO, you'd avoid every function being:
	if (some previous failure)
		LOG(something about previous)
		goto out

	setup something completely unrelated to previous

	out:
		next step

In the specific case of nics you would probably build the done function
into the multidev thing with a separate callback for success vs failure.

We already have the start+done pairing in some cases (e.g. the dm
started case).

>   Perhaps we could just fix the logging to DTRT.
> 
> How about a new logging macro which takes an argument for the context
> to print instead of the function name ?
> 
>    static void domcreate_attach_vtpms(libxl__egc *egc,
>                                       libxl__multidev *multidev,
>                                       int ret)
>    {
> ...
>       libxl_domain_config* const d_config = dcs->guest_config;
> 
>       if (ret) {
> -         LOG(ERROR, "unable to add nic devices");
> +         CLOG("domcreate nics", ERROR, "unable to add nic devices");
>           goto error_out;
>       }
> 
> We'd probably only need to change about one call in each callback
> function - the one which prints an error message for the previous
> operation.
> 
> Ian.

Re: Unable to create VM with nic device on Arndale

[Dennis Lan (dlan)]

On Wed, Dec 18, 2013 at 10:49 PM, Eugene Fedotov <e.fedotov@samsung.com> wrote:
>> On Wed, 2013-12-18 at 17:23 +0400, Eugene Fedotov wrote:
>>
>>> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1b37078
>>> wpath=/local/domain/0/backend/vif/12/0/state token=3/0: event
>>> epath=/local/domain/0/backend/vif/12/0/state
>>> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend
>>> /local/domain/0/backend/vif/12/0/state wanted state 2 still waiting state
>>> 1
>>
>> Do you have netback enabled in your dom0 kernel?
>>
>>
> Yes, 3.9.1 kernel.
> Can it be outdated ?
>
> Best regards,
> Eugene.
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


Hi ALL

I'm also hit this problem, using xen-4.4.0-rc2 release,  on x86_64 machine.
but I've tested with xen-4.3.1 the same settings, and have no problem

if I remove "vif    = ['mac=00:16:3E:8E:84:00']", domU can boot up successfully
but network is not usable.

kernel settings: 3.12.6
# zcat /proc/config.gz |grep XEN
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_DEBUG_FS=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_XEN_WDT=y
CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_XEN_BALLOON=y
# CONFIG_XEN_SELFBALLOONING is not set
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=m
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_ACPI_PROCESSOR=y
CONFIG_XEN_MCE_LOG=y
CONFIG_XEN_HAVE_PVMMU=y

# cat test1_stable
kernel = "kernel/kernel-genkernel-x86_64-3.12.6"
ramdisk = "kernel/initramfs-genkernel-x86_64-3.12.6"
extra = "root=/dev/xvda1 ro rootfstype=ext4 console=hvc0"
memory = 2048
vcpus = 2
name = "test1_stable"
vif    = ['mac=00:16:3E:8E:84:00']
disk   = ['file:/mnt/proj/xen/gen1_stable_ext4.img,xvda1,w']

# xl -vvvv create -c test1_stable
Parsing config from test1_stable
libxl: debug: libxl_create.c:1322:do_domain_create: ao 0x62a640:
create: how=(nil) callback=(nil) poller=0x629f90
libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk
vdev=xvda1 spec.backend=unknown
libxl: debug: libxl_device.c:197:disk_try_backend: Disk vdev=xvda1,
backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:286:libxl__device_disk_set_backend: Disk
vdev=xvda1, using backend qdisk
libxl: debug: libxl_create.c:777:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no
bootloader configured, using user supplied kernel
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch
w=0x628858: deregister unregistered
libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best
NUMA placement candidate found: nr_nodes=1, nr_cpus=4, nr_vcpus=6,
free_memkb=2288
libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement
candidate with 1 nodes, 4 cpus and 2288 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvda1 ro
rootfstype=ext4 console=hvc0", features="(null)"
libxl: debug: libxl_dom.c:357:libxl__build_pv: pv kernel mapped 0 path
kernel/kernel-genkernel-x86_64-3.12.6
domainbuilder: detail: xc_dom_kernel_file:
filename="kernel/kernel-genkernel-x86_64-3.12.6"
domainbuilder: detail: xc_dom_malloc_filemap    : 8185 kB
domainbuilder: detail: xc_dom_ramdisk_file:
filename="kernel/initramfs-genkernel-x86_64-3.12.6"
domainbuilder: detail: xc_dom_malloc_filemap    : 4624 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps
xen-3.0-x86_64 xen-3.0-x86_32p
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_malloc            : 25202 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x7f557c -> 0x189ca00
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x1000000 memsz=0x1141000
xc: detail: elf_parse_binary: phdr: paddr=0x2200000 memsz=0x11c0f0
xc: detail: elf_parse_binary: phdr: paddr=0x231d000 memsz=0x14c80
xc: detail: elf_parse_binary: phdr: paddr=0x2332000 memsz=0x700000
xc: detail: elf_parse_binary: memory: 0x1000000 -> 0x2a32000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff823321e0
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
xc: detail: elf_xen_parse_note: FEATURES =
"!writable_page_tables|pae_pgdir_above_4gb"
xc: detail: elf_xen_parse_note: PAE_MODE = "yes"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0xffffffff80000000
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0xffffffff80000000
xc: detail:     virt_kstart      = 0xffffffff81000000
xc: detail:     virt_kend        = 0xffffffff82a32000
xc: detail:     virt_entry       = 0xffffffff823321e0
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64:
0xffffffff81000000 -> 0xffffffff82a32000
domainbuilder: detail: xc_dom_mem_init: mem 2048 MB, pages 0x80000
pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x80000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc            : 4096 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment:   kernel       :
0xffffffff81000000 -> 0xffffffff82a32000  (pfn 0x1000 + 0x1a32 pages)
domainbuilder: detail: xc_dom_malloc            : 157 kB
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x1000+0x1a32 at 0x7f8e1573a000
xc: detail: elf_load_binary: phdr 0 at 0x7f8e1573a000 -> 0x7f8e1687b000
xc: detail: elf_load_binary: phdr 1 at 0x7f8e1693a000 -> 0x7f8e16a560f0
xc: detail: elf_load_binary: phdr 2 at 0x7f8e16a57000 -> 0x7f8e16a6bc80
xc: detail: elf_load_binary: phdr 3 at 0x7f8e16a6c000 -> 0x7f8e16bd6000
domainbuilder: detail: xc_dom_alloc_segment:   ramdisk      :
0xffffffff82a32000 -> 0xffffffff82eb7000  (pfn 0x2a32 + 0x485 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x2a32+0x485 at 0x7f8e152b5000
domainbuilder: detail: xc_dom_alloc_segment:   phys2mach    :
0xffffffff82eb7000 -> 0xffffffff832b7000  (pfn 0x2eb7 + 0x400 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x2eb7+0x400 at 0x7f8e14eb5000
domainbuilder: detail: xc_dom_alloc_page   :   start info   :
0xffffffff832b7000 (pfn 0x32b7)
domainbuilder: detail: xc_dom_alloc_page   :   xenstore     :
0xffffffff832b8000 (pfn 0x32b8)
domainbuilder: detail: xc_dom_alloc_page   :   console      :
0xffffffff832b9000 (pfn 0x32b9)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48:
0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39:
0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30:
0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21:
0xffffffff80000000 -> 0xffffffff833fffff, 26 table(s)
domainbuilder: detail: xc_dom_alloc_segment:   page tables  :
0xffffffff832ba000 -> 0xffffffff832d7000  (pfn 0x32ba + 0x1d pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x32ba+0x1d at 0x7f8e1bf5c000
domainbuilder: detail: xc_dom_alloc_page   :   boot stack   :
0xffffffff832d7000 (pfn 0x32d7)
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0xffffffff832d8000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0xffffffff83400000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type:
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x80000
domainbuilder: detail: clear_page: pfn 0x32b9, mfn 0x141305
domainbuilder: detail: clear_page: pfn 0x32b8, mfn 0x1bcc66
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
0x32b7+0x1 at 0x7f8e1bfc5000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff81001000 pfn=0x1001
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 29510 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 12809 kB
domainbuilder: detail:       domU mmap          : 34 MB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xba9eb
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x32ba mfn 0x141304
domainbuilder: detail: launch_vm: called, ctxt=0x7f8e1bfc6004
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk
vdev=xvda1 spec.backend=qdisk
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch
w=0x629a80: deregister unregistered
libxl: debug: libxl_dm.c:1303:libxl__spawn_local_dm: Spawning
device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:
/usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   5
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-5,server,nowait
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:
chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -nodefaults
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -xen-attach
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   test1_stable
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -nographic
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -machine
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   xenpv
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   2049
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch
w=0x628a90 wpath=/local/domain/0/device-model/5/state token=3/0:
register slotnum=3
libxl: debug: libxl_create.c:1336:do_domain_create: ao 0x62a640:
inprogress: poller=0x629f90, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x628a90
wpath=/local/domain/0/device-model/5/state token=3/0: event
epath=/local/domain/0/device-model/5/state
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x628a90
wpath=/local/domain/0/device-model/5/state token=3/0: event
epath=/local/domain/0/device-model/5/state
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch
w=0x628a90 wpath=/local/domain/0/device-model/5/state token=3/0:
deregister slotnum=3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch
w=0x628a90: deregister unregistered
libxl: debug: libxl_qmp.c:696:libxl__qmp_initialize: connected to
/var/run/xen/qmp-libxl-5
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: '{
    "execute": "qmp_capabilities",
    "id": 1
}
'
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: '{
    "execute": "query-chardev",
    "id": 2
}
'
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: '{
    "execute": "query-vnc",
    "id": 3
}
'
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch
w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1:
register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8
wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event
epath=/local/domain/0/backend/vif/5/0/state
libxl: debug: libxl_event.c:646:devstate_watch_callback: backend
/local/domain/0/backend/vif/5/0/state wanted state 2 still waiting
state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8
wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event
epath=/local/domain/0/backend/vif/5/0/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend
/local/domain/0/backend/vif/5/0/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch
w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1:
deregister slotnum=3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch
w=0x62acd8: deregister unregistered
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch
w=0x62ad60: deregister unregistered
libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to
add nic devices
libxl: debug: libxl_dm.c:1478:kill_device_model: Device Model signaled
libxl: error: libxl_device.c:780:libxl__initiate_device_remove: unable
to get my domid
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch
w=0x62b800: deregister unregistered
libxl: error: libxl_device.c:780:libxl__initiate_device_remove: unable
to get my domid
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch
w=0x62bb10: deregister unregistered
libxl: error: libxl.c:1461:devices_destroy_cb: libxl__devices_destroy
failed for 5
libxl: debug: libxl_event.c:1560:libxl__ao_complete: ao 0x62a640:
complete, rc=-3
libxl: debug: libxl_event.c:1532:libxl__ao__destroy: ao 0x62a640: destroy
xc: debug: hypercall buffer: total allocations:1850 total releases:1850
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:1839 misses:4 toobig:7

Re: Unable to create VM with nic device on Arndale

[Ian Campbell]

On Fri, 2014-01-17 at 17:33 +0800, Dennis Lan (dlan) wrote:
> I'm also hit this problem, using xen-4.4.0-rc2 release,  on x86_64 machine.
> but I've tested with xen-4.3.1 the same settings, and have no problem

Do you have a bridge configured? What is the output of "brctl show" and
"ifconfig -a" while the guest is running?

> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: register slotnum=3
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 still waiting state 1
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
> libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 ok
> libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: deregister slotnum=3
> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62acd8: deregister unregistered
> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62ad60: deregister unregistered
> libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to add nic devices

It seems to have waited for the backend to hit state 2, which happens,
and then it has failed for some reason, which I can't see here. It might
be worth adding some debug to the vif hotplug script to see if it is
running at all.

Are you using the vif script's from Xen 4.4? Perhaps you have got some
stale 4.3.1 scripts sitting around?

Ian.

Re: Unable to create VM with nic device on Arndale

[Dennis Lan (dlan)]

On Fri, Jan 17, 2014 at 5:46 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-01-17 at 17:33 +0800, Dennis Lan (dlan) wrote:
>> I'm also hit this problem, using xen-4.4.0-rc2 release,  on x86_64 machine.
>> but I've tested with xen-4.3.1 the same settings, and have no problem
>
> Do you have a bridge configured? What is the output of "brctl show" and
> "ifconfig -a" while the guest is running?
>
>> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: register slotnum=3
>> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
>> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 still waiting state 1
>> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
>> libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 ok
>> libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: deregister slotnum=3
>> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62acd8: deregister unregistered
>> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62ad60: deregister unregistered
>> libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to add nic devices
>
> It seems to have waited for the backend to hit state 2, which happens,
> and then it has failed for some reason, which I can't see here. It might
> be worth adding some debug to the vif hotplug script to see if it is
> running at all.
>
> Are you using the vif script's from Xen 4.4? Perhaps you have got some
> stale 4.3.1 scripts sitting around?
>
> Ian.
>
I think there is no problem with bridge and script(exactly same as 4.4)
btw, If I install xen-4.3.1, then everyting works fine.


ofire xen-4.4.0-rc2 # ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 379790  bytes 215154226 (205.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 379790  bytes 215154226 (205.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::221:9bff:fe74:2cd1  prefixlen 64  scopeid 0x20<link>
        ether 00:21:9b:74:2c:d1  txqueuelen 1000  (Ethernet)
        RX packets 1017893  bytes 174572838 (166.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 577828  bytes 627464455 (598.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 21  memory 0xf7ae0000-f7b00000

xenbr0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet 192.168.90.122  netmask 255.255.254.0  broadcast 192.168.90.255
        inet6 fe80::221:9bff:fe74:2cd1  prefixlen 64  scopeid 0x20<link>
        ether 00:21:9b:74:2c:d1  txqueuelen 0  (Ethernet)
        RX packets 1015312  bytes 156011479 (148.7 MiB)
        RX errors 0  dropped 10  overruns 0  frame 0
        TX packets 216345  bytes 600980890 (573.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ofire xen-4.4.0-rc2 # br
brctl       break       bridge      brushtopbm
ofire xen-4.4.0-rc2 # brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.00219b742cd1       no              net0


ofire xen-4.4.0-rc2 # find . -name "vif-*" -and -type f | sort -u | xargs md5sum
f144e27878d656226202a2ec3f797167  ./tools/hotplug/Linux/vif-bridge
6dbe122d5bb8cc82cea01c3928f6361d  ./tools/hotplug/Linux/vif-common.sh
02a2614f1451b4207c82305a93bfa060  ./tools/hotplug/Linux/vif-nat
478061920ef927f570d6f8970be34926  ./tools/hotplug/Linux/vif-openvswitch
b8417ff8aa76ea6aef8fdd57fffe7677  ./tools/hotplug/Linux/vif-route
96d12dbf85c3823b2a644e58bf3fbc73  ./tools/hotplug/Linux/vif-setup
295971747eb386c8cc42b9f6907a6e8e  ./tools/hotplug/NetBSD/vif-bridge
b9ea5011dc0f72a69c8fa572de997954  ./tools/hotplug/NetBSD/vif-ip
ofire xen-4.4.0-rc2 # ls /etc/xen/scripts/vif-* |sort -u |xargs md5sum
f144e27878d656226202a2ec3f797167  /etc/xen/scripts/vif-bridge
6dbe122d5bb8cc82cea01c3928f6361d  /etc/xen/scripts/vif-common.sh
02a2614f1451b4207c82305a93bfa060  /etc/xen/scripts/vif-nat
478061920ef927f570d6f8970be34926  /etc/xen/scripts/vif-openvswitch
b8417ff8aa76ea6aef8fdd57fffe7677  /etc/xen/scripts/vif-route
96d12dbf85c3823b2a644e58bf3fbc73  /etc/xen/scripts/vif-setup

Re: Unable to create VM with nic device on Arndale

[Ian Campbell]

On Fri, 2014-01-17 at 18:37 +0800, Dennis Lan (dlan) wrote:
> On Fri, Jan 17, 2014 at 5:46 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Fri, 2014-01-17 at 17:33 +0800, Dennis Lan (dlan) wrote:
> >> I'm also hit this problem, using xen-4.4.0-rc2 release,  on x86_64 machine.
> >> but I've tested with xen-4.3.1 the same settings, and have no problem
> >
> > Do you have a bridge configured? What is the output of "brctl show" and
> > "ifconfig -a" while the guest is running?
> >
> >> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: register slotnum=3
> >> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
> >> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 still waiting state 1
> >> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
> >> libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 ok
> >> libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: deregister slotnum=3
> >> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62acd8: deregister unregistered
> >> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62ad60: deregister unregistered
> >> libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to add nic devices
> >
> > It seems to have waited for the backend to hit state 2, which happens,
> > and then it has failed for some reason, which I can't see here. It might
> > be worth adding some debug to the vif hotplug script to see if it is
> > running at all.
> >
> > Are you using the vif script's from Xen 4.4? Perhaps you have got some
> > stale 4.3.1 scripts sitting around?
> >
> > Ian.
> >
> I think there is no problem with bridge and script(exactly same as 4.4)

I still think the next step is to instrument the scripts and see what is
going on.

> btw, If I install xen-4.3.1, then everyting works fine.

Have you looked into the differences?

Ian.

Re: Unable to create VM with nic device on Arndale

[Dennis Lan (dlan)]

On Fri, Jan 17, 2014 at 6:44 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-01-17 at 18:37 +0800, Dennis Lan (dlan) wrote:
>> On Fri, Jan 17, 2014 at 5:46 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > On Fri, 2014-01-17 at 17:33 +0800, Dennis Lan (dlan) wrote:
>> >> I'm also hit this problem, using xen-4.4.0-rc2 release,  on x86_64 machine.
>> >> but I've tested with xen-4.3.1 the same settings, and have no problem
>> >
>> > Do you have a bridge configured? What is the output of "brctl show" and
>> > "ifconfig -a" while the guest is running?
>> >
>> >> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: register slotnum=3
>> >> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
>> >> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 still waiting state 1
>> >> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
>> >> libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 ok
>> >> libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: deregister slotnum=3
>> >> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62acd8: deregister unregistered
>> >> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62ad60: deregister unregistered
>> >> libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to add nic devices
>> >
>> > It seems to have waited for the backend to hit state 2, which happens,
>> > and then it has failed for some reason, which I can't see here. It might
>> > be worth adding some debug to the vif hotplug script to see if it is
>> > running at all.
>> >
>> > Are you using the vif script's from Xen 4.4? Perhaps you have got some
>> > stale 4.3.1 scripts sitting around?
>> >
>> > Ian.
>> >
>> I think there is no problem with bridge and script(exactly same as 4.4)
>
> I still think the next step is to instrument the scripts and see what is
> going on.
do you have any suggestion of how I should locate this problem?
which script or where I can add debug info.
headless here.

>
>> btw, If I install xen-4.3.1, then everyting works fine.
>
> Have you looked into the differences?
>
Not really, I copy the vif-* script from 4.3.1 to /etc/xen/scripts/
and still got the same problem

> Ian.
>

Re: Unable to create VM with nic device on Arndale

[Ian Campbell]

On Fri, 2014-01-17 at 18:55 +0800, Dennis Lan (dlan) wrote:
> On Fri, Jan 17, 2014 at 6:44 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Fri, 2014-01-17 at 18:37 +0800, Dennis Lan (dlan) wrote:
> >> On Fri, Jan 17, 2014 at 5:46 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > On Fri, 2014-01-17 at 17:33 +0800, Dennis Lan (dlan) wrote:
> >> >> I'm also hit this problem, using xen-4.4.0-rc2 release,  on x86_64 machine.
> >> >> but I've tested with xen-4.3.1 the same settings, and have no problem
> >> >
> >> > Do you have a bridge configured? What is the output of "brctl show" and
> >> > "ifconfig -a" while the guest is running?
> >> >
> >> >> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: register slotnum=3
> >> >> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
> >> >> libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 still waiting state 1
> >> >> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: event epath=/local/domain/0/backend/vif/5/0/state
> >> >> libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 ok
> >> >> libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/5/0/state token=3/1: deregister slotnum=3
> >> >> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62acd8: deregister unregistered
> >> >> libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62ad60: deregister unregistered
> >> >> libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to add nic devices
> >> >
> >> > It seems to have waited for the backend to hit state 2, which happens,
> >> > and then it has failed for some reason, which I can't see here. It might
> >> > be worth adding some debug to the vif hotplug script to see if it is
> >> > running at all.
> >> >
> >> > Are you using the vif script's from Xen 4.4? Perhaps you have got some
> >> > stale 4.3.1 scripts sitting around?
> >> >
> >> > Ian.
> >> >
> >> I think there is no problem with bridge and script(exactly same as 4.4)
> >
> > I still think the next step is to instrument the scripts and see what is
> > going on.
> do you have any suggestion of how I should locate this problem?
> which script or where I can add debug info.

vif-bridge and the common scripts which it includes would be a good
start. Just an echo at the top to confirm that the script is running
would be useful.

I used to do "exec 1>/tmp/hotplug.log 2>&1" at the top to aid debugging
when these scripts were launched by udev, but now that libxl runs them
you may find that the debug from the script comes out on stdout/err of
the xl create command so perhaps that isn't needed any more.

> headless here.

That shouldn't matter, you are looking for output from userspace
scripts, not kernel or hypervisor logs.

Ian.

Re: Unable to create VM with nic device on Arndale

[Dennis Lan (dlan)]

On Fri, Jan 17, 2014 at 7:01 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:


> vif-bridge and the common scripts which it includes would be a good
> start. Just an echo at the top to confirm that the script is running
> would be useful.
>
> I used to do "exec 1>/tmp/hotplug.log 2>&1" at the top to aid debugging
> when these scripts were launched by udev, but now that libxl runs them
> you may find that the debug from the script comes out on stdout/err of
> the xl create command so perhaps that isn't needed any more.
>
>> headless here.
>
> That shouldn't matter, you are looking for output from userspace
> scripts, not kernel or hypervisor logs.
>
> Ian.
>

Hi Ian
I suspect for 4.4.0, the network devices even was not detected.
this is output from 4.3.1, notes follow lines.

libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug
script: /etc/xen/scripts/vif-bridge online
dlan: vif-bridge start
dlan: vif-common start

dlan: vif-bridge start -> output from vif-bridge script
dlan: vif-common start -> output from vif-common.sh script


-------- logs -----
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch
w=0x62ab58 wpath=/local/domain/0/backend/vif/7/0/
state token=3/1: register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62ab58
wpath=/local/domain/0/backend/vif/7/0/state toke
n=3/1: event epath=/local/domain/0/backend/vif/7/0/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend
/local/domain/0/backend/vif/7/0/state wanted state
 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62ab58
wpath=/local/domain/0/backend/vif/7/0/state toke
n=3/1: event epath=/local/domain/0/backend/vif/7/0/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend
/local/domain/0/backend/vif/7/0/state wanted state
 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch
w=0x62ab58 wpath=/local/domain/0/backend/vif/7/
0/state token=3/1: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch
w=0x62ab58: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug
script: /etc/xen/scripts/vif-bridge online
dlan: vif-bridge start
dlan: vif-common start
libxl: debug: libxl_event.c:1747:libxl__ao_progress_report: ao
0x62a570: progress report: callback queued aop=0x62b
560
libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x62a570: complete, rc=0
libxl: debug: libxl_event.c:1160:egc_run_callbacks: ao 0x62a570:
progress report: callback aop=0x62b560

Re: Unable to create VM with nic device on Arndale

[Ian Campbell]

On Fri, 2014-01-17 at 19:43 +0800, Dennis Lan (dlan) wrote:
> On Fri, Jan 17, 2014 at 7:01 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> 
> 
> > vif-bridge and the common scripts which it includes would be a good
> > start. Just an echo at the top to confirm that the script is running
> > would be useful.
> >
> > I used to do "exec 1>/tmp/hotplug.log 2>&1" at the top to aid debugging
> > when these scripts were launched by udev, but now that libxl runs them
> > you may find that the debug from the script comes out on stdout/err of
> > the xl create command so perhaps that isn't needed any more.
> >
> >> headless here.
> >
> > That shouldn't matter, you are looking for output from userspace
> > scripts, not kernel or hypervisor logs.
> >
> > Ian.
> >
> 
> Hi Ian
> I suspect for 4.4.0, the network devices even was not detected.
> this is output from 4.3.1, notes follow lines.
> 
> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug
> script: /etc/xen/scripts/vif-bridge online
> dlan: vif-bridge start
> dlan: vif-common start
> 
> dlan: vif-bridge start -> output from vif-bridge script
> dlan: vif-common start -> output from vif-common.sh script

So these are the 4.3 logs? Have you tried 4.4 and found that it doesn't
produce the same output?

(please can you try and set the text type to "preformatted" for the logs
-- having them wrapped makes them very hard to read).

The lack of 
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online
in your original logs is a bit concerning.

Roger -- any ideas?
> 
> 
> -------- logs -----
> libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
> libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch
> w=0x62ab58 wpath=/local/domain/0/backend/vif/7/0/
> state token=3/1: register slotnum=3
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62ab58
> wpath=/local/domain/0/backend/vif/7/0/state toke
> n=3/1: event epath=/local/domain/0/backend/vif/7/0/state
> libxl: debug: libxl_event.c:647:devstate_watch_callback: backend
> /local/domain/0/backend/vif/7/0/state wanted state
>  2 still waiting state 1
> libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62ab58
> wpath=/local/domain/0/backend/vif/7/0/state toke
> n=3/1: event epath=/local/domain/0/backend/vif/7/0/state
> libxl: debug: libxl_event.c:643:devstate_watch_callback: backend
> /local/domain/0/backend/vif/7/0/state wanted state
>  2 ok
> libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch
> w=0x62ab58 wpath=/local/domain/0/backend/vif/7/
> 0/state token=3/1: deregister slotnum=3
> libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch
> w=0x62ab58: deregister unregistered
> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug
> script: /etc/xen/scripts/vif-bridge online
> dlan: vif-bridge start
> dlan: vif-common start
> libxl: debug: libxl_event.c:1747:libxl__ao_progress_report: ao
> 0x62a570: progress report: callback queued aop=0x62b
> 560
> libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x62a570: complete, rc=0
> libxl: debug: libxl_event.c:1160:egc_run_callbacks: ao 0x62a570:
> progress report: callback aop=0x62b560

Re: Unable to create VM with nic device on Arndale

[Roger Pau Monné]

On 17/01/14 12:59, Ian Campbell wrote:
> On Fri, 2014-01-17 at 19:43 +0800, Dennis Lan (dlan) wrote:
>> On Fri, Jan 17, 2014 at 7:01 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>>
>>
>>> vif-bridge and the common scripts which it includes would be a good
>>> start. Just an echo at the top to confirm that the script is running
>>> would be useful.
>>>
>>> I used to do "exec 1>/tmp/hotplug.log 2>&1" at the top to aid debugging
>>> when these scripts were launched by udev, but now that libxl runs them
>>> you may find that the debug from the script comes out on stdout/err of
>>> the xl create command so perhaps that isn't needed any more.
>>>
>>>> headless here.
>>>
>>> That shouldn't matter, you are looking for output from userspace
>>> scripts, not kernel or hypervisor logs.
>>>
>>> Ian.
>>>
>>
>> Hi Ian
>> I suspect for 4.4.0, the network devices even was not detected.
>> this is output from 4.3.1, notes follow lines.
>>
>> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug
>> script: /etc/xen/scripts/vif-bridge online
>> dlan: vif-bridge start
>> dlan: vif-common start
>>
>> dlan: vif-bridge start -> output from vif-bridge script
>> dlan: vif-common start -> output from vif-common.sh script
> 
> So these are the 4.3 logs? Have you tried 4.4 and found that it doesn't
> produce the same output?
> 
> (please can you try and set the text type to "preformatted" for the logs
> -- having them wrapped makes them very hard to read).
> 
> The lack of 
> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online
> in your original logs is a bit concerning.
> 
> Roger -- any ideas?

My first guess would be that libxl__get_domid failed, however I'm not 
able to reproduce this. I'm attaching a patch to add an error message 
if libxl__get_domid fails, and also prevent the removal of xenstore 
entries so we can see what's going on. Dennis/Eugene, could you try the 
attached patch and send the output of xl -vvv create <...> and 
xenstore-ls -fp after the failed creation?

---
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index a604cd8..03f9fe9 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1296,6 +1296,9 @@ static void domcreate_complete(libxl__egc *egc,
         rc = xc_flask_relabel_domain(CTX->xch, dcs->guest_domid, d_config->b_info.exec_ssidref);
 
     if (rc) {
+        LOG(ERROR, "domain creation failed, not doing removal of xs entries");
+        dcs->callback(egc, dcs, rc, -1);
+        return;
         if (dcs->guest_domid) {
             dcs->dds.ao = ao;
             dcs->dds.domid = dcs->guest_domid;
diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
index ba7d100..56d8162 100644
--- a/tools/libxl/libxl_device.c
+++ b/tools/libxl/libxl_device.c
@@ -965,7 +965,10 @@ static void device_hotplug(libxl__egc *egc, libxl__ao_device *aodev)
      * hotplug scripts
      */
     rc = libxl__get_domid(gc, &domid);
-    if (rc) goto out;
+    if (rc) {
+        LOG(ERROR, "unable to get domain id, error: %d", rc);
+        goto out;
+    }
     if (aodev->dev->backend_domid != domid) {
         if (aodev->action != LIBXL__DEVICE_ACTION_REMOVE)
             goto out;

Re: Unable to create VM with nic device on Arndale

[Dennis Lan (dlan)]

[-- Attachment #1: Type: text/plain, Size: 4058 bytes --]

On Fri, Jan 17, 2014 at 9:31 PM, Roger Pau Monné <roger.pau@citrix.com> wrote:
> On 17/01/14 12:59, Ian Campbell wrote:
>> On Fri, 2014-01-17 at 19:43 +0800, Dennis Lan (dlan) wrote:
>>> On Fri, Jan 17, 2014 at 7:01 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>>>
>>>
>>>> vif-bridge and the common scripts which it includes would be a good
>>>> start. Just an echo at the top to confirm that the script is running
>>>> would be useful.
>>>>
>>>> I used to do "exec 1>/tmp/hotplug.log 2>&1" at the top to aid debugging
>>>> when these scripts were launched by udev, but now that libxl runs them
>>>> you may find that the debug from the script comes out on stdout/err of
>>>> the xl create command so perhaps that isn't needed any more.
>>>>
>>>>> headless here.
>>>>
>>>> That shouldn't matter, you are looking for output from userspace
>>>> scripts, not kernel or hypervisor logs.
>>>>
>>>> Ian.
>>>>
>>>
>>> Hi Ian
>>> I suspect for 4.4.0, the network devices even was not detected.
>>> this is output from 4.3.1, notes follow lines.
>>>
>>> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug
>>> script: /etc/xen/scripts/vif-bridge online
>>> dlan: vif-bridge start
>>> dlan: vif-common start
>>>
>>> dlan: vif-bridge start -> output from vif-bridge script
>>> dlan: vif-common start -> output from vif-common.sh script
>>
>> So these are the 4.3 logs? Have you tried 4.4 and found that it doesn't
>> produce the same output?
>>
>> (please can you try and set the text type to "preformatted" for the logs
>> -- having them wrapped makes them very hard to read).
>>
>> The lack of
>> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online
>> in your original logs is a bit concerning.
>>
>> Roger -- any ideas?
>
> My first guess would be that libxl__get_domid failed, however I'm not
> able to reproduce this. I'm attaching a patch to add an error message
> if libxl__get_domid fails, and also prevent the removal of xenstore
> entries so we can see what's going on. Dennis/Eugene, could you try the
> attached patch and send the output of xl -vvv create <...> and
> xenstore-ls -fp after the failed creation?
>
> ---
> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> index a604cd8..03f9fe9 100644
> --- a/tools/libxl/libxl_create.c
> +++ b/tools/libxl/libxl_create.c
> @@ -1296,6 +1296,9 @@ static void domcreate_complete(libxl__egc *egc,
>          rc = xc_flask_relabel_domain(CTX->xch, dcs->guest_domid, d_config->b_info.exec_ssidref);
>
>      if (rc) {
> +        LOG(ERROR, "domain creation failed, not doing removal of xs entries");
> +        dcs->callback(egc, dcs, rc, -1);
> +        return;
>          if (dcs->guest_domid) {
>              dcs->dds.ao = ao;
>              dcs->dds.domid = dcs->guest_domid;
> diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
> index ba7d100..56d8162 100644
> --- a/tools/libxl/libxl_device.c
> +++ b/tools/libxl/libxl_device.c
> @@ -965,7 +965,10 @@ static void device_hotplug(libxl__egc *egc, libxl__ao_device *aodev)
>       * hotplug scripts
>       */
>      rc = libxl__get_domid(gc, &domid);
> -    if (rc) goto out;
> +    if (rc) {
> +        LOG(ERROR, "unable to get domain id, error: %d", rc);
> +        goto out;
> +    }
>      if (aodev->dev->backend_domid != domid) {
>          if (aodev->action != LIBXL__DEVICE_ACTION_REMOVE)
>              goto out;
>
with this patch applied, I got following err. or see attached file for more info

ofire configs # xl create -c test1_stable
Parsing config from test1_stable
libxl: error: libxl_device.c:969:device_hotplug: unable to get domain
id, error: -3
libxl: error: libxl_device.c:969:device_hotplug: unable to get domain
id, error: -3
libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to
add nic devices
libxl: error: libxl_create.c:1279:domcreate_complete: domain creation
failed, not doing removal of xs entries

[-- Attachment #2: log.txt --]
[-- Type: text/plain, Size: 12641 bytes --]

libxl: debug: libxl_create.c:1325:do_domain_create: ao 0x62a640: create: how=(nil) callback=(nil) poller=0x629f90
libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=unknown
libxl: debug: libxl_device.c:197:disk_try_backend: Disk vdev=xvda1, backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:286:libxl__device_disk_set_backend: Disk vdev=xvda1, using backend qdisk
libxl: debug: libxl_create.c:777:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no bootloader configured, using user supplied kernel
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x628858: deregister unregistered
libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=4, nr_vcpus=6, free_memkb=4336
libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement candidate with 1 nodes, 4 cpus and 4336 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvda1 ro rootfstype=ext4 console=hvc0", features="(null)"
libxl: debug: libxl_dom.c:357:libxl__build_pv: pv kernel mapped 0 path kernel/kernel-genkernel-x86_64-3.12.6
domainbuilder: detail: xc_dom_kernel_file: filename="kernel/kernel-genkernel-x86_64-3.12.6"
domainbuilder: detail: xc_dom_malloc_filemap    : 8185 kB
domainbuilder: detail: xc_dom_ramdisk_file: filename="kernel/initramfs-genkernel-x86_64-3.12.6"
domainbuilder: detail: xc_dom_malloc_filemap    : 4624 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.4, caps xen-3.0-x86_64 xen-3.0-x86_32p 
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... 
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ... 
domainbuilder: detail: xc_dom_malloc            : 25202 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x7f557c -> 0x189ca00
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x1000000 memsz=0x1141000
xc: detail: elf_parse_binary: phdr: paddr=0x2200000 memsz=0x11c0f0
xc: detail: elf_parse_binary: phdr: paddr=0x231d000 memsz=0x14c80
xc: detail: elf_parse_binary: phdr: paddr=0x2332000 memsz=0x700000
xc: detail: elf_parse_binary: memory: 0x1000000 -> 0x2a32000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff823321e0
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
xc: detail: elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"
xc: detail: elf_xen_parse_note: PAE_MODE = "yes"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0xffffffff80000000
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0xffffffff80000000
xc: detail:     virt_kstart      = 0xffffffff81000000
xc: detail:     virt_kend        = 0xffffffff82a32000
xc: detail:     virt_entry       = 0xffffffff823321e0
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff81000000 -> 0xffffffff82a32000
domainbuilder: detail: xc_dom_mem_init: mem 2048 MB, pages 0x80000 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x80000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc            : 4096 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0xffffffff81000000 -> 0xffffffff82a32000  (pfn 0x1000 + 0x1a32 pages)
domainbuilder: detail: xc_dom_malloc            : 157 kB
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1000+0x1a32 at 0x7fa4564e5000
xc: detail: elf_load_binary: phdr 0 at 0x7fa4564e5000 -> 0x7fa457626000
xc: detail: elf_load_binary: phdr 1 at 0x7fa4576e5000 -> 0x7fa4578010f0
xc: detail: elf_load_binary: phdr 2 at 0x7fa457802000 -> 0x7fa457816c80
xc: detail: elf_load_binary: phdr 3 at 0x7fa457817000 -> 0x7fa457981000
domainbuilder: detail: xc_dom_alloc_segment:   ramdisk      : 0xffffffff82a32000 -> 0xffffffff82eb7000  (pfn 0x2a32 + 0x485 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x2a32+0x485 at 0x7fa456060000
domainbuilder: detail: xc_dom_alloc_segment:   phys2mach    : 0xffffffff82eb7000 -> 0xffffffff832b7000  (pfn 0x2eb7 + 0x400 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x2eb7+0x400 at 0x7fa455c60000
domainbuilder: detail: xc_dom_alloc_page   :   start info   : 0xffffffff832b7000 (pfn 0x32b7)
domainbuilder: detail: xc_dom_alloc_page   :   xenstore     : 0xffffffff832b8000 (pfn 0x32b8)
domainbuilder: detail: xc_dom_alloc_page   :   console      : 0xffffffff832b9000 (pfn 0x32b9)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 -> 0xffffffff833fffff, 26 table(s)
domainbuilder: detail: xc_dom_alloc_segment:   page tables  : 0xffffffff832ba000 -> 0xffffffff832d7000  (pfn 0x32ba + 0x1d pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x32ba+0x1d at 0x7fa45cd07000
domainbuilder: detail: xc_dom_alloc_page   :   boot stack   : 0xffffffff832d7000 (pfn 0x32d7)
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0xffffffff832d8000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0xffffffff83400000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x80000
domainbuilder: detail: clear_page: pfn 0x32b9, mfn 0x22af33
domainbuilder: detail: clear_page: pfn 0x32b8, mfn 0x22af34
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x32b7+0x1 at 0x7fa45cd70000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff81001000 pfn=0x1001
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 29510 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 12809 kB
domainbuilder: detail:       domU mmap          : 34 MB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xb768d
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x32ba mfn 0x22af32
domainbuilder: detail: launch_vm: called, ctxt=0x7fa45cd71004
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:251:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=qdisk
libxl: error: libxl_device.c:969:device_hotplug: unable to get domain id, error: -3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x629a80: deregister unregistered
libxl: debug: libxl_dm.c:1303:libxl__spawn_local_dm: Spawning device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   /usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   6
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-6,server,nowait
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -nodefaults
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -xen-attach
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   test1_stable
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -nographic
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -machine
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   xenpv
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1305:libxl__spawn_local_dm:   2049
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x628a90 wpath=/local/domain/0/device-model/6/state token=3/0: register slotnum=3
libxl: debug: libxl_create.c:1339:do_domain_create: ao 0x62a640: inprogress: poller=0x629f90, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x628a90 wpath=/local/domain/0/device-model/6/state token=3/0: event epath=/local/domain/0/device-model/6/state
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x628a90 wpath=/local/domain/0/device-model/6/state token=3/0: event epath=/local/domain/0/device-model/6/state
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x628a90 wpath=/local/domain/0/device-model/6/state token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x628a90: deregister unregistered
libxl: debug: libxl_qmp.c:696:libxl__qmp_initialize: connected to /var/run/xen/qmp-libxl-6
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: qmp
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: '{
    "execute": "qmp_capabilities",
    "id": 1
}
'
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: '{
    "execute": "query-chardev",
    "id": 2
}
'
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:546:qmp_send_prepare: next qmp command: '{
    "execute": "query-vnc",
    "id": 3
}
'
libxl: debug: libxl_qmp.c:296:qmp_handle_response: message type: return
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/6/0/state token=3/1: register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/6/0/state token=3/1: event epath=/local/domain/0/backend/vif/6/0/state
libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vif/6/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/6/0/state token=3/1: event epath=/local/domain/0/backend/vif/6/0/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/6/0/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x62acd8 wpath=/local/domain/0/backend/vif/6/0/state token=3/1: deregister slotnum=3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62acd8: deregister unregistered
libxl: error: libxl_device.c:969:device_hotplug: unable to get domain id, error: -3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x62ad60: deregister unregistered
libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to add nic devices
libxl: error: libxl_create.c:1279:domcreate_complete: domain creation failed, not doing removal of xs entries
libxl: debug: libxl_event.c:1560:libxl__ao_complete: ao 0x62a640: complete, rc=-3
libxl: debug: libxl_event.c:1532:libxl__ao__destroy: ao 0x62a640: destroy
xc: debug: hypercall buffer: total allocations:217 total releases:217
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:206 misses:4 toobig:7
Parsing config from test1_stable

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Unable to create VM with nic device on Arndale

[Roger Pau Monné]

On 18/01/14 03:35, Dennis Lan (dlan) wrote:
> On Fri, Jan 17, 2014 at 9:31 PM, Roger Pau Monné <roger.pau@citrix.com> wrote:
>> On 17/01/14 12:59, Ian Campbell wrote:
>>> On Fri, 2014-01-17 at 19:43 +0800, Dennis Lan (dlan) wrote:
>>>> On Fri, Jan 17, 2014 at 7:01 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>>>>
>>>>
>>>>> vif-bridge and the common scripts which it includes would be a good
>>>>> start. Just an echo at the top to confirm that the script is running
>>>>> would be useful.
>>>>>
>>>>> I used to do "exec 1>/tmp/hotplug.log 2>&1" at the top to aid debugging
>>>>> when these scripts were launched by udev, but now that libxl runs them
>>>>> you may find that the debug from the script comes out on stdout/err of
>>>>> the xl create command so perhaps that isn't needed any more.
>>>>>
>>>>>> headless here.
>>>>>
>>>>> That shouldn't matter, you are looking for output from userspace
>>>>> scripts, not kernel or hypervisor logs.
>>>>>
>>>>> Ian.
>>>>>
>>>>
>>>> Hi Ian
>>>> I suspect for 4.4.0, the network devices even was not detected.
>>>> this is output from 4.3.1, notes follow lines.
>>>>
>>>> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug
>>>> script: /etc/xen/scripts/vif-bridge online
>>>> dlan: vif-bridge start
>>>> dlan: vif-common start
>>>>
>>>> dlan: vif-bridge start -> output from vif-bridge script
>>>> dlan: vif-common start -> output from vif-common.sh script
>>>
>>> So these are the 4.3 logs? Have you tried 4.4 and found that it doesn't
>>> produce the same output?
>>>
>>> (please can you try and set the text type to "preformatted" for the logs
>>> -- having them wrapped makes them very hard to read).
>>>
>>> The lack of
>>> libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online
>>> in your original logs is a bit concerning.
>>>
>>> Roger -- any ideas?
>>
>> My first guess would be that libxl__get_domid failed, however I'm not
>> able to reproduce this. I'm attaching a patch to add an error message
>> if libxl__get_domid fails, and also prevent the removal of xenstore
>> entries so we can see what's going on. Dennis/Eugene, could you try the
>> attached patch and send the output of xl -vvv create <...> and
>> xenstore-ls -fp after the failed creation?
>>
>> ---
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index a604cd8..03f9fe9 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -1296,6 +1296,9 @@ static void domcreate_complete(libxl__egc *egc,
>>          rc = xc_flask_relabel_domain(CTX->xch, dcs->guest_domid, d_config->b_info.exec_ssidref);
>>
>>      if (rc) {
>> +        LOG(ERROR, "domain creation failed, not doing removal of xs entries");
>> +        dcs->callback(egc, dcs, rc, -1);
>> +        return;
>>          if (dcs->guest_domid) {
>>              dcs->dds.ao = ao;
>>              dcs->dds.domid = dcs->guest_domid;
>> diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
>> index ba7d100..56d8162 100644
>> --- a/tools/libxl/libxl_device.c
>> +++ b/tools/libxl/libxl_device.c
>> @@ -965,7 +965,10 @@ static void device_hotplug(libxl__egc *egc, libxl__ao_device *aodev)
>>       * hotplug scripts
>>       */
>>      rc = libxl__get_domid(gc, &domid);
>> -    if (rc) goto out;
>> +    if (rc) {
>> +        LOG(ERROR, "unable to get domain id, error: %d", rc);
>> +        goto out;
>> +    }
>>      if (aodev->dev->backend_domid != domid) {
>>          if (aodev->action != LIBXL__DEVICE_ACTION_REMOVE)
>>              goto out;
>>
> with this patch applied, I got following err. or see attached file for more info
> 
> ofire configs # xl create -c test1_stable
> Parsing config from test1_stable
> libxl: error: libxl_device.c:969:device_hotplug: unable to get domain
> id, error: -3
> libxl: error: libxl_device.c:969:device_hotplug: unable to get domain
> id, error: -3
> libxl: error: libxl_create.c:1206:domcreate_attach_vtpms: unable to
> add nic devices
> libxl: error: libxl_create.c:1279:domcreate_complete: domain creation
> failed, not doing removal of xs entries
> 

Hello,

Thanks for the log, could you please post the output of xenstore-ls -fp
after a failed domain creation?

My first guess is that your xencommons init script is outdated, could
you check if your xencommons init script has the following line:

113                 ${BINDIR}/xenstore-write "/local/domain/0/domid" 0

Roger.

Re: Unable to create VM with nic device on Arndale

[Dennis Lan (dlan)]

On Sat, Jan 18, 2014 at 4:02 PM, Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> Hello,
>
> Thanks for the log, could you please post the output of xenstore-ls -fp
> after a failed domain creation?
>
> My first guess is that your xencommons init script is outdated, could
> you check if your xencommons init script has the following line:
>
> 113                 ${BINDIR}/xenstore-write "/local/domain/0/domid" 0
>
> Roger.

Hi Roger:
   Thanks, adding this line solve my problem ;-)

   And  I didn't make it clear that I'm using Gentoo Linux.
we slightly rewrote the init.d script to make it fit into the system,
and this changes (compare to 4.3.1) cause the problem.
since we inherit the script from previous version 4.3.1(thus old script).
    I wonder if it's worthy to make the error or warning more useful.
your previous patch looks good, and could be useful.
(not sure others will hit the same problem..)

Lan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel