Bug nestedhvm: staging-4.5, stable-4.5, staging and master. xen in xen and hvm

Bug nestedhvm: staging-4.5, stable-4.5, staging and master. xen in xen and hvm

[Don Slutz]

I have tried several of these xen versions both as host and as guest.
They all do the same thing for me.  Not sure I have tried all possible.

So far I have not found a way to use nestedhvm and xen.  When I try and
start a HVM guest, I get:

[root@xen-in-xen ~]# xl -vvv cre -V /home/don/C63-min-tools.list-nocdrom.xfg
Parsing config from /home/don/C63-min-tools.list-nocdrom.xfg
got a tsc mode string: "native_paravirt"
WARNING: ignoring "vncviewer" option. Use "-V" option of "xl create" to
automatically spawn vncviewer.
libxl: debug: libxl_create.c:1504:do_domain_create: ao 0x2022990:
create: how=(nil) callback=(nil) poller=0x2022630
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:215:disk_try_backend: Disk vdev=xvda,
backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
vdev=xvda, using backend qdisk
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=xvdb spec.backend=unknown
libxl: debug: libxl_device.c:215:disk_try_backend: Disk vdev=xvdb,
backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
vdev=xvdb, using backend qdisk
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=xvdc spec.backend=unknown
libxl: debug: libxl_device.c:215:disk_try_backend: Disk vdev=xvdc,
backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
vdev=xvdc, using backend qdisk
libxl: debug: libxl_create.c:907:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:323:libxl__bootloader_run: not a PV
domain, skipping bootloader
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x20197d0: deregister unregistered
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0xbf7e8
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x1bf7e8
xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail:   Loader:   0000000000100000->00000000001bf7e8
xc: detail:   Modules:  0000000000000000->0000000000000000
xc: detail:   TOTAL:    0000000000000000->000000003e000000
xc: detail:   ENTRY:    0000000000100000
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000001ef
xc: detail:   1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7fc13ae93000 -> 0x7fc13af49670
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=qdisk
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x201a300: deregister unregistered
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=xvdb spec.backend=qdisk
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x201b1f0: deregister unregistered
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=xvdc spec.backend=qdisk
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x201be60: deregister unregistered
libxl: debug: libxl_dm.c:1413:libxl__spawn_local_dm: Spawning
device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
/usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   6
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-6,server,nowait
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -nodefaults
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   C63-min-tools-b
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -vnc
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   127.0.0.1:0,to=99
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -serial
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   pty
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
cirrus-vga,vgamem_mb=32
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   order=cda
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
vmxnet3,id=nic0,netdev=net0,mac=00:0c:29:8a:44:a0
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -netdev
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
type=tap,id=net0,ifname=vif6.0-emu,script=no,downscript=no
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
e1000,id=nic1,netdev=net1,mac=00:0c:29:8a:44:aa
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -netdev
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
type=tap,id=net1,ifname=vif6.1-emu,script=no,downscript=no
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
vmxnet3,id=nic2,netdev=net2,mac=00:0c:29:8a:44:b4
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -netdev
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
type=tap,id=net2,ifname=vif6.2-emu,script=no,downscript=no
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -machine
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   pc,accel=xen
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -monitor
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   pty
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   menu=on
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   992
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
file=/home/don/qemu-img/C63-min-tools-disk1.raw,if=ide,index=0,media=disk,format=raw,cache=writeback
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
file=/home/don/qemu-img/C63-min-tools-disk2.raw,if=ide,index=1,media=disk,format=raw,cache=writeback
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1415:libxl__spawn_local_dm:
file=/home/don/qemu-img/C63-min-tools-disk3.raw,if=ide,index=2,media=disk,format=raw,cache=writeback
libxl: debug: libxl_event.c:577:libxl__ev_xswatch_register: watch
w=0x2019a30 wpath=/local/domain/0/device-model/6/state token=3/0:
register slotnum=3
libxl: debug: libxl_create.c:1520:do_domain_create: ao 0x2022990:
inprogress: poller=0x2022630, flags=i
libxl: debug: libxl_event.c:514:watchfd_callback: watch w=0x2019a30
wpath=/local/domain/0/device-model/6/state token=3/0: event
epath=/local/domain/0/device-model/6/state
libxl: debug: libxl_aoutils.c:87:xswait_timeout_callback: domain 6
device model startup: xswait timeout
(path=/local/domain/0/device-model/6/state)
libxl: debug: libxl_event.c:615:libxl__ev_xswatch_deregister: watch
w=0x2019a30 wpath=/local/domain/0/device-model/6/state token=3/0:
deregister slotnum=3
libxl: error: libxl_exec.c:393:spawn_watch_event: domain 6 device model:
startup timed out
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x2019a30: deregister unregistered
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x2019a30: deregister unregistered
libxl: error: libxl_dm.c:1482:device_model_spawn_outcome: domain 6
device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1322:domcreate_devmodel_started: device
model did not start: -3
libxl: debug: libxl_dm.c:1589:kill_device_model: Device Model signaled
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x201f920: deregister unregistered
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x201fb10: deregister unregistered
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x201fd00: deregister unregistered
libxl: debug: libxl_event.c:629:libxl__ev_xswatch_deregister: watch
w=0x201ff50: deregister unregistered
libxl: debug: libxl_event.c:1765:libxl__ao_complete: ao 0x2022990:
complete, rc=-3
libxl: debug: libxl_event.c:1737:libxl__ao__destroy: ao 0x2022990: destroy
xc: debug: hypercall buffer: total allocations:1056 total releases:1056
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:1040 misses:4 toobig:12


At which time I get a lot of messages on the serial console:


Fedora release 17 (Beefy Miracle)
Kernel 3.8.11-100.fc17.x86_64 on an x86_64 (hvc0)

xen-in-xen login: [ 2555.655873] tun: Universal TUN/TAP device driver, 1.6
[ 2555.675257] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
(XEN) [2015-02-16 19:55:05] mm.c:2004:d0v3 Error pfn 3c823e:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 19:55:05] mm.c:2004:d0v3 Error pfn 3c823e:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 19:55:05] mm.c:2004:d0v3 Error pfn 3c823d:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 19:55:05] mm.c:2004:d0v3 Error pfn 3c823d:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 19:55:05] mm.c:2004:d0v3 Error pfn 3c823c:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 19:55:05] mm.c:2004:d0v3 Error pfn 3c823c:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
...
(XEN) [2015-02-16 20:03:32] mm.c:2004:d0v5 Error pfn 3c64d4:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 20:03:32] mm.c:2004:d0v5 Error pfn 3c64d4:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 20:03:32] mm.c:2004:d0v5 Error pfn 3c64d3:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 20:03:32] mm.c:2004:d0v5 Error pfn 3c64d3:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 20:03:32] mm.c:2004:d0v5 Error pfn 3c64d2:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001
(XEN) [2015-02-16 20:03:32] mm.c:2004:d0v5 Error pfn 3c64d2:
rd=ffff83040de26000, od=0000000000000000, caf=180000000000000,
taf=0000000000000001




Nothing in QEMU log:
xen-in-xen:~>cat /var/log/xen/qemu-dm-C63-min-tools-b.log
char device redirected to /dev/pts/3 (label compat_monitor0)
char device redirected to /dev/pts/4 (label serial0)

I have some logs.  Let me know what you might like:

don-760:~/xen-in-xen>ls -lh
total 7.0M
-rw-rw-r-- 1 don don  750 Feb 16 15:03 C63-min-tools.list-nocdrom.xfg
-rw-rw-r-- 1 don don  71K Feb 16 14:33 dmesg.C63-min-tools.list-nocdrom
-rw-rw-r-- 1 don don  81K Feb 16 14:41 dmesg.xen-in-xen
-rw-rw-r-- 1 don don 6.1K Feb 16 15:11 host-cpuinfo
-rw-r--r-- 1 don don 4.1M Feb 16 14:42 host-xenstored-access.log
-rw-r--r-- 1 don don 9.0K Feb 16 14:42 host-xenstored.log
-rw-rw-r-- 1 don don 4.2K Feb 16 15:10 xen-in-xen-cpuinfo
-rw-r--r-- 1 don don 2.3M Feb 16 15:04 xen-in-xen-staging-4.5-2
-rw-r--r-- 1 don don 406K Feb 16 14:41 xen-in-xen-xenstored-access.log
-rw-r--r-- 1 don don  576 Feb 16 14:41 xen-in-xen-xenstored.log
-rw-rw-r-- 1 don don 1.2K Feb 16 15:01 xen-in-xen.list-skip.xfg
-rw-rw-r-- 1 don don  19K Feb 16 14:33 xl-dmesg.C63-min-tools.list-nocdrom
-rw-rw-r-- 1 don don  44K Feb 16 14:41 xl-dmesg.xen-in-xen


I do not know the best way to proceed.

   -Don Slutz