* Nested virtualization instabilities
@ 2013-12-16 11:05 Etzion Bar-Noy
2013-12-16 13:51 ` Andrew Cooper
2013-12-18 1:33 ` Zhang, Yang Z
0 siblings, 2 replies; 10+ messages in thread
From: Etzion Bar-Noy @ 2013-12-16 11:05 UTC (permalink / raw)
To: xen-devel; +Cc: eddie.dong, jun.nakajima
[-- Attachment #1.1: Type: text/plain, Size: 16691 bytes --]
Hi.
I am using the unstable build-from-source of Xen community. It reports as
4.4, and was pulled from Git few days ago. The problem I will describe
below appears on version 4.3.1, compiled from source as well. In both cases
- I used spec file to create my own RPMs, because it's the right method of
doing it.
Problem: When VM running under HVM which is a hypervisor (XenServer 6.2 or
VMware ESXi5.5) attempts to start a (HVM) virtual machine, the following
happens, 100% reproducible on this server.
XenServer: domain crash. XenServer restarts itself. It doesn't happen for
PV on XenServer VM
VMware: An "unknown failure", which leads me to think that the nested
virtualization properties are not forwarded correctly to DomU
Just to clarify:
Physical hardware: host
Nested hypervisor: level 1 virtualization: XenServer or VMware ESXi
Nested VM: level 2 virtualization: some generic VM inside XenServer or ESXi
(level 1)
This is the configuration used for XS:
name = "XenServer"
uuid = "a0b8c668-afaf-887a-1f4b-856dac4c8401"
maxmem = 4096
memory = 4096
vcpus = 2
builder = "hvm"
boot = "c"
pae = 1
acpi = 1
apic = 1
hap = 0
viridian = 0
rtc_timeoffset = 0
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
sdl = 0
vnc = 1
vncunused = 1
keymap = "en-us"
disk = [ "tap2:aio:/dev/share/VMS/xenserver01.lun,hda,w", ",hdc:cdrom,r" ]
vif = [ "mac=00:16:3e:14:0d:72,bridge=lan,script=vif-bridge,model=e1000" ]
parallel = "none"
serial = "pty"
nestedhvm = 1
nx=1
# A note - this problem appears with and without the 'nx' flag.
A snippet from strace to qemu process:
poll([{fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=18,
events=POLLIN|POLLERR|POLLHUP}, {fd=29, events=POLLIN|POLLERR|POLLHUP},
{fd=21,
events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN|POLLERR|POLLHUP},
{fd=5, events=
POLLIN|POLLERR|POLLHUP}, {fd=7, events=POLLIN}, {fd=11, events=POLLIN},
{fd=28, events=POLLIN}, {fd=6, events=POLLIN}], 10, 307) = ?
ERESTART_RESTARTBLOCK (To be restarted)
--- SIGHUP (Hangup) @ 0 (0) ---
write(6, "\1\0\0\0\0\0\0\0", 8) = 8
rt_sigreturn(0x2) = -1 EINTR (Interrupted system call)
gettimeofday({1386761621, 705266}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC, {1840, 693236049}) = 0
clock_gettime(CLOCK_MONOTONIC, {1840, 693333096}) = 0
gettimeofday({1386761621, 705546}, NULL) = 0
write(2, "qemu: terminating on signal 1", 29) = 29
write(2, " from pid 6933\n", 15) = 15
>From the xl log for this domain:
Waiting for domain XenServer (domid 5) to die [pid 6933]
Domain 5 has shut down, reason code 3 0x3
Action for shutdown reason code 3 is restart
Domain 5 needs to be cleaned up: destroying the domain
Done. Rebooting now
Waiting for domain XenServer (domid 6) to die [pid 6933]
>From 'xl dmesg'
(XEN) <vm_launch_fail> error code 7
(XEN) domain_crash_sync called from vmcs.c:1293
(XEN) Domain 8 (vcpu#0) crashed on cpu#4:
(XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0000:[<0000000000000000>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80
(XEN) rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
(XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8: 0000000000000004
(XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70 r11: 0000000000000000
(XEN) r12: ffff830824f9ff50 r13: ffff830654f9e000 r14: ffff830824f9ff30
(XEN) r15: ffff82d080187e43 cr0: 0000000000000039 cr4: 0000000000002050
(XEN) cr3: 00000000feffa000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: 0000
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(d9) HVM Loader
(d9) Detected Xen v4.4-unstable
(d9) Xenbus rings @0xfeffc000, event channel 4
(d9) System requested SeaBIOS
(d9) CPU speed is 3400 MHz
(d9) Relocating guest memory for lowmem MMIO space disabled
(XEN) irq.c:270: Dom9 PCI link 0 changed 0 -> 5
(d9) PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom9 PCI link 1 changed 0 -> 10
(d9) PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom9 PCI link 2 changed 0 -> 11
(d9) PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5
(d9) PCI-ISA link 3 routed to IRQ5
(d9) pci dev 01:3 INTA->IRQ10
(d9) pci dev 02:0 INTA->IRQ11
(d9) pci dev 04:0 INTA->IRQ5
(d9) RAM in high memory; setting high_mem resource base to 10f800000
(d9) pci dev 03:0 bar 10 size 002000000: 0f0000008
(d9) pci dev 02:0 bar 14 size 001000000: 0f2000008
(d9) pci dev 04:0 bar 30 size 000040000: 0f3000000
(d9) pci dev 04:0 bar 10 size 000020000: 0f3040000
(d9) pci dev 03:0 bar 30 size 000010000: 0f3060000
(d9) pci dev 03:0 bar 14 size 000001000: 0f3070000
(d9) pci dev 02:0 bar 10 size 000000100: 00000c001
(d9) pci dev 04:0 bar 14 size 000000040: 00000c101
(d9) pci dev 01:1 bar 20 size 000000010: 00000c141
(d9) Multiprocessor initialisation:
(d9) - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d9) - CPU1 ... 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d9) Testing HVM environment:
(d9) - REP INSB across page boundaries ... passed
(d9) - GS base MSRs and SWAPGS ... passed
(d9) Passed 2 of 2 tests
(d9) Writing SMBIOS tables ...
(d9) Loading SeaBIOS ...
(d9) Creating MP tables ...
(d9) Loading ACPI ...
(d9) vm86 TSS at fc00a080
(d9) BIOS map:
(d9) 10000-100d3: Scratch space
(d9) e0000-fffff: Main BIOS
(d9) E820 table:
(d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d9) HOLE: 00000000:000a0000 - 00000000:000e0000
(d9) [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d9) [02]: 00000000:00100000 - 00000000:f0000000: RAM
(d9) HOLE: 00000000:f0000000 - 00000000:fc000000
(d9) [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM
(d9) Invoking SeaBIOS ...
(d9) SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
(d9)
(d9) Found Xen hypervisor signature at 40000000
(d9) xen: copy e820...
(d9) Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784)
(d9) CPU Mhz=3400
(d9) Found 7 PCI devices (max PCI bus is 00)
(d9) Allocated Xen hypercall page at effff000
(d9) Detected Xen v4.4-unstable
(d9) xen: copy BIOS tables...
(d9) Copying SMBIOS entry point from 0x00010010 to 0x000f1020
(d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20
(d9) Copying PIR from 0x00010030 to 0x000f0ea0
(d9) Copying ACPI RSDP from 0x000100b0 to 0x000f0e70
(d9) Using pmtimer, ioport 0xb008, freq 3579 kHz
(d9) Scan for VGA option rom
(d9) WARNING! Found unaligned PCI rom (vd=1013:00b8)
(d9) Running option rom at c000:0003
(XEN) stdvga.c:147:d9 entering stdvga and caching modes
(d9) Turning on vga text mode console
(d9) SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
(d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401
(d9) Found 0 lpt ports
(d9) Found 1 serial ports
(d9) ATA controller 1 at 1f0/3f4/c140 (irq 14 dev 9)
(d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9)
(d9) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes)
(d9) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
(d9) DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
(d9) Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
(d9) PS2 keyboard initialized
(d9) All threads complete.
(d9) Scan for option roms
(d9) Running option rom at c900:0003
(d9) pmm call arg1=1
(d9) pmm call arg1=0
(d9) pmm call arg1=1
(d9) pmm call arg1=0
(d9) Searching bootorder for: /pci@i0cf8/*@4
(d9)
(d9) Press F12 for boot menu.
(d9)
(d9) Searching bootorder for: HALT
(d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
s=29360128
(d9) Space available for UMB: ca000-ee800, f0000-f0dc0
(d9) Returned 61440 bytes of ZoneHigh
(d9) e820 map has 7 items:
(d9) 0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d9) 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d9) 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d9) 3: 0000000000100000 - 00000000effff000 = 1 RAM
(d9) 4: 00000000effff000 - 00000000f0000000 = 2 RESERVED
(d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
(d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM
(d9) enter handle_19:
(d9) NULL
(d9) Booting from Hard Disk...
(d9) Booting from 0000:7c00
[root@monster ~]# xl dmesg
(XEN) <vm_launch_fail> error code 7
(XEN) domain_crash_sync called from vmcs.c:1293
(XEN) Domain 8 (vcpu#0) crashed on cpu#4:
(XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: 0000:[<0000000000000000>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80
(XEN) rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
(XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8: 0000000000000004
(XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70 r11: 0000000000000000
(XEN) r12: ffff830824f9ff50 r13: ffff830654f9e000 r14: ffff830824f9ff30
(XEN) r15: ffff82d080187e43 cr0: 0000000000000039 cr4: 0000000000002050
(XEN) cr3: 00000000feffa000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: 0000
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 9, error -22
(d9) HVM Loader
(d9) Detected Xen v4.4-unstable
(d9) Xenbus rings @0xfeffc000, event channel 4
(d9) System requested SeaBIOS
(d9) CPU speed is 3400 MHz
(d9) Relocating guest memory for lowmem MMIO space disabled
(XEN) irq.c:270: Dom9 PCI link 0 changed 0 -> 5
(d9) PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom9 PCI link 1 changed 0 -> 10
(d9) PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom9 PCI link 2 changed 0 -> 11
(d9) PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5
(d9) PCI-ISA link 3 routed to IRQ5
(d9) pci dev 01:3 INTA->IRQ10
(d9) pci dev 02:0 INTA->IRQ11
(d9) pci dev 04:0 INTA->IRQ5
(d9) RAM in high memory; setting high_mem resource base to 10f800000
(d9) pci dev 03:0 bar 10 size 002000000: 0f0000008
(d9) pci dev 02:0 bar 14 size 001000000: 0f2000008
(d9) pci dev 04:0 bar 30 size 000040000: 0f3000000
(d9) pci dev 04:0 bar 10 size 000020000: 0f3040000
(d9) pci dev 03:0 bar 30 size 000010000: 0f3060000
(d9) pci dev 03:0 bar 14 size 000001000: 0f3070000
(d9) pci dev 02:0 bar 10 size 000000100: 00000c001
(d9) pci dev 04:0 bar 14 size 000000040: 00000c101
(d9) pci dev 01:1 bar 20 size 000000010: 00000c141
(d9) Multiprocessor initialisation:
(d9) - CPU0 ... 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d9) - CPU1 ... 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d9) Testing HVM environment:
(d9) - REP INSB across page boundaries ... passed
(d9) - GS base MSRs and SWAPGS ... passed
(d9) Passed 2 of 2 tests
(d9) Writing SMBIOS tables ...
(d9) Loading SeaBIOS ...
(d9) Creating MP tables ...
(d9) Loading ACPI ...
(d9) vm86 TSS at fc00a080
(d9) BIOS map:
(d9) 10000-100d3: Scratch space
(d9) e0000-fffff: Main BIOS
(d9) E820 table:
(d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d9) HOLE: 00000000:000a0000 - 00000000:000e0000
(d9) [01]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d9) [02]: 00000000:00100000 - 00000000:f0000000: RAM
(d9) HOLE: 00000000:f0000000 - 00000000:fc000000
(d9) [03]: 00000000:fc000000 - 00000001:00000000: RESERVED
(d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM
(d9) Invoking SeaBIOS ...
(d9) SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
(d9)
(d9) Found Xen hypervisor signature at 40000000
(d9) xen: copy e820...
(d9) Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784)
(d9) CPU Mhz=3400
(d9) Found 7 PCI devices (max PCI bus is 00)
(d9) Allocated Xen hypercall page at effff000
(d9) Detected Xen v4.4-unstable
(d9) xen: copy BIOS tables...
(d9) Copying SMBIOS entry point from 0x00010010 to 0x000f1020
(d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20
(d9) Copying PIR from 0x00010030 to 0x000f0ea0
(d9) Copying ACPI RSDP from 0x000100b0 to 0x000f0e70
(d9) Using pmtimer, ioport 0xb008, freq 3579 kHz
(d9) Scan for VGA option rom
(d9) WARNING! Found unaligned PCI rom (vd=1013:00b8)
(d9) Running option rom at c000:0003
(XEN) stdvga.c:147:d9 entering stdvga and caching modes
(d9) Turning on vga text mode console
(d9) SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
(d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401
(d9) Found 0 lpt ports
(d9) Found 1 serial ports
(d9) ATA controller 1 at 1f0/3f4/c140 (irq 14 dev 9)
(d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9)
(d9) ata0-0: QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes)
(d9) Searching bootorder for: /pci@i0cf8/*@1,1/drive@0/disk@0
(d9) DVD/CD [ata1-0: QEMU DVD-ROM ATAPI-4 DVD/CD]
(d9) Searching bootorder for: /pci@i0cf8/*@1,1/drive@1/disk@0
(d9) PS2 keyboard initialized
(d9) All threads complete.
(d9) Scan for option roms
(d9) Running option rom at c900:0003
(d9) pmm call arg1=1
(d9) pmm call arg1=0
(d9) pmm call arg1=1
(d9) pmm call arg1=0
(d9) Searching bootorder for: /pci@i0cf8/*@4
(d9)
(d9) Press F12 for boot menu.
(d9)
(d9) Searching bootorder for: HALT
(d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
s=29360128
(d9) Space available for UMB: ca000-ee800, f0000-f0dc0
(d9) Returned 61440 bytes of ZoneHigh
(d9) e820 map has 7 items:
(d9) 0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d9) 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d9) 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d9) 3: 0000000000100000 - 00000000effff000 = 1 RAM
(d9) 4: 00000000effff000 - 00000000f0000000 = 2 RESERVED
(d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
(d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM
(d9) enter handle_19:
(d9) NULL
(d9) Booting from Hard Disk...
(d9) Booting from 0000:7c00
System description:
1. Centos 6.4 x86_64, latest updates
2. Xen4centos packages - current kernel is 3.10.23-11.el6.centos.alt.x86_64
3. System with 32GB RAM
4. Virtual disks are ZFS volumes (aka - block devices). Same problem
appeared when on local ext3 disks
5. Using OpenVSwitch version 2.0, compiled to RPM, again.
6. Intel CPU:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
stepping : 4
microcode : 0x416
cpu MHz : 3400.158
cache size : 12288 KB
physical id : 0
siblings : 12
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush
acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl
nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2
popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm ida
arat epb xsaveopt pln pts dtherm fsgsbase erms
bogomips : 6800.31
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
It was suggested by Mr. Ian Campbell to CC both Mr. Jun Nakajima and Mr.
Eddie Dong, so, my apologies if this action is improper.
Any tip or advice would be great. Any request for further info would be
also great!
Thanks!
Etzion Bar-Noy
[-- Attachment #1.2: Type: text/html, Size: 18693 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-16 11:05 Nested virtualization instabilities Etzion Bar-Noy
@ 2013-12-16 13:51 ` Andrew Cooper
2013-12-18 1:33 ` Zhang, Yang Z
1 sibling, 0 replies; 10+ messages in thread
From: Andrew Cooper @ 2013-12-16 13:51 UTC (permalink / raw)
To: Etzion Bar-Noy, xen-devel; +Cc: eddie.dong, jun.nakajima
[-- Attachment #1.1: Type: text/plain, Size: 2308 bytes --]
On 16/12/2013 11:05, Etzion Bar-Noy wrote:
> Hi.
>
> I am using the unstable build-from-source of Xen community. It reports
> as 4.4, and was pulled from Git few days ago. The problem I will
> describe below appears on version 4.3.1, compiled from source as well.
> In both cases - I used spec file to create my own RPMs, because it's
> the right method of doing it.
>
> Problem: When VM running under HVM which is a hypervisor (XenServer
> 6.2 or VMware ESXi5.5) attempts to start a (HVM) virtual machine, the
> following happens, 100% reproducible on this server.
> XenServer: domain crash. XenServer restarts itself. It doesn't happen
> for PV on XenServer VM
> VMware: An "unknown failure", which leads me to think that the nested
> virtualization properties are not forwarded correctly to DomU
>
>
<snip>
> (XEN) <vm_launch_fail> error code 7
> (XEN) domain_crash_sync called from vmcs.c:1293
> (XEN) Domain 8 (vcpu#0) crashed on cpu#4:
> (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
> (XEN) CPU: 4
> (XEN) RIP: 0000:[<0000000000000000>]
> (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
> (XEN) rax: 0000000000000000 rbx: ffff830824f98000 rcx:
> ffff830824f9ff80
> (XEN) rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi:
> ffff82d0801de7fc
> (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
> 0000000000000004
> (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70 r11:
> 0000000000000000
> (XEN) r12: ffff830824f9ff50 r13: ffff830654f9e000 r14:
> ffff830824f9ff30
> (XEN) r15: ffff82d080187e43 cr0: 0000000000000039 cr4:
> 0000000000002050
> (XEN) cr3: 00000000feffa000 cr2: 0000000000000000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: 0000
This is the problem.
Error code 7 is defined as "VM entry with invalid control field(s)"
At a guess, I would think that the L1 hypervisor is probing the control
fields (as that seems to be the only way of detecting whether a feature
is available or not), and this error might need bouncing back to the VM.
Otherwise, the L1 hypervisor has decided it can use a particular
control, and Xen has insufficient validation if the L1 state.
Either way, it would be useful to know if there is a way for the
hardware to identify which control field it has an issue with...
~Andrew
[-- Attachment #1.2: Type: text/html, Size: 6585 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-16 11:05 Nested virtualization instabilities Etzion Bar-Noy
2013-12-16 13:51 ` Andrew Cooper
@ 2013-12-18 1:33 ` Zhang, Yang Z
2013-12-18 9:46 ` Etzion Bar-Noy
1 sibling, 1 reply; 10+ messages in thread
From: Zhang, Yang Z @ 2013-12-18 1:33 UTC (permalink / raw)
To: Etzion Bar-Noy, xen-devel@lists.xen.org; +Cc: Dong, Eddie, Nakajima, Jun
[-- Attachment #1: Type: text/plain, Size: 18680 bytes --]
Etzion Bar-Noy wrote on 2013-12-16:
> Hi.
>
Hi Etzion Bar-Noy
It's glad to know someone is using Nested Xen and report bugs to us. Also, if there are more nested bugs you found, please CC me too.
>
> I am using the unstable build-from-source of Xen community. It reports
> as 4.4, and was pulled from Git few days ago. The problem I will
> describe below appears on version 4.3.1, compiled from source as well.
> In both cases - I used spec file to create my own RPMs, because it's the right method of doing it.
>
>
> Problem: When VM running under HVM which is a hypervisor (XenServer
> 6.2 or VMware ESXi5.5) attempts to start a (HVM) virtual machine, the
> following happens, 100% reproducible on this server.
>
> XenServer: domain crash. XenServer restarts itself. It doesn't happen
> for PV on XenServer VM
>
> VMware: An "unknown failure", which leads me to think that the nested
> virtualization properties are not forwarded correctly to DomU
To using nested, you need to add the set bit 31 of cupid.1.ecx to 0, add the following to your config file:
cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
For VMware ESX server, the attached patch is missing in current Xen which will caused the guest boot up fail. Also, I know some windows guests will boot fail on ESX.
I never tried XenServer. But latest Xen on Xen is working well. So can you try the upsteam Xen to see whether it fails too?
BTW, which L2 VM boot fail?
>
>
> Just to clarify:
>
> Physical hardware: host
>
> Nested hypervisor: level 1 virtualization: XenServer or VMware ESXi
>
> Nested VM: level 2 virtualization: some generic VM inside XenServer or
> ESXi (level 1)
>
>
> This is the configuration used for XS:
> name = "XenServer"
> uuid = "a0b8c668-afaf-887a-1f4b-856dac4c8401"
> maxmem = 4096
> memory = 4096
> vcpus = 2
> builder = "hvm"
> boot = "c"
> pae = 1
> acpi = 1
> apic = 1
> hap = 0
> viridian = 0
> rtc_timeoffset = 0
> localtime = 0
> on_poweroff = "destroy"
> on_reboot = "restart"
> on_crash = "restart"
> sdl = 0
> vnc = 1
> vncunused = 1
> keymap = "en-us"
> disk = [ "tap2:aio:/dev/share/VMS/xenserver01.lun,hda,w",
> ",hdc:cdrom,r" ] vif = [
> "mac=00:16:3e:14:0d:72,bridge=lan,script=vif-bridge,model=e1000" ] parallel = "none"
> serial = "pty"
> nestedhvm = 1
>
> nx=1
>
>
> # A note - this problem appears with and without the 'nx' flag.
>
> A snippet from strace to qemu process:
>
> poll([{fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=18,
> events=POLLIN|POLLERR|POLLHUP}, {fd=29,
> events=POLLIN|POLLERR|POLLHUP}, {fd=21,
> events=POLLIN|POLLERR|POLLHUP}, {fd=15,
> events=POLLIN|POLLERR|POLLHUP}, {fd=5, events=
> POLLIN|POLLERR|POLLHUP}, {fd=7, events=POLLIN}, {fd=11, POLLIN|POLLERR|events=POLLIN}, {fd=28, events=POLLIN}, {fd=6, POLLIN|POLLERR|events=POLLIN}], 10, 307) = ?
> ERESTART_RESTARTBLOCK (To POLLIN|POLLERR|be restarted) --- SIGHUP
> (Hangup) @ 0 (0) --- write(6, "\1\0\0\0\0\0\0\0", 8) = 8
> rt_sigreturn(0x2) = -1 EINTR (Interrupted system
> call) gettimeofday({1386761621, 705266}, NULL) = 0
> clock_gettime(CLOCK_MONOTONIC, {1840, 693236049}) = 0
> clock_gettime(CLOCK_MONOTONIC, {1840, 693333096}) = 0
> gettimeofday({1386761621, 705546}, NULL) = 0 write(2, "qemu: terminating
> on signal 1", 29) = 29 write(2, " from pid 6933\n", 15) = 15
>
>
> From the xl log for this domain:
> Waiting for domain XenServer (domid 5) to die [pid 6933] Domain 5 has
> shut down, reason code 3 0x3 Action for shutdown reason code 3 is
> restart Domain
> 5 needs to be cleaned up: destroying the domain Done. Rebooting now
> Waiting for domain XenServer (domid 6) to die [pid 6933]
>
>
> From 'xl dmesg' (XEN) <vm_launch_fail> error code 7 (XEN)
> domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8 (vcpu#0)
> crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not
> tainted ]---- (XEN) CPU: 4 (XEN) RIP: 0000:[<0000000000000000>]
> (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax:
> 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80 (XEN)
> rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
> (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
> 0000000000000004 (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70
> r11: 0000000000000000 (XEN) r12: ffff830824f9ff50 r13:
> ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
> cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3:
> 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
> 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
> Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4 (d9)
> System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9) Relocating
> guest memory for lowmem MMIO space disabled (XEN) irq.c:270: Dom9 PCI
> link 0 changed 0 -> 5 (d9) PCI-ISA link 0 routed to IRQ5 (XEN)
> irq.c:270: Dom9 PCI link 1 changed 0 -> 10 (d9) PCI-ISA link 1 routed
> to IRQ10 (XEN)
> irq.c:270: Dom9 PCI link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed
> to
> IRQ11 (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
> link
> 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
> INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
> setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar 10
> size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size 001000000:
> 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d9) pci
> dev 04:0 bar 10 size 000020000: 0f3040000 (d9) pci dev 03:0 bar 30
> size
> 000010000: 0f3060000 (d9) pci dev 03:0 bar 14 size 000001000:
> 0f3070000
> (d9) pci dev 02:0 bar 10 size 000000100: 00000c001 (d9) pci dev 04:0
> bar
> 14 size 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
> 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
> phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
> 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) Testing
> HVM environment: (d9) - REP INSB across page boundaries ... passed
> (d9)
> - GS base MSRs and SWAPGS ... passed (d9) Passed 2 of 2 tests (d9)
> Writing SMBIOS tables ... (d9) Loading SeaBIOS ... (d9) Creating MP
> tables ... (d9) Loading ACPI ... (d9) vm86 TSS at fc00a080 (d9) BIOS
> map: (d9) 10000-100d3: Scratch space (d9) e0000-fffff: Main BIOS
> (d9)
> E820 table: (d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
> (d9)
> HOLE: 00000000:000a0000 - 00000000:000e0000 (d9) [01]:
> 00000000:000e0000 - 00000000:00100000: RESERVED (d9) [02]:
> 00000000:00100000 - 00000000:f0000000: RAM (d9) HOLE:
> 00000000:f0000000
> - 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
> RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM (d9)
> Invoking SeaBIOS ... (d9) SeaBIOS (version
> rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
> hypervisor signature at 40000000 (d9) xen: copy e820... (d9)
> Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
> Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
> Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
> xen: copy BIOS tables... (d9) Copying SMBIOS entry point from
> 0x00010010 to 0x000f1020
> (d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20 (d9)
> Copying PIR from 0x00010030 to 0x000f0ea0 (d9) Copying ACPI RSDP from
> 0x000100b0 to 0x000f0e70 (d9) Using pmtimer, ioport 0xb008, freq 3579
> kHz (d9) Scan for VGA option rom (d9) WARNING! Found unaligned PCI rom
> (vd=1013:00b8)
> (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
> stdvga and caching modes (d9) Turning on vga text mode console (d9)
> SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
> (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
> lpt ports
> (d9) Found 1 serial ports (d9) ATA controller 1 at 1f0/3f4/c140 (irq
> 14 dev 9) (d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9) (d9) ata0-0:
> QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes) (d9) Searching bootorder
> for: /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
> ATAPI-4 DVD/CD] (d9) Searching bootorder for:
> /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
> threads complete. (d9) Scan for option roms (d9) Running option rom at
> c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm call
> arg1=1
> (d9) pmm call arg1=0 (d9) Searching bootorder for: /pci@i0cf8/*@4 (d9)
> (d9) Press F12 for boot menu. (d9) (d9) Searching bootorder for: HALT
> (d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba
> LCHS=1024/255/63
> s=29360128 (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
> Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
> 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1: 000000000009fc00 -
> 00000000000a0000 = 2 RESERVED (d9) 2: 00000000000f0000 -
> 0000000000100000 = 2 RESERVED (d9) 3: 0000000000100000 -
> 00000000effff000 = 1 RAM (d9) 4: 00000000effff000 - 00000000f0000000 =
> 2 RESERVED (d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
> (d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
> handle_19: (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
> 0000:7c00 [root@monster ~]# xl dmesg (XEN) <vm_launch_fail> error code
> 7
> (XEN) domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8
> (vcpu#0) crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not
> tainted ]---- (XEN) CPU: 4 (XEN) RIP: 0000:[<0000000000000000>]
> (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax:
> 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80 (XEN)
> rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
> (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
> 0000000000000004 (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70
> r11: 0000000000000000 (XEN) r12: ffff830824f9ff50 r13:
> ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
> cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3:
> 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
> 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
> Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4 (d9)
> System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9) Relocating
> guest memory for lowmem MMIO space disabled (XEN) irq.c:270: Dom9 PCI
> link 0 changed 0 -> 5 (d9) PCI-ISA link 0 routed to IRQ5 (XEN)
> irq.c:270: Dom9 PCI link 1 changed 0 -> 10 (d9) PCI-ISA link 1 routed
> to IRQ10 (XEN)
> irq.c:270: Dom9 PCI link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed
> to
> IRQ11 (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
> link
> 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
> INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
> setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar 10
> size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size 001000000:
> 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d9) pci
> dev 04:0 bar 10 size 000020000: 0f3040000 (d9) pci dev 03:0 bar 30
> size
> 000010000: 0f3060000 (d9) pci dev 03:0 bar 14 size 000001000:
> 0f3070000
> (d9) pci dev 02:0 bar 10 size 000000100: 00000c001 (d9) pci dev 04:0
> bar
> 14 size 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
> 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
> phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
> 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) Testing
> HVM environment: (d9) - REP INSB across page boundaries ... passed
> (d9)
> - GS base MSRs and SWAPGS ... passed (d9) Passed 2 of 2 tests (d9)
> Writing SMBIOS tables ... (d9) Loading SeaBIOS ... (d9) Creating MP
> tables ... (d9) Loading ACPI ... (d9) vm86 TSS at fc00a080 (d9) BIOS
> map: (d9) 10000-100d3: Scratch space (d9) e0000-fffff: Main BIOS
> (d9)
> E820 table: (d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
> (d9)
> HOLE: 00000000:000a0000 - 00000000:000e0000 (d9) [01]:
> 00000000:000e0000 - 00000000:00100000: RESERVED (d9) [02]:
> 00000000:00100000 - 00000000:f0000000: RAM (d9) HOLE:
> 00000000:f0000000
> - 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
> RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM (d9)
> Invoking SeaBIOS ... (d9) SeaBIOS (version
> rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
> hypervisor signature at 40000000 (d9) xen: copy e820... (d9)
> Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
> Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
> Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
> xen: copy BIOS tables... (d9) Copying SMBIOS entry point from
> 0x00010010 to 0x000f1020
> (d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20 (d9)
> Copying PIR from 0x00010030 to 0x000f0ea0 (d9) Copying ACPI RSDP from
> 0x000100b0 to 0x000f0e70 (d9) Using pmtimer, ioport 0xb008, freq 3579
> kHz (d9) Scan for VGA option rom (d9) WARNING! Found unaligned PCI rom
> (vd=1013:00b8)
> (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
> stdvga and caching modes (d9) Turning on vga text mode console (d9)
> SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
> (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
> lpt ports
> (d9) Found 1 serial ports (d9) ATA controller 1 at 1f0/3f4/c140 (irq
> 14 dev 9) (d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9) (d9) ata0-0:
> QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes) (d9) Searching bootorder
> for: /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
> ATAPI-4 DVD/CD] (d9) Searching bootorder for:
> /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
> threads complete. (d9) Scan for option roms (d9) Running option rom at
> c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm call
> arg1=1
> (d9) pmm call arg1=0 (d9) Searching bootorder for: /pci@i0cf8/*@4 (d9)
> (d9) Press F12 for boot menu. (d9) (d9) Searching bootorder for: HALT
> (d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba
> LCHS=1024/255/63
> s=29360128 (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
> Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
> 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1: 000000000009fc00 -
> 00000000000a0000 = 2 RESERVED (d9) 2: 00000000000f0000 -
> 0000000000100000 = 2 RESERVED (d9) 3: 0000000000100000 -
> 00000000effff000 = 1 RAM (d9) 4: 00000000effff000 - 00000000f0000000 =
> 2 RESERVED (d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
> (d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
> handle_19: (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
> 0000:7c00
>
>
> System description:
>
> 1. Centos 6.4 x86_64, latest updates
>
> 2. Xen4centos packages - current kernel is
> 3.10.23-11.el6.centos.alt.x86_64
>
> 3. System with 32GB RAM
>
> 4. Virtual disks are ZFS volumes (aka - block devices). Same problem
> appeared when on local ext3 disks
>
> 5. Using OpenVSwitch version 2.0, compiled to RPM, again.
>
> 6. Intel CPU:
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 62
> model name : Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
> stepping : 4
> microcode : 0x416
> cpu MHz : 3400.158
> cache size : 12288 KB
> physical id : 0
> siblings : 12
> core id : 0
> cpu cores : 6
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi
> mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl
> nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 cx16 sse4_1
> sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor
> lahf_lm ida arat epb xsaveopt pln pts dtherm fsgsbase erms
> bogomips : 6800.31
> clflush size : 64
> cache_alignment : 64
> address sizes : 46 bits physical, 48 bits virtual
> power management:
>
>
> It was suggested by Mr. Ian Campbell to CC both Mr. Jun Nakajima and Mr.
> Eddie Dong, so, my apologies if this action is improper.
>
>
> Any tip or advice would be great. Any request for further info would
> be also great!
>
> Thanks!
>
> Etzion Bar-Noy
Best regards,
Yang
[-- Attachment #2: 0001-VMX-Eliminate-cr3-save-loading-exiting-when-UG-enabl.patch --]
[-- Type: application/octet-stream, Size: 2771 bytes --]
From fd9ec90b9f49d6da74558dfaed26b893f577e2ab Mon Sep 17 00:00:00 2001
From: Yang Zhang <yang.z.zhang@Intel.com>
Date: Wed, 23 Oct 2013 12:39:00 -0400
Subject: [PATCH v2] VMX: Eliminate cr3 save/loading exiting when UG enabled
With the feature of unrestricted guest, there should no vmexit
be triggered when guest accesses the cr3 in non-paging mode.
Signed-off-by: Yang Zhang <yang.z.zhang@Intel.com>
---
changes in v2:
Fix the guest boot failure on non-UG platform.
The previous patch doesn't consider the non-UG platform and will cause guest boot
failure on non-ug platform. During developing this patch, i was thinking whether
the reading of GUEST_CR3 is necessary on each vmexit. I thought the answer should
be no. But current implemention relies the {hw,guest}_cr3 in many places and it
hard to do a cleanup.(If anyone interesting, welcome :))
BTW: this patch will improve performance, especially for context switching intensive
workload, we saw about 1% improvement. So please backport it to 4.3.
---
xen/arch/x86/hvm/vmx/vmx.c | 9 +++++----
1 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
index dfff628..f6409d6 100644
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -1157,7 +1157,7 @@ static void vmx_update_guest_cr(struct vcpu *v, unsigned int cr)
uint32_t cr3_ctls = (CPU_BASED_CR3_LOAD_EXITING |
CPU_BASED_CR3_STORE_EXITING);
v->arch.hvm_vmx.exec_control &= ~cr3_ctls;
- if ( !hvm_paging_enabled(v) )
+ if ( !hvm_paging_enabled(v) && !vmx_unrestricted_guest(v) )
v->arch.hvm_vmx.exec_control |= cr3_ctls;
/* Trap CR3 updates if CR3 memory events are enabled. */
@@ -1231,7 +1231,7 @@ static void vmx_update_guest_cr(struct vcpu *v, unsigned int cr)
case 3:
if ( paging_mode_hap(v->domain) )
{
- if ( !hvm_paging_enabled(v) )
+ if ( !hvm_paging_enabled(v) && !vmx_unrestricted_guest(v) )
v->arch.hvm_vcpu.hw_cr[3] =
v->domain->arch.hvm_domain.params[HVM_PARAM_IDENT_PT];
vmx_load_pdptrs(v);
@@ -2487,10 +2487,11 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
hvm_invalidate_regs_fields(regs);
- if ( paging_mode_hap(v->domain) && hvm_paging_enabled(v) )
+ if ( paging_mode_hap(v->domain) )
{
__vmread(GUEST_CR3, &v->arch.hvm_vcpu.hw_cr[3]);
- v->arch.hvm_vcpu.guest_cr[3] = v->arch.hvm_vcpu.hw_cr[3];
+ if ( vmx_unrestricted_guest(v) || hvm_paging_enabled(v) )
+ v->arch.hvm_vcpu.guest_cr[3] = v->arch.hvm_vcpu.hw_cr[3];
}
__vmread(VM_EXIT_REASON, &exit_reason);
--
1.7.1
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-18 1:33 ` Zhang, Yang Z
@ 2013-12-18 9:46 ` Etzion Bar-Noy
2013-12-19 19:40 ` Etzion Bar-Noy
0 siblings, 1 reply; 10+ messages in thread
From: Etzion Bar-Noy @ 2013-12-18 9:46 UTC (permalink / raw)
To: Zhang, Yang Z; +Cc: Dong, Eddie, Nakajima, Jun, xen-devel@lists.xen.org
[-- Attachment #1.1: Type: text/plain, Size: 19758 bytes --]
Hi Mr. Yang.
I did attempt to use your suggested configuration with the XenServer VM,
but to no avail. The L2 VM keeps on crushing. I defined it now to leave a
dump file when crushing, and if you desire so, I will find some method of
uploading it for your viewing pleasure. As I am not a programmer, but an
integrator, my abilities in this direction are limited at best.
The L3 VM type is 'generic VM', because I wanted to enforce the system to
use HVM emulation.
I have just applied the patch you supplied me to my 'staging' git checkout,
and just finished compiling it. I will soon apply the newly created RPMs on
my test (and hopefully - to be production soon) server, and will check
VMware.
Thanks for the input. I will try and then respond with results.
Etzion
On Wed, Dec 18, 2013 at 3:33 AM, Zhang, Yang Z <yang.z.zhang@intel.com>wrote:
> Etzion Bar-Noy wrote on 2013-12-16:
> > Hi.
> >
>
> Hi Etzion Bar-Noy
>
> It's glad to know someone is using Nested Xen and report bugs to us. Also,
> if there are more nested bugs you found, please CC me too.
>
> >
> > I am using the unstable build-from-source of Xen community. It reports
> > as 4.4, and was pulled from Git few days ago. The problem I will
> > describe below appears on version 4.3.1, compiled from source as well.
> > In both cases - I used spec file to create my own RPMs, because it's the
> right method of doing it.
> >
> >
> > Problem: When VM running under HVM which is a hypervisor (XenServer
> > 6.2 or VMware ESXi5.5) attempts to start a (HVM) virtual machine, the
> > following happens, 100% reproducible on this server.
> >
> > XenServer: domain crash. XenServer restarts itself. It doesn't happen
> > for PV on XenServer VM
> >
> > VMware: An "unknown failure", which leads me to think that the nested
> > virtualization properties are not forwarded correctly to DomU
>
> To using nested, you need to add the set bit 31 of cupid.1.ecx to 0, add
> the following to your config file:
> cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
>
> For VMware ESX server, the attached patch is missing in current Xen which
> will caused the guest boot up fail. Also, I know some windows guests will
> boot fail on ESX.
> I never tried XenServer. But latest Xen on Xen is working well. So can you
> try the upsteam Xen to see whether it fails too?
>
> BTW, which L2 VM boot fail?
>
> >
> >
> > Just to clarify:
> >
> > Physical hardware: host
> >
> > Nested hypervisor: level 1 virtualization: XenServer or VMware ESXi
> >
> > Nested VM: level 2 virtualization: some generic VM inside XenServer or
> > ESXi (level 1)
> >
> >
> > This is the configuration used for XS:
> > name = "XenServer"
> > uuid = "a0b8c668-afaf-887a-1f4b-856dac4c8401"
> > maxmem = 4096
> > memory = 4096
> > vcpus = 2
> > builder = "hvm"
> > boot = "c"
> > pae = 1
> > acpi = 1
> > apic = 1
> > hap = 0
> > viridian = 0
> > rtc_timeoffset = 0
> > localtime = 0
> > on_poweroff = "destroy"
> > on_reboot = "restart"
> > on_crash = "restart"
> > sdl = 0
> > vnc = 1
> > vncunused = 1
> > keymap = "en-us"
> > disk = [ "tap2:aio:/dev/share/VMS/xenserver01.lun,hda,w",
> > ",hdc:cdrom,r" ] vif = [
> > "mac=00:16:3e:14:0d:72,bridge=lan,script=vif-bridge,model=e1000" ]
> parallel = "none"
> > serial = "pty"
> > nestedhvm = 1
> >
> > nx=1
> >
> >
> > # A note - this problem appears with and without the 'nx' flag.
> >
> > A snippet from strace to qemu process:
> >
> > poll([{fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=18,
> > events=POLLIN|POLLERR|POLLHUP}, {fd=29,
> > events=POLLIN|POLLERR|POLLHUP}, {fd=21,
> > events=POLLIN|POLLERR|POLLHUP}, {fd=15,
> > events=POLLIN|POLLERR|POLLHUP}, {fd=5, events=
> > POLLIN|POLLERR|POLLHUP}, {fd=7, events=POLLIN}, {fd=11,
> POLLIN|POLLERR|events=POLLIN}, {fd=28, events=POLLIN}, {fd=6,
> POLLIN|POLLERR|events=POLLIN}], 10, 307) = ?
> > ERESTART_RESTARTBLOCK (To POLLIN|POLLERR|be restarted) --- SIGHUP
> > (Hangup) @ 0 (0) --- write(6, "\1\0\0\0\0\0\0\0", 8) = 8
> > rt_sigreturn(0x2) = -1 EINTR (Interrupted system
> > call) gettimeofday({1386761621, 705266}, NULL) = 0
> > clock_gettime(CLOCK_MONOTONIC, {1840, 693236049}) = 0
> > clock_gettime(CLOCK_MONOTONIC, {1840, 693333096}) = 0
> > gettimeofday({1386761621, 705546}, NULL) = 0 write(2, "qemu: terminating
> > on signal 1", 29) = 29 write(2, " from pid 6933\n", 15) = 15
> >
> >
> > From the xl log for this domain:
> > Waiting for domain XenServer (domid 5) to die [pid 6933] Domain 5 has
> > shut down, reason code 3 0x3 Action for shutdown reason code 3 is
> > restart Domain
> > 5 needs to be cleaned up: destroying the domain Done. Rebooting now
> > Waiting for domain XenServer (domid 6) to die [pid 6933]
> >
> >
> > From 'xl dmesg' (XEN) <vm_launch_fail> error code 7 (XEN)
> > domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8 (vcpu#0)
> > crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not
> > tainted ]---- (XEN) CPU: 4 (XEN) RIP: 0000:[<0000000000000000>]
> > (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax:
> > 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80 (XEN)
> > rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
> > (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
> > 0000000000000004 (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70
> > r11: 0000000000000000 (XEN) r12: ffff830824f9ff50 r13:
> > ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
> > cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3:
> > 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
> > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
> > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4 (d9)
> > System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9) Relocating
> > guest memory for lowmem MMIO space disabled (XEN) irq.c:270: Dom9 PCI
> > link 0 changed 0 -> 5 (d9) PCI-ISA link 0 routed to IRQ5 (XEN)
> > irq.c:270: Dom9 PCI link 1 changed 0 -> 10 (d9) PCI-ISA link 1 routed
> > to IRQ10 (XEN)
> > irq.c:270: Dom9 PCI link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed
> > to
> > IRQ11 (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
> > link
> > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
> > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
> > setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar 10
> > size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size 001000000:
> > 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d9) pci
> > dev 04:0 bar 10 size 000020000: 0f3040000 (d9) pci dev 03:0 bar 30
> > size
> > 000010000: 0f3060000 (d9) pci dev 03:0 bar 14 size 000001000:
> > 0f3070000
> > (d9) pci dev 02:0 bar 10 size 000000100: 00000c001 (d9) pci dev 04:0
> > bar
> > 14 size 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
> > 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
> > phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
> > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) Testing
> > HVM environment: (d9) - REP INSB across page boundaries ... passed
> > (d9)
> > - GS base MSRs and SWAPGS ... passed (d9) Passed 2 of 2 tests (d9)
> > Writing SMBIOS tables ... (d9) Loading SeaBIOS ... (d9) Creating MP
> > tables ... (d9) Loading ACPI ... (d9) vm86 TSS at fc00a080 (d9) BIOS
> > map: (d9) 10000-100d3: Scratch space (d9) e0000-fffff: Main BIOS
> > (d9)
> > E820 table: (d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
> > (d9)
> > HOLE: 00000000:000a0000 - 00000000:000e0000 (d9) [01]:
> > 00000000:000e0000 - 00000000:00100000: RESERVED (d9) [02]:
> > 00000000:00100000 - 00000000:f0000000: RAM (d9) HOLE:
> > 00000000:f0000000
> > - 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
> > RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM (d9)
> > Invoking SeaBIOS ... (d9) SeaBIOS (version
> > rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
> > hypervisor signature at 40000000 (d9) xen: copy e820... (d9)
> > Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
> > Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
> > Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
> > xen: copy BIOS tables... (d9) Copying SMBIOS entry point from
> > 0x00010010 to 0x000f1020
> > (d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20 (d9)
> > Copying PIR from 0x00010030 to 0x000f0ea0 (d9) Copying ACPI RSDP from
> > 0x000100b0 to 0x000f0e70 (d9) Using pmtimer, ioport 0xb008, freq 3579
> > kHz (d9) Scan for VGA option rom (d9) WARNING! Found unaligned PCI rom
> > (vd=1013:00b8)
> > (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
> > stdvga and caching modes (d9) Turning on vga text mode console (d9)
> > SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
> > (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
> > lpt ports
> > (d9) Found 1 serial ports (d9) ATA controller 1 at 1f0/3f4/c140 (irq
> > 14 dev 9) (d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9) (d9)
> ata0-0:
> > QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes) (d9) Searching bootorder
> > for: /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
> > ATAPI-4 DVD/CD] (d9) Searching bootorder for:
> > /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
> > threads complete. (d9) Scan for option roms (d9) Running option rom at
> > c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm call
> > arg1=1
> > (d9) pmm call arg1=0 (d9) Searching bootorder for: /pci@i0cf8/*@4 (d9)
> > (d9) Press F12 for boot menu. (d9) (d9) Searching bootorder for: HALT
> > (d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba
> > LCHS=1024/255/63
> > s=29360128 (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
> > Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
> > 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1: 000000000009fc00 -
> > 00000000000a0000 = 2 RESERVED (d9) 2: 00000000000f0000 -
> > 0000000000100000 = 2 RESERVED (d9) 3: 0000000000100000 -
> > 00000000effff000 = 1 RAM (d9) 4: 00000000effff000 - 00000000f0000000 =
> > 2 RESERVED (d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
> > (d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
> > handle_19: (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
> > 0000:7c00 [root@monster ~]# xl dmesg (XEN) <vm_launch_fail> error code
> > 7
> > (XEN) domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8
> > (vcpu#0) crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64 debug=y
> Not
> > tainted ]---- (XEN) CPU: 4 (XEN) RIP: 0000:[<0000000000000000>]
> > (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax:
> > 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80 (XEN)
> > rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
> > (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
> > 0000000000000004 (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70
> > r11: 0000000000000000 (XEN) r12: ffff830824f9ff50 r13:
> > ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
> > cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3:
> > 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
> > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
> > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4 (d9)
> > System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9) Relocating
> > guest memory for lowmem MMIO space disabled (XEN) irq.c:270: Dom9 PCI
> > link 0 changed 0 -> 5 (d9) PCI-ISA link 0 routed to IRQ5 (XEN)
> > irq.c:270: Dom9 PCI link 1 changed 0 -> 10 (d9) PCI-ISA link 1 routed
> > to IRQ10 (XEN)
> > irq.c:270: Dom9 PCI link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed
> > to
> > IRQ11 (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
> > link
> > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
> > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
> > setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar 10
> > size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size 001000000:
> > 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d9) pci
> > dev 04:0 bar 10 size 000020000: 0f3040000 (d9) pci dev 03:0 bar 30
> > size
> > 000010000: 0f3060000 (d9) pci dev 03:0 bar 14 size 000001000:
> > 0f3070000
> > (d9) pci dev 02:0 bar 10 size 000000100: 00000c001 (d9) pci dev 04:0
> > bar
> > 14 size 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
> > 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
> > phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
> > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) Testing
> > HVM environment: (d9) - REP INSB across page boundaries ... passed
> > (d9)
> > - GS base MSRs and SWAPGS ... passed (d9) Passed 2 of 2 tests (d9)
> > Writing SMBIOS tables ... (d9) Loading SeaBIOS ... (d9) Creating MP
> > tables ... (d9) Loading ACPI ... (d9) vm86 TSS at fc00a080 (d9) BIOS
> > map: (d9) 10000-100d3: Scratch space (d9) e0000-fffff: Main BIOS
> > (d9)
> > E820 table: (d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
> > (d9)
> > HOLE: 00000000:000a0000 - 00000000:000e0000 (d9) [01]:
> > 00000000:000e0000 - 00000000:00100000: RESERVED (d9) [02]:
> > 00000000:00100000 - 00000000:f0000000: RAM (d9) HOLE:
> > 00000000:f0000000
> > - 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
> > RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM (d9)
> > Invoking SeaBIOS ... (d9) SeaBIOS (version
> > rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
> > hypervisor signature at 40000000 (d9) xen: copy e820... (d9)
> > Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
> > Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
> > Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
> > xen: copy BIOS tables... (d9) Copying SMBIOS entry point from
> > 0x00010010 to 0x000f1020
> > (d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20 (d9)
> > Copying PIR from 0x00010030 to 0x000f0ea0 (d9) Copying ACPI RSDP from
> > 0x000100b0 to 0x000f0e70 (d9) Using pmtimer, ioport 0xb008, freq 3579
> > kHz (d9) Scan for VGA option rom (d9) WARNING! Found unaligned PCI rom
> > (vd=1013:00b8)
> > (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
> > stdvga and caching modes (d9) Turning on vga text mode console (d9)
> > SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
> > (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
> > lpt ports
> > (d9) Found 1 serial ports (d9) ATA controller 1 at 1f0/3f4/c140 (irq
> > 14 dev 9) (d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9) (d9)
> ata0-0:
> > QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes) (d9) Searching bootorder
> > for: /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
> > ATAPI-4 DVD/CD] (d9) Searching bootorder for:
> > /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
> > threads complete. (d9) Scan for option roms (d9) Running option rom at
> > c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm call
> > arg1=1
> > (d9) pmm call arg1=0 (d9) Searching bootorder for: /pci@i0cf8/*@4 (d9)
> > (d9) Press F12 for boot menu. (d9) (d9) Searching bootorder for: HALT
> > (d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba
> > LCHS=1024/255/63
> > s=29360128 (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
> > Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
> > 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1: 000000000009fc00 -
> > 00000000000a0000 = 2 RESERVED (d9) 2: 00000000000f0000 -
> > 0000000000100000 = 2 RESERVED (d9) 3: 0000000000100000 -
> > 00000000effff000 = 1 RAM (d9) 4: 00000000effff000 - 00000000f0000000 =
> > 2 RESERVED (d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
> > (d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
> > handle_19: (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
> > 0000:7c00
> >
> >
> > System description:
> >
> > 1. Centos 6.4 x86_64, latest updates
> >
> > 2. Xen4centos packages - current kernel is
> > 3.10.23-11.el6.centos.alt.x86_64
> >
> > 3. System with 32GB RAM
> >
> > 4. Virtual disks are ZFS volumes (aka - block devices). Same problem
> > appeared when on local ext3 disks
> >
> > 5. Using OpenVSwitch version 2.0, compiled to RPM, again.
> >
> > 6. Intel CPU:
> > processor : 0
> > vendor_id : GenuineIntel
> > cpu family : 6
> > model : 62
> > model name : Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
> > stepping : 4
> > microcode : 0x416
> > cpu MHz : 3400.158
> > cache size : 12288 KB
> > physical id : 0
> > siblings : 12
> > core id : 0
> > cpu cores : 6
> > apicid : 0
> > initial apicid : 0
> > fpu : yes
> > fpu_exception : yes
> > cpuid level : 13
> > wp : yes
> > flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush
> acpi
> > mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl
> > nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 cx16 sse4_1
> > sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor
> > lahf_lm ida arat epb xsaveopt pln pts dtherm fsgsbase erms
> > bogomips : 6800.31
> > clflush size : 64
> > cache_alignment : 64
> > address sizes : 46 bits physical, 48 bits virtual
> > power management:
> >
> >
> > It was suggested by Mr. Ian Campbell to CC both Mr. Jun Nakajima and Mr.
> > Eddie Dong, so, my apologies if this action is improper.
> >
> >
> > Any tip or advice would be great. Any request for further info would
> > be also great!
> >
> > Thanks!
> >
> > Etzion Bar-Noy
>
>
> Best regards,
> Yang
>
>
[-- Attachment #1.2: Type: text/html, Size: 22734 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-18 9:46 ` Etzion Bar-Noy
@ 2013-12-19 19:40 ` Etzion Bar-Noy
2013-12-19 19:42 ` Etzion Bar-Noy
0 siblings, 1 reply; 10+ messages in thread
From: Etzion Bar-Noy @ 2013-12-19 19:40 UTC (permalink / raw)
To: Zhang, Yang Z; +Cc: Dong, Eddie, Nakajima, Jun, xen-devel@lists.xen.org
[-- Attachment #1.1: Type: text/plain, Size: 22669 bytes --]
Hi.
With the added flag suggested by you, the VM (nested hypervisor) crashed.
Now ESXi will attempt to start the VM and crash as well (unlike before,
where it just failed to start it with 'unknown error').
XenServer 6.2 still crashes, just the same.
I am attempting to deploy a Centos6 machine (and clone it) so that I will
be able to test both KVM and Xen community (latest stable RPM release for
Centos6).
xl dmesg looks much like before:
(XEN) stdvga.c:151:d12 leaving stdvga
(XEN) common.c:3525:d0 tracking VRAM f0000 - f0800
(XEN) stdvga.c:147:d12 entering stdvga and caching modes
(XEN) realmode.c:115:d12 Failed to emulate insn.
(XEN) realmode.c:167:d12 Real-mode emulation failed @ f000:0000fff0: 00 10
23 00 c0 82
(XEN) domain_crash called from realmode.c:168
(XEN) Domain 12 (vcpu#0) crashed on cpu#4:
(XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 4
(XEN) RIP: f000:[<000000000000fff0>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx: 0000000000000000
(XEN) rdx: 00000000078bfbff rsi: 0000000000000000 rdi: 0000000000000000
(XEN) rbp: 0000000000000000 rsp: 0000000000000000 r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 0000000000000030 cr4: 0000000000002040
(XEN) cr3: 0000000000000000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: f000
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
Thanks!
Etzion
On Wed, Dec 18, 2013 at 11:46 AM, Etzion Bar-Noy <etzion@barnoy.co.il>wrote:
> Hi Mr. Yang.
> I did attempt to use your suggested configuration with the XenServer VM,
> but to no avail. The L2 VM keeps on crushing. I defined it now to leave a
> dump file when crushing, and if you desire so, I will find some method of
> uploading it for your viewing pleasure. As I am not a programmer, but an
> integrator, my abilities in this direction are limited at best.
> The L3 VM type is 'generic VM', because I wanted to enforce the system to
> use HVM emulation.
>
> I have just applied the patch you supplied me to my 'staging' git
> checkout, and just finished compiling it. I will soon apply the newly
> created RPMs on my test (and hopefully - to be production soon) server, and
> will check VMware.
>
> Thanks for the input. I will try and then respond with results.
>
> Etzion
>
>
> On Wed, Dec 18, 2013 at 3:33 AM, Zhang, Yang Z <yang.z.zhang@intel.com>wrote:
>
>> Etzion Bar-Noy wrote on 2013-12-16:
>> > Hi.
>> >
>>
>> Hi Etzion Bar-Noy
>>
>> It's glad to know someone is using Nested Xen and report bugs to us.
>> Also, if there are more nested bugs you found, please CC me too.
>>
>> >
>> > I am using the unstable build-from-source of Xen community. It reports
>> > as 4.4, and was pulled from Git few days ago. The problem I will
>> > describe below appears on version 4.3.1, compiled from source as well.
>> > In both cases - I used spec file to create my own RPMs, because it's
>> the right method of doing it.
>> >
>> >
>> > Problem: When VM running under HVM which is a hypervisor (XenServer
>> > 6.2 or VMware ESXi5.5) attempts to start a (HVM) virtual machine, the
>> > following happens, 100% reproducible on this server.
>> >
>> > XenServer: domain crash. XenServer restarts itself. It doesn't happen
>> > for PV on XenServer VM
>> >
>> > VMware: An "unknown failure", which leads me to think that the nested
>> > virtualization properties are not forwarded correctly to DomU
>>
>> To using nested, you need to add the set bit 31 of cupid.1.ecx to 0, add
>> the following to your config file:
>> cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
>>
>> For VMware ESX server, the attached patch is missing in current Xen which
>> will caused the guest boot up fail. Also, I know some windows guests will
>> boot fail on ESX.
>> I never tried XenServer. But latest Xen on Xen is working well. So can
>> you try the upsteam Xen to see whether it fails too?
>>
>> BTW, which L2 VM boot fail?
>>
>> >
>> >
>> > Just to clarify:
>> >
>> > Physical hardware: host
>> >
>> > Nested hypervisor: level 1 virtualization: XenServer or VMware ESXi
>> >
>> > Nested VM: level 2 virtualization: some generic VM inside XenServer or
>> > ESXi (level 1)
>> >
>> >
>> > This is the configuration used for XS:
>> > name = "XenServer"
>> > uuid = "a0b8c668-afaf-887a-1f4b-856dac4c8401"
>> > maxmem = 4096
>> > memory = 4096
>> > vcpus = 2
>> > builder = "hvm"
>> > boot = "c"
>> > pae = 1
>> > acpi = 1
>> > apic = 1
>> > hap = 0
>> > viridian = 0
>> > rtc_timeoffset = 0
>> > localtime = 0
>> > on_poweroff = "destroy"
>> > on_reboot = "restart"
>> > on_crash = "restart"
>> > sdl = 0
>> > vnc = 1
>> > vncunused = 1
>> > keymap = "en-us"
>> > disk = [ "tap2:aio:/dev/share/VMS/xenserver01.lun,hda,w",
>> > ",hdc:cdrom,r" ] vif = [
>> > "mac=00:16:3e:14:0d:72,bridge=lan,script=vif-bridge,model=e1000" ]
>> parallel = "none"
>> > serial = "pty"
>> > nestedhvm = 1
>> >
>> > nx=1
>> >
>> >
>> > # A note - this problem appears with and without the 'nx' flag.
>> >
>> > A snippet from strace to qemu process:
>> >
>> > poll([{fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=18,
>> > events=POLLIN|POLLERR|POLLHUP}, {fd=29,
>> > events=POLLIN|POLLERR|POLLHUP}, {fd=21,
>> > events=POLLIN|POLLERR|POLLHUP}, {fd=15,
>> > events=POLLIN|POLLERR|POLLHUP}, {fd=5, events=
>> > POLLIN|POLLERR|POLLHUP}, {fd=7, events=POLLIN}, {fd=11,
>> POLLIN|POLLERR|events=POLLIN}, {fd=28, events=POLLIN}, {fd=6,
>> POLLIN|POLLERR|events=POLLIN}], 10, 307) = ?
>> > ERESTART_RESTARTBLOCK (To POLLIN|POLLERR|be restarted) --- SIGHUP
>> > (Hangup) @ 0 (0) --- write(6, "\1\0\0\0\0\0\0\0", 8) = 8
>> > rt_sigreturn(0x2) = -1 EINTR (Interrupted system
>> > call) gettimeofday({1386761621, 705266}, NULL) = 0
>> > clock_gettime(CLOCK_MONOTONIC, {1840, 693236049}) = 0
>> > clock_gettime(CLOCK_MONOTONIC, {1840, 693333096}) = 0
>> > gettimeofday({1386761621, 705546}, NULL) = 0 write(2, "qemu: terminating
>> > on signal 1", 29) = 29 write(2, " from pid 6933\n", 15) = 15
>> >
>> >
>> > From the xl log for this domain:
>> > Waiting for domain XenServer (domid 5) to die [pid 6933] Domain 5 has
>> > shut down, reason code 3 0x3 Action for shutdown reason code 3 is
>> > restart Domain
>> > 5 needs to be cleaned up: destroying the domain Done. Rebooting now
>> > Waiting for domain XenServer (domid 6) to die [pid 6933]
>> >
>> >
>> > From 'xl dmesg' (XEN) <vm_launch_fail> error code 7 (XEN)
>> > domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8 (vcpu#0)
>> > crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not
>> > tainted ]---- (XEN) CPU: 4 (XEN) RIP: 0000:[<0000000000000000>]
>> > (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax:
>> > 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80 (XEN)
>> > rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
>> > (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
>> > 0000000000000004 (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70
>> > r11: 0000000000000000 (XEN) r12: ffff830824f9ff50 r13:
>> > ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
>> > cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3:
>> > 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
>> > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
>> > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4 (d9)
>> > System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9) Relocating
>> > guest memory for lowmem MMIO space disabled (XEN) irq.c:270: Dom9 PCI
>> > link 0 changed 0 -> 5 (d9) PCI-ISA link 0 routed to IRQ5 (XEN)
>> > irq.c:270: Dom9 PCI link 1 changed 0 -> 10 (d9) PCI-ISA link 1 routed
>> > to IRQ10 (XEN)
>> > irq.c:270: Dom9 PCI link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed
>> > to
>> > IRQ11 (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
>> > link
>> > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
>> > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
>> > setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar 10
>> > size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size 001000000:
>> > 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d9) pci
>> > dev 04:0 bar 10 size 000020000: 0f3040000 (d9) pci dev 03:0 bar 30
>> > size
>> > 000010000: 0f3060000 (d9) pci dev 03:0 bar 14 size 000001000:
>> > 0f3070000
>> > (d9) pci dev 02:0 bar 10 size 000000100: 00000c001 (d9) pci dev 04:0
>> > bar
>> > 14 size 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
>> > 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
>> > phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
>> > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) Testing
>> > HVM environment: (d9) - REP INSB across page boundaries ... passed
>> > (d9)
>> > - GS base MSRs and SWAPGS ... passed (d9) Passed 2 of 2 tests (d9)
>> > Writing SMBIOS tables ... (d9) Loading SeaBIOS ... (d9) Creating MP
>> > tables ... (d9) Loading ACPI ... (d9) vm86 TSS at fc00a080 (d9) BIOS
>> > map: (d9) 10000-100d3: Scratch space (d9) e0000-fffff: Main BIOS
>> > (d9)
>> > E820 table: (d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
>> > (d9)
>> > HOLE: 00000000:000a0000 - 00000000:000e0000 (d9) [01]:
>> > 00000000:000e0000 - 00000000:00100000: RESERVED (d9) [02]:
>> > 00000000:00100000 - 00000000:f0000000: RAM (d9) HOLE:
>> > 00000000:f0000000
>> > - 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
>> > RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM (d9)
>> > Invoking SeaBIOS ... (d9) SeaBIOS (version
>> > rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
>> > hypervisor signature at 40000000 (d9) xen: copy e820... (d9)
>> > Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
>> > Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
>> > Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
>> > xen: copy BIOS tables... (d9) Copying SMBIOS entry point from
>> > 0x00010010 to 0x000f1020
>> > (d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20 (d9)
>> > Copying PIR from 0x00010030 to 0x000f0ea0 (d9) Copying ACPI RSDP from
>> > 0x000100b0 to 0x000f0e70 (d9) Using pmtimer, ioport 0xb008, freq 3579
>> > kHz (d9) Scan for VGA option rom (d9) WARNING! Found unaligned PCI rom
>> > (vd=1013:00b8)
>> > (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
>> > stdvga and caching modes (d9) Turning on vga text mode console (d9)
>> > SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
>> > (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
>> > lpt ports
>> > (d9) Found 1 serial ports (d9) ATA controller 1 at 1f0/3f4/c140 (irq
>> > 14 dev 9) (d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9) (d9)
>> ata0-0:
>> > QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes) (d9) Searching bootorder
>> > for: /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
>> > ATAPI-4 DVD/CD] (d9) Searching bootorder for:
>> > /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
>> > threads complete. (d9) Scan for option roms (d9) Running option rom at
>> > c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm call
>> > arg1=1
>> > (d9) pmm call arg1=0 (d9) Searching bootorder for: /pci@i0cf8/*@4 (d9)
>> > (d9) Press F12 for boot menu. (d9) (d9) Searching bootorder for: HALT
>> > (d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba
>> > LCHS=1024/255/63
>> > s=29360128 (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
>> > Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
>> > 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1: 000000000009fc00 -
>> > 00000000000a0000 = 2 RESERVED (d9) 2: 00000000000f0000 -
>> > 0000000000100000 = 2 RESERVED (d9) 3: 0000000000100000 -
>> > 00000000effff000 = 1 RAM (d9) 4: 00000000effff000 - 00000000f0000000 =
>> > 2 RESERVED (d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
>> > (d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
>> > handle_19: (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
>> > 0000:7c00 [root@monster ~]# xl dmesg (XEN) <vm_launch_fail> error code
>> > 7
>> > (XEN) domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8
>> > (vcpu#0) crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64
>> debug=y Not
>> > tainted ]---- (XEN) CPU: 4 (XEN) RIP: 0000:[<0000000000000000>]
>> > (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax:
>> > 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80 (XEN)
>> > rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
>> > (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
>> > 0000000000000004 (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70
>> > r11: 0000000000000000 (XEN) r12: ffff830824f9ff50 r13:
>> > ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
>> > cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3:
>> > 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
>> > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>> > EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
>> > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4 (d9)
>> > System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9) Relocating
>> > guest memory for lowmem MMIO space disabled (XEN) irq.c:270: Dom9 PCI
>> > link 0 changed 0 -> 5 (d9) PCI-ISA link 0 routed to IRQ5 (XEN)
>> > irq.c:270: Dom9 PCI link 1 changed 0 -> 10 (d9) PCI-ISA link 1 routed
>> > to IRQ10 (XEN)
>> > irq.c:270: Dom9 PCI link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed
>> > to
>> > IRQ11 (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
>> > link
>> > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
>> > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
>> > setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar 10
>> > size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size 001000000:
>> > 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d9) pci
>> > dev 04:0 bar 10 size 000020000: 0f3040000 (d9) pci dev 03:0 bar 30
>> > size
>> > 000010000: 0f3060000 (d9) pci dev 03:0 bar 14 size 000001000:
>> > 0f3070000
>> > (d9) pci dev 02:0 bar 10 size 000000100: 00000c001 (d9) pci dev 04:0
>> > bar
>> > 14 size 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
>> > 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
>> > phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
>> > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) Testing
>> > HVM environment: (d9) - REP INSB across page boundaries ... passed
>> > (d9)
>> > - GS base MSRs and SWAPGS ... passed (d9) Passed 2 of 2 tests (d9)
>> > Writing SMBIOS tables ... (d9) Loading SeaBIOS ... (d9) Creating MP
>> > tables ... (d9) Loading ACPI ... (d9) vm86 TSS at fc00a080 (d9) BIOS
>> > map: (d9) 10000-100d3: Scratch space (d9) e0000-fffff: Main BIOS
>> > (d9)
>> > E820 table: (d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
>> > (d9)
>> > HOLE: 00000000:000a0000 - 00000000:000e0000 (d9) [01]:
>> > 00000000:000e0000 - 00000000:00100000: RESERVED (d9) [02]:
>> > 00000000:00100000 - 00000000:f0000000: RAM (d9) HOLE:
>> > 00000000:f0000000
>> > - 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
>> > RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM (d9)
>> > Invoking SeaBIOS ... (d9) SeaBIOS (version
>> > rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
>> > hypervisor signature at 40000000 (d9) xen: copy e820... (d9)
>> > Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
>> > Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
>> > Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
>> > xen: copy BIOS tables... (d9) Copying SMBIOS entry point from
>> > 0x00010010 to 0x000f1020
>> > (d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20 (d9)
>> > Copying PIR from 0x00010030 to 0x000f0ea0 (d9) Copying ACPI RSDP from
>> > 0x000100b0 to 0x000f0e70 (d9) Using pmtimer, ioport 0xb008, freq 3579
>> > kHz (d9) Scan for VGA option rom (d9) WARNING! Found unaligned PCI rom
>> > (vd=1013:00b8)
>> > (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
>> > stdvga and caching modes (d9) Turning on vga text mode console (d9)
>> > SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
>> > (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
>> > lpt ports
>> > (d9) Found 1 serial ports (d9) ATA controller 1 at 1f0/3f4/c140 (irq
>> > 14 dev 9) (d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9) (d9)
>> ata0-0:
>> > QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes) (d9) Searching bootorder
>> > for: /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
>> > ATAPI-4 DVD/CD] (d9) Searching bootorder for:
>> > /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
>> > threads complete. (d9) Scan for option roms (d9) Running option rom at
>> > c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm call
>> > arg1=1
>> > (d9) pmm call arg1=0 (d9) Searching bootorder for: /pci@i0cf8/*@4 (d9)
>> > (d9) Press F12 for boot menu. (d9) (d9) Searching bootorder for: HALT
>> > (d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba
>> > LCHS=1024/255/63
>> > s=29360128 (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
>> > Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
>> > 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1: 000000000009fc00 -
>> > 00000000000a0000 = 2 RESERVED (d9) 2: 00000000000f0000 -
>> > 0000000000100000 = 2 RESERVED (d9) 3: 0000000000100000 -
>> > 00000000effff000 = 1 RAM (d9) 4: 00000000effff000 - 00000000f0000000 =
>> > 2 RESERVED (d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
>> > (d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
>> > handle_19: (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
>> > 0000:7c00
>> >
>> >
>> > System description:
>> >
>> > 1. Centos 6.4 x86_64, latest updates
>> >
>> > 2. Xen4centos packages - current kernel is
>> > 3.10.23-11.el6.centos.alt.x86_64
>> >
>> > 3. System with 32GB RAM
>> >
>> > 4. Virtual disks are ZFS volumes (aka - block devices). Same problem
>> > appeared when on local ext3 disks
>> >
>> > 5. Using OpenVSwitch version 2.0, compiled to RPM, again.
>> >
>> > 6. Intel CPU:
>> > processor : 0
>> > vendor_id : GenuineIntel
>> > cpu family : 6
>> > model : 62
>> > model name : Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
>> > stepping : 4
>> > microcode : 0x416
>> > cpu MHz : 3400.158
>> > cache size : 12288 KB
>> > physical id : 0
>> > siblings : 12
>> > core id : 0
>> > cpu cores : 6
>> > apicid : 0
>> > initial apicid : 0
>> > fpu : yes
>> > fpu_exception : yes
>> > cpuid level : 13
>> > wp : yes
>> > flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush
>> acpi
>> > mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl
>> > nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 cx16 sse4_1
>> > sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor
>> > lahf_lm ida arat epb xsaveopt pln pts dtherm fsgsbase erms
>> > bogomips : 6800.31
>> > clflush size : 64
>> > cache_alignment : 64
>> > address sizes : 46 bits physical, 48 bits virtual
>> > power management:
>> >
>> >
>> > It was suggested by Mr. Ian Campbell to CC both Mr. Jun Nakajima and Mr.
>> > Eddie Dong, so, my apologies if this action is improper.
>> >
>> >
>> > Any tip or advice would be great. Any request for further info would
>> > be also great!
>> >
>> > Thanks!
>> >
>> > Etzion Bar-Noy
>>
>>
>> Best regards,
>> Yang
>>
>>
>
[-- Attachment #1.2: Type: text/html, Size: 25817 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-19 19:40 ` Etzion Bar-Noy
@ 2013-12-19 19:42 ` Etzion Bar-Noy
2013-12-20 0:50 ` Zhang, Yang Z
0 siblings, 1 reply; 10+ messages in thread
From: Etzion Bar-Noy @ 2013-12-19 19:42 UTC (permalink / raw)
To: Zhang, Yang Z; +Cc: Dong, Eddie, Nakajima, Jun, xen-devel@lists.xen.org
[-- Attachment #1.1: Type: text/plain, Size: 26381 bytes --]
Sorry for spamming. Actually - it is a bit different - two results which
appear (on XenServer and on VMware). The previous messages were for Vmware.
The one below is for XenServer:
xl dmesg:
(XEN) stdvga.c:151:d10 leaving stdvga
(XEN) common.c:3525:d0 tracking VRAM f0000 - f0800
(XEN) stdvga.c:147:d10 entering stdvga and caching modes
(XEN) realmode.c:115:d10 Failed to emulate insn.
(XEN) realmode.c:167:d10 Real-mode emulation failed @ f000:0000fff0: 00 90
23 00 c0 82
(XEN) domain_crash called from realmode.c:168
(XEN) Domain 10 (vcpu#0) crashed on cpu#8:
(XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 8
(XEN) RIP: f000:[<000000000000fff0>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx: 0000000000000000
(XEN) rdx: 00000000078bfbff rsi: 0000000000000000 rdi: 0000000000000000
(XEN) rbp: 0000000000000000 rsp: 0000000000000000 r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 0000000000000030 cr4: 0000000000002040
(XEN) cr3: 0000000000000000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: f000
(XEN) <vm_launch_fail> error code 7
(XEN) domain_crash_sync called from vmcs.c:1293
(XEN) Domain 9 (vcpu#0) crashed on cpu#5:
(XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: 0000:[<0000000000000000>]
(XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
(XEN) rax: 0000000000000000 rbx: ffff830824f90000 rcx: ffff830824f97f80
(XEN) rdx: ffff82d0801d0cb9 rsi: 0000000000000000 rdi: ffff82d0801de7ec
(XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8: 0000000000000004
(XEN) r9: ffff82d080105b20 r10: ffff830824f97f70 r11: 0000000000000000
(XEN) r12: ffff830824f97f50 r13: ffff83080191c000 r14: 0000000000000004
(XEN) r15: ffff82d080187e15 cr0: 0000000000000039 cr4: 0000000000002050
(XEN) cr3: 00000000feffa000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: 0000
(XEN) grant_table.c:289:d0 Increased maptrack size to 2 frames
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
(XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
Thanks
Etzion
On Thu, Dec 19, 2013 at 9:40 PM, Etzion Bar-Noy <etzion@barnoy.co.il> wrote:
> Hi.
> With the added flag suggested by you, the VM (nested hypervisor) crashed.
> Now ESXi will attempt to start the VM and crash as well (unlike before,
> where it just failed to start it with 'unknown error').
> XenServer 6.2 still crashes, just the same.
>
> I am attempting to deploy a Centos6 machine (and clone it) so that I will
> be able to test both KVM and Xen community (latest stable RPM release for
> Centos6).
>
> xl dmesg looks much like before:
>
> (XEN) stdvga.c:151:d12 leaving stdvga
> (XEN) common.c:3525:d0 tracking VRAM f0000 - f0800
> (XEN) stdvga.c:147:d12 entering stdvga and caching modes
> (XEN) realmode.c:115:d12 Failed to emulate insn.
> (XEN) realmode.c:167:d12 Real-mode emulation failed @ f000:0000fff0: 00 10
> 23 00 c0 82
> (XEN) domain_crash called from realmode.c:168
> (XEN) Domain 12 (vcpu#0) crashed on cpu#4:
>
> (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
> (XEN) CPU: 4
> (XEN) RIP: f000:[<000000000000fff0>]
>
> (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
> (XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx: 0000000000000000
> (XEN) rdx: 00000000078bfbff rsi: 0000000000000000 rdi: 0000000000000000
> (XEN) rbp: 0000000000000000 rsp: 0000000000000000 r8: 0000000000000000
> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> (XEN) r15: 0000000000000000 cr0: 0000000000000030 cr4: 0000000000002040
> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: f000
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN) event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
>
> Thanks!
> Etzion
>
>
> On Wed, Dec 18, 2013 at 11:46 AM, Etzion Bar-Noy <etzion@barnoy.co.il>wrote:
>
>> Hi Mr. Yang.
>> I did attempt to use your suggested configuration with the XenServer VM,
>> but to no avail. The L2 VM keeps on crushing. I defined it now to leave a
>> dump file when crushing, and if you desire so, I will find some method of
>> uploading it for your viewing pleasure. As I am not a programmer, but an
>> integrator, my abilities in this direction are limited at best.
>> The L3 VM type is 'generic VM', because I wanted to enforce the system to
>> use HVM emulation.
>>
>> I have just applied the patch you supplied me to my 'staging' git
>> checkout, and just finished compiling it. I will soon apply the newly
>> created RPMs on my test (and hopefully - to be production soon) server, and
>> will check VMware.
>>
>> Thanks for the input. I will try and then respond with results.
>>
>> Etzion
>>
>>
>> On Wed, Dec 18, 2013 at 3:33 AM, Zhang, Yang Z <yang.z.zhang@intel.com>wrote:
>>
>>> Etzion Bar-Noy wrote on 2013-12-16:
>>> > Hi.
>>> >
>>>
>>> Hi Etzion Bar-Noy
>>>
>>> It's glad to know someone is using Nested Xen and report bugs to us.
>>> Also, if there are more nested bugs you found, please CC me too.
>>>
>>> >
>>> > I am using the unstable build-from-source of Xen community. It reports
>>> > as 4.4, and was pulled from Git few days ago. The problem I will
>>> > describe below appears on version 4.3.1, compiled from source as well.
>>> > In both cases - I used spec file to create my own RPMs, because it's
>>> the right method of doing it.
>>> >
>>> >
>>> > Problem: When VM running under HVM which is a hypervisor (XenServer
>>> > 6.2 or VMware ESXi5.5) attempts to start a (HVM) virtual machine, the
>>> > following happens, 100% reproducible on this server.
>>> >
>>> > XenServer: domain crash. XenServer restarts itself. It doesn't happen
>>> > for PV on XenServer VM
>>> >
>>> > VMware: An "unknown failure", which leads me to think that the nested
>>> > virtualization properties are not forwarded correctly to DomU
>>>
>>> To using nested, you need to add the set bit 31 of cupid.1.ecx to 0, add
>>> the following to your config file:
>>> cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
>>>
>>> For VMware ESX server, the attached patch is missing in current Xen
>>> which will caused the guest boot up fail. Also, I know some windows guests
>>> will boot fail on ESX.
>>> I never tried XenServer. But latest Xen on Xen is working well. So can
>>> you try the upsteam Xen to see whether it fails too?
>>>
>>> BTW, which L2 VM boot fail?
>>>
>>> >
>>> >
>>> > Just to clarify:
>>> >
>>> > Physical hardware: host
>>> >
>>> > Nested hypervisor: level 1 virtualization: XenServer or VMware ESXi
>>> >
>>> > Nested VM: level 2 virtualization: some generic VM inside XenServer or
>>> > ESXi (level 1)
>>> >
>>> >
>>> > This is the configuration used for XS:
>>> > name = "XenServer"
>>> > uuid = "a0b8c668-afaf-887a-1f4b-856dac4c8401"
>>> > maxmem = 4096
>>> > memory = 4096
>>> > vcpus = 2
>>> > builder = "hvm"
>>> > boot = "c"
>>> > pae = 1
>>> > acpi = 1
>>> > apic = 1
>>> > hap = 0
>>> > viridian = 0
>>> > rtc_timeoffset = 0
>>> > localtime = 0
>>> > on_poweroff = "destroy"
>>> > on_reboot = "restart"
>>> > on_crash = "restart"
>>> > sdl = 0
>>> > vnc = 1
>>> > vncunused = 1
>>> > keymap = "en-us"
>>> > disk = [ "tap2:aio:/dev/share/VMS/xenserver01.lun,hda,w",
>>> > ",hdc:cdrom,r" ] vif = [
>>> > "mac=00:16:3e:14:0d:72,bridge=lan,script=vif-bridge,model=e1000" ]
>>> parallel = "none"
>>> > serial = "pty"
>>> > nestedhvm = 1
>>> >
>>> > nx=1
>>> >
>>> >
>>> > # A note - this problem appears with and without the 'nx' flag.
>>> >
>>> > A snippet from strace to qemu process:
>>> >
>>> > poll([{fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=18,
>>> > events=POLLIN|POLLERR|POLLHUP}, {fd=29,
>>> > events=POLLIN|POLLERR|POLLHUP}, {fd=21,
>>> > events=POLLIN|POLLERR|POLLHUP}, {fd=15,
>>> > events=POLLIN|POLLERR|POLLHUP}, {fd=5, events=
>>> > POLLIN|POLLERR|POLLHUP}, {fd=7, events=POLLIN}, {fd=11,
>>> POLLIN|POLLERR|events=POLLIN}, {fd=28, events=POLLIN}, {fd=6,
>>> POLLIN|POLLERR|events=POLLIN}], 10, 307) = ?
>>> > ERESTART_RESTARTBLOCK (To POLLIN|POLLERR|be restarted) --- SIGHUP
>>> > (Hangup) @ 0 (0) --- write(6, "\1\0\0\0\0\0\0\0", 8) = 8
>>> > rt_sigreturn(0x2) = -1 EINTR (Interrupted system
>>> > call) gettimeofday({1386761621, 705266}, NULL) = 0
>>> > clock_gettime(CLOCK_MONOTONIC, {1840, 693236049}) = 0
>>> > clock_gettime(CLOCK_MONOTONIC, {1840, 693333096}) = 0
>>> > gettimeofday({1386761621, 705546}, NULL) = 0 write(2, "qemu:
>>> terminating
>>> > on signal 1", 29) = 29 write(2, " from pid 6933\n", 15) = 15
>>> >
>>> >
>>> > From the xl log for this domain:
>>> > Waiting for domain XenServer (domid 5) to die [pid 6933] Domain 5 has
>>> > shut down, reason code 3 0x3 Action for shutdown reason code 3 is
>>> > restart Domain
>>> > 5 needs to be cleaned up: destroying the domain Done. Rebooting now
>>> > Waiting for domain XenServer (domid 6) to die [pid 6933]
>>> >
>>> >
>>> > From 'xl dmesg' (XEN) <vm_launch_fail> error code 7 (XEN)
>>> > domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8 (vcpu#0)
>>> > crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not
>>> > tainted ]---- (XEN) CPU: 4 (XEN) RIP: 0000:[<0000000000000000>]
>>> > (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax:
>>> > 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80 (XEN)
>>> > rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
>>> > (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
>>> > 0000000000000004 (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70
>>> > r11: 0000000000000000 (XEN) r12: ffff830824f9ff50 r13:
>>> > ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
>>> > cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3:
>>> > 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000
>>> fs:
>>> > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
>>> > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4 (d9)
>>> > System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9) Relocating
>>> > guest memory for lowmem MMIO space disabled (XEN) irq.c:270: Dom9 PCI
>>> > link 0 changed 0 -> 5 (d9) PCI-ISA link 0 routed to IRQ5 (XEN)
>>> > irq.c:270: Dom9 PCI link 1 changed 0 -> 10 (d9) PCI-ISA link 1 routed
>>> > to IRQ10 (XEN)
>>> > irq.c:270: Dom9 PCI link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed
>>> > to
>>> > IRQ11 (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
>>> > link
>>> > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
>>> > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
>>> > setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar 10
>>> > size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size 001000000:
>>> > 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d9) pci
>>> > dev 04:0 bar 10 size 000020000: 0f3040000 (d9) pci dev 03:0 bar 30
>>> > size
>>> > 000010000: 0f3060000 (d9) pci dev 03:0 bar 14 size 000001000:
>>> > 0f3070000
>>> > (d9) pci dev 02:0 bar 10 size 000000100: 00000c001 (d9) pci dev 04:0
>>> > bar
>>> > 14 size 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
>>> > 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
>>> > phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
>>> > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) Testing
>>> > HVM environment: (d9) - REP INSB across page boundaries ... passed
>>> > (d9)
>>> > - GS base MSRs and SWAPGS ... passed (d9) Passed 2 of 2 tests (d9)
>>> > Writing SMBIOS tables ... (d9) Loading SeaBIOS ... (d9) Creating MP
>>> > tables ... (d9) Loading ACPI ... (d9) vm86 TSS at fc00a080 (d9) BIOS
>>> > map: (d9) 10000-100d3: Scratch space (d9) e0000-fffff: Main BIOS
>>> > (d9)
>>> > E820 table: (d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
>>> > (d9)
>>> > HOLE: 00000000:000a0000 - 00000000:000e0000 (d9) [01]:
>>> > 00000000:000e0000 - 00000000:00100000: RESERVED (d9) [02]:
>>> > 00000000:00100000 - 00000000:f0000000: RAM (d9) HOLE:
>>> > 00000000:f0000000
>>> > - 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
>>> > RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM (d9)
>>> > Invoking SeaBIOS ... (d9) SeaBIOS (version
>>> > rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
>>> > hypervisor signature at 40000000 (d9) xen: copy e820... (d9)
>>> > Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
>>> > Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
>>> > Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
>>> > xen: copy BIOS tables... (d9) Copying SMBIOS entry point from
>>> > 0x00010010 to 0x000f1020
>>> > (d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20 (d9)
>>> > Copying PIR from 0x00010030 to 0x000f0ea0 (d9) Copying ACPI RSDP from
>>> > 0x000100b0 to 0x000f0e70 (d9) Using pmtimer, ioport 0xb008, freq 3579
>>> > kHz (d9) Scan for VGA option rom (d9) WARNING! Found unaligned PCI rom
>>> > (vd=1013:00b8)
>>> > (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
>>> > stdvga and caching modes (d9) Turning on vga text mode console (d9)
>>> > SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
>>> > (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
>>> > lpt ports
>>> > (d9) Found 1 serial ports (d9) ATA controller 1 at 1f0/3f4/c140 (irq
>>> > 14 dev 9) (d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9) (d9)
>>> ata0-0:
>>> > QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes) (d9) Searching bootorder
>>> > for: /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
>>> > ATAPI-4 DVD/CD] (d9) Searching bootorder for:
>>> > /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
>>> > threads complete. (d9) Scan for option roms (d9) Running option rom at
>>> > c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm call
>>> > arg1=1
>>> > (d9) pmm call arg1=0 (d9) Searching bootorder for: /pci@i0cf8/*@4 (d9)
>>> > (d9) Press F12 for boot menu. (d9) (d9) Searching bootorder for: HALT
>>> > (d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba
>>> > LCHS=1024/255/63
>>> > s=29360128 (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
>>> > Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
>>> > 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1: 000000000009fc00
>>> -
>>> > 00000000000a0000 = 2 RESERVED (d9) 2: 00000000000f0000 -
>>> > 0000000000100000 = 2 RESERVED (d9) 3: 0000000000100000 -
>>> > 00000000effff000 = 1 RAM (d9) 4: 00000000effff000 - 00000000f0000000
>>> =
>>> > 2 RESERVED (d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
>>> > (d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
>>> > handle_19: (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
>>> > 0000:7c00 [root@monster ~]# xl dmesg (XEN) <vm_launch_fail> error code
>>> > 7
>>> > (XEN) domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8
>>> > (vcpu#0) crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64
>>> debug=y Not
>>> > tainted ]---- (XEN) CPU: 4 (XEN) RIP: 0000:[<0000000000000000>]
>>> > (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax:
>>> > 0000000000000000 rbx: ffff830824f98000 rcx: ffff830824f9ff80 (XEN)
>>> > rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: ffff82d0801de7fc
>>> > (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8:
>>> > 0000000000000004 (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70
>>> > r11: 0000000000000000 (XEN) r12: ffff830824f9ff50 r13:
>>> > ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
>>> > cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3:
>>> > 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000
>>> fs:
>>> > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
>>> > EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
>>> > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4 (d9)
>>> > System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9) Relocating
>>> > guest memory for lowmem MMIO space disabled (XEN) irq.c:270: Dom9 PCI
>>> > link 0 changed 0 -> 5 (d9) PCI-ISA link 0 routed to IRQ5 (XEN)
>>> > irq.c:270: Dom9 PCI link 1 changed 0 -> 10 (d9) PCI-ISA link 1 routed
>>> > to IRQ10 (XEN)
>>> > irq.c:270: Dom9 PCI link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed
>>> > to
>>> > IRQ11 (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
>>> > link
>>> > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
>>> > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
>>> > setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar 10
>>> > size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size 001000000:
>>> > 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000: 0f3000000 (d9) pci
>>> > dev 04:0 bar 10 size 000020000: 0f3040000 (d9) pci dev 03:0 bar 30
>>> > size
>>> > 000010000: 0f3060000 (d9) pci dev 03:0 bar 14 size 000001000:
>>> > 0f3070000
>>> > (d9) pci dev 02:0 bar 10 size 000000100: 00000c001 (d9) pci dev 04:0
>>> > bar
>>> > 14 size 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
>>> > 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
>>> > phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
>>> > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) Testing
>>> > HVM environment: (d9) - REP INSB across page boundaries ... passed
>>> > (d9)
>>> > - GS base MSRs and SWAPGS ... passed (d9) Passed 2 of 2 tests (d9)
>>> > Writing SMBIOS tables ... (d9) Loading SeaBIOS ... (d9) Creating MP
>>> > tables ... (d9) Loading ACPI ... (d9) vm86 TSS at fc00a080 (d9) BIOS
>>> > map: (d9) 10000-100d3: Scratch space (d9) e0000-fffff: Main BIOS
>>> > (d9)
>>> > E820 table: (d9) [00]: 00000000:00000000 - 00000000:000a0000: RAM
>>> > (d9)
>>> > HOLE: 00000000:000a0000 - 00000000:000e0000 (d9) [01]:
>>> > 00000000:000e0000 - 00000000:00100000: RESERVED (d9) [02]:
>>> > 00000000:00100000 - 00000000:f0000000: RAM (d9) HOLE:
>>> > 00000000:f0000000
>>> > - 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
>>> > RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM (d9)
>>> > Invoking SeaBIOS ... (d9) SeaBIOS (version
>>> > rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
>>> > hypervisor signature at 40000000 (d9) xen: copy e820... (d9)
>>> > Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
>>> > Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
>>> > Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
>>> > xen: copy BIOS tables... (d9) Copying SMBIOS entry point from
>>> > 0x00010010 to 0x000f1020
>>> > (d9) Copying MPTABLE from 0xfc001170/fc001180 to 0x000f0f20 (d9)
>>> > Copying PIR from 0x00010030 to 0x000f0ea0 (d9) Copying ACPI RSDP from
>>> > 0x000100b0 to 0x000f0e70 (d9) Using pmtimer, ioport 0xb008, freq 3579
>>> > kHz (d9) Scan for VGA option rom (d9) WARNING! Found unaligned PCI rom
>>> > (vd=1013:00b8)
>>> > (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
>>> > stdvga and caching modes (d9) Turning on vga text mode console (d9)
>>> > SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
>>> > (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
>>> > lpt ports
>>> > (d9) Found 1 serial ports (d9) ATA controller 1 at 1f0/3f4/c140 (irq
>>> > 14 dev 9) (d9) ATA controller 2 at 170/374/c148 (irq 15 dev 9) (d9)
>>> ata0-0:
>>> > QEMU HARDDISK ATA-7 Hard-Disk (14336 MiBytes) (d9) Searching bootorder
>>> > for: /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
>>> > ATAPI-4 DVD/CD] (d9) Searching bootorder for:
>>> > /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
>>> > threads complete. (d9) Scan for option roms (d9) Running option rom at
>>> > c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm call
>>> > arg1=1
>>> > (d9) pmm call arg1=0 (d9) Searching bootorder for: /pci@i0cf8/*@4 (d9)
>>> > (d9) Press F12 for boot menu. (d9) (d9) Searching bootorder for: HALT
>>> > (d9) drive 0x000f0e20: PCHS=16383/16/63 translation=lba
>>> > LCHS=1024/255/63
>>> > s=29360128 (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
>>> > Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
>>> > 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1: 000000000009fc00
>>> -
>>> > 00000000000a0000 = 2 RESERVED (d9) 2: 00000000000f0000 -
>>> > 0000000000100000 = 2 RESERVED (d9) 3: 0000000000100000 -
>>> > 00000000effff000 = 1 RAM (d9) 4: 00000000effff000 - 00000000f0000000
>>> =
>>> > 2 RESERVED (d9) 5: 00000000fc000000 - 0000000100000000 = 2 RESERVED
>>> > (d9) 6: 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
>>> > handle_19: (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
>>> > 0000:7c00
>>> >
>>> >
>>> > System description:
>>> >
>>> > 1. Centos 6.4 x86_64, latest updates
>>> >
>>> > 2. Xen4centos packages - current kernel is
>>> > 3.10.23-11.el6.centos.alt.x86_64
>>> >
>>> > 3. System with 32GB RAM
>>> >
>>> > 4. Virtual disks are ZFS volumes (aka - block devices). Same problem
>>> > appeared when on local ext3 disks
>>> >
>>> > 5. Using OpenVSwitch version 2.0, compiled to RPM, again.
>>> >
>>> > 6. Intel CPU:
>>> > processor : 0
>>> > vendor_id : GenuineIntel
>>> > cpu family : 6
>>> > model : 62
>>> > model name : Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
>>> > stepping : 4
>>> > microcode : 0x416
>>> > cpu MHz : 3400.158
>>> > cache size : 12288 KB
>>> > physical id : 0
>>> > siblings : 12
>>> > core id : 0
>>> > cpu cores : 6
>>> > apicid : 0
>>> > initial apicid : 0
>>> > fpu : yes
>>> > fpu_exception : yes
>>> > cpuid level : 13
>>> > wp : yes
>>> > flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat
>>> clflush acpi
>>> > mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl
>>> > nonstop_tsc eagerfpu pni pclmulqdq monitor est ssse3 cx16 sse4_1
>>> > sse4_2 popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor
>>> > lahf_lm ida arat epb xsaveopt pln pts dtherm fsgsbase erms
>>> > bogomips : 6800.31
>>> > clflush size : 64
>>> > cache_alignment : 64
>>> > address sizes : 46 bits physical, 48 bits virtual
>>> > power management:
>>> >
>>> >
>>> > It was suggested by Mr. Ian Campbell to CC both Mr. Jun Nakajima and
>>> Mr.
>>> > Eddie Dong, so, my apologies if this action is improper.
>>> >
>>> >
>>> > Any tip or advice would be great. Any request for further info would
>>> > be also great!
>>> >
>>> > Thanks!
>>> >
>>> > Etzion Bar-Noy
>>>
>>>
>>> Best regards,
>>> Yang
>>>
>>>
>>
>
[-- Attachment #1.2: Type: text/html, Size: 29634 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-19 19:42 ` Etzion Bar-Noy
@ 2013-12-20 0:50 ` Zhang, Yang Z
2013-12-20 11:24 ` Etzion Bar-Noy
0 siblings, 1 reply; 10+ messages in thread
From: Zhang, Yang Z @ 2013-12-20 0:50 UTC (permalink / raw)
To: Etzion Bar-Noy; +Cc: Dong, Eddie, Nakajima, Jun, xen-devel@lists.xen.org
Etzion Bar-Noy wrote on 2013-12-20:
> Sorry for spamming. Actually - it is a bit different - two results
> which appear (on XenServer and on VMware). The previous messages were
> for Vmware. The one below is for XenServer:
I guess you are using shadow mode for L1 VMM. With shadow L1, this is a known issue to me. Can you try to use L1 EPT to see whether it is reproducible?
>
> xl dmesg:
>
> (XEN) stdvga.c:151:d10 leaving stdvga (XEN) common.c:3525:d0 tracking
> VRAM f0000 - f0800 (XEN) stdvga.c:147:d10 entering stdvga and caching
> modes (XEN) realmode.c:115:d10 Failed to emulate insn. (XEN)
> realmode.c:167:d10 Real-mode emulation failed @ f000:0000fff0: 00 90 23
> 00 c0 82 (XEN) domain_crash called from realmode.c:168 (XEN) Domain 10
> (vcpu#0) crashed on cpu#8: (XEN) ----[ Xen-4.4-unstable x86_64 debug=y
> Not tainted ]---- (XEN) CPU: 8 (XEN) RIP:
> f000:[<000000000000fff0>] (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm
> guest (XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx:
> 0000000000000000 (XEN) rdx: 00000000078bfbff rsi: 0000000000000000
> rdi: 0000000000000000 (XEN) rbp: 0000000000000000 rsp:
> 0000000000000000 r8: 0000000000000000 (XEN) r9: 0000000000000000
> r10: 0000000000000000 r11: 0000000000000000 (XEN) r12:
> 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN)
> r15: 0000000000000000 cr0: 0000000000000030 cr4: 0000000000002040
> (XEN) cr3: 0000000000000000 cr2: 0000000000000000 (XEN) ds: 0000 es:
> 0000 fs: 0000 gs: 0000 ss: 0000 cs: f000 (XEN) <vm_launch_fail>
> error code 7 (XEN) domain_crash_sync called from vmcs.c:1293 (XEN)
> Domain 9 (vcpu#0) crashed on cpu#5: (XEN) ----[ Xen-4.4-unstable x86_64
> debug=y Not tainted ]---- (XEN) CPU: 5 (XEN) RIP:
> 0000:[<0000000000000000>] (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm
> guest (XEN) rax: 0000000000000000 rbx: ffff830824f90000 rcx:
> ffff830824f97f80 (XEN) rdx: ffff82d0801d0cb9 rsi: 0000000000000000
> rdi: ffff82d0801de7ec (XEN) rbp: ffff82d080105ab1 rsp:
> 0000000000000000 r8: 0000000000000004 (XEN) r9: ffff82d080105b20
> r10: ffff830824f97f70 r11: 0000000000000000 (XEN) r12:
> ffff830824f97f50 r13: ffff83080191c000 r14: 0000000000000004 (XEN)
> r15: ffff82d080187e15 cr0: 0000000000000039 cr4: 0000000000002050
> (XEN) cr3: 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es:
> 0000 fs: 0000 gs: 0000 ss: 0000 cs: 0000 (XEN)
> grant_table.c:289:d0 Increased maptrack size to 2 frames (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
>
>
> Thanks
> Etzion
>
>
>
> On Thu, Dec 19, 2013 at 9:40 PM, Etzion Bar-Noy <etzion@barnoy.co.il> wrote:
>
>
> Hi.
>
> With the added flag suggested by you, the VM (nested hypervisor) crashed.
> Now ESXi will attempt to start the VM and crash as well (unlike
> before, where it just failed to start it with 'unknown error').
>
> XenServer 6.2 still crashes, just the same.
>
>
> I am attempting to deploy a Centos6 machine (and clone it) so that I
> will be able to test both KVM and Xen community (latest stable RPM
> release for Centos6).
>
>
> xl dmesg looks much like before:
>
> (XEN) stdvga.c:151:d12 leaving stdvga
> (XEN) common.c:3525:d0 tracking VRAM f0000 - f0800
> (XEN) stdvga.c:147:d12 entering stdvga and caching modes
> (XEN) realmode.c:115:d12 Failed to emulate insn.
> (XEN) realmode.c:167:d12 Real-mode emulation failed @ f000:0000fff0:
> 00 10 23 00 c0 82
> (XEN) domain_crash called from realmode.c:168
> (XEN) Domain 12 (vcpu#0) crashed on cpu#4:
>
> (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
> (XEN) CPU: 4
>
> (XEN) RIP: f000:[<000000000000fff0>]
>
> (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
>
> (XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx:
> 0000000000000000 (XEN) rdx: 00000000078bfbff rsi: 0000000000000000
> rdi: 0000000000000000 (XEN) rbp: 0000000000000000 rsp:
> 0000000000000000 r8: 0000000000000000 (XEN) r9: 0000000000000000
> r10: 0000000000000000 r11: 0000000000000000 (XEN) r12:
> 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN)
> r15: 0000000000000000 cr0: 0000000000000030 cr4: 0000000000002040
> (XEN) cr3: 0000000000000000 cr2: 0000000000000000 (XEN) ds: 0000
> es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: f000 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22 (XEN)
> event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
>
>
> Thanks!
> Etzion
>
>
>
> On Wed, Dec 18, 2013 at 11:46 AM, Etzion Bar-Noy <etzion@barnoy.co.il>
> wrote:
>
>
> Hi Mr. Yang.
>
> I did attempt to use your suggested configuration with the XenServer
> VM, but to no avail. The L2 VM keeps on crushing. I defined it now to
> leave a dump file when crushing, and if you desire so, I will find
> some method of uploading it for your viewing pleasure. As I am not a
> programmer, but an integrator, my abilities in this direction are limited at best.
>
> The L3 VM type is 'generic VM', because I wanted to enforce the
> system to use HVM emulation.
>
>
> I have just applied the patch you supplied me to my 'staging' git
> checkout, and just finished compiling it. I will soon apply the newly
> created RPMs on my test (and hopefully - to be production soon)
> server, and will check VMware.
>
>
> Thanks for the input. I will try and then respond with results.
>
> Etzion
>
>
>
> On Wed, Dec 18, 2013 at 3:33 AM, Zhang, Yang Z
> <yang.z.zhang@intel.com> wrote:
>
>
> Etzion Bar-Noy wrote on 2013-12-16:
> > Hi.
> >
>
> Hi Etzion Bar-Noy
>
> It's glad to know someone is using Nested Xen and report bugs to
> us. Also, if there are more nested bugs you found, please CC me too.
>
>
> > > I am using the unstable build-from-source of Xen community. It
> reports > as 4.4, and was pulled from Git few days ago. The problem I
> will > describe below appears on version 4.3.1, compiled from source
> as well. > In both cases - I used spec file to create my own RPMs,
> because it's the right method of doing it. > > > Problem: When
> VM running under HVM which is a hypervisor (XenServer > 6.2 or VMware
> ESXi5.5) attempts to start a (HVM) virtual machine, the > following
> happens, 100% reproducible on this server. > > XenServer: domain
> crash. XenServer restarts itself. It doesn't happen > for PV on
> XenServer VM > > VMware: An "unknown failure", which leads me to
> think that the nested > virtualization properties are not forwarded
> correctly to DomU
>
>
> To using nested, you need to add the set bit 31 of cupid.1.ecx to
> 0, add the following to your config file:
> cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
>
> For VMware ESX server, the attached patch is missing in current Xen
> which will caused the guest boot up fail. Also, I know some windows
> guests will boot fail on ESX.
> I never tried XenServer. But latest Xen on Xen is working well. So
> can you try the upsteam Xen to see whether it fails too?
>
> BTW, which L2 VM boot fail?
>
>
> > > > Just to clarify: > > Physical hardware: host >
> > Nested hypervisor: level 1 virtualization: XenServer or VMware ESXi
> > > Nested VM: level 2 virtualization: some generic VM inside
> XenServer or > ESXi (level 1) > > > This is the
> configuration used for XS: > name = "XenServer" > uuid =
> "a0b8c668-afaf-887a-1f4b-856dac4c8401" > maxmem = 4096 > memory =
> 4096 > vcpus = 2 > builder = "hvm" > boot = "c" > pae = 1
> > acpi = 1 > apic = 1 > hap = 0 > viridian = 0 >
> rtc_timeoffset = 0 > localtime = 0 > on_poweroff = "destroy" >
> on_reboot = "restart" > on_crash = "restart" > sdl = 0 > vnc =
> 1 > vncunused = 1 > keymap = "en-us" > disk = [
> "tap2:aio:/dev/share/VMS/xenserver01.lun,hda,w", > ",hdc:cdrom,r" ]
> vif = [ >
> "mac=00:16:3e:14:0d:72,bridge=lan,script=vif-bridge,model=e1000" ]
> parallel = "none" > serial = "pty" > nestedhvm = 1 > > nx=1
> > > > # A note - this problem appears with and without the 'nx'
> flag. > > A snippet from strace to qemu process: > >
> poll([{fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=18, >
> events=POLLIN|POLLERR|POLLHUP}, {fd=29, >
> events=POLLIN|POLLERR|POLLHUP}, {fd=21, >
> events=POLLIN|POLLERR|POLLHUP}, {fd=15, >
> events=POLLIN|POLLERR|POLLHUP}, {fd=5, events=
>
> > POLLIN|POLLERR|POLLHUP}, {fd=7, events=POLLIN}, {fd=11,
> POLLIN|POLLERR|events=POLLIN}, {fd=28, events=POLLIN}, {fd=6,
> POLLIN|POLLERR|events=POLLIN}], 10, 307) = ?
> > ERESTART_RESTARTBLOCK (To POLLIN|POLLERR|be restarted)
> --- SIGHUP
>
> > (Hangup) @ 0 (0) --- write(6, "\1\0\0\0\0\0\0\0", 8) = 8 >
> rt_sigreturn(0x2) = -1 EINTR (Interrupted system
> > call) gettimeofday({1386761621, 705266}, NULL) = 0 >
> clock_gettime(CLOCK_MONOTONIC, {1840, 693236049}) = 0 >
> clock_gettime(CLOCK_MONOTONIC, {1840, 693333096}) = 0 >
> gettimeofday({1386761621, 705546}, NULL) = 0 write(2, "qemu: terminating
> > on signal 1", 29) = 29 write(2, " from pid 6933\n", 15) = 15 >
> > > From the xl log for this domain: > Waiting for domain
> XenServer (domid 5) to die [pid 6933] Domain 5 has > shut down,
> reason code 3 0x3 Action for shutdown reason code 3 is > restart
> Domain > 5 needs to be cleaned up: destroying the domain Done.
> Rebooting now > Waiting for domain XenServer (domid 6) to die [pid
> 6933] > > > From 'xl dmesg' (XEN) <vm_launch_fail> error code 7
> (XEN) > domain_crash_sync called from vmcs.c:1293 (XEN) Domain 8
> (vcpu#0) > crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64
> debug=y Not > tainted ]---- (XEN) CPU: 4 (XEN) RIP:
> 0000:[<0000000000000000>] > (XEN) RFLAGS: 0000000000000002 CONTEXT:
> hvm guest (XEN) rax: > 0000000000000000 rbx: ffff830824f98000
> rcx: ffff830824f9ff80 (XEN) > rdx: ffff82d0801d0cf9 rsi:
> 0000000000000000 rdi: ffff82d0801de7fc > (XEN) rbp:
> ffff82d080105ab1 rsp: 0000000000000000 r8: > 0000000000000004
> (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70 > r11:
> 0000000000000000 (XEN) r12: ffff830824f9ff50 r13: >
> ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
> > cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3: >
> 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
> > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
> > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4
> (d9) > System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9)
> Relocating > guest memory for lowmem MMIO space disabled (XEN)
> irq.c:270: Dom9 PCI > link 0 changed 0 -> 5 (d9) PCI-ISA link 0
> routed to IRQ5 (XEN) > irq.c:270: Dom9 PCI link 1 changed 0 -> 10
> (d9) PCI-ISA link 1 routed > to IRQ10 (XEN) > irq.c:270: Dom9 PCI
> link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed > to > IRQ11
> (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA > link
> > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
> > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
> > setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar
> 10 > size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size
> 001000000: > 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000:
> 0f3000000 (d9) pci > dev 04:0 bar 10 size 000020000: 0f3040000 (d9)
> pci dev 03:0 bar 30 > size > 000010000: 0f3060000 (d9) pci dev
> 03:0 bar 14 size 000001000: > 0f3070000 > (d9) pci dev 02:0 bar 10
> size 000000100: 00000c001 (d9) pci dev 04:0 > bar > 14 size
> 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010: >
> 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
> > phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
> > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9)
> Testing > HVM environment: (d9) - REP INSB across page boundaries
> ... passed > (d9) > - GS base MSRs and SWAPGS ... passed (d9)
> Passed 2 of 2 tests (d9) > Writing SMBIOS tables ... (d9) Loading
> SeaBIOS ... (d9) Creating MP > tables ... (d9) Loading ACPI ... (d9)
> vm86 TSS at fc00a080 (d9) BIOS > map: (d9) 10000-100d3: Scratch
> space (d9) e0000-fffff: Main BIOS > (d9) > E820 table: (d9)
> [00]: 00000000:00000000 - 00000000:000a0000: RAM > (d9) > HOLE:
> 00000000:000a0000 - 00000000:000e0000 (d9) [01]: > 00000000:000e0000
> - 00000000:00100000: RESERVED (d9) [02]: > 00000000:00100000 -
> 00000000:f0000000: RAM (d9) HOLE: > 00000000:f0000000 > -
> 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
> > RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM
> (d9) > Invoking SeaBIOS ... (d9) SeaBIOS (version >
> rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
> > hypervisor signature at 40000000 (d9) xen: copy e820... (d9) >
> Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU >
> Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
> > Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
> > xen: copy BIOS tables... (d9) Copying SMBIOS entry point from >
> 0x00010010 to 0x000f1020 > (d9) Copying MPTABLE from
> 0xfc001170/fc001180 to 0x000f0f20 (d9) > Copying PIR from 0x00010030
> to 0x000f0ea0 (d9) Copying ACPI RSDP from > 0x000100b0 to 0x000f0e70
> (d9) Using pmtimer, ioport 0xb008, freq 3579 > kHz (d9) Scan for VGA
> option rom (d9) WARNING! Found unaligned PCI rom > (vd=1013:00b8)
> > (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
> > stdvga and caching modes (d9) Turning on vga text mode console (d9)
> > SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
> > (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
> > lpt ports > (d9) Found 1 serial ports (d9) ATA controller 1 at
> 1f0/3f4/c140 (irq > 14 dev 9) (d9) ATA controller 2 at 170/374/c148
> (irq 15 dev 9) (d9) ata0-0: > QEMU HARDDISK ATA-7 Hard-Disk (14336
> MiBytes) (d9) Searching bootorder > for:
> /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM >
> ATAPI-4 DVD/CD] (d9) Searching bootorder for: >
> /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
> > threads complete. (d9) Scan for option roms (d9) Running option rom
> at > c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm
> call > arg1=1 > (d9) pmm call arg1=0 (d9) Searching bootorder for:
> /pci@i0cf8/*@4 (d9) > (d9) Press F12 for boot menu. (d9) (d9)
> Searching bootorder for: HALT > (d9) drive 0x000f0e20:
> PCHS=16383/16/63 translation=lba > LCHS=1024/255/63 > s=29360128
> (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9) >
> Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
> > 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1:
> 000000000009fc00 - > 00000000000a0000 = 2 RESERVED (d9) 2:
> 00000000000f0000 - > 0000000000100000 = 2 RESERVED (d9) 3:
> 0000000000100000 - > 00000000effff000 = 1 RAM (d9) 4:
> 00000000effff000 - 00000000f0000000 = > 2 RESERVED (d9) 5:
> 00000000fc000000 - 0000000100000000 = 2 RESERVED > (d9) 6:
> 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter > handle_19:
> (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from >
> 0000:7c00 [root@monster ~]# xl dmesg (XEN) <vm_launch_fail> error code
> > 7 > (XEN) domain_crash_sync called from vmcs.c:1293 (XEN) Domain
> 8 > (vcpu#0) crashed on cpu#4: (XEN) ----[ Xen-4.4-unstable x86_64
> debug=y Not > tainted ]---- (XEN) CPU: 4 (XEN) RIP:
> 0000:[<0000000000000000>] > (XEN) RFLAGS: 0000000000000002 CONTEXT:
> hvm guest (XEN) rax: > 0000000000000000 rbx: ffff830824f98000
> rcx: ffff830824f9ff80 (XEN) > rdx: ffff82d0801d0cf9 rsi:
> 0000000000000000 rdi: ffff82d0801de7fc > (XEN) rbp:
> ffff82d080105ab1 rsp: 0000000000000000 r8: > 0000000000000004
> (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70 > r11:
> 0000000000000000 (XEN) r12: ffff830824f9ff50 r13: >
> ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
> > cr0: 0000000000000039 cr4: 0000000000002050 (XEN) cr3: >
> 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
> > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (XEN) event_channel.c:270:d0
> > EVTCHNOP failure: domain 9, error -22 (d9) HVM Loader (d9) Detected
> > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000, event channel 4
> (d9) > System requested SeaBIOS (d9) CPU speed is 3400 MHz (d9)
> Relocating > guest memory for lowmem MMIO space disabled (XEN)
> irq.c:270: Dom9 PCI > link 0 changed 0 -> 5 (d9) PCI-ISA link 0
> routed to IRQ5 (XEN) > irq.c:270: Dom9 PCI link 1 changed 0 -> 10
> (d9) PCI-ISA link 1 routed > to IRQ10 (XEN) > irq.c:270: Dom9 PCI
> link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed > to > IRQ11
> (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA > link
> > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10 (d9) pci dev 02:0
> > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9) RAM in high memory;
> > setting high_mem resource base to 10f800000 (d9) pci dev 03:0 bar
> 10 > size 002000000: 0f0000008 (d9) pci dev 02:0 bar 14 size
> 001000000: > 0f2000008 (d9) pci dev 04:0 bar 30 size 000040000:
> 0f3000000 (d9) pci > dev 04:0 bar 10 size 000020000: 0f3040000 (d9)
> pci dev 03:0 bar 30 > size > 000010000: 0f3060000 (d9) pci dev
> 03:0 bar 14 size 000001000: > 0f3070000 > (d9) pci dev 02:0 bar 10
> size 000000100: 00000c001 (d9) pci dev 04:0 > bar > 14 size
> 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010: >
> 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
> > phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9) - CPU1 ...
> > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (d9)
> Testing > HVM environment: (d9) - REP INSB across page boundaries
> ... passed > (d9) > - GS base MSRs and SWAPGS ... passed (d9)
> Passed 2 of 2 tests (d9) > Writing SMBIOS tables ... (d9) Loading
> SeaBIOS ... (d9) Creating MP > tables ... (d9) Loading ACPI ... (d9)
> vm86 TSS at fc00a080 (d9) BIOS > map: (d9) 10000-100d3: Scratch
> space (d9) e0000-fffff: Main BIOS > (d9) > E820 table: (d9)
> [00]: 00000000:00000000 - 00000000:000a0000: RAM > (d9) > HOLE:
> 00000000:000a0000 - 00000000:000e0000 (d9) [01]: > 00000000:000e0000
> - 00000000:00100000: RESERVED (d9) [02]: > 00000000:00100000 -
> 00000000:f0000000: RAM (d9) HOLE: > 00000000:f0000000 > -
> 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
> > RESERVED (d9) [04]: 00000001:00000000 - 00000001:0f800000: RAM
> (d9) > Invoking SeaBIOS ... (d9) SeaBIOS (version >
> rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
> > hypervisor signature at 40000000 (d9) xen: copy e820... (d9) >
> Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU >
> Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
> > Xen hypercall page at effff000 (d9) Detected Xen v4.4-unstable (d9)
> > xen: copy BIOS tables... (d9) Copying SMBIOS entry point from >
> 0x00010010 to 0x000f1020 > (d9) Copying MPTABLE from
> 0xfc001170/fc001180 to 0x000f0f20 (d9) > Copying PIR from 0x00010030
> to 0x000f0ea0 (d9) Copying ACPI RSDP from > 0x000100b0 to 0x000f0e70
> (d9) Using pmtimer, ioport 0xb008, freq 3579 > kHz (d9) Scan for VGA
> option rom (d9) WARNING! Found unaligned PCI rom > (vd=1013:00b8)
> > (d9) Running option rom at c000:0003 (XEN) stdvga.c:147:d9 entering
> > stdvga and caching modes (d9) Turning on vga text mode console (d9)
> > SeaBIOS (version rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
> > (d9) Machine UUID a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
> > lpt ports > (d9) Found 1 serial ports (d9) ATA controller 1 at
> 1f0/3f4/c140 (irq > 14 dev 9) (d9) ATA controller 2 at 170/374/c148
> (irq 15 dev 9) (d9) ata0-0: > QEMU HARDDISK ATA-7 Hard-Disk (14336
> MiBytes) (d9) Searching bootorder > for:
> /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM >
> ATAPI-4 DVD/CD] (d9) Searching bootorder for: >
> /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
> > threads complete. (d9) Scan for option roms (d9) Running option rom
> at > c900:0003 (d9) pmm call arg1=1 (d9) pmm call arg1=0 (d9) pmm
> call > arg1=1 > (d9) pmm call arg1=0 (d9) Searching bootorder for:
> /pci@i0cf8/*@4 (d9) > (d9) Press F12 for boot menu. (d9) (d9)
> Searching bootorder for: HALT > (d9) drive 0x000f0e20:
> PCHS=16383/16/63 translation=lba > LCHS=1024/255/63 > s=29360128
> (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9) >
> Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
> > 0000000000000000 - 000000000009fc00 = 1 RAM (d9) 1:
> 000000000009fc00 - > 00000000000a0000 = 2 RESERVED (d9) 2:
> 00000000000f0000 - > 0000000000100000 = 2 RESERVED (d9) 3:
> 0000000000100000 - > 00000000effff000 = 1 RAM (d9) 4:
> 00000000effff000 - 00000000f0000000 = > 2 RESERVED (d9) 5:
> 00000000fc000000 - 0000000100000000 = 2 RESERVED > (d9) 6:
> 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter > handle_19:
> (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from >
> 0000:7c00 > > > System description: > > 1. Centos 6.4
> x86_64, latest updates > > 2. Xen4centos packages - current kernel
> is > 3.10.23-11.el6.centos.alt.x86_64 > > 3. System with 32GB
> RAM > > 4. Virtual disks are ZFS volumes (aka - block devices).
> Same problem > appeared when on local ext3 disks > > 5. Using
> OpenVSwitch version 2.0, compiled to RPM, again. > > 6. Intel CPU:
> > processor : 0 > vendor_id : GenuineIntel > cpu family
> : 6 > model : 62 > model name : Intel(R) Core(TM)
> i7-4930K CPU @ 3.40GHz > stepping : 4 > microcode : 0x416
> > cpu MHz : 3400.158 > cache size : 12288 KB >
> physical id : 0 > siblings : 12 > core id : 0 >
> cpu cores : 6 > apicid : 0 > initial apicid : 0 >
> fpu : yes > fpu_exception : yes > cpuid level : 13
> > wp : yes > flags : fpu de tsc msr pae mce cx8 apic
> sep mca cmov pat clflush acpi > mmx fxsr sse sse2 ss ht syscall nx lm
> constant_tsc rep_good nopl > nonstop_tsc eagerfpu pni pclmulqdq
> monitor est ssse3 cx16 sse4_1 > sse4_2 popcnt tsc_deadline_timer aes
> xsave avx f16c rdrand hypervisor > lahf_lm ida arat epb xsaveopt pln
> pts dtherm fsgsbase erms > bogomips : 6800.31 > clflush size
> : 64 > cache_alignment : 64 > address sizes : 46 bits
> physical, 48 bits virtual > power management: > > > It was
> suggested by Mr. Ian Campbell to CC both Mr. Jun Nakajima and Mr. >
> Eddie Dong, so, my apologies if this action is improper. > > >
> Any tip or advice would be great. Any request for further info would
> > be also great! > > Thanks! > > Etzion Bar-Noy
>
>
>
> Best regards,
> Yang
>
>
>
>
>
Best regards,
Yang
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-20 0:50 ` Zhang, Yang Z
@ 2013-12-20 11:24 ` Etzion Bar-Noy
2013-12-23 1:09 ` Zhang, Yang Z
0 siblings, 1 reply; 10+ messages in thread
From: Etzion Bar-Noy @ 2013-12-20 11:24 UTC (permalink / raw)
To: Yang Z Zhang; +Cc: Eddie Dong, Jun Nakajima, xen-devel@lists.xen.org
[-- Attachment #1.1: Type: text/plain, Size: 34761 bytes --]
Hi.
It probably seems trivial to you, however, I have no idea (and didn't find
much documentation about) how to set this this parameter in the config file.
Etzion
On Dec 20, 2013 2:50 AM, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote:
> Etzion Bar-Noy wrote on 2013-12-20:
> > Sorry for spamming. Actually - it is a bit different - two results
> > which appear (on XenServer and on VMware). The previous messages were
> > for Vmware. The one below is for XenServer:
>
> I guess you are using shadow mode for L1 VMM. With shadow L1, this is a
> known issue to me. Can you try to use L1 EPT to see whether it is
> reproducible?
>
> >
> > xl dmesg:
> >
> > (XEN) stdvga.c:151:d10 leaving stdvga (XEN) common.c:3525:d0 tracking
> > VRAM f0000 - f0800 (XEN) stdvga.c:147:d10 entering stdvga and caching
> > modes (XEN) realmode.c:115:d10 Failed to emulate insn. (XEN)
> > realmode.c:167:d10 Real-mode emulation failed @ f000:0000fff0: 00 90 23
> > 00 c0 82 (XEN) domain_crash called from realmode.c:168 (XEN) Domain 10
> > (vcpu#0) crashed on cpu#8: (XEN) ----[ Xen-4.4-unstable x86_64 debug=y
> > Not tainted ]---- (XEN) CPU: 8 (XEN) RIP:
> > f000:[<000000000000fff0>] (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm
> > guest (XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx:
> > 0000000000000000 (XEN) rdx: 00000000078bfbff rsi: 0000000000000000
> > rdi: 0000000000000000 (XEN) rbp: 0000000000000000 rsp:
> > 0000000000000000 r8: 0000000000000000 (XEN) r9: 0000000000000000
> > r10: 0000000000000000 r11: 0000000000000000 (XEN) r12:
> > 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN)
> > r15: 0000000000000000 cr0: 0000000000000030 cr4: 0000000000002040
> > (XEN) cr3: 0000000000000000 cr2: 0000000000000000 (XEN) ds: 0000 es:
> > 0000 fs: 0000 gs: 0000 ss: 0000 cs: f000 (XEN) <vm_launch_fail>
> > error code 7 (XEN) domain_crash_sync called from vmcs.c:1293 (XEN)
> > Domain 9 (vcpu#0) crashed on cpu#5: (XEN) ----[ Xen-4.4-unstable x86_64
> > debug=y Not tainted ]---- (XEN) CPU: 5 (XEN) RIP:
> > 0000:[<0000000000000000>] (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm
> > guest (XEN) rax: 0000000000000000 rbx: ffff830824f90000 rcx:
> > ffff830824f97f80 (XEN) rdx: ffff82d0801d0cb9 rsi: 0000000000000000
> > rdi: ffff82d0801de7ec (XEN) rbp: ffff82d080105ab1 rsp:
> > 0000000000000000 r8: 0000000000000004 (XEN) r9: ffff82d080105b20
> > r10: ffff830824f97f70 r11: 0000000000000000 (XEN) r12:
> > ffff830824f97f50 r13: ffff83080191c000 r14: 0000000000000004 (XEN)
> > r15: ffff82d080187e15 cr0: 0000000000000039 cr4: 0000000000002050
> > (XEN) cr3: 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es:
> > 0000 fs: 0000 gs: 0000 ss: 0000 cs: 0000 (XEN)
> > grant_table.c:289:d0 Increased maptrack size to 2 frames (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 11, error -22
> >
> >
> > Thanks
> > Etzion
> >
> >
> >
> > On Thu, Dec 19, 2013 at 9:40 PM, Etzion Bar-Noy <etzion@barnoy.co.il>
> wrote:
> >
> >
> > Hi.
> >
> > With the added flag suggested by you, the VM (nested hypervisor)
> crashed.
> > Now ESXi will attempt to start the VM and crash as well (unlike
> > before, where it just failed to start it with 'unknown error').
> >
> > XenServer 6.2 still crashes, just the same.
> >
> >
> > I am attempting to deploy a Centos6 machine (and clone it) so that
> I
> > will be able to test both KVM and Xen community (latest stable RPM
> > release for Centos6).
> >
> >
> > xl dmesg looks much like before:
> >
> > (XEN) stdvga.c:151:d12 leaving stdvga
> > (XEN) common.c:3525:d0 tracking VRAM f0000 - f0800
> > (XEN) stdvga.c:147:d12 entering stdvga and caching modes
> > (XEN) realmode.c:115:d12 Failed to emulate insn.
> > (XEN) realmode.c:167:d12 Real-mode emulation failed @
> f000:0000fff0:
> > 00 10 23 00 c0 82
> > (XEN) domain_crash called from realmode.c:168
> > (XEN) Domain 12 (vcpu#0) crashed on cpu#4:
> >
> > (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
> > (XEN) CPU: 4
> >
> > (XEN) RIP: f000:[<000000000000fff0>]
> >
> > (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest
> >
> > (XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx:
> > 0000000000000000 (XEN) rdx: 00000000078bfbff rsi: 0000000000000000
> > rdi: 0000000000000000 (XEN) rbp: 0000000000000000 rsp:
> > 0000000000000000 r8: 0000000000000000 (XEN) r9: 0000000000000000
> > r10: 0000000000000000 r11: 0000000000000000 (XEN) r12:
> > 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> (XEN)
> > r15: 0000000000000000 cr0: 0000000000000030 cr4: 0000000000002040
> > (XEN) cr3: 0000000000000000 cr2: 0000000000000000 (XEN) ds:
> 0000
> > es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: f000 (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> (XEN)
> > event_channel.c:270:d0 EVTCHNOP failure: domain 14, error -22
> >
> >
> > Thanks!
> > Etzion
> >
> >
> >
> > On Wed, Dec 18, 2013 at 11:46 AM, Etzion Bar-Noy <
> etzion@barnoy.co.il>
> > wrote:
> >
> >
> > Hi Mr. Yang.
> >
> > I did attempt to use your suggested configuration with the
> XenServer
> > VM, but to no avail. The L2 VM keeps on crushing. I defined it now to
> > leave a dump file when crushing, and if you desire so, I will find
> > some method of uploading it for your viewing pleasure. As I am not a
> > programmer, but an integrator, my abilities in this direction are
> limited at best.
> >
> > The L3 VM type is 'generic VM', because I wanted to
> enforce the
> > system to use HVM emulation.
> >
> >
> > I have just applied the patch you supplied me to my
> 'staging' git
> > checkout, and just finished compiling it. I will soon apply the newly
> > created RPMs on my test (and hopefully - to be production soon)
> > server, and will check VMware.
> >
> >
> > Thanks for the input. I will try and then respond with
> results.
> >
> > Etzion
> >
> >
> >
> > On Wed, Dec 18, 2013 at 3:33 AM, Zhang, Yang Z
> > <yang.z.zhang@intel.com> wrote:
> >
> >
> > Etzion Bar-Noy wrote on 2013-12-16:
> > > Hi.
> > >
> >
> > Hi Etzion Bar-Noy
> >
> > It's glad to know someone is using Nested Xen and
> report bugs to
> > us. Also, if there are more nested bugs you found, please CC me too.
> >
> >
> > > > I am using the unstable
> build-from-source of Xen community. It
> > reports > as 4.4, and was pulled from Git few days
> ago. The problem I
> > will > describe below appears on version 4.3.1,
> compiled from source
> > as well. > In both cases - I used spec file to
> create my own RPMs,
> > because it's the right method of doing it. >
> > > Problem: When
> > VM running under HVM which is a hypervisor (XenServer
> > 6.2 or VMware
> > ESXi5.5) attempts to start a (HVM) virtual machine, the
> > following
> > happens, 100% reproducible on this server. >
> > XenServer: domain
> > crash. XenServer restarts itself. It doesn't happen >
> for PV on
> > XenServer VM > > VMware: An
> "unknown failure", which leads me to
> > think that the nested > virtualization
> properties are not forwarded
> > correctly to DomU
> >
> >
> > To using nested, you need to add the set bit 31 of
> cupid.1.ecx to
> > 0, add the following to your config file:
> > cpuid =
> ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
> >
> > For VMware ESX server, the attached patch is
> missing in current Xen
> > which will caused the guest boot up fail. Also, I know some windows
> > guests will boot fail on ESX.
> > I never tried XenServer. But latest Xen on Xen is
> working well. So
> > can you try the upsteam Xen to see whether it fails too?
> >
> > BTW, which L2 VM boot fail?
> >
> >
> > > > >
> Just to clarify: > > Physical
> hardware: host >
> > > Nested hypervisor: level 1 virtualization:
> XenServer or VMware ESXi
> > > > Nested VM: level 2
> virtualization: some generic VM inside
> > XenServer or > ESXi (level 1) >
> > > This is the
> > configuration used for XS: > name = "XenServer"
> > uuid =
> > "a0b8c668-afaf-887a-1f4b-856dac4c8401" > maxmem =
> 4096 > memory =
> > 4096 > vcpus = 2 > builder = "hvm"
> > boot = "c" > pae = 1
> > > acpi = 1 > apic = 1
> > hap = 0 > viridian = 0
> >
> > rtc_timeoffset = 0 > localtime = 0
> > on_poweroff = "destroy" >
> > on_reboot = "restart" > on_crash = "restart"
> > sdl = 0 > vnc =
> > 1 > vncunused = 1 > keymap =
> "en-us" > disk = [
> > "tap2:aio:/dev/share/VMS/xenserver01.lun,hda,w", >
> ",hdc:cdrom,r" ]
> > vif = [ >
> > "mac=00:16:3e:14:0d:72,bridge=lan,script=vif-bridge,model=e1000" ]
> > parallel = "none" > serial = "pty"
> > nestedhvm = 1 > > nx=1
> > > > >
> # A note - this problem appears with and without the 'nx'
> > flag. > > A snippet from
> strace to qemu process: > >
> > poll([{fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=18,
> >
> > events=POLLIN|POLLERR|POLLHUP}, {fd=29, >
> > events=POLLIN|POLLERR|POLLHUP}, {fd=21, >
> > events=POLLIN|POLLERR|POLLHUP}, {fd=15, >
> > events=POLLIN|POLLERR|POLLHUP}, {fd=5, events=
> >
> > > POLLIN|POLLERR|POLLHUP}, {fd=7, events=POLLIN},
> {fd=11,
> > POLLIN|POLLERR|events=POLLIN}, {fd=28, events=POLLIN}, {fd=6,
> > POLLIN|POLLERR|events=POLLIN}], 10, 307) = ?
> > > ERESTART_RESTARTBLOCK (To POLLIN|POLLERR|be
> restarted)
> > --- SIGHUP
> >
> > > (Hangup) @ 0 (0) --- write(6,
> "\1\0\0\0\0\0\0\0", 8) = 8 >
> > rt_sigreturn(0x2) = -1 EINTR (Interrupted system
> > > call) gettimeofday({1386761621, 705266}, NULL) =
> 0 >
> > clock_gettime(CLOCK_MONOTONIC, {1840, 693236049}) = 0
> >
> > clock_gettime(CLOCK_MONOTONIC, {1840, 693333096}) = 0
> >
> > gettimeofday({1386761621, 705546}, NULL) = 0 write(2, "qemu: terminating
> > > on signal 1", 29) = 29 write(2, " from pid
> 6933\n", 15) = 15 >
> > > > From the xl log for this
> domain: > Waiting for domain
> > XenServer (domid 5) to die [pid 6933] Domain 5 has >
> shut down,
> > reason code 3 0x3 Action for shutdown reason code 3 is
> > restart
> > Domain > 5 needs to be cleaned up: destroying the
> domain Done.
> > Rebooting now > Waiting for domain XenServer
> (domid 6) to die [pid
> > 6933] > >
> > From 'xl dmesg' (XEN) <vm_launch_fail> error code 7
> > (XEN) > domain_crash_sync called from
> vmcs.c:1293 (XEN) Domain 8
> > (vcpu#0) > crashed on cpu#4: (XEN) ----[
> Xen-4.4-unstable x86_64
> > debug=y Not > tainted ]---- (XEN) CPU: 4 (XEN) RIP:
> > 0000:[<0000000000000000>] > (XEN) RFLAGS:
> 0000000000000002 CONTEXT:
> > hvm guest (XEN) rax: > 0000000000000000 rbx:
> ffff830824f98000
> > rcx: ffff830824f9ff80 (XEN) > rdx: ffff82d0801d0cf9
> rsi:
> > 0000000000000000 rdi: ffff82d0801de7fc > (XEN)
> rbp:
> > ffff82d080105ab1 rsp: 0000000000000000 r8: >
> 0000000000000004
> > (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70 >
> r11:
> > 0000000000000000 (XEN) r12: ffff830824f9ff50 r13: >
> > ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
> > > cr0: 0000000000000039 cr4: 0000000000002050
> (XEN) cr3: >
> > 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
> > > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (d9) HVM
> Loader (d9) Detected
> > > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000,
> event channel 4
> > (d9) > System requested SeaBIOS (d9) CPU speed is 3400
> MHz (d9)
> > Relocating > guest memory for lowmem MMIO space
> disabled (XEN)
> > irq.c:270: Dom9 PCI > link 0 changed 0 -> 5 (d9)
> PCI-ISA link 0
> > routed to IRQ5 (XEN) > irq.c:270: Dom9 PCI link 1
> changed 0 -> 10
> > (d9) PCI-ISA link 1 routed > to IRQ10 (XEN)
> > irq.c:270: Dom9 PCI
> > link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed >
> to > IRQ11
> > (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
> > link
> > > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10
> (d9) pci dev 02:0
> > > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9)
> RAM in high memory;
> > > setting high_mem resource base to 10f800000 (d9)
> pci dev 03:0 bar
> > 10 > size 002000000: 0f0000008 (d9) pci dev 02:0 bar
> 14 size
> > 001000000: > 0f2000008 (d9) pci dev 04:0 bar 30 size
> 000040000:
> > 0f3000000 (d9) pci > dev 04:0 bar 10 size 000020000:
> 0f3040000 (d9)
> > pci dev 03:0 bar 30 > size >
> 000010000: 0f3060000 (d9) pci dev
> > 03:0 bar 14 size 000001000: > 0f3070000
> > (d9) pci dev 02:0 bar 10
> > size 000000100: 00000c001 (d9) pci dev 04:0 > bar
> > 14 size
> > 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
> >
> > 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
> > > phys ... fixed MTRRs ... var MTRRs [2/8] ...
> done. (d9) - CPU1 ...
> > > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8]
> ... done. (d9)
> > Testing > HVM environment: (d9) - REP INSB across
> page boundaries
> > ... passed > (d9) > - GS base MSRs
> and SWAPGS ... passed (d9)
> > Passed 2 of 2 tests (d9) > Writing SMBIOS tables
> ... (d9) Loading
> > SeaBIOS ... (d9) Creating MP > tables ... (d9) Loading
> ACPI ... (d9)
> > vm86 TSS at fc00a080 (d9) BIOS > map: (d9)
> 10000-100d3: Scratch
> > space (d9) e0000-fffff: Main BIOS > (d9)
> > E820 table: (d9)
> > [00]: 00000000:00000000 - 00000000:000a0000: RAM >
> (d9) > HOLE:
> > 00000000:000a0000 - 00000000:000e0000 (d9) [01]: >
> 00000000:000e0000
> > - 00000000:00100000: RESERVED (d9) [02]: >
> 00000000:00100000 -
> > 00000000:f0000000: RAM (d9) HOLE: >
> 00000000:f0000000 > -
> > 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
> > > RESERVED (d9) [04]: 00000001:00000000 -
> 00000001:0f800000: RAM
> > (d9) > Invoking SeaBIOS ... (d9) SeaBIOS (version
> >
> > rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
> > > hypervisor signature at 40000000 (d9) xen: copy
> e820... (d9) >
> > Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
> >
> > Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
> > > Xen hypercall page at effff000 (d9) Detected Xen
> v4.4-unstable (d9)
> > > xen: copy BIOS tables... (d9) Copying SMBIOS
> entry point from >
> > 0x00010010 to 0x000f1020 > (d9) Copying MPTABLE from
> > 0xfc001170/fc001180 to 0x000f0f20 (d9) > Copying
> PIR from 0x00010030
> > to 0x000f0ea0 (d9) Copying ACPI RSDP from >
> 0x000100b0 to 0x000f0e70
> > (d9) Using pmtimer, ioport 0xb008, freq 3579 > kHz (d9)
> Scan for VGA
> > option rom (d9) WARNING! Found unaligned PCI rom >
> (vd=1013:00b8)
> > > (d9) Running option rom at c000:0003 (XEN)
> stdvga.c:147:d9 entering
> > > stdvga and caching modes (d9) Turning on vga
> text mode console (d9)
> > > SeaBIOS (version
> rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
> > > (d9) Machine UUID
> a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
> > > lpt ports > (d9) Found 1
> serial ports (d9) ATA controller 1 at
> > 1f0/3f4/c140 (irq > 14 dev 9) (d9) ATA controller 2
> at 170/374/c148
> > (irq 15 dev 9) (d9) ata0-0: > QEMU HARDDISK ATA-7
> Hard-Disk (14336
> > MiBytes) (d9) Searching bootorder > for:
> > /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
> >
> > ATAPI-4 DVD/CD] (d9) Searching bootorder for: >
> > /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
> > > threads complete. (d9) Scan for option roms (d9)
> Running option rom
> > at > c900:0003 (d9) pmm call arg1=1 (d9) pmm call
> arg1=0 (d9) pmm
> > call > arg1=1 > (d9) pmm call
> arg1=0 (d9) Searching bootorder for:
> > /pci@i0cf8/*@4 (d9) > (d9) Press F12 for boot menu.
> (d9) (d9)
> > Searching bootorder for: HALT > (d9) drive
> 0x000f0e20:
> > PCHS=16383/16/63 translation=lba > LCHS=1024/255/63
> > s=29360128
> > (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
> >
> > Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
> > > 0000000000000000 - 000000000009fc00 = 1 RAM (d9)
> 1:
> > 000000000009fc00 - > 00000000000a0000 = 2 RESERVED
> (d9) 2:
> > 00000000000f0000 - > 0000000000100000 = 2 RESERVED
> (d9) 3:
> > 0000000000100000 - > 00000000effff000 = 1 RAM (d9)
> 4:
> > 00000000effff000 - 00000000f0000000 = > 2
> RESERVED (d9) 5:
> > 00000000fc000000 - 0000000100000000 = 2 RESERVED >
> (d9) 6:
> > 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
> > handle_19:
> > (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
> >
> > 0000:7c00 [root@monster ~]# xl dmesg (XEN) <vm_launch_fail> error code
> > > 7 > (XEN) domain_crash_sync
> called from vmcs.c:1293 (XEN) Domain
> > 8 > (vcpu#0) crashed on cpu#4: (XEN) ----[
> Xen-4.4-unstable x86_64
> > debug=y Not > tainted ]---- (XEN) CPU: 4 (XEN) RIP:
> > 0000:[<0000000000000000>] > (XEN) RFLAGS:
> 0000000000000002 CONTEXT:
> > hvm guest (XEN) rax: > 0000000000000000 rbx:
> ffff830824f98000
> > rcx: ffff830824f9ff80 (XEN) > rdx: ffff82d0801d0cf9
> rsi:
> > 0000000000000000 rdi: ffff82d0801de7fc > (XEN)
> rbp:
> > ffff82d080105ab1 rsp: 0000000000000000 r8: >
> 0000000000000004
> > (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70 >
> r11:
> > 0000000000000000 (XEN) r12: ffff830824f9ff50 r13: >
> > ffff830654f9e000 r14: ffff830824f9ff30 (XEN) r15: ffff82d080187e43
> > > cr0: 0000000000000039 cr4: 0000000000002050
> (XEN) cr3: >
> > 00000000feffa000 cr2: 0000000000000000 (XEN) ds: 0000 es: 0000 fs:
> > > 0000 gs: 0000 ss: 0000 cs: 0000 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (XEN)
> event_channel.c:270:d0
> > > EVTCHNOP failure: domain 9, error -22 (d9) HVM
> Loader (d9) Detected
> > > Xen v4.4-unstable (d9) Xenbus rings @0xfeffc000,
> event channel 4
> > (d9) > System requested SeaBIOS (d9) CPU speed is 3400
> MHz (d9)
> > Relocating > guest memory for lowmem MMIO space
> disabled (XEN)
> > irq.c:270: Dom9 PCI > link 0 changed 0 -> 5 (d9)
> PCI-ISA link 0
> > routed to IRQ5 (XEN) > irq.c:270: Dom9 PCI link 1
> changed 0 -> 10
> > (d9) PCI-ISA link 1 routed > to IRQ10 (XEN)
> > irq.c:270: Dom9 PCI
> > link 2 changed 0 -> 11 (d9) PCI-ISA link 2 routed >
> to > IRQ11
> > (XEN) irq.c:270: Dom9 PCI link 3 changed 0 -> 5 (d9) PCI-ISA
> > link
> > > 3 routed to IRQ5 (d9) pci dev 01:3 INTA->IRQ10
> (d9) pci dev 02:0
> > > INTA->IRQ11 (d9) pci dev 04:0 INTA->IRQ5 (d9)
> RAM in high memory;
> > > setting high_mem resource base to 10f800000 (d9)
> pci dev 03:0 bar
> > 10 > size 002000000: 0f0000008 (d9) pci dev 02:0 bar
> 14 size
> > 001000000: > 0f2000008 (d9) pci dev 04:0 bar 30 size
> 000040000:
> > 0f3000000 (d9) pci > dev 04:0 bar 10 size 000020000:
> 0f3040000 (d9)
> > pci dev 03:0 bar 30 > size >
> 000010000: 0f3060000 (d9) pci dev
> > 03:0 bar 14 size 000001000: > 0f3070000
> > (d9) pci dev 02:0 bar 10
> > size 000000100: 00000c001 (d9) pci dev 04:0 > bar
> > 14 size
> > 000000040: 00000c101 (d9) pci dev 01:1 bar 20 size 000000010:
> >
> > 00000c141 (d9) Multiprocessor initialisation: (d9) - CPU0 ... 46-bit
> > > phys ... fixed MTRRs ... var MTRRs [2/8] ...
> done. (d9) - CPU1 ...
> > > 46-bit phys ... fixed MTRRs ... var MTRRs [2/8]
> ... done. (d9)
> > Testing > HVM environment: (d9) - REP INSB across
> page boundaries
> > ... passed > (d9) > - GS base MSRs
> and SWAPGS ... passed (d9)
> > Passed 2 of 2 tests (d9) > Writing SMBIOS tables
> ... (d9) Loading
> > SeaBIOS ... (d9) Creating MP > tables ... (d9) Loading
> ACPI ... (d9)
> > vm86 TSS at fc00a080 (d9) BIOS > map: (d9)
> 10000-100d3: Scratch
> > space (d9) e0000-fffff: Main BIOS > (d9)
> > E820 table: (d9)
> > [00]: 00000000:00000000 - 00000000:000a0000: RAM >
> (d9) > HOLE:
> > 00000000:000a0000 - 00000000:000e0000 (d9) [01]: >
> 00000000:000e0000
> > - 00000000:00100000: RESERVED (d9) [02]: >
> 00000000:00100000 -
> > 00000000:f0000000: RAM (d9) HOLE: >
> 00000000:f0000000 > -
> > 00000000:fc000000 (d9) [03]: 00000000:fc000000 - 00000001:00000000:
> > > RESERVED (d9) [04]: 00000001:00000000 -
> 00000001:0f800000: RAM
> > (d9) > Invoking SeaBIOS ... (d9) SeaBIOS (version
> >
> > rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm) (d9) (d9) Found Xen
> > > hypervisor signature at 40000000 (d9) xen: copy
> e820... (d9) >
> > Relocating init from 0x000e2e88 to 0xeffe1d80 (size 57784) (d9) CPU
> >
> > Mhz=3400 (d9) Found 7 PCI devices (max PCI bus is 00) (d9) Allocated
> > > Xen hypercall page at effff000 (d9) Detected Xen
> v4.4-unstable (d9)
> > > xen: copy BIOS tables... (d9) Copying SMBIOS
> entry point from >
> > 0x00010010 to 0x000f1020 > (d9) Copying MPTABLE from
> > 0xfc001170/fc001180 to 0x000f0f20 (d9) > Copying
> PIR from 0x00010030
> > to 0x000f0ea0 (d9) Copying ACPI RSDP from >
> 0x000100b0 to 0x000f0e70
> > (d9) Using pmtimer, ioport 0xb008, freq 3579 > kHz (d9)
> Scan for VGA
> > option rom (d9) WARNING! Found unaligned PCI rom >
> (vd=1013:00b8)
> > > (d9) Running option rom at c000:0003 (XEN)
> stdvga.c:147:d9 entering
> > > stdvga and caching modes (d9) Turning on vga
> text mode console (d9)
> > > SeaBIOS (version
> rel-1.7.3.1-0-g7d9cbe6-20131211_100730-compilevm)
> > > (d9) Machine UUID
> a0b8c668-afaf-887a-1f4b-856dac4c8401 (d9) Found 0
> > > lpt ports > (d9) Found 1
> serial ports (d9) ATA controller 1 at
> > 1f0/3f4/c140 (irq > 14 dev 9) (d9) ATA controller 2
> at 170/374/c148
> > (irq 15 dev 9) (d9) ata0-0: > QEMU HARDDISK ATA-7
> Hard-Disk (14336
> > MiBytes) (d9) Searching bootorder > for:
> > /pci@i0cf8/*@1,1/drive@0/disk@0 (d9) DVD/CD [ata1-0: QEMU DVD-ROM
> >
> > ATAPI-4 DVD/CD] (d9) Searching bootorder for: >
> > /pci@i0cf8/*@1,1/drive@1/disk@0 (d9) PS2 keyboard initialized (d9) All
> > > threads complete. (d9) Scan for option roms (d9)
> Running option rom
> > at > c900:0003 (d9) pmm call arg1=1 (d9) pmm call
> arg1=0 (d9) pmm
> > call > arg1=1 > (d9) pmm call
> arg1=0 (d9) Searching bootorder for:
> > /pci@i0cf8/*@4 (d9) > (d9) Press F12 for boot menu.
> (d9) (d9)
> > Searching bootorder for: HALT > (d9) drive
> 0x000f0e20:
> > PCHS=16383/16/63 translation=lba > LCHS=1024/255/63
> > s=29360128
> > (d9) Space available for UMB: ca000-ee800, f0000-f0dc0 (d9)
> >
> > Returned 61440 bytes of ZoneHigh (d9) e820 map has 7 items: (d9) 0:
> > > 0000000000000000 - 000000000009fc00 = 1 RAM (d9)
> 1:
> > 000000000009fc00 - > 00000000000a0000 = 2 RESERVED
> (d9) 2:
> > 00000000000f0000 - > 0000000000100000 = 2 RESERVED
> (d9) 3:
> > 0000000000100000 - > 00000000effff000 = 1 RAM (d9)
> 4:
> > 00000000effff000 - 00000000f0000000 = > 2
> RESERVED (d9) 5:
> > 00000000fc000000 - 0000000100000000 = 2 RESERVED >
> (d9) 6:
> > 0000000100000000 - 000000010f800000 = 1 RAM (d9) enter
> > handle_19:
> > (d9) NULL (d9) Booting from Hard Disk... (d9) Booting from
> >
> > 0000:7c00 > >
> > System description: > > 1.
> Centos 6.4
> > x86_64, latest updates > >
> 2. Xen4centos packages - current kernel
> > is > 3.10.23-11.el6.centos.alt.x86_64
> > > 3. System with 32GB
> > RAM > > 4. Virtual disks are ZFS
> volumes (aka - block devices).
> > Same problem > appeared when on local ext3 disks
> > > 5. Using
> > OpenVSwitch version 2.0, compiled to RPM, again. >
> > 6. Intel CPU:
> > > processor : 0 >
> vendor_id : GenuineIntel > cpu family
> > : 6 > model : 62 > model
> name : Intel(R) Core(TM)
> > i7-4930K CPU @ 3.40GHz > stepping : 4
> > microcode : 0x416
> > > cpu MHz : 3400.158 >
> cache size : 12288 KB >
> > physical id : 0 > siblings : 12
> > core id : 0 >
> > cpu cores : 6 > apicid : 0
> > initial apicid : 0 >
> > fpu : yes > fpu_exception : yes
> > cpuid level : 13
> > > wp : yes > flags
> : fpu de tsc msr pae mce cx8 apic
> > sep mca cmov pat clflush acpi > mmx fxsr sse
> sse2 ss ht syscall nx lm
> > constant_tsc rep_good nopl > nonstop_tsc eagerfpu pni
> pclmulqdq
> > monitor est ssse3 cx16 sse4_1 > sse4_2 popcnt
> tsc_deadline_timer aes
> > xsave avx f16c rdrand hypervisor > lahf_lm ida arat
> epb xsaveopt pln
> > pts dtherm fsgsbase erms > bogomips : 6800.31
> > clflush size
> > : 64 > cache_alignment : 64 >
> address sizes : 46 bits
> > physical, 48 bits virtual > power management:
> > > > It was
> > suggested by Mr. Ian Campbell to CC both Mr. Jun Nakajima and Mr.
> >
> > Eddie Dong, so, my apologies if this action is improper.
> > > >
> > Any tip or advice would be great. Any request for further info would
> > > be also great! >
> > Thanks! > >
> Etzion Bar-Noy
> >
> >
> >
> > Best regards,
> > Yang
> >
> >
> >
> >
> >
>
>
> Best regards,
> Yang
>
>
>
[-- Attachment #1.2: Type: text/html, Size: 39526 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-20 11:24 ` Etzion Bar-Noy
@ 2013-12-23 1:09 ` Zhang, Yang Z
2013-12-23 14:17 ` Etzion Bar-Noy
0 siblings, 1 reply; 10+ messages in thread
From: Zhang, Yang Z @ 2013-12-23 1:09 UTC (permalink / raw)
To: Etzion Bar-Noy; +Cc: Dong, Eddie, Nakajima, Jun, xen-devel@lists.xen.org
Etzion Bar-Noy wrote on 2013-12-20:
> Hi.
> It probably seems trivial to you, however, I have no idea (and didn't
> find much documentation about) how to set this this parameter in the config file.
>
What's your CPU model? If you didn't set anything in config file, then I guess your processor doesn't support EPT. Because EPT is used by default.
> Etzion
>
> On Dec 20, 2013 2:50 AM, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote:
>
>
> Etzion Bar-Noy wrote on 2013-12-20: > Sorry for spamming. Actually -
> it is a bit different - two results > which appear (on XenServer and on
> VMware). The previous messages were > for Vmware. The one below is for
> XenServer:
>
> I guess you are using shadow mode for L1 VMM. With shadow L1, this is
> a known issue to me. Can you try to use L1 EPT to see whether it is reproducible?
>
>
> Best regards,
> Yang
>
>
>
Best regards,
Yang
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Nested virtualization instabilities
2013-12-23 1:09 ` Zhang, Yang Z
@ 2013-12-23 14:17 ` Etzion Bar-Noy
0 siblings, 0 replies; 10+ messages in thread
From: Etzion Bar-Noy @ 2013-12-23 14:17 UTC (permalink / raw)
To: Zhang, Yang Z; +Cc: Dong, Eddie, Nakajima, Jun, xen-devel@lists.xen.org
[-- Attachment #1.1: Type: text/plain, Size: 2575 bytes --]
Hi.
CPU model: Intel(R) Core(TM) i7-4930K CPU @ 3.40GHz
Flags: fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx
fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc
eagerfpu pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2 popcnt
tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm ida arat
epb xsaveopt pln pts dtherm fsgsbase erms
Following your advice I googled a bit, and was able to find a *working*
configuration for my system. I added 'hap=1' which solved the problem, so
far. I was able to boot a Linux VM under ESXi 5.5, and was able to boot a
Linux VM under HVM under XenServer without crashes. I add my VM
configuration for future reference by whoever needs it.
name = "VMware"
uuid = "a0b8ce28-afaf-887a-1f4b-856dac4c8401"
maxmem = 4096
memory = 4096
vcpus = 1
builder = "hvm"
boot = "c"
pae = 1
acpi = 1
apic = 1
hap = 0
viridian = 0
rtc_timeoffset = 0
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "destroy"
sdl = 0
vnc = 1
vncunused = 1
keymap = "en-us"
disk = [ "tap2:aio:/dev/share/VMS/vmware.lun,hda,w",
"file:/share/ISO/VMware-VMvisor-Installer-5.5.0-1331820.x86_64.iso,hdc:cdrom,r"
]
vif = [ "mac=00:16:3e:12:1d:72,bridge=lan,script=vif-bridge,model=e1000" ]
parallel = "none"
serial = "pty"
nestedhvm = 1
nx = 1
hap = 1
cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']
I am currently only with a single VCPU, and would like to add additional
VCPUs. I hope it will behave nicely then.
Thanks!
Etzion
On Mon, Dec 23, 2013 at 3:09 AM, Zhang, Yang Z <yang.z.zhang@intel.com>wrote:
> Etzion Bar-Noy wrote on 2013-12-20:
> > Hi.
> > It probably seems trivial to you, however, I have no idea (and didn't
> > find much documentation about) how to set this this parameter in the
> config file.
> >
>
> What's your CPU model? If you didn't set anything in config file, then I
> guess your processor doesn't support EPT. Because EPT is used by default.
>
> > Etzion
> >
> > On Dec 20, 2013 2:50 AM, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote:
> >
> >
> > Etzion Bar-Noy wrote on 2013-12-20: > Sorry for spamming.
> Actually -
> > it is a bit different - two results > which appear (on XenServer and on
> > VMware). The previous messages were > for Vmware. The one below is for
> > XenServer:
> >
> > I guess you are using shadow mode for L1 VMM. With shadow L1, this
> is
> > a known issue to me. Can you try to use L1 EPT to see whether it is
> reproducible?
> >
> >
>
>
> > Best regards,
> > Yang
> >
> >
> >
>
>
> Best regards,
> Yang
>
>
[-- Attachment #1.2: Type: text/html, Size: 3584 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2013-12-23 14:17 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-16 11:05 Nested virtualization instabilities Etzion Bar-Noy
2013-12-16 13:51 ` Andrew Cooper
2013-12-18 1:33 ` Zhang, Yang Z
2013-12-18 9:46 ` Etzion Bar-Noy
2013-12-19 19:40 ` Etzion Bar-Noy
2013-12-19 19:42 ` Etzion Bar-Noy
2013-12-20 0:50 ` Zhang, Yang Z
2013-12-20 11:24 ` Etzion Bar-Noy
2013-12-23 1:09 ` Zhang, Yang Z
2013-12-23 14:17 ` Etzion Bar-Noy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.