Re: [BUG] invocation of xl create with insufficient memory for new domU causes system reboot

["Georgy V. Repin" <grep@informika.ru>]

02.04.2014 13:30, Ian Campbell пишет:
>
> On Tue, 2014-04-01 at 14:17 +0400, Georgy V. Repin wrote:
>> 01.04.2014 13:52, Ian Campbell пишет:
>>> On Tue, 2014-04-01 at 13:35 +0400, Georgy V. Repin wrote:
>>>> Hello!
>>>>
>>>> 1. Problem: invocation of xl create when there is not enough memory to
>>>> create new domU causes oom to select process to kiill; when oom selects
>>>> to kill quemu-system-* process, this causes kernel panic (it's my
>>>> suppose, because i see no message "kernel panic" in log, but i do see
>>>> "CPU: 0 PID: 7899 Comm: qemu-system-i38 Not tainted 3.10.17-gentoo #1" +
>>>> failed call trace), and, therefore (assuming default settings) kernel
>>>> reboot (see log at "4.")
>>>> Couple of remarks:
>>>> - fix checking of available memory for new domU in xl toolstack could
>>>> serve as workaround
>>>> - real cause is not clear to me, maybe it's not even a xen bug (quemu ?
>>>> kernel ?)
>>>
>>> Are you using dom0_mem= together with auto ballooning enabled in the
>>> toolstack?
>>
>> informika-xen-3 ~ # cat /proc/cmdline
>> placeholder root=/dev/sda1 ro
>
> dom0_mem is a hypervisor command line option, not a Linux one. Although
> given that you have show autobaloon=auto that is moot since it implies
> the automatic functionality was indeed added in the version you have.
>
>>
>> part of grub.cfg
>>
>> 		        if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
>> 		            xen_rm_opts=
>> 		        else
>> 		            xen_rm_opts="no-real-mode edd=off"
>> 		        fi
>> 			multiboot	/boot/xen-4.gz placeholder   ${xen_rm_opts}


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
this code from grub.cfg shows that there is no dom0_mem=  parameter for 
/boot/xen-4.gz


>> 			echo	'Loading Linux 3.10.17-gentoo ...'
>> 			module	/boot/vmlinuz-3.10.17-gentoo placeholder root=/dev/sda1 ro
>>
>> xl.conf:
>> # Control whether dom0 is ballooned down when xen doesn't have enough
>> # free memory to create a domain.  "auto" means only balloon if dom0
>> # starts with all the host's memory.
>> #autoballoon="auto"
>>
>> So:
>> 	- no dom0_mem= as boot param
>> 	- autobaloon is set to default ("auto", i suppose)
>
> Correct, the default is auto.
>
> Please can you show us the output of "xl info" and your guest cfg file
> as well as the full output of "xl -vvv create $cfg".

informika-xen-3 ~ # xl info
host                   : informika-xen-3
release                : 3.10.17-gentoo
version                : #1 SMP Sun Dec 8 13:22:31 MSK 2013
machine                : x86_64
nr_cpus                : 16
max_cpu_id             : 15
nr_nodes               : 2
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 2400
hw_caps                : 
bfebfbff:2c100800:00000000:00003f00:029ee3ff:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 24566
free_memory            : 418
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 3
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder
cc_compiler            : x86_64-pc-linux-gnu-gcc (Gentoo 4.7.3-r1 p1.3, 
pie-0.5.5) 4.7.3
cc_compile_by          :
cc_compile_domain      : internal
cc_compile_date        : Sun Jan  5 22:39:16 MSK 2014
xend_config_format     : 4


remark:
	- now there is no oom reaction, because i have reduced memory amount 
for one of other domU's (now it's enough memory for all domU's)
	- it seems that "xen_commandline" states that there is no "dom0_mem=" 
option.

informika-xen-3 ~ # xl -vvv create /etc/xen/vm/www.icsit.ru
Parsing config from /etc/xen/vm/www.icsit.ru
libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x24915a0: 
create: how=(nil) callback=(nil) poller=0x24903a0
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk 
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk 
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk 
vdev=(null) spec.backend=phy
libxl: debug: libxl.c:2604:libxl__device_disk_local_initiate_attach: 
locally attaching PHY disk /dev/storage-a/www.icsit.ru
libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config 
bootloader value: pygrub
libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: 
Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x24915a0: 
inprogress: poller=0x24903a0, flags=i
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch 
w=0x248f798 wpath=/local/domain/8 token=3/0: register slotnum=3
libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 
0x24915a0: progress report: ignored
libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing 
bootloader: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader 
arg: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader 
arg: --args=root=/dev/xvda fastboot
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader 
arg: --output=/var/run/xen/bootloader.8.out
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader 
arg: --output-format=simple0
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader 
arg: --output-directory=/var/run/xen/bootloader.8.d
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys:   bootloader 
arg: /dev/storage-a/www.icsit.ru
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248f798 
wpath=/local/domain/8 token=3/0: event epath=/local/domain/8

libxl: debug: libxl_bootloader.c:634:bootloader_finished: bootloader 
completed
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: 
bootloader output contained kernel 
/var/run/xen/bootloader.8.d/boot_kernel.VfLJyd
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: 
bootloader output contained ramdisk 
/var/run/xen/bootloader.8.d/boot_ramdisk.HBHXnw
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: 
bootloader output contained args root=/dev/xvda console=hvc0 ro 
root=/dev/xvda fastboot
libxl: debug: libxl_bootloader.c:647:bootloader_finished: bootloader 
execution successful
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch 
w=0x248f798 wpath=/local/domain/8 token=3/0: deregister slotnum=3
libxl: debug: libxl_numa.c:475:libxl__get_numa_candidate: New best NUMA 
placement candidate found: nr_nodes=2, nr_cpus=16, nr_vcpus=65, 
free_memkb=2466
libxl: detail: libxl_dom.c:195:numa_place_domain: NUMA placement 
candidate with 2 nodes, 16 cpus and 2466 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/xvda 
console=hvc0 ro root=/dev/xvda fastboot", features="(null)"
libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 1 path 
/var/run/xen/bootloader.8.d/boot_kernel.VfLJyd

domainbuilder: detail: xc_dom_kernel_mem: called
domainbuilder: detail: xc_dom_ramdisk_mem: called
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.3, caps 
xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p 
hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary 
loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_malloc            : 18250 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x4b30aa -> 0x11d2aa0
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x1000000 memsz=0xad6000
xc: detail: elf_parse_binary: phdr: paddr=0x1c00000 memsz=0xe30e0
xc: detail: elf_parse_binary: phdr: paddr=0x1ce4000 memsz=0x143c0
xc: detail: elf_parse_binary: phdr: paddr=0x1cf9000 memsz=0x35f000
xc: detail: elf_parse_binary: memory: 0x1000000 -> 0x2058000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff81cf9200
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
xc: detail: elf_xen_parse_note: FEATURES = 
"!writable_page_tables|pae_pgdir_above_4gb"
xc: detail: elf_xen_parse_note: PAE_MODE = "yes"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail:     virt_base        = 0xffffffff80000000
xc: detail:     elf_paddr_offset = 0x0
xc: detail:     virt_offset      = 0xffffffff80000000
xc: detail:     virt_kstart      = 0xffffffff81000000
xc: detail:     virt_kend        = 0xffffffff82058000
xc: detail:     virt_entry       = 0xffffffff81cf9200
xc: detail:     p2m_base         = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 
0xffffffff81000000 -> 0xffffffff82058000
domainbuilder: detail: xc_dom_mem_init: mem 2048 MB, pages 0x80000 
pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x80000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc            : 4096 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 
0xffffffff81000000 -> 0xffffffff82058000  (pfn 0x1000 + 0x1058 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x1000+0x1058 at 0x7f9aa40e4000
xc: detail: elf_load_binary: phdr 0 at 0x7f9aa40e4000 -> 0x7f9aa4bba000
xc: detail: elf_load_binary: phdr 1 at 0x7f9aa4ce4000 -> 0x7f9aa4dc70e0
xc: detail: elf_load_binary: phdr 2 at 0x7f9aa4dc8000 -> 0x7f9aa4ddc3c0
xc: detail: elf_load_binary: phdr 3 at 0x7f9aa4ddd000 -> 0x7f9aa4eb6000
domainbuilder: detail: xc_dom_alloc_segment:   ramdisk      : 
0xffffffff82058000 -> 0xffffffff82fa8000  (pfn 0x2058 + 0xf50 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x2058+0xf50 at 0x7f9a9f0b0000
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x5c1479 -> 0xf4fa10
domainbuilder: detail: xc_dom_alloc_segment:   phys2mach    : 
0xffffffff82fa8000 -> 0xffffffff833a8000  (pfn 0x2fa8 + 0x400 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x2fa8+0x400 at 0x7f9a9ecb0000
domainbuilder: detail: xc_dom_alloc_page   :   start info   : 
0xffffffff833a8000 (pfn 0x33a8)
domainbuilder: detail: xc_dom_alloc_page   :   xenstore     : 
0xffffffff833a9000 (pfn 0x33a9)
domainbuilder: detail: xc_dom_alloc_page   :   console      : 
0xffffffff833aa000 (pfn 0x33aa)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 
0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 
0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 
0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 
0xffffffff80000000 -> 0xffffffff837fffff, 28 table(s)
domainbuilder: detail: xc_dom_alloc_segment:   page tables  : 
0xffffffff833ab000 -> 0xffffffff833ca000  (pfn 0x33ab + 0x1f pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x33ab+0x1f at 0x7f9aa9696000
domainbuilder: detail: xc_dom_alloc_page   :   boot stack   : 
0xffffffff833ca000 (pfn 0x33ca)
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 
0xffffffff833cb000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 
0xffffffff83800000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x80000
domainbuilder: detail: clear_page: pfn 0x33aa, mfn 0x5d4935
domainbuilder: detail: clear_page: pfn 0x33a9, mfn 0x21c2da
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x33a8+0x1 at 0x7f9aa9693000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff81001000 
pfn=0x1001
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 22564 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 0 bytes
domainbuilder: detail:       domU mmap          : 35 MB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 
0x1f6b9
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x33ab mfn 0x21c2d9
domainbuilder: detail: launch_vm: called, ctxt=0x7fff07610e70
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk 
vdev=xvda spec.backend=phy
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch 
w=0x248dd08 wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: 
register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248dd08 
wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: event 
epath=/local/domain/0/backend/vbd/8/51712/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend 
/local/domain/0/backend/vbd/8/51712/state wanted state 2 still waiting 
state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248dd08 
wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: event 
epath=/local/domain/0/backend/vbd/8/51712/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend 
/local/domain/0/backend/vbd/8/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch 
w=0x248dd08 wpath=/local/domain/0/backend/vbd/8/51712/state token=3/1: 
deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch 
w=0x248dd08: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: 
/etc/xen/scripts/block add
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch 
w=0x248de18 wpath=/local/domain/0/backend/vif/8/0/state token=3/2: 
register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248de18 
wpath=/local/domain/0/backend/vif/8/0/state token=3/2: event 
epath=/local/domain/0/backend/vif/8/0/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend 
/local/domain/0/backend/vif/8/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x248de18 
wpath=/local/domain/0/backend/vif/8/0/state token=3/2: event 
epath=/local/domain/0/backend/vif/8/0/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend 
/local/domain/0/backend/vif/8/0/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch 
w=0x248de18 wpath=/local/domain/0/backend/vif/8/0/state token=3/2: 
deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch 
w=0x248de18: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: 
/etc/xen/scripts/vif-bridge online
libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 
0x24915a0: progress report: ignored
libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x24915a0: 
complete, rc=0
libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x24915a0: destroy
Daemon running with PID 6087
xc: debug: hypercall buffer: total allocations:346 total releases:346
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:336 misses:4 toobig:6

informika-xen-3 ~ # cat /etc/xen/vm/www.icsit.ru
bootloader = "pygrub"
memory = 2048
vcpus = 2
name = "www.icsit.ru"
vif = [ 'mac=00:16:3E:5A:04:01, bridge=v119, model=e1000' ]
disk = ['phy:/dev/storage-a/www.icsit.ru,xvda,w']
root = "/dev/xvda"
extra = "fastboot"

>
> Thanks,
> Ian.
>
>
>
>

-- 
	Georgy V. Repin


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel