XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

["Yuvraj Agarwal" <yuvraj@cs.ucsd.edu>]

[-- Attachment #1.1: Type: text/plain, Size: 1169 bytes --]

Hi All, 

 

We are setting up a system with a large number of very small VMs for a
project. We worked through a number of limitations, including those
imposed by blktap2 devices and the number of dynamic IRQ (set kernel
config NR_CPUS) etc. After these changes we were able to get to  154 domUs
(!), but as soon as we start up the 155th domU the system crashes. We
edited each domU config to have one blktap2 device (the disk image) and
two virtual network interfaces each. 

 

We are using the standard XEN 4.0 (stable/release version)  with the
2.6.31.13 (pvops) kernel. 

 

Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
but as far as we can see we don't quite know what might be going causing
the system to crash (no console access anymore and system becomes
unresponsive and needs to be power-cycled).  I have pasted only the
relevant bits of information (the last domU that did successfully start
and the next one that failed). It may be the case that all the log
messages weren't flushed before the system crashed. 

 

Does anyone know where this limit of 155 domU is coming from and how we
can fix/increase it? 


thanks

Yuvraj 


[-- Attachment #1.2: Type: text/html, Size: 3221 bytes --]

[-- Attachment #2: var-log-daemon.log.txt --]
[-- Type: text/plain, Size: 4796 bytes --]

Apr 26 20:08:08  BlackBox tapdisk2[10074]: Created /dev/xen/blktap-2/blktap152 device
Apr 26 20:08:08  BlackBox tapdisk2[10074]: Created /dev/xen/blktap-2/tapdev152 device
Apr 26 20:08:08  BlackBox tapdisk2[10074]: new interface: ring: 251, device: 253, minor: 152
Apr 26 20:08:08  BlackBox tapdisk2[10074]: I/O queue driver: lio
Apr 26 20:08:08  BlackBox tapdisk2[10074]: block-aio open('/home/xen/domains/testing-ss-155.ucsd.edu/disk.img')
Apr 26 20:08:08  BlackBox tapdisk2[10074]: open(/home/xen/domains/testing-ss-155.ucsd.edu/disk.img) with O_DIRECT
Apr 26 20:08:08  BlackBox tapdisk2[10074]: Image size: #012#011pre sector_shift  [1073741824]#012#011post sector_shift [2097152]
Apr 26 20:08:08  BlackBox tapdisk2[10074]: opened image /home/xen/domains/testing-ss-155.ucsd.edu/disk.img (1 users, state: 0x00000001, type: 0)
Apr 26 20:08:08  BlackBox tapdisk2[10074]: VBD CHAIN:
Apr 26 20:08:08  BlackBox tapdisk2[10074]: /home/xen/domains/testing-ss-155.ucsd.edu/disk.img: 0
Apr 26 20:08:09  BlackBox logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/153/770
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/0
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.0, bridge testbr.
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/0/hotplug-status connected to xenstore.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/physical-device fd:98 to xenstore.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/hotplug-status connected to xenstore.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/1
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.1, bridge dummy.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/1/hotplug-status connected to xenstore.


Apr 26 20:10:17  BlackBox tapdisk2[12749]: Created /dev/xen/blktap-2/blktap153 device
Apr 26 20:10:17  BlackBox tapdisk2[12749]: Created /dev/xen/blktap-2/tapdev153 device
Apr 26 20:10:17  BlackBox tapdisk2[12749]: new interface: ring: 251, device: 253, minor: 153
Apr 26 20:10:17  BlackBox tapdisk2[12749]: I/O queue driver: lio
Apr 26 20:10:17  BlackBox tapdisk2[12749]: block-aio open('/home/xen/domains/testing-ss-156.ucsd.edu/disk.img')
Apr 26 20:10:17  BlackBox tapdisk2[12749]: open(/home/xen/domains/testing-ss-156.ucsd.edu/disk.img) with O_DIRECT
Apr 26 20:10:17  BlackBox tapdisk2[12749]: Image size: #012#011pre sector_shift  [1073741824]#012#011post sector_shift [2097152]
Apr 26 20:10:17  BlackBox tapdisk2[12749]: opened image /home/xen/domains/testing-ss-156.ucsd.edu/disk.img (1 users, state: 0x00000001, type: 0)
Apr 26 20:10:17  BlackBox tapdisk2[12749]: VBD CHAIN:
Apr 26 20:10:17  BlackBox tapdisk2[12749]: /home/xen/domains/testing-ss-156.ucsd.edu/disk.img: 0
Apr 26 20:10:17  BlackBox logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/154/770
Apr 26 20:10:29  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/154/770/physical-device fd:99 to xenstore.
Apr 26 20:10:29  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/154/770/hotplug-status connected to xenstore.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/154/0
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif154.0, bridge testbr.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/154/0/hotplug-status connected to xenstore.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/154/1
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif154.1, bridge dummy.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/154/1/hotplug-status connected to xenstore.

[-- Attachment #3: var-log-xen-xend.log --]
[-- Type: application/octet-stream, Size: 11282 bytes --]

[2010-04-26 20:10:16 2359] DEBUG (XendDomainInfo:101) XendDomainInfo.create(['vm', ['name', 'testing-ss-156.ucsd.edu'], ['memory', '64'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 1], ['oos', 1], ['image', ['linux', ['kernel', '/home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen'], ['ramdisk', '/home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen'], ['root', '/dev/hda2 ro'], ['videoram', 4], ['args', '2 console=xvc0'], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['tap2', ['uname', 'tap:aio:/home/xen/domains/testing-ss-156.ucsd.edu/disk.img'], ['dev', 'hda2'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.200.156'], ['mac', '00:77:22:33:44:9C'], ['bridge', 'testbr']]], ['device', ['vif', ['bridge', 'dummy']]]])
[2010-04-26 20:10:16 2359] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain
[2010-04-26 20:10:16 2359] DEBUG (balloon:220) Balloon: 4022792 KiB free; need 16384; done.
[2010-04-26 20:10:16 2359] DEBUG (XendDomain:464) Adding Domain: 154
[2010-04-26 20:10:16 2359] DEBUG (XendDomainInfo:2818) XendDomainInfo.initDomain: 154 256
[2010-04-26 20:10:17 2359] DEBUG (XendDomainInfo:2845) _initDomain:shadow_memory=0x0, memory_static_max=0x4000000, memory_static_min=0x0.
[2010-04-26 20:10:17 2359] INFO (image:182) buildDomain os=linux dom=154 vcpus=1
[2010-04-26 20:10:17 2359] DEBUG (image:721) domid          = 154
[2010-04-26 20:10:17 2359] DEBUG (image:722) memsize        = 64
[2010-04-26 20:10:17 2359] DEBUG (image:723) image          = /home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen
[2010-04-26 20:10:17 2359] DEBUG (image:724) store_evtchn   = 1
[2010-04-26 20:10:17 2359] DEBUG (image:725) console_evtchn = 2
[2010-04-26 20:10:17 2359] DEBUG (image:726) cmdline        = root=/dev/hda2 ro 2 console=xvc0
[2010-04-26 20:10:17 2359] DEBUG (image:727) ramdisk        = /home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen
[2010-04-26 20:10:17 2359] DEBUG (image:728) vcpus          = 1
[2010-04-26 20:10:17 2359] DEBUG (image:729) features       = 
[2010-04-26 20:10:17 2359] DEBUG (image:730) flags          = 0
[2010-04-26 20:10:17 2359] DEBUG (image:731) superpages     = 0
[2010-04-26 20:10:17 2359] INFO (XendDomainInfo:2367) createDevice: tap2 : {'bootable': 1, 'uname': 'tap:aio:/home/xen/domains/testing-ss-156.ucsd.edu/disk.img', 'mode': 'w', 'dev': 'hda2', 'uuid': '532b3206-7802-8c02-3ca6-85ffae37fb21'}
[2010-04-26 20:10:17 2359] DEBUG (DevController:95) DevController: writing {'virtual-device': '770', 'device-type': 'disk', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/154/770'} to /local/domain/154/device/vbd/770.
[2010-04-26 20:10:17 2359] DEBUG (DevController:97) DevController: writing {'domain': 'testing-ss-156.ucsd.edu', 'frontend': '/local/domain/154/device/vbd/770', 'uuid': '532b3206-7802-8c02-3ca6-85ffae37fb21', 'bootable': '1', 'dev': 'hda2', 'state': '1', 'params': '/dev/xen/blktap-2/tapdev153', 'mode': 'w', 'online': '1', 'frontend-id': '154', 'type': 'phy', 'tapdisk-params': 'aio:/home/xen/domains/testing-ss-156.ucsd.edu/disk.img'} to /local/domain/0/backend/vbd/154/770.
[2010-04-26 20:10:17 2359] INFO (XendDomainInfo:2367) createDevice: vif : {'ip': '192.168.200.156', 'mac': '00:77:22:33:44:9C', 'uuid': 'd86cdace-35a6-fd85-5920-cbc74895535b', 'bridge': 'testbr'}
[2010-04-26 20:10:17 2359] DEBUG (DevController:95) DevController: writing {'mac': '00:77:22:33:44:9C', 'handle': '0', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/154/0'} to /local/domain/154/device/vif/0.
[2010-04-26 20:10:17 2359] DEBUG (DevController:97) DevController: writing {'bridge': 'testbr', 'domain': 'testing-ss-156.ucsd.edu', 'handle': '0', 'uuid': 'd86cdace-35a6-fd85-5920-cbc74895535b', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '192.168.200.156', 'mac': '00:77:22:33:44:9C', 'frontend-id': '154', 'state': '1', 'online': '1', 'frontend': '/local/domain/154/device/vif/0'} to /local/domain/0/backend/vif/154/0.
[2010-04-26 20:10:30 2359] INFO (XendDomainInfo:2367) createDevice: vif : {'bridge': 'dummy', 'mac': '00:16:3e:32:d5:2c', 'uuid': 'c3a244df-030d-ec04-63cd-6823e76d90a8'}
[2010-04-26 20:10:30 2359] DEBUG (DevController:95) DevController: writing {'mac': '00:16:3e:32:d5:2c', 'handle': '1', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/154/1'} to /local/domain/154/device/vif/1.
[2010-04-26 20:10:30 2359] DEBUG (DevController:97) DevController: writing {'bridge': 'dummy', 'domain': 'testing-ss-156.ucsd.edu', 'handle': '1', 'uuid': 'c3a244df-030d-ec04-63cd-6823e76d90a8', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3e:32:d5:2c', 'frontend-id': '154', 'state': '1', 'online': '1', 'frontend': '/local/domain/154/device/vif/1'} to /local/domain/0/backend/vif/154/1.
[2010-04-26 20:10:30 2359] DEBUG (XendDomainInfo:3400) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'a976f839-b5d4-2841-2ce9-b7b5168aabc9', 'on_reboot': 'restart', 'start_time': '1272337830.28', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '', 'image': "(linux (kernel /home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen) (ramdisk /home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen) (args 'root=/dev/hda2 ro 2 console=xvc0') (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 3221225472) (GUEST_OS linux) (HYPERCALL_PAGE 3222278144) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3222274048) (XEN_VERSION xen-3.0)))", 'name': 'testing-ss-156.ucsd.edu'}
[2010-04-26 20:10:30 2359] DEBUG (XendDomainInfo:1804) Storing domain details: {'console/ring-ref': '3622290', 'image/entry': '3222274048', 'console/port': '2', 'store/ring-ref': '3622291', 'image/loader': 'generic', 'vm': '/vm/a976f839-b5d4-2841-2ce9-b7b5168aabc9', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '3221225472', 'memory/target': '65536', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576', 'image/paddr-offset': '3221225472', 'image/hypercall-page': '3222278144', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'console/type': 'xenconsoled', 'image/features/auto-translated-physmap': '1', 'name': 'testing-ss-156.ucsd.edu', 'domid': '154', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2010-04-26 20:10:32 2359] DEBUG (DevController:95) DevController: writing {'protocol': 'x86_32-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/154/0'} to /local/domain/154/device/console/0.
[2010-04-26 20:10:32 2359] DEBUG (DevController:97) DevController: writing {'domain': 'testing-ss-156.ucsd.edu', 'frontend': '/local/domain/154/device/console/0', 'uuid': '0e439360-1b37-71a0-5826-4b8b0c1d7dc0', 'frontend-id': '154', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/154/0.
[2010-04-26 20:10:32 2359] DEBUG (XendDomainInfo:1891) XendDomainInfo.handleShutdownWatch
[2010-04-26 20:10:33 2359] DEBUG (DevController:139) Waiting for devices tap2.
[2010-04-26 20:10:33 2359] DEBUG (DevController:144) Waiting for 770.
[2010-04-26 20:10:34 2359] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/154/770/hotplug-status.
[2010-04-26 20:10:34 2359] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-26 20:10:34 2359] DEBUG (DevController:139) Waiting for devices vif.
[2010-04-26 20:10:34 2359] DEBUG (DevController:144) Waiting for 0.
[2010-04-26 20:10:34 2359] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/154/0/hotplug-status.
[2010-04-26 20:10:34 2359] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-26 20:10:34 2359] DEBUG (DevController:144) Waiting for 1.
[2010-04-26 20:10:34 2359] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/154/1/hotplug-status.
[2010-04-26 20:10:34 2359] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-26 20:10:34 2359] DEBUG (DevController:139) Waiting for devices vkbd.
[2010-04-26 20:10:34 2359] DEBUG (DevController:139) Waiting for devices ioports.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices tap.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vif2.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices console.
[2010-04-26 20:10:35 2359] DEBUG (DevController:144) Waiting for 0.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vscsi.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vbd.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices irq.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vfb.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices pci.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vusb.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vtpm.
[2010-04-26 20:10:35 2359] INFO (XendDomain:1206) Domain testing-ss-156.ucsd.edu (154) unpaused.


[2010-04-26 20:10:52 2359] DEBUG (XendDomainInfo:101) XendDomainInfo.create(['vm', ['name', 'testing-ss-157.ucsd.edu'], ['memory', '64'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 1], ['oos', 1], ['image', ['linux', ['kernel', '/home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen'], ['ramdisk', '/home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen'], ['root', '/dev/hda2 ro'], ['videoram', 4], ['args', '2 console=xvc0'], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['tap2', ['uname', 'tap:aio:/home/xen/domains/testing-ss-157.ucsd.edu/disk.img'], ['dev', 'hda2'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.200.157'], ['mac', '00:77:22:33:44:9D'], ['bridge', 'testbr']]], ['device', ['vif', ['bridge', 'dummy']]]])
[2010-04-26 20:10:52 2359] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain
[2010-04-26 20:10:52 2359] DEBUG (balloon:220) Balloon: 3957036 KiB free; need 16384; done.
[2010-04-26 20:10:52 2359] DEBUG (XendDomain:464) Adding Domain: 155
[2010-04-26 20:10:52 2359] DEBUG (XendDomainInfo:2818) XendDomainInfo.initDomain: 155 256
[2010-04-26 20:10:52 2359] DEBUG (XendDomainInfo:2845) _initDomain:shadow_memory=0x0, memory_static_max=0x4000000, memory_static_min=0x0.


[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel