XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

[-- Attachment #1.1: Type: text/plain, Size: 1169 bytes --]

Hi All, 

 

We are setting up a system with a large number of very small VMs for a
project. We worked through a number of limitations, including those
imposed by blktap2 devices and the number of dynamic IRQ (set kernel
config NR_CPUS) etc. After these changes we were able to get to  154 domUs
(!), but as soon as we start up the 155th domU the system crashes. We
edited each domU config to have one blktap2 device (the disk image) and
two virtual network interfaces each. 

 

We are using the standard XEN 4.0 (stable/release version)  with the
2.6.31.13 (pvops) kernel. 

 

Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
but as far as we can see we don't quite know what might be going causing
the system to crash (no console access anymore and system becomes
unresponsive and needs to be power-cycled).  I have pasted only the
relevant bits of information (the last domU that did successfully start
and the next one that failed). It may be the case that all the log
messages weren't flushed before the system crashed. 

 

Does anyone know where this limit of 155 domU is coming from and how we
can fix/increase it? 


thanks

Yuvraj 


[-- Attachment #1.2: Type: text/html, Size: 3221 bytes --]

[-- Attachment #2: var-log-daemon.log.txt --]
[-- Type: text/plain, Size: 4796 bytes --]

Apr 26 20:08:08  BlackBox tapdisk2[10074]: Created /dev/xen/blktap-2/blktap152 device
Apr 26 20:08:08  BlackBox tapdisk2[10074]: Created /dev/xen/blktap-2/tapdev152 device
Apr 26 20:08:08  BlackBox tapdisk2[10074]: new interface: ring: 251, device: 253, minor: 152
Apr 26 20:08:08  BlackBox tapdisk2[10074]: I/O queue driver: lio
Apr 26 20:08:08  BlackBox tapdisk2[10074]: block-aio open('/home/xen/domains/testing-ss-155.ucsd.edu/disk.img')
Apr 26 20:08:08  BlackBox tapdisk2[10074]: open(/home/xen/domains/testing-ss-155.ucsd.edu/disk.img) with O_DIRECT
Apr 26 20:08:08  BlackBox tapdisk2[10074]: Image size: #012#011pre sector_shift  [1073741824]#012#011post sector_shift [2097152]
Apr 26 20:08:08  BlackBox tapdisk2[10074]: opened image /home/xen/domains/testing-ss-155.ucsd.edu/disk.img (1 users, state: 0x00000001, type: 0)
Apr 26 20:08:08  BlackBox tapdisk2[10074]: VBD CHAIN:
Apr 26 20:08:08  BlackBox tapdisk2[10074]: /home/xen/domains/testing-ss-155.ucsd.edu/disk.img: 0
Apr 26 20:08:09  BlackBox logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/153/770
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/0
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.0, bridge testbr.
Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/0/hotplug-status connected to xenstore.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/physical-device fd:98 to xenstore.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/hotplug-status connected to xenstore.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/1
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.1, bridge dummy.
Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/1/hotplug-status connected to xenstore.


Apr 26 20:10:17  BlackBox tapdisk2[12749]: Created /dev/xen/blktap-2/blktap153 device
Apr 26 20:10:17  BlackBox tapdisk2[12749]: Created /dev/xen/blktap-2/tapdev153 device
Apr 26 20:10:17  BlackBox tapdisk2[12749]: new interface: ring: 251, device: 253, minor: 153
Apr 26 20:10:17  BlackBox tapdisk2[12749]: I/O queue driver: lio
Apr 26 20:10:17  BlackBox tapdisk2[12749]: block-aio open('/home/xen/domains/testing-ss-156.ucsd.edu/disk.img')
Apr 26 20:10:17  BlackBox tapdisk2[12749]: open(/home/xen/domains/testing-ss-156.ucsd.edu/disk.img) with O_DIRECT
Apr 26 20:10:17  BlackBox tapdisk2[12749]: Image size: #012#011pre sector_shift  [1073741824]#012#011post sector_shift [2097152]
Apr 26 20:10:17  BlackBox tapdisk2[12749]: opened image /home/xen/domains/testing-ss-156.ucsd.edu/disk.img (1 users, state: 0x00000001, type: 0)
Apr 26 20:10:17  BlackBox tapdisk2[12749]: VBD CHAIN:
Apr 26 20:10:17  BlackBox tapdisk2[12749]: /home/xen/domains/testing-ss-156.ucsd.edu/disk.img: 0
Apr 26 20:10:17  BlackBox logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/154/770
Apr 26 20:10:29  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/154/770/physical-device fd:99 to xenstore.
Apr 26 20:10:29  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/154/770/hotplug-status connected to xenstore.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/154/0
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif154.0, bridge testbr.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/154/0/hotplug-status connected to xenstore.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/154/1
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif154.1, bridge dummy.
Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/154/1/hotplug-status connected to xenstore.

[-- Attachment #3: var-log-xen-xend.log --]
[-- Type: application/octet-stream, Size: 11282 bytes --]

[2010-04-26 20:10:16 2359] DEBUG (XendDomainInfo:101) XendDomainInfo.create(['vm', ['name', 'testing-ss-156.ucsd.edu'], ['memory', '64'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 1], ['oos', 1], ['image', ['linux', ['kernel', '/home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen'], ['ramdisk', '/home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen'], ['root', '/dev/hda2 ro'], ['videoram', 4], ['args', '2 console=xvc0'], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['tap2', ['uname', 'tap:aio:/home/xen/domains/testing-ss-156.ucsd.edu/disk.img'], ['dev', 'hda2'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.200.156'], ['mac', '00:77:22:33:44:9C'], ['bridge', 'testbr']]], ['device', ['vif', ['bridge', 'dummy']]]])
[2010-04-26 20:10:16 2359] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain
[2010-04-26 20:10:16 2359] DEBUG (balloon:220) Balloon: 4022792 KiB free; need 16384; done.
[2010-04-26 20:10:16 2359] DEBUG (XendDomain:464) Adding Domain: 154
[2010-04-26 20:10:16 2359] DEBUG (XendDomainInfo:2818) XendDomainInfo.initDomain: 154 256
[2010-04-26 20:10:17 2359] DEBUG (XendDomainInfo:2845) _initDomain:shadow_memory=0x0, memory_static_max=0x4000000, memory_static_min=0x0.
[2010-04-26 20:10:17 2359] INFO (image:182) buildDomain os=linux dom=154 vcpus=1
[2010-04-26 20:10:17 2359] DEBUG (image:721) domid          = 154
[2010-04-26 20:10:17 2359] DEBUG (image:722) memsize        = 64
[2010-04-26 20:10:17 2359] DEBUG (image:723) image          = /home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen
[2010-04-26 20:10:17 2359] DEBUG (image:724) store_evtchn   = 1
[2010-04-26 20:10:17 2359] DEBUG (image:725) console_evtchn = 2
[2010-04-26 20:10:17 2359] DEBUG (image:726) cmdline        = root=/dev/hda2 ro 2 console=xvc0
[2010-04-26 20:10:17 2359] DEBUG (image:727) ramdisk        = /home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen
[2010-04-26 20:10:17 2359] DEBUG (image:728) vcpus          = 1
[2010-04-26 20:10:17 2359] DEBUG (image:729) features       = 
[2010-04-26 20:10:17 2359] DEBUG (image:730) flags          = 0
[2010-04-26 20:10:17 2359] DEBUG (image:731) superpages     = 0
[2010-04-26 20:10:17 2359] INFO (XendDomainInfo:2367) createDevice: tap2 : {'bootable': 1, 'uname': 'tap:aio:/home/xen/domains/testing-ss-156.ucsd.edu/disk.img', 'mode': 'w', 'dev': 'hda2', 'uuid': '532b3206-7802-8c02-3ca6-85ffae37fb21'}
[2010-04-26 20:10:17 2359] DEBUG (DevController:95) DevController: writing {'virtual-device': '770', 'device-type': 'disk', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/154/770'} to /local/domain/154/device/vbd/770.
[2010-04-26 20:10:17 2359] DEBUG (DevController:97) DevController: writing {'domain': 'testing-ss-156.ucsd.edu', 'frontend': '/local/domain/154/device/vbd/770', 'uuid': '532b3206-7802-8c02-3ca6-85ffae37fb21', 'bootable': '1', 'dev': 'hda2', 'state': '1', 'params': '/dev/xen/blktap-2/tapdev153', 'mode': 'w', 'online': '1', 'frontend-id': '154', 'type': 'phy', 'tapdisk-params': 'aio:/home/xen/domains/testing-ss-156.ucsd.edu/disk.img'} to /local/domain/0/backend/vbd/154/770.
[2010-04-26 20:10:17 2359] INFO (XendDomainInfo:2367) createDevice: vif : {'ip': '192.168.200.156', 'mac': '00:77:22:33:44:9C', 'uuid': 'd86cdace-35a6-fd85-5920-cbc74895535b', 'bridge': 'testbr'}
[2010-04-26 20:10:17 2359] DEBUG (DevController:95) DevController: writing {'mac': '00:77:22:33:44:9C', 'handle': '0', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/154/0'} to /local/domain/154/device/vif/0.
[2010-04-26 20:10:17 2359] DEBUG (DevController:97) DevController: writing {'bridge': 'testbr', 'domain': 'testing-ss-156.ucsd.edu', 'handle': '0', 'uuid': 'd86cdace-35a6-fd85-5920-cbc74895535b', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '192.168.200.156', 'mac': '00:77:22:33:44:9C', 'frontend-id': '154', 'state': '1', 'online': '1', 'frontend': '/local/domain/154/device/vif/0'} to /local/domain/0/backend/vif/154/0.
[2010-04-26 20:10:30 2359] INFO (XendDomainInfo:2367) createDevice: vif : {'bridge': 'dummy', 'mac': '00:16:3e:32:d5:2c', 'uuid': 'c3a244df-030d-ec04-63cd-6823e76d90a8'}
[2010-04-26 20:10:30 2359] DEBUG (DevController:95) DevController: writing {'mac': '00:16:3e:32:d5:2c', 'handle': '1', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/154/1'} to /local/domain/154/device/vif/1.
[2010-04-26 20:10:30 2359] DEBUG (DevController:97) DevController: writing {'bridge': 'dummy', 'domain': 'testing-ss-156.ucsd.edu', 'handle': '1', 'uuid': 'c3a244df-030d-ec04-63cd-6823e76d90a8', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3e:32:d5:2c', 'frontend-id': '154', 'state': '1', 'online': '1', 'frontend': '/local/domain/154/device/vif/1'} to /local/domain/0/backend/vif/154/1.
[2010-04-26 20:10:30 2359] DEBUG (XendDomainInfo:3400) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'a976f839-b5d4-2841-2ce9-b7b5168aabc9', 'on_reboot': 'restart', 'start_time': '1272337830.28', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '', 'image': "(linux (kernel /home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen) (ramdisk /home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen) (args 'root=/dev/hda2 ro 2 console=xvc0') (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 3221225472) (GUEST_OS linux) (HYPERCALL_PAGE 3222278144) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3222274048) (XEN_VERSION xen-3.0)))", 'name': 'testing-ss-156.ucsd.edu'}
[2010-04-26 20:10:30 2359] DEBUG (XendDomainInfo:1804) Storing domain details: {'console/ring-ref': '3622290', 'image/entry': '3222274048', 'console/port': '2', 'store/ring-ref': '3622291', 'image/loader': 'generic', 'vm': '/vm/a976f839-b5d4-2841-2ce9-b7b5168aabc9', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '3221225472', 'memory/target': '65536', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576', 'image/paddr-offset': '3221225472', 'image/hypercall-page': '3222278144', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'console/type': 'xenconsoled', 'image/features/auto-translated-physmap': '1', 'name': 'testing-ss-156.ucsd.edu', 'domid': '154', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2010-04-26 20:10:32 2359] DEBUG (DevController:95) DevController: writing {'protocol': 'x86_32-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/154/0'} to /local/domain/154/device/console/0.
[2010-04-26 20:10:32 2359] DEBUG (DevController:97) DevController: writing {'domain': 'testing-ss-156.ucsd.edu', 'frontend': '/local/domain/154/device/console/0', 'uuid': '0e439360-1b37-71a0-5826-4b8b0c1d7dc0', 'frontend-id': '154', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/154/0.
[2010-04-26 20:10:32 2359] DEBUG (XendDomainInfo:1891) XendDomainInfo.handleShutdownWatch
[2010-04-26 20:10:33 2359] DEBUG (DevController:139) Waiting for devices tap2.
[2010-04-26 20:10:33 2359] DEBUG (DevController:144) Waiting for 770.
[2010-04-26 20:10:34 2359] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/154/770/hotplug-status.
[2010-04-26 20:10:34 2359] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-26 20:10:34 2359] DEBUG (DevController:139) Waiting for devices vif.
[2010-04-26 20:10:34 2359] DEBUG (DevController:144) Waiting for 0.
[2010-04-26 20:10:34 2359] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/154/0/hotplug-status.
[2010-04-26 20:10:34 2359] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-26 20:10:34 2359] DEBUG (DevController:144) Waiting for 1.
[2010-04-26 20:10:34 2359] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/154/1/hotplug-status.
[2010-04-26 20:10:34 2359] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-26 20:10:34 2359] DEBUG (DevController:139) Waiting for devices vkbd.
[2010-04-26 20:10:34 2359] DEBUG (DevController:139) Waiting for devices ioports.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices tap.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vif2.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices console.
[2010-04-26 20:10:35 2359] DEBUG (DevController:144) Waiting for 0.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vscsi.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vbd.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices irq.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vfb.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices pci.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vusb.
[2010-04-26 20:10:35 2359] DEBUG (DevController:139) Waiting for devices vtpm.
[2010-04-26 20:10:35 2359] INFO (XendDomain:1206) Domain testing-ss-156.ucsd.edu (154) unpaused.


[2010-04-26 20:10:52 2359] DEBUG (XendDomainInfo:101) XendDomainInfo.create(['vm', ['name', 'testing-ss-157.ucsd.edu'], ['memory', '64'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 1], ['oos', 1], ['image', ['linux', ['kernel', '/home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen'], ['ramdisk', '/home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen'], ['root', '/dev/hda2 ro'], ['videoram', 4], ['args', '2 console=xvc0'], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['tap2', ['uname', 'tap:aio:/home/xen/domains/testing-ss-157.ucsd.edu/disk.img'], ['dev', 'hda2'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.200.157'], ['mac', '00:77:22:33:44:9D'], ['bridge', 'testbr']]], ['device', ['vif', ['bridge', 'dummy']]]])
[2010-04-26 20:10:52 2359] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain
[2010-04-26 20:10:52 2359] DEBUG (balloon:220) Balloon: 3957036 KiB free; need 16384; done.
[2010-04-26 20:10:52 2359] DEBUG (XendDomain:464) Adding Domain: 155
[2010-04-26 20:10:52 2359] DEBUG (XendDomainInfo:2818) XendDomainInfo.initDomain: 155 256
[2010-04-26 20:10:52 2359] DEBUG (XendDomainInfo:2845) _initDomain:shadow_memory=0x0, memory_static_max=0x4000000, memory_static_min=0x0.


[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Pasi Kärkkäinen]

On Tue, Apr 27, 2010 at 12:41:07AM -0700, Yuvraj Agarwal wrote:
>    Hi All,
> 
> 
> 
>    We are setting up a system with a large number of very small VMs for a
>    project. We worked through a number of limitations, including those
>    imposed by blktap2 devices and the number of dynamic IRQ (set kernel
>    config NR_CPUS) etc. After these changes we were able to get to  154 domUs
>    (!), but as soon as we start up the 155^th domU the system crashes. We
>    edited each domU config to have one blktap2 device (the disk image) and
>    two virtual network interfaces each.
> 
> 
> 
>    We are using the standard XEN 4.0 (stable/release version)  with the
>    2.6.31.13 (pvops) kernel.
> 
> 
> 
>    Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
>    but as far as we can see we don't quite know what might be going causing
>    the system to crash (no console access anymore and system becomes
>    unresponsive and needs to be power-cycled).  I have pasted only the
>    relevant bits of information (the last domU that did successfully start
>    and the next one that failed). It may be the case that all the log
>    messages weren't flushed before the system crashed...
> 
> 
> 
>    Does anyone know where this limit of 155 domU is coming from and how we
>    can fix/increase it?
> 

Please paste your dom0 grub.conf.
Are you using memory ballooning? 

-- Pasi

>    thanks
> 
>    Yuvraj

> Apr 26 20:08:08  BlackBox tapdisk2[10074]: Created /dev/xen/blktap-2/blktap152 device
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: Created /dev/xen/blktap-2/tapdev152 device
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: new interface: ring: 251, device: 253, minor: 152
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: I/O queue driver: lio
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: block-aio open('/home/xen/domains/testing-ss-155.ucsd.edu/disk.img')
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: open(/home/xen/domains/testing-ss-155.ucsd.edu/disk.img) with O_DIRECT
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: Image size: #012#011pre sector_shift  [1073741824]#012#011post sector_shift [2097152]
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: opened image /home/xen/domains/testing-ss-155.ucsd.edu/disk.img (1 users, state: 0x00000001, type: 0)
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: VBD CHAIN:
> Apr 26 20:08:08  BlackBox tapdisk2[10074]: /home/xen/domains/testing-ss-155.ucsd.edu/disk.img: 0
> Apr 26 20:08:09  BlackBox logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/153/770
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/0
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.0, bridge testbr.
> Apr 26 20:08:10  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/0/hotplug-status connected to xenstore.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/physical-device fd:98 to xenstore.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/hotplug-status connected to xenstore.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/1
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.1, bridge dummy.
> Apr 26 20:08:21  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/1/hotplug-status connected to xenstore.
> 
> 
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: Created /dev/xen/blktap-2/blktap153 device
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: Created /dev/xen/blktap-2/tapdev153 device
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: new interface: ring: 251, device: 253, minor: 153
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: I/O queue driver: lio
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: block-aio open('/home/xen/domains/testing-ss-156.ucsd.edu/disk.img')
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: open(/home/xen/domains/testing-ss-156.ucsd.edu/disk.img) with O_DIRECT
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: Image size: #012#011pre sector_shift  [1073741824]#012#011post sector_shift [2097152]
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: opened image /home/xen/domains/testing-ss-156.ucsd.edu/disk.img (1 users, state: 0x00000001, type: 0)
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: VBD CHAIN:
> Apr 26 20:10:17  BlackBox tapdisk2[12749]: /home/xen/domains/testing-ss-156.ucsd.edu/disk.img: 0
> Apr 26 20:10:17  BlackBox logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/154/770
> Apr 26 20:10:29  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/154/770/physical-device fd:99 to xenstore.
> Apr 26 20:10:29  BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/154/770/hotplug-status connected to xenstore.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/154/0
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif154.0, bridge testbr.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/154/0/hotplug-status connected to xenstore.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/154/1
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif154.1, bridge dummy.
> Apr 26 20:10:30  BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/154/1/hotplug-status connected to xenstore.

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Konrad Rzeszutek Wilk]

On Tue, Apr 27, 2010 at 12:41:07AM -0700, Yuvraj Agarwal wrote:
> Hi All, 
> 
>  
> 
> We are setting up a system with a large number of very small VMs for a
> project. We worked through a number of limitations, including those
> imposed by blktap2 devices and the number of dynamic IRQ (set kernel
> config NR_CPUS) etc. After these changes we were able to get to  154 domUs
> (!), but as soon as we start up the 155th domU the system crashes. We
> edited each domU config to have one blktap2 device (the disk image) and
> two virtual network interfaces each. 

This sounds strangly familiar. I believe somebody posted a question
about this a couple of months ago on the xen-devel mailing list and
found the answer. I think part of it was CONFIG_LEGACY_PTY_COUNT had to
be high. But I don't remember the exact details - you might want to
search the archive.

RE: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

I am not using grub2; using grub-legacy instead. 

Also, once the dom0 boots up I do have to set the dom0 mem to:

(1) xm mem-set 0 12000   (otherwise when starting up a lot of domU it
would run out of memory) 
and 
(2) echo 1548576 > /proc/sys/fs/aio-max-nr 


My /boot/grub/menu.lst  (pasted relevant lines) 
**************************************************

root@MESL-BlackBox:/usr/src# cat /boot/grub/menu.lst                 
# menu.lst - See: grub(8), info grub, update-grub(8)
#            grub-install(8), grub-floppy(8),
#            grub-md5-crypt, /usr/share/doc/grub
#            and /usr/share/doc/grub-doc/.

## default num
# Set the default entry to the entry number NUM. Numbering starts from 0,
and
# the entry number 0 is the default if the command is not used.
#
# You can specify 'saved' instead of a number. In this case, the default
entry
# is the entry saved with the command 'savedefault'.
# WARNING: If you are using dmraid do not use 'savedefault' or your
# array will desync and will not let you boot your system.
default         0

## timeout sec
# Set a timeout, in SEC seconds, before automatically booting the default
entry
# (normally the first entry defined).
timeout         20

### BEGIN AUTOMAGIC KERNELS LIST
## lines between the AUTOMAGIC KERNELS LIST markers will be modified
## by the debian update-grub script except for the default options below

## DO NOT UNCOMMENT THEM, Just edit them to your needs

## ## Start Default Options ##
## default kernel options
## default kernel options for automagic boot options
## If you want special options for specific kernels use kopt_x_y_z
## where x.y.z is kernel version. Minor versions can be omitted.
## e.g. kopt=root=/dev/hda1 ro
##      kopt_2_6_8=root=/dev/hdc1 ro
##      kopt_2_6_8_2_686=root=/dev/hdc2 ro
# kopt=root=UUID=909f7c32-639a-469d-b34b-b418d2b6a2dc ro

## default grub root device
## e.g. groot=(hd0,0)
# groot=909f7c32-639a-469d-b34b-b418d2b6a2dc

## should update-grub create alternative automagic boot options
## e.g. alternative=true
##      alternative=false
# alternative=true

## should update-grub lock alternative automagic boot options
## e.g. lockalternative=true
##      lockalternative=false
# lockalternative=false

## additional options to use with the default boot option, but not with
the
## alternatives
## e.g. defoptions=vga=791 resume=/dev/hda5
# defoptions=quiet splash

## should update-grub lock old automagic boot options
## e.g. lockold=false
##      lockold=true
# lockold=false

## Xen hypervisor options to use with the default Xen boot option
# xenhopt=dom0_max_vcpus=1 dom0_mem=8192

## Xen Linux kernel options to use with the default Xen boot option
# xenkopt=console=tty0

## altoption boot targets option
## multiple altoptions lines are allowed
## e.g. altoptions=(extra menu suffix) extra boot options
##      altoptions=(recovery) single
# altoptions=(recovery mode) single

## controls how many kernels should be put into the menu.lst
## only counts the first occurence of a kernel, not the
## alternative kernel options
## e.g. howmany=all
##      howmany=7
# howmany=all

## specify if running in Xen domU or have grub detect automatically
## update-grub will ignore non-xen kernels when running in domU and vice
versa
## e.g. indomU=detect
##      indomU=true
##      indomU=false
# indomU=detect

## should update-grub create memtest86 boot option
## e.g. memtest86=true
##      memtest86=false
# memtest86=true

## should update-grub adjust the value of the default booted system
## can be true or false
# updatedefaultentry=false

## should update-grub add savedefault to the default options
## can be true or false
# savedefault=false

## ## End Default Options ##

title           Xen 4.0.0 / Debian GNU/Linux, kernel 2.6.31.13
root            (hd0,0)
kernel          /boot/xen-4.0.0.gz 
module          /boot/vmlinuz-2.6.31.13 root=/dev/sda1 ro 

title           Xen 3.4.2 / Debian GNU/Linux, kernel 2.6.31.8-xenapr2010
root            (hd0,0)
kernel          /boot/xen-3.4.2.gz 
module          /boot/vmlinuz-2.6.31.8-xenapr2010 root=/dev/sda1 ro
console=tty0 
module          /boot/initrd.img-2.6.31.8-xenapr2010

### END DEBIAN AUTOMAGIC KERNELS LIST


Please paste your dom0 grub.conf.
Are you using memory ballooning? 

-- Pasi

RE: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

Thank you for the pointer. Unfortunately, I was unable to find the message
you mentioned after looking through the archives...  

In my current kernel config (default 2.6.31.13 pvops kernel) I don't have
CONFIG_LEGACY_PTY_COUNT defined, perhaps because I have the following in
the .config ?

# CONFIG_LEGACY_PTYS is not set

Do you think setting the variable above and setting the
CONFIG_LEGACY_PTY_COUNT to a high value will help in this case?

--Yuvraj


-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
Sent: Tuesday, April 27, 2010 6:59 AM
To: Yuvraj Agarwal
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
on starting 155th domU

On Tue, Apr 27, 2010 at 12:41:07AM -0700, Yuvraj Agarwal wrote:
> Hi All, 
> 
>  
> 
> We are setting up a system with a large number of very small VMs for a
> project. We worked through a number of limitations, including those
> imposed by blktap2 devices and the number of dynamic IRQ (set kernel
> config NR_CPUS) etc. After these changes we were able to get to  154
domUs
> (!), but as soon as we start up the 155th domU the system crashes. We
> edited each domU config to have one blktap2 device (the disk image) and
> two virtual network interfaces each. 

This sounds strangly familiar. I believe somebody posted a question
about this a couple of months ago on the xen-devel mailing list and
found the answer. I think part of it was CONFIG_LEGACY_PTY_COUNT had to
be high. But I don't remember the exact details - you might want to
search the archive.

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Pasi Kärkkäinen]

On Tue, Apr 27, 2010 at 10:14:40AM -0700, Yuvraj Agarwal wrote:
> I am not using grub2; using grub-legacy instead. 
> 
> Also, once the dom0 boots up I do have to set the dom0 mem to:
> 
> (1) xm mem-set 0 12000   (otherwise when starting up a lot of domU it
> would run out of memory) 
> and 
> (2) echo 1548576 > /proc/sys/fs/aio-max-nr 
> 

You might want to use dom0_mem=<X>M option for xen.gz instead.

See:
http://wiki.xensource.com/xenwiki/XenBestPractices

-- Pasi

> 
> My /boot/grub/menu.lst  (pasted relevant lines) 
> **************************************************
> 
> root@MESL-BlackBox:/usr/src# cat /boot/grub/menu.lst                 
> # menu.lst - See: grub(8), info grub, update-grub(8)
> #            grub-install(8), grub-floppy(8),
> #            grub-md5-crypt, /usr/share/doc/grub
> #            and /usr/share/doc/grub-doc/.
> 
> ## default num
> # Set the default entry to the entry number NUM. Numbering starts from 0,
> and
> # the entry number 0 is the default if the command is not used.
> #
> # You can specify 'saved' instead of a number. In this case, the default
> entry
> # is the entry saved with the command 'savedefault'.
> # WARNING: If you are using dmraid do not use 'savedefault' or your
> # array will desync and will not let you boot your system.
> default         0
> 
> ## timeout sec
> # Set a timeout, in SEC seconds, before automatically booting the default
> entry
> # (normally the first entry defined).
> timeout         20
> 
> ### BEGIN AUTOMAGIC KERNELS LIST
> ## lines between the AUTOMAGIC KERNELS LIST markers will be modified
> ## by the debian update-grub script except for the default options below
> 
> ## DO NOT UNCOMMENT THEM, Just edit them to your needs
> 
> ## ## Start Default Options ##
> ## default kernel options
> ## default kernel options for automagic boot options
> ## If you want special options for specific kernels use kopt_x_y_z
> ## where x.y.z is kernel version. Minor versions can be omitted.
> ## e.g. kopt=root=/dev/hda1 ro
> ##      kopt_2_6_8=root=/dev/hdc1 ro
> ##      kopt_2_6_8_2_686=root=/dev/hdc2 ro
> # kopt=root=UUID=909f7c32-639a-469d-b34b-b418d2b6a2dc ro
> 
> ## default grub root device
> ## e.g. groot=(hd0,0)
> # groot=909f7c32-639a-469d-b34b-b418d2b6a2dc
> 
> ## should update-grub create alternative automagic boot options
> ## e.g. alternative=true
> ##      alternative=false
> # alternative=true
> 
> ## should update-grub lock alternative automagic boot options
> ## e.g. lockalternative=true
> ##      lockalternative=false
> # lockalternative=false
> 
> ## additional options to use with the default boot option, but not with
> the
> ## alternatives
> ## e.g. defoptions=vga=791 resume=/dev/hda5
> # defoptions=quiet splash
> 
> ## should update-grub lock old automagic boot options
> ## e.g. lockold=false
> ##      lockold=true
> # lockold=false
> 
> ## Xen hypervisor options to use with the default Xen boot option
> # xenhopt=dom0_max_vcpus=1 dom0_mem=8192
> 
> ## Xen Linux kernel options to use with the default Xen boot option
> # xenkopt=console=tty0
> 
> ## altoption boot targets option
> ## multiple altoptions lines are allowed
> ## e.g. altoptions=(extra menu suffix) extra boot options
> ##      altoptions=(recovery) single
> # altoptions=(recovery mode) single
> 
> ## controls how many kernels should be put into the menu.lst
> ## only counts the first occurence of a kernel, not the
> ## alternative kernel options
> ## e.g. howmany=all
> ##      howmany=7
> # howmany=all
> 
> ## specify if running in Xen domU or have grub detect automatically
> ## update-grub will ignore non-xen kernels when running in domU and vice
> versa
> ## e.g. indomU=detect
> ##      indomU=true
> ##      indomU=false
> # indomU=detect
> 
> ## should update-grub create memtest86 boot option
> ## e.g. memtest86=true
> ##      memtest86=false
> # memtest86=true
> 
> ## should update-grub adjust the value of the default booted system
> ## can be true or false
> # updatedefaultentry=false
> 
> ## should update-grub add savedefault to the default options
> ## can be true or false
> # savedefault=false
> 
> ## ## End Default Options ##
> 
> title           Xen 4.0.0 / Debian GNU/Linux, kernel 2.6.31.13
> root            (hd0,0)
> kernel          /boot/xen-4.0.0.gz 
> module          /boot/vmlinuz-2.6.31.13 root=/dev/sda1 ro 
> 
> title           Xen 3.4.2 / Debian GNU/Linux, kernel 2.6.31.8-xenapr2010
> root            (hd0,0)
> kernel          /boot/xen-3.4.2.gz 
> module          /boot/vmlinuz-2.6.31.8-xenapr2010 root=/dev/sda1 ro
> console=tty0 
> module          /boot/initrd.img-2.6.31.8-xenapr2010
> 
> ### END DEBIAN AUTOMAGIC KERNELS LIST
> 
> 
> Please paste your dom0 grub.conf.
> Are you using memory ballooning? 
> 
> -- Pasi
> 

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Jeremy Fitzhardinge]

On 04/27/2010 12:41 AM, Yuvraj Agarwal wrote:
>
> Hi All,
>
>  
>
> We are setting up a system with a large number of very small VMs for a
> project. We worked through a number of limitations, including those
> imposed by blktap2 devices and the number of dynamic IRQ (set kernel
> config NR_CPUS) etc. After these changes we were able to get to  154
> domUs (!), but as soon as we start up the 155^th domU the system crashes.
>

How does the system crash?  You mean the dom0 kernel crashes?

> Does anyone know where this limit of 155 domU is coming from and how
> we can fix/increase it?
>

What does /proc/interrupts look like before the crash?  How many network
and block devices do you have in dom0?

    J

RE: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

[-- Attachment #1: Type: text/plain, Size: 6431 bytes --]

I did make that change (dom0_mem=8192M), but I still get the same error.
When I started the 154th domU (last time it was 155th) the dom0 crashes
(it kills network connections and I have to manually go and reboot it). I
did get a little bit more information on xend.log and daemon.log
(attached).  After the last successful VM startup I did check and make
sure the dom0 did indeed have enough memory.

root@MESL-BlackBox:/home/xen/noswap-configs# xm list | grep testing | wc
-l
153
root@MESL-BlackBox:/home/xen/noswap-configs# xm info | grep mem

total_memory           : 24490
free_memory            : 9254
node_to_memory         : node0:2076
node_to_dma32_mem      : node0:2076
xen_commandline        : dom0_mem=8192M
root@MESL-BlackBox:/home/xen/noswap-configs# xm create
testing-ss-157.ucsd.edu
Using config file "./testing-ss-157.ucsd.edu".
Started domain testing-ss-157.ucsd.edu (id=154)

/var/log/daemon.log  --> daemon.log <attached>
/var/log/xend.log    --> xend.log   <attached>

We'd appreciate any pointers to fix this...

Thank you
--Yuvraj

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Pasi
Kärkkäinen
Sent: Tuesday, April 27, 2010 10:19 AM
To: Yuvraj Agarwal
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
on starting 155th domU

On Tue, Apr 27, 2010 at 10:14:40AM -0700, Yuvraj Agarwal wrote:
> I am not using grub2; using grub-legacy instead.
>
> Also, once the dom0 boots up I do have to set the dom0 mem to:
>
> (1) xm mem-set 0 12000   (otherwise when starting up a lot of domU it
> would run out of memory)
> and
> (2) echo 1548576 > /proc/sys/fs/aio-max-nr
>

You might want to use dom0_mem=<X>M option for xen.gz instead.

See:
http://wiki.xensource.com/xenwiki/XenBestPractices

-- Pasi

>
> My /boot/grub/menu.lst  (pasted relevant lines)
> **************************************************
>
> root@MESL-BlackBox:/usr/src# cat /boot/grub/menu.lst
> # menu.lst - See: grub(8), info grub, update-grub(8)
> #            grub-install(8), grub-floppy(8),
> #            grub-md5-crypt, /usr/share/doc/grub
> #            and /usr/share/doc/grub-doc/.
>
> ## default num
> # Set the default entry to the entry number NUM. Numbering starts from
0,
> and
> # the entry number 0 is the default if the command is not used.
> #
> # You can specify 'saved' instead of a number. In this case, the default
> entry
> # is the entry saved with the command 'savedefault'.
> # WARNING: If you are using dmraid do not use 'savedefault' or your
> # array will desync and will not let you boot your system.
> default         0
>
> ## timeout sec
> # Set a timeout, in SEC seconds, before automatically booting the
default
> entry
> # (normally the first entry defined).
> timeout         20
>
> ### BEGIN AUTOMAGIC KERNELS LIST
> ## lines between the AUTOMAGIC KERNELS LIST markers will be modified
> ## by the debian update-grub script except for the default options below
>
> ## DO NOT UNCOMMENT THEM, Just edit them to your needs
>
> ## ## Start Default Options ##
> ## default kernel options
> ## default kernel options for automagic boot options
> ## If you want special options for specific kernels use kopt_x_y_z
> ## where x.y.z is kernel version. Minor versions can be omitted.
> ## e.g. kopt=root=/dev/hda1 ro
> ##      kopt_2_6_8=root=/dev/hdc1 ro
> ##      kopt_2_6_8_2_686=root=/dev/hdc2 ro
> # kopt=root=UUID=909f7c32-639a-469d-b34b-b418d2b6a2dc ro
>
> ## default grub root device
> ## e.g. groot=(hd0,0)
> # groot=909f7c32-639a-469d-b34b-b418d2b6a2dc
>
> ## should update-grub create alternative automagic boot options
> ## e.g. alternative=true
> ##      alternative=false
> # alternative=true
>
> ## should update-grub lock alternative automagic boot options
> ## e.g. lockalternative=true
> ##      lockalternative=false
> # lockalternative=false
>
> ## additional options to use with the default boot option, but not with
> the
> ## alternatives
> ## e.g. defoptions=vga=791 resume=/dev/hda5
> # defoptions=quiet splash
>
> ## should update-grub lock old automagic boot options
> ## e.g. lockold=false
> ##      lockold=true
> # lockold=false
>
> ## Xen hypervisor options to use with the default Xen boot option
> # xenhopt=dom0_max_vcpus=1 dom0_mem=8192
>
> ## Xen Linux kernel options to use with the default Xen boot option
> # xenkopt=console=tty0
>
> ## altoption boot targets option
> ## multiple altoptions lines are allowed
> ## e.g. altoptions=(extra menu suffix) extra boot options
> ##      altoptions=(recovery) single
> # altoptions=(recovery mode) single
>
> ## controls how many kernels should be put into the menu.lst
> ## only counts the first occurence of a kernel, not the
> ## alternative kernel options
> ## e.g. howmany=all
> ##      howmany=7
> # howmany=all
>
> ## specify if running in Xen domU or have grub detect automatically
> ## update-grub will ignore non-xen kernels when running in domU and vice
> versa
> ## e.g. indomU=detect
> ##      indomU=true
> ##      indomU=false
> # indomU=detect
>
> ## should update-grub create memtest86 boot option
> ## e.g. memtest86=true
> ##      memtest86=false
> # memtest86=true
>
> ## should update-grub adjust the value of the default booted system
> ## can be true or false
> # updatedefaultentry=false
>
> ## should update-grub add savedefault to the default options
> ## can be true or false
> # savedefault=false
>
> ## ## End Default Options ##
>
> title           Xen 4.0.0 / Debian GNU/Linux, kernel 2.6.31.13
> root            (hd0,0)
> kernel          /boot/xen-4.0.0.gz
> module          /boot/vmlinuz-2.6.31.13 root=/dev/sda1 ro
>
> title           Xen 3.4.2 / Debian GNU/Linux, kernel 2.6.31.8-xenapr2010
> root            (hd0,0)
> kernel          /boot/xen-3.4.2.gz
> module          /boot/vmlinuz-2.6.31.8-xenapr2010 root=/dev/sda1 ro
> console=tty0
> module          /boot/initrd.img-2.6.31.8-xenapr2010
>
> ### END DEBIAN AUTOMAGIC KERNELS LIST
>
>
> Please paste your dom0 grub.conf.
> Are you using memory ballooning?
>
> -- Pasi
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #2: daemon.log --]
[-- Type: application/octet-stream, Size: 6057 bytes --]

Apr 27 11:46:10 BlackBox tapdisk2[11178]: Created /dev/xen/blktap-2/blktap152 device
Apr 27 11:46:10 BlackBox tapdisk2[11178]: Created /dev/xen/blktap-2/tapdev152 device
Apr 27 11:46:10 BlackBox tapdisk2[11178]: new interface: ring: 251, device: 253, minor: 152
Apr 27 11:46:10 BlackBox tapdisk2[11178]: I/O queue driver: lio
Apr 27 11:46:10 BlackBox tapdisk2[11178]: block-aio open('/home/xen/domains/testing-ss-156.ucsd.edu/disk.img')
Apr 27 11:46:10 BlackBox tapdisk2[11178]: open(/home/xen/domains/testing-ss-156.ucsd.edu/disk.img) with O_DIRECT
Apr 27 11:46:10 BlackBox tapdisk2[11178]: Image size: #012#011pre sector_shift  [1073741824]#012#011post sector_shift [2097152]
Apr 27 11:46:10 BlackBox tapdisk2[11178]: opened image /home/xen/domains/testing-ss-156.ucsd.edu/disk.img (1 users, state: 0x00000001, type: 0)
Apr 27 11:46:10 BlackBox tapdisk2[11178]: VBD CHAIN:
Apr 27 11:46:10 BlackBox tapdisk2[11178]: /home/xen/domains/testing-ss-156.ucsd.edu/disk.img: 0
Apr 27 11:46:10 BlackBox logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/153/770
Apr 27 11:46:22 BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/physical-device fd:98 to xenstore.
Apr 27 11:46:22 BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/hotplug-status connected to xenstore.
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/0
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.0, bridge testbr.
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/0/hotplug-status connected to xenstore.
Apr 27 11:46:24 BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/1
Apr 27 11:46:24 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:46:24 BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.1, bridge dummy.
Apr 27 11:46:24 BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/1/hotplug-status connected to xenstore.



^C
root@BlackBox:/home/yuvraj# tail -f /var/log/daemon.log
root@BlackBox:/home/yuvraj# tail -f /var/log/daemon.log
Apr 27 11:46:22 BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/153/770/hotplug-status connected to xenstore.
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/0
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.0, bridge testbr.
Apr 27 11:46:23 BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/0/hotplug-status connected to xenstore.
Apr 27 11:46:24 BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/153/1
Apr 27 11:46:24 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:46:24 BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif153.1, bridge dummy.
Apr 27 11:46:24 BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/153/1/hotplug-status connected to xenstore.


Apr 27 11:48:25 BlackBox tapdisk2[11975]: Created /dev/xen/blktap-2/blktap153 device
Apr 27 11:48:25 BlackBox tapdisk2[11975]: Created /dev/xen/blktap-2/tapdev153 device
Apr 27 11:48:25 BlackBox tapdisk2[11975]: new interface: ring: 251, device: 253, minor: 153
Apr 27 11:48:25 BlackBox tapdisk2[11975]: I/O queue driver: lio
Apr 27 11:48:25 BlackBox tapdisk2[11975]: block-aio open('/home/xen/domains/testing-ss-157.ucsd.edu/disk.img')
Apr 27 11:48:25 BlackBox tapdisk2[11975]: open(/home/xen/domains/testing-ss-157.ucsd.edu/disk.img) with O_DIRECT
Apr 27 11:48:25 BlackBox tapdisk2[11975]: Image size: #012#011pre sector_shift  [1073741824]#012#011post sector_shift [2097152]
Apr 27 11:48:25 BlackBox tapdisk2[11975]: opened image /home/xen/domains/testing-ss-157.ucsd.edu/disk.img (1 users, state: 0x00000001, type: 0)
Apr 27 11:48:25 BlackBox tapdisk2[11975]: VBD CHAIN:
Apr 27 11:48:25 BlackBox tapdisk2[11975]: /home/xen/domains/testing-ss-157.ucsd.edu/disk.img: 0
Apr 27 11:48:26 BlackBox logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/154/770
Apr 27 11:48:26 BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/154/0
Apr 27 11:48:26 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:48:26 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:48:26 BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif154.0, bridge testbr.
Apr 27 11:48:26 BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/154/0/hotplug-status connected to xenstore.
Apr 27 11:48:27 BlackBox logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/154/1
Apr 27 11:48:27 BlackBox logger: /etc/xen/scripts/vif-bridge: iptables setup failed. This may affect guest networking.
Apr 27 11:48:27 BlackBox logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif154.1, bridge dummy.
Apr 27 11:48:27 BlackBox logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/154/1/hotplug-status connected to xenstore.
Apr 27 11:48:39 BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/154/770/physical-device fd:99 to xenstore.
Apr 27 11:48:39 BlackBox logger: /etc/xen/scripts/block: Writing backend/vbd/154/770/hotplug-status connected to xenstore.

[-- Attachment #3: xend.log --]
[-- Type: application/octet-stream, Size: 10452 bytes --]

root@MESL-BlackBox:/home/yuvraj# tail -f /var/log/xen/xend.log
[2010-04-27 11:46:29 2360] DEBUG (DevController:139) Waiting for devices console.
[2010-04-27 11:46:29 2360] DEBUG (DevController:144) Waiting for 0.
[2010-04-27 11:46:29 2360] DEBUG (DevController:139) Waiting for devices vscsi.
[2010-04-27 11:46:29 2360] DEBUG (DevController:139) Waiting for devices vbd.
[2010-04-27 11:46:29 2360] DEBUG (DevController:139) Waiting for devices irq.
[2010-04-27 11:46:29 2360] DEBUG (DevController:139) Waiting for devices vfb.
[2010-04-27 11:46:29 2360] DEBUG (DevController:139) Waiting for devices pci.
[2010-04-27 11:46:29 2360] DEBUG (DevController:139) Waiting for devices vusb.
[2010-04-27 11:46:30 2360] DEBUG (DevController:139) Waiting for devices vtpm.
[2010-04-27 11:46:30 2360] INFO (XendDomain:1206) Domain testing-ss-156.ucsd.edu (153) unpaused.

[2010-04-27 11:48:24 2360] DEBUG (XendDomainInfo:101) XendDomainInfo.create(['vm', ['name', 'testing-ss-157.ucsd.edu'], ['memory', '64'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 1], ['oos', 1], ['image', ['linux', ['kernel', '/home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen'], ['ramdisk', '/home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen'], ['root', '/dev/hda2 ro'], ['videoram', 4], ['args', '2 console=xvc0'], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['tap2', ['uname', 'tap:aio:/home/xen/domains/testing-ss-157.ucsd.edu/disk.img'], ['dev', 'hda2'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.200.157'], ['mac', '00:77:22:33:44:9D'], ['bridge', 'testbr']]], ['device', ['vif', ['bridge', 'dummy']]]])
[2010-04-27 11:48:24 2360] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain
[2010-04-27 11:48:24 2360] DEBUG (balloon:220) Balloon: 9477100 KiB free; need 16384; done.
[2010-04-27 11:48:24 2360] DEBUG (XendDomain:464) Adding Domain: 154
[2010-04-27 11:48:24 2360] DEBUG (XendDomainInfo:2818) XendDomainInfo.initDomain: 154 256
[2010-04-27 11:48:25 2360] DEBUG (XendDomainInfo:2845) _initDomain:shadow_memory=0x0, memory_static_max=0x4000000, memory_static_min=0x0.
[2010-04-27 11:48:25 2360] INFO (image:182) buildDomain os=linux dom=154 vcpus=1
[2010-04-27 11:48:25 2360] DEBUG (image:721) domid          = 154
[2010-04-27 11:48:25 2360] DEBUG (image:722) memsize        = 64
[2010-04-27 11:48:25 2360] DEBUG (image:723) image          = /home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen
[2010-04-27 11:48:25 2360] DEBUG (image:724) store_evtchn   = 1
[2010-04-27 11:48:25 2360] DEBUG (image:725) console_evtchn = 2
[2010-04-27 11:48:25 2360] DEBUG (image:726) cmdline        = root=/dev/hda2 ro 2 console=xvc0
[2010-04-27 11:48:25 2360] DEBUG (image:727) ramdisk        = /home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen
[2010-04-27 11:48:25 2360] DEBUG (image:728) vcpus          = 1
[2010-04-27 11:48:25 2360] DEBUG (image:729) features       = 
[2010-04-27 11:48:25 2360] DEBUG (image:730) flags          = 0
[2010-04-27 11:48:25 2360] DEBUG (image:731) superpages     = 0
[2010-04-27 11:48:25 2360] INFO (XendDomainInfo:2367) createDevice: tap2 : {'bootable': 1, 'uname': 'tap:aio:/home/xen/domains/testing-ss-157.ucsd.edu/disk.img', 'mode': 'w', 'dev': 'hda2', 'uuid': '7d5b73e7-fefc-8156-140b-3e6def7fe01e'}
[2010-04-27 11:48:25 2360] DEBUG (DevController:95) DevController: writing {'virtual-device': '770', 'device-type': 'disk', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/154/770'} to /local/domain/154/device/vbd/770.
[2010-04-27 11:48:25 2360] DEBUG (DevController:97) DevController: writing {'domain': 'testing-ss-157.ucsd.edu', 'frontend': '/local/domain/154/device/vbd/770', 'uuid': '7d5b73e7-fefc-8156-140b-3e6def7fe01e', 'bootable': '1', 'dev': 'hda2', 'state': '1', 'params': '/dev/xen/blktap-2/tapdev153', 'mode': 'w', 'online': '1', 'frontend-id': '154', 'type': 'phy', 'tapdisk-params': 'aio:/home/xen/domains/testing-ss-157.ucsd.edu/disk.img'} to /local/domain/0/backend/vbd/154/770.
[2010-04-27 11:48:25 2360] INFO (XendDomainInfo:2367) createDevice: vif : {'ip': '192.168.200.157', 'mac': '00:77:22:33:44:9D', 'uuid': '3fe1879a-dce1-2ea5-dad6-3ef1d27b4866', 'bridge': 'testbr'}
[2010-04-27 11:48:26 2360] DEBUG (DevController:95) DevController: writing {'mac': '00:77:22:33:44:9D', 'handle': '0', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/154/0'} to /local/domain/154/device/vif/0.
[2010-04-27 11:48:26 2360] DEBUG (DevController:97) DevController: writing {'bridge': 'testbr', 'domain': 'testing-ss-157.ucsd.edu', 'handle': '0', 'uuid': '3fe1879a-dce1-2ea5-dad6-3ef1d27b4866', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '192.168.200.157', 'mac': '00:77:22:33:44:9D', 'frontend-id': '154', 'state': '1', 'online': '1', 'frontend': '/local/domain/154/device/vif/0'} to /local/domain/0/backend/vif/154/0.
[2010-04-27 11:48:26 2360] INFO (XendDomainInfo:2367) createDevice: vif : {'bridge': 'dummy', 'mac': '00:16:3e:4d:0f:6e', 'uuid': '5afdbbdd-b25d-19f5-14ff-0357d96dfae9'}
[2010-04-27 11:48:26 2360] DEBUG (DevController:95) DevController: writing {'mac': '00:16:3e:4d:0f:6e', 'handle': '1', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/154/1'} to /local/domain/154/device/vif/1.
[2010-04-27 11:48:26 2360] DEBUG (DevController:97) DevController: writing {'bridge': 'dummy', 'domain': 'testing-ss-157.ucsd.edu', 'handle': '1', 'uuid': '5afdbbdd-b25d-19f5-14ff-0357d96dfae9', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3e:4d:0f:6e', 'frontend-id': '154', 'state': '1', 'online': '1', 'frontend': '/local/domain/154/device/vif/1'} to /local/domain/0/backend/vif/154/1.
[2010-04-27 11:48:26 2360] DEBUG (XendDomainInfo:3400) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'b396dd83-c66a-e224-e4b8-91684fe1b964', 'on_reboot': 'restart', 'start_time': '1272394106.84', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '', 'image': "(linux (kernel /home/xen/xen-install/x32bit-fromkapaki/vmlinuz-2.6.24-21-xen) (ramdisk /home/xen/xen-install/x32bit-fromkapaki/initrd.img-2.6.24-21-xen) (args 'root=/dev/hda2 ro 2 console=xvc0') (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (notes (HV_START_LOW 4118806528) (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 3221225472) (GUEST_OS linux) (HYPERCALL_PAGE 3222278144) (LOADER generic) (SUSPEND_CANCEL 1) (PAE_MODE yes) (ENTRY 3222274048) (XEN_VERSION xen-3.0)))", 'name': 'testing-ss-157.ucsd.edu'}
[2010-04-27 11:48:39 2360] DEBUG (XendDomainInfo:1804) Storing domain details: {'console/ring-ref': '5435922', 'image/entry': '3222274048', 'console/port': '2', 'store/ring-ref': '5435923', 'image/loader': 'generic', 'vm': '/vm/b396dd83-c66a-e224-e4b8-91684fe1b964', 'control/platform-feature-multiprocessor-suspend': '1', 'image/hv-start-low': '4118806528', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '3221225472', 'memory/target': '65536', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576', 'image/paddr-offset': '3221225472', 'image/hypercall-page': '3222278144', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'console/type': 'xenconsoled', 'image/features/auto-translated-physmap': '1', 'name': 'testing-ss-157.ucsd.edu', 'domid': '154', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2010-04-27 11:48:41 2360] DEBUG (DevController:95) DevController: writing {'protocol': 'x86_32-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/154/0'} to /local/domain/154/device/console/0.
[2010-04-27 11:48:41 2360] DEBUG (DevController:97) DevController: writing {'domain': 'testing-ss-157.ucsd.edu', 'frontend': '/local/domain/154/device/console/0', 'uuid': '7026bbc4-841f-dc6a-e865-431ced12d8a8', 'frontend-id': '154', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/154/0.
[2010-04-27 11:48:42 2360] DEBUG (XendDomainInfo:1891) XendDomainInfo.handleShutdownWatch
[2010-04-27 11:48:42 2360] DEBUG (DevController:139) Waiting for devices tap2.
[2010-04-27 11:48:42 2360] DEBUG (DevController:144) Waiting for 770.
[2010-04-27 11:48:43 2360] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vbd/154/770/hotplug-status.
[2010-04-27 11:48:43 2360] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-27 11:48:43 2360] DEBUG (DevController:139) Waiting for devices vif.
[2010-04-27 11:48:43 2360] DEBUG (DevController:144) Waiting for 0.
[2010-04-27 11:48:43 2360] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/154/0/hotplug-status.
[2010-04-27 11:48:43 2360] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-27 11:48:43 2360] DEBUG (DevController:144) Waiting for 1.
[2010-04-27 11:48:43 2360] DEBUG (DevController:628) hotplugStatusCallback /local/domain/0/backend/vif/154/1/hotplug-status.
[2010-04-27 11:48:43 2360] DEBUG (DevController:642) hotplugStatusCallback 1.
[2010-04-27 11:48:43 2360] DEBUG (DevController:139) Waiting for devices vkbd.
[2010-04-27 11:48:43 2360] DEBUG (DevController:139) Waiting for devices ioports.
[2010-04-27 11:48:44 2360] DEBUG (DevController:139) Waiting for devices tap.
[2010-04-27 11:48:44 2360] DEBUG (DevController:139) Waiting for devices vif2.
[2010-04-27 11:48:44 2360] DEBUG (DevController:139) Waiting for devices console.
[2010-04-27 11:48:44 2360] DEBUG (DevController:144) Waiting for 0.
[2010-04-27 11:48:44 2360] DEBUG (DevController:139) Waiting for devices vscsi.
[2010-04-27 11:48:44 2360] DEBUG (DevController:139) Waiting for devices vbd.
[2010-04-27 11:48:44 2360] DEBUG (DevController:139) Waiting for devices irq.
[2010-04-27 11:48:44 2360] DEBUG (DevController:139) Waiting for devices vfb.

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

Dom0 crashes (the network connections are all killed, and I cant log in
locally on the machine anymore either). 
I'll start up the 154 domU again and report back  /proc/interrupts and the
number of blktap2 devices. I do recall that when I checked last (cat
/sys/class/blktap2/*/name | wc -l ) I used to get 1 blktap2 device per
domU (which means that I had 154 blktap devices). Each domU has two
network interfaces.  I'll report back what we find when we start up all
the domUs again.  

On another related note it takes a LONG time to start up all these 150
domains (>20-30mins), and I believe the culprit is the xenstored since it
has to write the entire xenstore for each domU. We tried to edit to build
the OCAML version instead which is supposed to be faster (?), but
xenstored did not start in that case. We edited the Config.mk file to

CONFIG_OCAML_XENSTORED ?= y 

We did clean out the "dist" directory and rebuild and reinstall xen, but
xenstored did not start up. Are we missing obvious?

--Yuvraj 

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jeremy
Fitzhardinge
Sent: Tuesday, April 27, 2010 11:52 AM
To: Yuvraj Agarwal
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
on starting 155th domU

On 04/27/2010 12:41 AM, Yuvraj Agarwal wrote:
>
> Hi All,
>
>  
>
> We are setting up a system with a large number of very small VMs for a
> project. We worked through a number of limitations, including those
> imposed by blktap2 devices and the number of dynamic IRQ (set kernel
> config NR_CPUS) etc. After these changes we were able to get to  154
> domUs (!), but as soon as we start up the 155^th domU the system
crashes.
>

How does the system crash?  You mean the dom0 kernel crashes?

> Does anyone know where this limit of 155 domU is coming from and how
> we can fix/increase it?
>

What does /proc/interrupts look like before the crash?  How many network
and block devices do you have in dom0?

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Jeremy Fitzhardinge]

On 04/27/2010 12:10 PM, Yuvraj Agarwal wrote:
> Dom0 crashes (the network connections are all killed, and I cant log in
> locally on the machine anymore either). 
>   

Does it display anything on the console?  Do you have a serial console
setup?

> On another related note it takes a LONG time to start up all these 150
> domains (>20-30mins), and I believe the culprit is the xenstored since it
> has to write the entire xenstore for each domU.

That should be fairly easy to verify with "top" in dom0.

>  We tried to edit to build
> the OCAML version instead which is supposed to be faster (?), but
> xenstored did not start in that case. We edited the Config.mk file to
>
> CONFIG_OCAML_XENSTORED ?= y 
>
> We did clean out the "dist" directory and rebuild and reinstall xen, but
> xenstored did not start up. Are we missing obvious?
>   

What happens if you manually start xenstored (before xend)?  Does it
complain, crash, something else?

I think this is a very under-tested configuration, so it wouldn't
surprise me if there are problems with it.

    J

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Pasi Kärkkäinen]

On Tue, Apr 27, 2010 at 11:58:42AM -0700, Yuvraj Agarwal wrote:
> I did make that change (dom0_mem=8192M), but I still get the same error.
> When I started the 154th domU (last time it was 155th) the dom0 crashes
> (it kills network connections and I have to manually go and reboot it). 
>

Hmm.. interesting. It should never crash.. 

Do you have a serial console so you could capture the error/crash messages? 
See: http://wiki.xensource.com/xenwiki/XenSerialConsole

> I did get a little bit more information on xend.log and daemon.log
> (attached).  After the last successful VM startup I did check and make
> sure the dom0 did indeed have enough memory.
> 
> root@MESL-BlackBox:/home/xen/noswap-configs# xm list | grep testing | wc
> -l
> 153
> root@MESL-BlackBox:/home/xen/noswap-configs# xm info | grep mem
> 
> total_memory           : 24490
> free_memory            : 9254

Ok so over 9 GB of free memory in the hypervisor.
How's the memory in dom0? It still has 8 GB and most of it is free?

> node_to_memory         : node0:2076
> node_to_dma32_mem      : node0:2076
> xen_commandline        : dom0_mem=8192M
> root@MESL-BlackBox:/home/xen/noswap-configs# xm create
> testing-ss-157.ucsd.edu
> Using config file "./testing-ss-157.ucsd.edu".
> Started domain testing-ss-157.ucsd.edu (id=154)
> 

How much memory does that ss-157 domU have configured? 

> /var/log/daemon.log  --> daemon.log <attached>
> /var/log/xend.log    --> xend.log   <attached>
> 
> We'd appreciate any pointers to fix this...
> 

Did you disable dom0 ballooning from xend-config.sxp? 
Did you make sure dom0 is NOT ballooned? Or does this crash happen when dom0 is ballooned? 

Please capture Xen and dom0 messages when it crashes..

-- Pasi

> Thank you
> --Yuvraj
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Pasi
> Kärkkäinen
> Sent: Tuesday, April 27, 2010 10:19 AM
> To: Yuvraj Agarwal
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
> on starting 155th domU
> 
> On Tue, Apr 27, 2010 at 10:14:40AM -0700, Yuvraj Agarwal wrote:
> > I am not using grub2; using grub-legacy instead.
> >
> > Also, once the dom0 boots up I do have to set the dom0 mem to:
> >
> > (1) xm mem-set 0 12000   (otherwise when starting up a lot of domU it
> > would run out of memory)
> > and
> > (2) echo 1548576 > /proc/sys/fs/aio-max-nr
> >
> 
> You might want to use dom0_mem=<X>M option for xen.gz instead.
> 
> See:
> http://wiki.xensource.com/xenwiki/XenBestPractices
> 
> -- Pasi
> 
> >
> > My /boot/grub/menu.lst  (pasted relevant lines)
> > **************************************************
> >
> > root@MESL-BlackBox:/usr/src# cat /boot/grub/menu.lst
> > # menu.lst - See: grub(8), info grub, update-grub(8)
> > #            grub-install(8), grub-floppy(8),
> > #            grub-md5-crypt, /usr/share/doc/grub
> > #            and /usr/share/doc/grub-doc/.
> >
> > ## default num
> > # Set the default entry to the entry number NUM. Numbering starts from
> 0,
> > and
> > # the entry number 0 is the default if the command is not used.
> > #
> > # You can specify 'saved' instead of a number. In this case, the default
> > entry
> > # is the entry saved with the command 'savedefault'.
> > # WARNING: If you are using dmraid do not use 'savedefault' or your
> > # array will desync and will not let you boot your system.
> > default         0
> >
> > ## timeout sec
> > # Set a timeout, in SEC seconds, before automatically booting the
> default
> > entry
> > # (normally the first entry defined).
> > timeout         20
> >
> > ### BEGIN AUTOMAGIC KERNELS LIST
> > ## lines between the AUTOMAGIC KERNELS LIST markers will be modified
> > ## by the debian update-grub script except for the default options below
> >
> > ## DO NOT UNCOMMENT THEM, Just edit them to your needs
> >
> > ## ## Start Default Options ##
> > ## default kernel options
> > ## default kernel options for automagic boot options
> > ## If you want special options for specific kernels use kopt_x_y_z
> > ## where x.y.z is kernel version. Minor versions can be omitted.
> > ## e.g. kopt=root=/dev/hda1 ro
> > ##      kopt_2_6_8=root=/dev/hdc1 ro
> > ##      kopt_2_6_8_2_686=root=/dev/hdc2 ro
> > # kopt=root=UUID=909f7c32-639a-469d-b34b-b418d2b6a2dc ro
> >
> > ## default grub root device
> > ## e.g. groot=(hd0,0)
> > # groot=909f7c32-639a-469d-b34b-b418d2b6a2dc
> >
> > ## should update-grub create alternative automagic boot options
> > ## e.g. alternative=true
> > ##      alternative=false
> > # alternative=true
> >
> > ## should update-grub lock alternative automagic boot options
> > ## e.g. lockalternative=true
> > ##      lockalternative=false
> > # lockalternative=false
> >
> > ## additional options to use with the default boot option, but not with
> > the
> > ## alternatives
> > ## e.g. defoptions=vga=791 resume=/dev/hda5
> > # defoptions=quiet splash
> >
> > ## should update-grub lock old automagic boot options
> > ## e.g. lockold=false
> > ##      lockold=true
> > # lockold=false
> >
> > ## Xen hypervisor options to use with the default Xen boot option
> > # xenhopt=dom0_max_vcpus=1 dom0_mem=8192
> >
> > ## Xen Linux kernel options to use with the default Xen boot option
> > # xenkopt=console=tty0
> >
> > ## altoption boot targets option
> > ## multiple altoptions lines are allowed
> > ## e.g. altoptions=(extra menu suffix) extra boot options
> > ##      altoptions=(recovery) single
> > # altoptions=(recovery mode) single
> >
> > ## controls how many kernels should be put into the menu.lst
> > ## only counts the first occurence of a kernel, not the
> > ## alternative kernel options
> > ## e.g. howmany=all
> > ##      howmany=7
> > # howmany=all
> >
> > ## specify if running in Xen domU or have grub detect automatically
> > ## update-grub will ignore non-xen kernels when running in domU and vice
> > versa
> > ## e.g. indomU=detect
> > ##      indomU=true
> > ##      indomU=false
> > # indomU=detect
> >
> > ## should update-grub create memtest86 boot option
> > ## e.g. memtest86=true
> > ##      memtest86=false
> > # memtest86=true
> >
> > ## should update-grub adjust the value of the default booted system
> > ## can be true or false
> > # updatedefaultentry=false
> >
> > ## should update-grub add savedefault to the default options
> > ## can be true or false
> > # savedefault=false
> >
> > ## ## End Default Options ##
> >
> > title           Xen 4.0.0 / Debian GNU/Linux, kernel 2.6.31.13
> > root            (hd0,0)
> > kernel          /boot/xen-4.0.0.gz
> > module          /boot/vmlinuz-2.6.31.13 root=/dev/sda1 ro
> >
> > title           Xen 3.4.2 / Debian GNU/Linux, kernel 2.6.31.8-xenapr2010
> > root            (hd0,0)
> > kernel          /boot/xen-3.4.2.gz
> > module          /boot/vmlinuz-2.6.31.8-xenapr2010 root=/dev/sda1 ro
> > console=tty0
> > module          /boot/initrd.img-2.6.31.8-xenapr2010
> >
> > ### END DEBIAN AUTOMAGIC KERNELS LIST
> >
> >
> > Please paste your dom0 grub.conf.
> > Are you using memory ballooning?
> >
> > -- Pasi
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

Does it display anything on the console?  Do you have a serial console
setup? 

>> YA: No, this machine does not have a dedicated serial console, but I
can 
>> try and find another machine that does.  

That should be fairly easy to verify with "top" in dom0.

>> YA: Sorry I was unclear. We have already used top to see that it is
indeed 
>> xenstored that takes up a lot of CPU cycles as we increase the number
of domUs.  
>> That is why we wanted to try out the OCAML version since it is supposed
to be better. 

What happens if you manually start xenstored (before xend)?  Does it
complain, crash, something else?

>> YA: I will test this once we have the other problem worked out (number
of VMs) and report back. 
>> right now the system automatically starts up xend, so I can make that
manual and test it. 

I think this is a very under-tested configuration, so it wouldn't
surprise me if there are problems with it.

>> YA: that's what I figured. I can help with the testing. :)

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Keir Fraser]

On 27/04/2010 08:41, "Yuvraj Agarwal" <yuvraj@cs.ucsd.edu> wrote:

> Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log, but
> as far as we can see we don¹t quite know what might be going causing the
> system to crash (no console access anymore and system becomes unresponsive and
> needs to be power-cycled).  I have pasted only the relevant bits of
> information (the last domU that did successfully start and the next one that
> failed). It may be the case that all the log messages weren¹t flushed before
> the system crashedŠ
>  
> Does anyone know where this limit of 155 domU is coming from and how we can
> fix/increase it? 

Get a serial line on a test box, and capture Xen logging output on it. You
can both see if any crash messages come from Xen when the 155th domain is
created, and also try the serial debug keys (e.g., try 'h' to get help to
start with) to see whether Xen itself is still alive.

 -- Keir

RE: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

Actually, I did identify the problem (don’t know the fix) at least from
the console logs. Its related to running out of nr_irq's  (attached JPG
for the console log).


-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
Sent: Tuesday, April 27, 2010 5:44 PM
To: Yuvraj Agarwal; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
on starting 155th domU

On 27/04/2010 08:41, "Yuvraj Agarwal" <yuvraj@cs.ucsd.edu> wrote:

> Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
but
> as far as we can see we don¹t quite know what might be going causing the
> system to crash (no console access anymore and system becomes
unresponsive and
> needs to be power-cycled).  I have pasted only the relevant bits of
> information (the last domU that did successfully start and the next one
that
> failed). It may be the case that all the log messages weren¹t flushed
before
> the system crashedŠ
>
> Does anyone know where this limit of 155 domU is coming from and how we
can
> fix/increase it?

Get a serial line on a test box, and capture Xen logging output on it. You
can both see if any crash messages come from Xen when the 155th domain is
created, and also try the serial debug keys (e.g., try 'h' to get help to
start with) to see whether Xen itself is still alive.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #2: console-log-xen.JPG --]
[-- Type: image/jpeg, Size: 34222 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Keir Fraser]

I think nr_irqs is specifiable on the command line on newer kernels. You may
be able to do nr_irqs=65536 as a kernel boot parameter, or something like
that, without needing to rebuild the kernel.

 -- Keir

On 28/04/2010 02:02, "Yuvraj Agarwal" <yuvraj@cs.ucsd.edu> wrote:

> Actually, I did identify the problem (don’t know the fix) at least from
> the console logs. Its related to running out of nr_irq's  (attached JPG
> for the console log).
> 
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
> Sent: Tuesday, April 27, 2010 5:44 PM
> To: Yuvraj Agarwal; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
> on starting 155th domU
> 
> On 27/04/2010 08:41, "Yuvraj Agarwal" <yuvraj@cs.ucsd.edu> wrote:
> 
>> Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
> but
>> as far as we can see we don¹t quite know what might be going causing the
>> system to crash (no console access anymore and system becomes
> unresponsive and
>> needs to be power-cycled).  I have pasted only the relevant bits of
>> information (the last domU that did successfully start and the next one
> that
>> failed). It may be the case that all the log messages weren¹t flushed
> before
>> the system crashedŠ
>> 
>> Does anyone know where this limit of 155 domU is coming from and how we
> can
>> fix/increase it?
> 
> Get a serial line on a test box, and capture Xen logging output on it. You
> can both see if any crash messages come from Xen when the 155th domain is
> created, and also try the serial debug keys (e.g., try 'h' to get help to
> start with) to see whether Xen itself is still alive.
> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Keir Fraser]

Cc'ing Ian and Jeremy -- one of them should be able to answer definitively
on this. I'm *pretty* sure there's an easy way to bump the irq limit you're
hitting, on the pv_ops kernels, but 'nr_irqs=' on dom0 cmdline clearly isn't
it as it had no effect at all!

 -- Keir

On 28/04/2010 07:47, "John McCullough" <jmccullo@cs.ucsd.edu> wrote:

> I did a little testing.
> 
> With no kernel option:
> # dmesg | grep -i nr_irqs
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] NR_IRQS:4352 nr_irqs:256
> 
> w/nr_irqs=65536:
> # dmesg | grep -i nr_irqs
> [    0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0
> nr_irqs=65536
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] Kernel command line: root=/dev/sda1 ro quiet console=hvc0
> nr_irqs=65536
> [    0.000000] NR_IRQS:4352 nr_irqs:256
> 
> tweaking the NR_IRQS macro in the kernel will change the NR_IRQS output,
> but unfortunately that doesn't change nr_irqs and I run into the same
> limit (36 domus on a less-beefy dual core machine).
> 
> I did find this:
> http://blogs.sun.com/fvdl/entry/a_million_vms
> which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops
> kernel.
> 
> Watching /proc/interrupts, the domain irqs seem to be getting allocated
> from 248 downward until they hit some other limit:
> ...
>   64:      59104  xen-pirq-ioapic-level  ioc0
>   89:          1   xen-dyn-event     evtchn:xenconsoled
>   90:          1   xen-dyn-event     evtchn:xenstored
>   91:          6   xen-dyn-event     vif36.0
>   92:        140   xen-dyn-event     blkif-backend
>   93:         97   xen-dyn-event     evtchn:xenconsoled
>   94:        139   xen-dyn-event     evtchn:xenstored
>   95:          7   xen-dyn-event     vif35.0
>   96:        301   xen-dyn-event     blkif-backend
>   97:        261   xen-dyn-event     evtchn:xenconsoled
>   98:        145   xen-dyn-event     evtchn:xenstored
>   99:          7   xen-dyn-event     vif34.0
> ...
> Perhaps the xen irqs are getting allocated out of the nr_irqs pool,
> while they could be allocated from the NR_IRQS pool?
> 
> -John
> 
> 
> 
> 
> On 04/27/2010 08:45 PM, Keir Fraser wrote:
>> I think nr_irqs is specifiable on the command line on newer kernels. You may
>> be able to do nr_irqs=65536 as a kernel boot parameter, or something like
>> that, without needing to rebuild the kernel.
>> 
>>   -- Keir
>> 
>> On 28/04/2010 02:02, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
>> 
>>    
>>> Actually, I did identify the problem (don’t know the fix) at least from
>>> the console logs. Its related to running out of nr_irq's  (attached JPG
>>> for the console log).
>>> 
>>> 
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com
>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
>>> Sent: Tuesday, April 27, 2010 5:44 PM
>>> To: Yuvraj Agarwal; xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
>>> on starting 155th domU
>>> 
>>> On 27/04/2010 08:41, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
>>> 
>>>      
>>>> Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
>>>>        
>>> but
>>>      
>>>> as far as we can see we don¹t quite know what might be going causing the
>>>> system to crash (no console access anymore and system becomes
>>>>        
>>> unresponsive and
>>>      
>>>> needs to be power-cycled).  I have pasted only the relevant bits of
>>>> information (the last domU that did successfully start and the next one
>>>>        
>>> that
>>>      
>>>> failed). It may be the case that all the log messages weren¹t flushed
>>>>        
>>> before
>>>      
>>>> the system crashedŠ
>>>> 
>>>> Does anyone know where this limit of 155 domU is coming from and how we
>>>>        
>>> can
>>>      
>>>> fix/increase it?
>>>>        
>>> Get a serial line on a test box, and capture Xen logging output on it. You
>>> can both see if any crash messages come from Xen when the 155th domain is
>>> created, and also try the serial debug keys (e.g., try 'h' to get help to
>>> start with) to see whether Xen itself is still alive.
>>> 
>>>   -- Keir
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>      
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>    
> 

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[John McCullough]

I did a little testing.

With no kernel option:
# dmesg | grep -i nr_irqs
[    0.000000] nr_irqs_gsi: 88
[    0.000000] NR_IRQS:4352 nr_irqs:256

w/nr_irqs=65536:
# dmesg | grep -i nr_irqs
[    0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0 
nr_irqs=65536
[    0.000000] nr_irqs_gsi: 88
[    0.000000] Kernel command line: root=/dev/sda1 ro quiet console=hvc0 
nr_irqs=65536
[    0.000000] NR_IRQS:4352 nr_irqs:256

tweaking the NR_IRQS macro in the kernel will change the NR_IRQS output, 
but unfortunately that doesn't change nr_irqs and I run into the same 
limit (36 domus on a less-beefy dual core machine).

I did find this:
http://blogs.sun.com/fvdl/entry/a_million_vms
which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops 
kernel.

Watching /proc/interrupts, the domain irqs seem to be getting allocated 
from 248 downward until they hit some other limit:
...
  64:      59104  xen-pirq-ioapic-level  ioc0
  89:          1   xen-dyn-event     evtchn:xenconsoled
  90:          1   xen-dyn-event     evtchn:xenstored
  91:          6   xen-dyn-event     vif36.0
  92:        140   xen-dyn-event     blkif-backend
  93:         97   xen-dyn-event     evtchn:xenconsoled
  94:        139   xen-dyn-event     evtchn:xenstored
  95:          7   xen-dyn-event     vif35.0
  96:        301   xen-dyn-event     blkif-backend
  97:        261   xen-dyn-event     evtchn:xenconsoled
  98:        145   xen-dyn-event     evtchn:xenstored
  99:          7   xen-dyn-event     vif34.0
...
Perhaps the xen irqs are getting allocated out of the nr_irqs pool, 
while they could be allocated from the NR_IRQS pool?

-John




On 04/27/2010 08:45 PM, Keir Fraser wrote:
> I think nr_irqs is specifiable on the command line on newer kernels. You may
> be able to do nr_irqs=65536 as a kernel boot parameter, or something like
> that, without needing to rebuild the kernel.
>
>   -- Keir
>
> On 28/04/2010 02:02, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
>
>    
>> Actually, I did identify the problem (don’t know the fix) at least from
>> the console logs. Its related to running out of nr_irq's  (attached JPG
>> for the console log).
>>
>>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com
>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
>> Sent: Tuesday, April 27, 2010 5:44 PM
>> To: Yuvraj Agarwal; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
>> on starting 155th domU
>>
>> On 27/04/2010 08:41, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
>>
>>      
>>> Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
>>>        
>> but
>>      
>>> as far as we can see we don¹t quite know what might be going causing the
>>> system to crash (no console access anymore and system becomes
>>>        
>> unresponsive and
>>      
>>> needs to be power-cycled).  I have pasted only the relevant bits of
>>> information (the last domU that did successfully start and the next one
>>>        
>> that
>>      
>>> failed). It may be the case that all the log messages weren¹t flushed
>>>        
>> before
>>      
>>> the system crashedŠ
>>>
>>> Does anyone know where this limit of 155 domU is coming from and how we
>>>        
>> can
>>      
>>> fix/increase it?
>>>        
>> Get a serial line on a test box, and capture Xen logging output on it. You
>> can both see if any crash messages come from Xen when the 155th domain is
>> created, and also try the serial debug keys (e.g., try 'h' to get help to
>> start with) to see whether Xen itself is still alive.
>>
>>   -- Keir
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>      
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>    

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Konrad Rzeszutek Wilk]

On Tue, Apr 27, 2010 at 11:47:30PM -0700, John McCullough wrote:
> I did a little testing.
>
> With no kernel option:
> # dmesg | grep -i nr_irqs
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] NR_IRQS:4352 nr_irqs:256
>
> w/nr_irqs=65536:
> # dmesg | grep -i nr_irqs
> [    0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0  
> nr_irqs=65536
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] Kernel command line: root=/dev/sda1 ro quiet console=hvc0  
> nr_irqs=65536
> [    0.000000] NR_IRQS:4352 nr_irqs:256
>
> tweaking the NR_IRQS macro in the kernel will change the NR_IRQS output,  
> but unfortunately that doesn't change nr_irqs and I run into the same  
> limit (36 domus on a less-beefy dual core machine).

If you have CONFIG_SPARSE_IRQ defined in your .config, it gets
overwritten by some code that figures out how many IRQs you need based
on your CPU count.

So can you change NR_VECTORS in arch/x86/include/asm/irq_vectors.h to a
higher value and see what happens?

>
> I did find this:
> http://blogs.sun.com/fvdl/entry/a_million_vms
> which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops  
> kernel.
>
> Watching /proc/interrupts, the domain irqs seem to be getting allocated  
> from 248 downward until they hit some other limit:

Yeah. They hit the nr_irqs_gsi and don't go below that.

> ...
>  64:      59104  xen-pirq-ioapic-level  ioc0
>  89:          1   xen-dyn-event     evtchn:xenconsoled
>  90:          1   xen-dyn-event     evtchn:xenstored
>  91:          6   xen-dyn-event     vif36.0
>  92:        140   xen-dyn-event     blkif-backend
>  93:         97   xen-dyn-event     evtchn:xenconsoled
>  94:        139   xen-dyn-event     evtchn:xenstored
>  95:          7   xen-dyn-event     vif35.0
>  96:        301   xen-dyn-event     blkif-backend
>  97:        261   xen-dyn-event     evtchn:xenconsoled
>  98:        145   xen-dyn-event     evtchn:xenstored
>  99:          7   xen-dyn-event     vif34.0
> ...
> Perhaps the xen irqs are getting allocated out of the nr_irqs pool,  
> while they could be allocated from the NR_IRQS pool?
>
> -John
>
>
>
>
> On 04/27/2010 08:45 PM, Keir Fraser wrote:
>> I think nr_irqs is specifiable on the command line on newer kernels. You may
>> be able to do nr_irqs=65536 as a kernel boot parameter, or something like
>> that, without needing to rebuild the kernel.
>>
>>   -- Keir
>>
>> On 28/04/2010 02:02, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
>>
>>    
>>> Actually, I did identify the problem (don’t know the fix) at least from
>>> the console logs. Its related to running out of nr_irq's  (attached JPG
>>> for the console log).
>>>
>>>
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com
>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
>>> Sent: Tuesday, April 27, 2010 5:44 PM
>>> To: Yuvraj Agarwal; xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
>>> on starting 155th domU
>>>
>>> On 27/04/2010 08:41, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
>>>
>>>      
>>>> Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
>>>>        
>>> but
>>>      
>>>> as far as we can see we don¹t quite know what might be going causing the
>>>> system to crash (no console access anymore and system becomes
>>>>        
>>> unresponsive and
>>>      
>>>> needs to be power-cycled).  I have pasted only the relevant bits of
>>>> information (the last domU that did successfully start and the next one
>>>>        
>>> that
>>>      
>>>> failed). It may be the case that all the log messages weren¹t flushed
>>>>        
>>> before
>>>      
>>>> the system crashedŠ
>>>>
>>>> Does anyone know where this limit of 155 domU is coming from and how we
>>>>        
>>> can
>>>      
>>>> fix/increase it?
>>>>        
>>> Get a serial line on a test box, and capture Xen logging output on it. You
>>> can both see if any crash messages come from Xen when the 155th domain is
>>> created, and also try the serial debug keys (e.g., try 'h' to get help to
>>> start with) to see whether Xen itself is still alive.
>>>
>>>   -- Keir
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>      
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>    
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Ian Campbell]

On Wed, 2010-04-28 at 15:04 +0100, Konrad Rzeszutek Wilk wrote:
> On Tue, Apr 27, 2010 at 11:47:30PM -0700, John McCullough wrote:
> > I did a little testing.
> >
> > With no kernel option:
> > # dmesg | grep -i nr_irqs
> > [    0.000000] nr_irqs_gsi: 88
> > [    0.000000] NR_IRQS:4352 nr_irqs:256
> >
> > w/nr_irqs=65536:
> > # dmesg | grep -i nr_irqs
> > [    0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0  
> > nr_irqs=65536
> > [    0.000000] nr_irqs_gsi: 88
> > [    0.000000] Kernel command line: root=/dev/sda1 ro quiet console=hvc0  
> > nr_irqs=65536
> > [    0.000000] NR_IRQS:4352 nr_irqs:256
> >
> > tweaking the NR_IRQS macro in the kernel will change the NR_IRQS output,  
> > but unfortunately that doesn't change nr_irqs and I run into the same  
> > limit (36 domus on a less-beefy dual core machine).
> 
> If you have CONFIG_SPARSE_IRQ defined in your .config, it gets
> overwritten by some code that figures out how many IRQs you need based
> on your CPU count.
> 
> So can you change NR_VECTORS in arch/x86/include/asm/irq_vectors.h to a
> higher value and see what happens?

Jeremy applied a patch of mine which added some extra space for dynamic
IRQs at the start of march:
        commit 6d4a9168207ade237098a401270959ecc0bdd1e9
        Author: Ian Campbell <ian.campbell@citrix.com>
        Date:   Mon Mar 1 11:21:15 2010 +0000
        
            xen: allow some overhead in IRQ space for dynamic IRQs
        
If you have this patch then you can edit NR_DYNAMIC_IRQS in
arch/x86/include/asm/irq_vectors.h to increase the number of extra IRQs.

Ian.

> 
> >
> > I did find this:
> > http://blogs.sun.com/fvdl/entry/a_million_vms
> > which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops  
> > kernel.
> >
> > Watching /proc/interrupts, the domain irqs seem to be getting allocated  
> > from 248 downward until they hit some other limit:
> 
> Yeah. They hit the nr_irqs_gsi and don't go below that.
> 
> > ...
> >  64:      59104  xen-pirq-ioapic-level  ioc0
> >  89:          1   xen-dyn-event     evtchn:xenconsoled
> >  90:          1   xen-dyn-event     evtchn:xenstored
> >  91:          6   xen-dyn-event     vif36.0
> >  92:        140   xen-dyn-event     blkif-backend
> >  93:         97   xen-dyn-event     evtchn:xenconsoled
> >  94:        139   xen-dyn-event     evtchn:xenstored
> >  95:          7   xen-dyn-event     vif35.0
> >  96:        301   xen-dyn-event     blkif-backend
> >  97:        261   xen-dyn-event     evtchn:xenconsoled
> >  98:        145   xen-dyn-event     evtchn:xenstored
> >  99:          7   xen-dyn-event     vif34.0
> > ...
> > Perhaps the xen irqs are getting allocated out of the nr_irqs pool,  
> > while they could be allocated from the NR_IRQS pool?
> >
> > -John
> >
> >
> >
> >
> > On 04/27/2010 08:45 PM, Keir Fraser wrote:
> >> I think nr_irqs is specifiable on the command line on newer kernels. You may
> >> be able to do nr_irqs=65536 as a kernel boot parameter, or something like
> >> that, without needing to rebuild the kernel.
> >>
> >>   -- Keir
> >>
> >> On 28/04/2010 02:02, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
> >>
> >>    
> >>> Actually, I did identify the problem (don’t know the fix) at least from
> >>> the console logs. Its related to running out of nr_irq's  (attached JPG
> >>> for the console log).
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: xen-devel-bounces@lists.xensource.com
> >>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
> >>> Sent: Tuesday, April 27, 2010 5:44 PM
> >>> To: Yuvraj Agarwal; xen-devel@lists.xensource.com
> >>> Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes
> >>> on starting 155th domU
> >>>
> >>> On 27/04/2010 08:41, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
> >>>
> >>>      
> >>>> Attached is the output of /var/log/daemon.log and /var/log/xen/xend.log,
> >>>>        
> >>> but
> >>>      
> >>>> as far as we can see we don¹t quite know what might be going causing the
> >>>> system to crash (no console access anymore and system becomes
> >>>>        
> >>> unresponsive and
> >>>      
> >>>> needs to be power-cycled).  I have pasted only the relevant bits of
> >>>> information (the last domU that did successfully start and the next one
> >>>>        
> >>> that
> >>>      
> >>>> failed). It may be the case that all the log messages weren¹t flushed
> >>>>        
> >>> before
> >>>      
> >>>> the system crashedŠ
> >>>>
> >>>> Does anyone know where this limit of 155 domU is coming from and how we
> >>>>        
> >>> can
> >>>      
> >>>> fix/increase it?
> >>>>        
> >>> Get a serial line on a test box, and capture Xen logging output on it. You
> >>> can both see if any crash messages come from Xen when the 155th domain is
> >>> created, and also try the serial debug keys (e.g., try 'h' to get help to
> >>> start with) to see whether Xen itself is still alive.
> >>>
> >>>   -- Keir
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Xen-devel mailing list
> >>> Xen-devel@lists.xensource.com
> >>> http://lists.xensource.com/xen-devel
> >>>      
> >>
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >>    
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Jeremy Fitzhardinge]

On 04/27/2010 11:47 PM, John McCullough wrote:
> I did a little testing.
>
> With no kernel option:
> # dmesg | grep -i nr_irqs
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] NR_IRQS:4352 nr_irqs:256
>
> w/nr_irqs=65536:
> # dmesg | grep -i nr_irqs
> [    0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0
> nr_irqs=65536
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] Kernel command line: root=/dev/sda1 ro quiet
> console=hvc0 nr_irqs=65536
> [    0.000000] NR_IRQS:4352 nr_irqs:256
>
> tweaking the NR_IRQS macro in the kernel will change the NR_IRQS
> output, but unfortunately that doesn't change nr_irqs and I run into
> the same limit (36 domus on a less-beefy dual core machine).

Yes, NR_IRQS is the hard limit (for any statically defined irq arrays,
which are deprecated now), but nr_irqs is the amount it decides to
actually allocate for dynamic irq arrays, and so represents the actual
runtime limit.

nr_irqs is computed in arch_probe_nr_irqs(), and its a function of the
number of cpus, with a bump to deal with dynamically allocated MSI
interrupts.  I should probably add something to specifically add more if
we're running under Xen, at least as a workaround (ultimately the plan
is to make all irqs completely dynamically allocated so there is no hard
limit).

>
> I did find this:
> http://blogs.sun.com/fvdl/entry/a_million_vms
> which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops
> kernel.

I'm pretty sure that's referring to Solaris dom0, so the fact that
there's a similarly named symbol is coincidence.  (But the root problem
is the same.)

    J

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Jeremy Fitzhardinge]

On 04/28/2010 09:57 AM, Ian Campbell wrote:
> Jeremy applied a patch of mine which added some extra space for dynamic
> IRQs at the start of march:
>         commit 6d4a9168207ade237098a401270959ecc0bdd1e9
>         Author: Ian Campbell <ian.campbell@citrix.com>
>         Date:   Mon Mar 1 11:21:15 2010 +0000
>         
>             xen: allow some overhead in IRQ space for dynamic IRQs
>         
> If you have this patch then you can edit NR_DYNAMIC_IRQS in
> arch/x86/include/asm/irq_vectors.h to increase the number of extra IRQs.
>   

That's only present in 2.6.32, not .31.  But it would be easy to backport.

    J

RE: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

I tried making the change in 
linux-2.6-pvops.git/arch/x86/include/asm/irq_vectors.h

It was:
#define NR_VECTORS                     256
I changed it to
#define NR_VECTORS                       1024

I still get the same number of nr_irqs  (dmesg | grep -i nr_irq) before and 
after the change.

[    0.000000] nr_irqs_gsi: 48
[    0.500076] NR_IRQS:5120 nr_irqs:944

Also, as earlier it crashes on the same number of domU (154). I didn’t 
mention earlier, this a dual core Nehalem machine  -- 2 (sockets) * 4 cores 
per CPU * 2 (hyperthreading)

--Yuvraj

-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
Sent: Wednesday, April 28, 2010 7:05 AM
To: John McCullough
Cc: Keir Fraser; xen-devel@lists.xensource.com; Yuvraj Agarwal
Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system crashes 
on starting 155th domU

On Tue, Apr 27, 2010 at 11:47:30PM -0700, John McCullough wrote:
> I did a little testing.
>
> With no kernel option:
> # dmesg | grep -i nr_irqs
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] NR_IRQS:4352 nr_irqs:256
>
> w/nr_irqs=65536:
> # dmesg | grep -i nr_irqs
> [    0.000000] Command line: root=/dev/sda1 ro quiet console=hvc0
> nr_irqs=65536
> [    0.000000] nr_irqs_gsi: 88
> [    0.000000] Kernel command line: root=/dev/sda1 ro quiet console=hvc0
> nr_irqs=65536
> [    0.000000] NR_IRQS:4352 nr_irqs:256
>
> tweaking the NR_IRQS macro in the kernel will change the NR_IRQS output,
> but unfortunately that doesn't change nr_irqs and I run into the same
> limit (36 domus on a less-beefy dual core machine).

If you have CONFIG_SPARSE_IRQ defined in your .config, it gets
overwritten by some code that figures out how many IRQs you need based
on your CPU count.

So can you change NR_VECTORS in arch/x86/include/asm/irq_vectors.h to a
higher value and see what happens?

>
> I did find this:
> http://blogs.sun.com/fvdl/entry/a_million_vms
> which references NR_DYNIRQS, which is in 2.6.18, but not in the pvops
> kernel.
>
> Watching /proc/interrupts, the domain irqs seem to be getting allocated
> from 248 downward until they hit some other limit:

Yeah. They hit the nr_irqs_gsi and don't go below that.

> ...
>  64:      59104  xen-pirq-ioapic-level  ioc0
>  89:          1   xen-dyn-event     evtchn:xenconsoled
>  90:          1   xen-dyn-event     evtchn:xenstored
>  91:          6   xen-dyn-event     vif36.0
>  92:        140   xen-dyn-event     blkif-backend
>  93:         97   xen-dyn-event     evtchn:xenconsoled
>  94:        139   xen-dyn-event     evtchn:xenstored
>  95:          7   xen-dyn-event     vif35.0
>  96:        301   xen-dyn-event     blkif-backend
>  97:        261   xen-dyn-event     evtchn:xenconsoled
>  98:        145   xen-dyn-event     evtchn:xenstored
>  99:          7   xen-dyn-event     vif34.0
> ...
> Perhaps the xen irqs are getting allocated out of the nr_irqs pool,
> while they could be allocated from the NR_IRQS pool?
>
> -John
>
>
>
>
> On 04/27/2010 08:45 PM, Keir Fraser wrote:
>> I think nr_irqs is specifiable on the command line on newer kernels. You 
>> may
>> be able to do nr_irqs=65536 as a kernel boot parameter, or something like
>> that, without needing to rebuild the kernel.
>>
>>   -- Keir
>>
>> On 28/04/2010 02:02, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
>>
>>
>>> Actually, I did identify the problem (don’t know the fix) at least from
>>> the console logs. Its related to running out of nr_irq's  (attached JPG
>>> for the console log).
>>>
>>>
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com
>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
>>> Sent: Tuesday, April 27, 2010 5:44 PM
>>> To: Yuvraj Agarwal; xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] XEN 4.0 + 2.6.31.13 pvops kernel : system 
>>> crashes
>>> on starting 155th domU
>>>
>>> On 27/04/2010 08:41, "Yuvraj Agarwal"<yuvraj@cs.ucsd.edu>  wrote:
>>>
>>>
>>>> Attached is the output of /var/log/daemon.log and 
>>>> /var/log/xen/xend.log,
>>>>
>>> but
>>>
>>>> as far as we can see we don¹t quite know what might be going causing 
>>>> the
>>>> system to crash (no console access anymore and system becomes
>>>>
>>> unresponsive and
>>>
>>>> needs to be power-cycled).  I have pasted only the relevant bits of
>>>> information (the last domU that did successfully start and the next one
>>>>
>>> that
>>>
>>>> failed). It may be the case that all the log messages weren¹t flushed
>>>>
>>> before
>>>
>>>> the system crashedŠ
>>>>
>>>> Does anyone know where this limit of 155 domU is coming from and how we
>>>>
>>> can
>>>
>>>> fix/increase it?
>>>>
>>> Get a serial line on a test box, and capture Xen logging output on it. 
>>> You
>>> can both see if any crash messages come from Xen when the 155th domain 
>>> is
>>> created, and also try the serial debug keys (e.g., try 'h' to get help 
>>> to
>>> start with) to see whether Xen itself is still alive.
>>>
>>>   -- Keir
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Konrad Rzeszutek Wilk]

On Wed, Apr 28, 2010 at 03:51:19PM -0700, Yuvraj Agarwal wrote:
> I tried making the change in 
> linux-2.6-pvops.git/arch/x86/include/asm/irq_vectors.h
> 
> It was:
> #define NR_VECTORS                     256
> I changed it to
> #define NR_VECTORS                       1024
> 
> I still get the same number of nr_irqs  (dmesg | grep -i nr_irq) before and 
> after the change.
> 
> [    0.000000] nr_irqs_gsi: 48
> [    0.500076] NR_IRQS:5120 nr_irqs:944

That looks to be different from the previous bootup:
[    0.000000] NR_IRQS:4352 nr_irqs:256

?
> 
> Also, as earlier it crashes on the same number of domU (154). I didn’t 
> mention earlier, this a dual core Nehalem machine  -- 2 (sockets) * 4 cores 
> per CPU * 2 (hyperthreading)

Lots of logical CPUs, weird that your nr_irqs initially was that much
lower.

Anyhow, you mentioned that you narrowed it down to not being enough IRQs
- how did you find that out? Was there an kernel message when you
  started the 155th guest?
 

Oh, also you say that the /proc/interrupts showed the number descending
from 255 down to 89. With this it should have started at 944 and gone
down to 49? Which roughly means 175 guests?

RE: XEN 4.0 + 2.6.31.13 pvops kernel : system crashes on starting 155th domU

[Yuvraj Agarwal]

[-- Attachment #1: Type: text/plain, Size: 2184 bytes --]


> [    0.000000] nr_irqs_gsi: 48
> [    0.500076] NR_IRQS:5120 nr_irqs:944

That looks to be different from the previous bootup:
[    0.000000] NR_IRQS:4352 nr_irqs:256

>> YA: I think you may have mixed up the post by John. He has a different 
>> dual core machine
>> and he is trying the same edits as me. He was getting nr_irqs as 256 
>> while I was getting 944.
>> [    0.000000] nr_irqs_gsi: 48
>> [    0.500270] NR_IRQS:5120 nr_irqs:944
>> Also it did not matter if I changed the VECTORS to 1024 instead of the 
>> original 256.

>
> Also, as earlier it crashes on the same number of domU (154). I didn’t
> mention earlier, this a dual core Nehalem machine  -- 2 (sockets) * 4 
> cores
> per CPU * 2 (hyperthreading)

Lots of logical CPUs, weird that your nr_irqs initially was that much lower.

>> No, as mentioned above, my machine has the larger number of CPUs while 
>> John McCullough
>> machine has lesser number of CPUs.

Anyhow, you mentioned that you narrowed it down to not being enough IRQs
- how did you find that out? Was there an kernel message when you
  started the 155th guest?

>>> yes, as I mentioned in an email, right after I started up the 154th 
>>> guest there was
>>> a log message printed out as below:
>>> Kernel Panic - not syncing: No available IRQs to bind to: increase 
>>> nr_irqs!

Oh, also you say that the /proc/interrupts showed the number descending
from 255 down to 89. With this it should have started at 944 and gone
down to 49? Which roughly means 175 guests?

>> Yes, the interrupts started counting down from 944 as I increased the 
>> number of domUs;
>> I am attaching the /proc/interrupts log file for three cases (no domU, 
>> after 1 domU,
>> and then after 150 domU).  The machine crashed after the 154th domU was 
>> started.

Thank you all for helping out to debug this. As I mentioned at the XEN 
summit, once I do fix these issues I’d really like to create  wiki stub for 
any one else trying to do this. It looks like in the original 2.6.18 kernel 
people did indeed try this, but I guess some of the edits/changes didn’t 
make it to the 2.6.31 pvops branch....

[-- Attachment #2: interrupts-after1domU --]
[-- Type: application/octet-stream, Size: 29750 bytes --]

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      
  3:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge
  4:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge
  8:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge  rtc0
  9:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  acpi
 16:        103          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb2, uhci_hcd:usb6, uhci_hcd:usb7, uhci_hcd:usb8
 18:      99267          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ata_piix
 19:        188          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb1, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
 21:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ata_piix
811:          8          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif1.1
812:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif1.0
813:        558          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
814:        328          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
815:        219          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
816:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
817:        634          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
818:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1
819:     307038          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-3
820:     418043          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-2
821:     452331          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-1
822:    3362424          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-0
823:       1390          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-3
824:        542          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-2
825:        451          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-1
826:        547          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-0
840:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
841:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
842:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
843:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
844:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
845:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
846:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      pcpu
847:       1076          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     xenbus
848:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         13   xen-dyn-ipi       callfuncsingle15
849:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug15
850:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         34   xen-dyn-ipi       callfunc15
851:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     516348   xen-dyn-ipi       resched15
852:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         13   xen-dyn-ipi       spinlock15
853:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0       7333   xen-dyn-virq      timer15
854:          0          0          0          0          0          0          0          0          0          0          0          0          0          0         37          0   xen-dyn-ipi       callfuncsingle14
855:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug14
856:          0          0          0          0          0          0          0          0          0          0          0          0          0          0         38          0   xen-dyn-ipi       callfunc14
857:          0          0          0          0          0          0          0          0          0          0          0          0          0          0     516391          0   xen-dyn-ipi       resched14
858:          0          0          0          0          0          0          0          0          0          0          0          0          0          0         14          0   xen-dyn-ipi       spinlock14
859:          0          0          0          0          0          0          0          0          0          0          0          0          0          0       4132          0   xen-dyn-virq      timer14
860:          0          0          0          0          0          0          0          0          0          0          0          0          0         60          0          0   xen-dyn-ipi       callfuncsingle13
861:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug13
862:          0          0          0          0          0          0          0          0          0          0          0          0          0         39          0          0   xen-dyn-ipi       callfunc13
863:          0          0          0          0          0          0          0          0          0          0          0          0          0     516813          0          0   xen-dyn-ipi       resched13
864:          0          0          0          0          0          0          0          0          0          0          0          0          0         17          0          0   xen-dyn-ipi       spinlock13
865:          0          0          0          0          0          0          0          0          0          0          0          0          0      25829          0          0   xen-dyn-virq      timer13
866:          0          0          0          0          0          0          0          0          0          0          0          0         55          0          0          0   xen-dyn-ipi       callfuncsingle12
867:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug12
868:          0          0          0          0          0          0          0          0          0          0          0          0         37          0          0          0   xen-dyn-ipi       callfunc12
869:          0          0          0          0          0          0          0          0          0          0          0          0     516757          0          0          0   xen-dyn-ipi       resched12
870:          0          0          0          0          0          0          0          0          0          0          0          0         10          0          0          0   xen-dyn-ipi       spinlock12
871:          0          0          0          0          0          0          0          0          0          0          0          0       2873          0          0          0   xen-dyn-virq      timer12
872:          0          0          0          0          0          0          0          0          0          0          0         25          0          0          0          0   xen-dyn-ipi       callfuncsingle11
873:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug11
874:          0          0          0          0          0          0          0          0          0          0          0         34          0          0          0          0   xen-dyn-ipi       callfunc11
875:          0          0          0          0          0          0          0          0          0          0          0     517665          0          0          0          0   xen-dyn-ipi       resched11
876:          0          0          0          0          0          0          0          0          0          0          0         12          0          0          0          0   xen-dyn-ipi       spinlock11
877:          0          0          0          0          0          0          0          0          0          0          0       1832          0          0          0          0   xen-dyn-virq      timer11
878:          0          0          0          0          0          0          0          0          0          0         93          0          0          0          0          0   xen-dyn-ipi       callfuncsingle10
879:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug10
880:          0          0          0          0          0          0          0          0          0          0         54          0          0          0          0          0   xen-dyn-ipi       callfunc10
881:          0          0          0          0          0          0          0          0          0          0     518998          0          0          0          0          0   xen-dyn-ipi       resched10
882:          0          0          0          0          0          0          0          0          0          0         10          0          0          0          0          0   xen-dyn-ipi       spinlock10
883:          0          0          0          0          0          0          0          0          0          0       2586          0          0          0          0          0   xen-dyn-virq      timer10
884:          0          0          0          0          0          0          0          0          0        231          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle9
885:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug9
886:          0          0          0          0          0          0          0          0          0         59          0          0          0          0          0          0   xen-dyn-ipi       callfunc9
887:          0          0          0          0          0          0          0          0          0     508499          0          0          0          0          0          0   xen-dyn-ipi       resched9
888:          0          0          0          0          0          0          0          0          0          5          0          0          0          0          0          0   xen-dyn-ipi       spinlock9
889:          0          0          0          0          0          0          0          0          0      13387          0          0          0          0          0          0   xen-dyn-virq      timer9
890:          0          0          0          0          0          0          0          0        412          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle8
891:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug8
892:          0          0          0          0          0          0          0          0         44          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc8
893:          0          0          0          0          0          0          0          0     253187          0          0          0          0          0          0          0   xen-dyn-ipi       resched8
894:          0          0          0          0          0          0          0          0         78          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock8
895:          0          0          0          0          0          0          0          0     741332          0          0          0          0          0          0          0   xen-dyn-virq      timer8
896:          0          0          0          0          0          0          0          5          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle7
897:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug7
898:          0          0          0          0          0          0          0         32          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc7
899:          0          0          0          0          0          0          0     516214          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched7
900:          0          0          0          0          0          0          0          2          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock7
901:          0          0          0          0          0          0          0       1854          0          0          0          0          0          0          0          0   xen-dyn-virq      timer7
902:          0          0          0          0          0          0         11          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle6
903:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug6
904:          0          0          0          0          0          0         34          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc6
905:          0          0          0          0          0          0     516217          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched6
906:          0          0          0          0          0          0          5          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock6
907:          0          0          0          0          0          0       1025          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer6
908:          0          0          0          0          0         12          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle5
909:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug5
910:          0          0          0          0          0         38          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc5
911:          0          0          0          0          0     516547          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched5
912:          0          0          0          0          0          7          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock5
913:          0          0          0          0          0       1574          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer5
914:          0          0          0          0          7          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle4
915:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug4
916:          0          0          0          0         33          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc4
917:          0          0          0          0     516432          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched4
918:          0          0          0          0         10          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock4
919:          0          0          0          0        967          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer4
920:          0          0          0          5          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle3
921:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug3
922:          0          0          0         33          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc3
923:          0          0          0     516313          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched3
924:          0          0          0         13          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock3
925:          0          0          0      20394          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer3
926:          0          0         32          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle2
927:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug2
928:          0          0         44          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc2
929:          0          0     517469          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched2
930:          0          0         10          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock2
931:          0          0     171467          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer2
932:          0         65          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle1
933:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug1
934:          0         48          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc1
935:          0     513305          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched1
936:          0         11          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock1
937:          0     614200          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer1
938:        165          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle0
939:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug0
940:         31          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc0
941:     319172          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched0
942:         43          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock0
943:     544363          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer0
NMI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
CNT:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance counter interrupts
PND:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance pending work
RES:     319172     513305     517469     516313     516432     516547     516217     516214     253187     508499     518998     517665     516757     516813     516391     516348   Rescheduling interrupts
CAL:        196        113         76         38         40         50         45         37        456        290        147         59         92         99         75         47   Function call interrupts
TLB:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:        216        216        216        216        216        216        216        216        216        216        216        216        216        216        216        216   Machine check polls
ERR:          0
MIS:          0

[-- Attachment #3: interrupts-after150domUs --]
[-- Type: application/octet-stream, Size: 190160 bytes --]

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      
  3:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge
  4:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge
  8:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge  rtc0
  9:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  acpi
 16:        103          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb2, uhci_hcd:usb6, uhci_hcd:usb7, uhci_hcd:usb8
 18:     491197          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ata_piix
 19:        188          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb1, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
 21:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ata_piix
 66:         11          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif150.1
 67:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif150.0
 68:        622          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
 69:        253          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
 70:        228          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
 71:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif149.1
 72:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif149.0
 73:        603          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
 74:        246          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
 75:        205          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
 76:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif148.1
 77:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif148.0
 78:        595          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
 79:        249          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
 80:        202          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
 81:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif147.1
 82:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif147.0
 83:        605          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
 84:        253          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
 85:        200          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
 86:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif146.1
 87:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif146.0
 88:        580          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
 89:        255          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
 90:        193          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
 91:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif145.1
 92:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif145.0
 93:        588          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
 94:        239          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
 95:        193          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
 96:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif144.1
 97:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif144.0
 98:        588          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
 99:        241          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
100:        218          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
101:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif143.1
102:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif143.0
103:        586          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
104:        255          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
105:        235          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
106:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif142.1
107:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif142.0
108:        599          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
109:        257          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
110:        216          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
111:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif141.1
112:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif141.0
113:        591          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
114:        251          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
115:        205          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
116:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif140.1
117:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif140.0
118:        587          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
119:        260          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
120:        216          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
121:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif139.1
122:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif139.0
123:        598          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
124:        257          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
125:        191          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
126:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif138.1
127:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif138.0
128:        587          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
129:        255          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
130:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
131:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif137.1
132:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif137.0
133:        581          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
134:        256          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
135:        232          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
136:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif136.1
137:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif136.0
138:        588          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
139:        242          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
140:        197          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
141:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif135.1
142:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif135.0
143:        593          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
144:        256          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
145:        233          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
146:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif134.1
147:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif134.0
148:        584          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
149:        257          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
150:        213          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
151:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif133.1
152:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif133.0
153:        590          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
154:        256          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
155:        188          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
156:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif132.1
157:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif132.0
158:        603          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
159:        257          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
160:        200          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
161:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif131.1
162:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif131.0
163:        609          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
164:        253          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
165:        200          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
166:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif130.1
167:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif130.0
168:        586          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
169:        260          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
170:        217          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
171:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif129.1
172:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif129.0
173:        602          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
174:        254          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
175:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
176:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif128.1
177:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif128.0
178:        589          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
179:        256          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
180:        205          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
181:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif127.1
182:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif127.0
183:        609          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
184:        255          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
185:        219          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
186:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif126.1
187:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif126.0
188:        591          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
189:        262          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
190:        210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
191:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif125.1
192:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif125.0
193:        598          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
194:        257          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
195:        213          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
196:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif124.1
197:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif124.0
198:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
199:        247          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
200:        203          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
201:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif123.1
202:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif123.0
203:        620          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
204:        270          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
205:        208          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
206:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif122.1
207:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif122.0
208:        596          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
209:        257          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
210:        233          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
211:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif121.1
212:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif121.0
213:        595          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
214:        263          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
215:        208          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
216:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif120.1
217:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif120.0
218:        586          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
219:        267          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
220:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
221:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif119.1
222:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif119.0
223:        596          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
224:        266          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
225:        190          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
226:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif118.1
227:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif118.0
228:        607          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
229:        270          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
230:        197          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
231:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif117.1
232:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif117.0
233:        601          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
234:        263          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
235:        216          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
236:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif116.1
237:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif116.0
238:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
239:        257          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
240:        196          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
241:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif115.1
242:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif115.0
243:        579          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
244:        261          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
245:        211          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
246:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif114.1
247:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif114.0
248:        579          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
249:        268          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
250:        194          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
251:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif113.1
252:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif113.0
253:        590          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
254:        266          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
255:        212          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
256:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif112.1
257:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif112.0
258:        592          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
259:        260          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
260:        203          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
261:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif111.1
262:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif111.0
263:        598          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
264:        268          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
265:        201          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
266:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif110.1
267:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif110.0
268:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
269:        266          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
270:        207          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
271:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif109.1
272:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif109.0
273:        593          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
274:        264          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
275:        223          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
276:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif108.1
277:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif108.0
278:        584          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
279:        266          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
280:        210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
281:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif107.1
282:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif107.0
283:        599          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
284:        265          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
285:        201          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
286:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif106.1
287:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif106.0
288:        602          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
289:        273          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
290:        193          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
291:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif105.1
292:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif105.0
293:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
294:        269          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
295:        218          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
296:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif104.1
297:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif104.0
298:        592          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
299:        273          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
300:        210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
301:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif103.1
302:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif103.0
303:        610          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
304:        265          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
305:        201          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
306:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif102.1
307:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif102.0
308:        592          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
309:        267          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
310:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
311:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif101.1
312:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif101.0
313:        595          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
314:        275          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
315:        187          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
316:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif100.1
317:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif100.0
318:        588          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
319:        264          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
320:        182          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
321:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif99.1
322:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif99.0
323:        631          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
324:        267          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
325:        202          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
326:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif98.1
327:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif98.0
328:        580          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
329:        267          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
330:        211          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
331:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif97.1
332:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif97.0
333:        590          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
334:        288          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
335:        199          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
336:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif96.1
337:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif96.0
338:        580          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
339:        265          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
340:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
341:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif95.1
342:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif95.0
343:        603          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
344:        271          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
345:        224          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
346:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif94.1
347:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif94.0
348:        608          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
349:        272          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
350:        205          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
351:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif93.1
352:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif93.0
353:        589          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
354:        276          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
355:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
356:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif92.1
357:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif92.0
358:       2472          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
359:        328          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
360:        224          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
361:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif91.1
362:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif91.0
363:        580          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
364:        272          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
365:        198          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
366:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif90.1
367:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif90.0
368:        624          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
369:        273          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
370:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
371:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif89.1
372:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif89.0
373:        613          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
374:        266          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
375:        196          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
376:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif88.1
377:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif88.0
378:        594          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
379:        270          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
380:        204          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
381:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif87.1
382:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif87.0
383:        602          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
384:        291          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
385:        209          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
386:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif86.1
387:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif86.0
388:        601          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
389:        278          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
390:        195          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
391:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif85.1
392:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif85.0
393:        596          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
394:        275          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
395:        204          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
396:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif84.1
397:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif84.0
398:        574          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
399:        277          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
400:        216          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
401:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif83.1
402:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif83.0
403:       2642          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
404:        344          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
405:        194          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
406:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif82.1
407:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif82.0
408:        588          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
409:        274          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
410:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
411:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif81.1
412:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif81.0
413:        590          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
414:        272          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
415:        211          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
416:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif80.1
417:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif80.0
418:        577          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
419:        279          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
420:        219          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
421:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif79.1
422:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif79.0
423:        603          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
424:        272          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
425:        210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
426:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif78.1
427:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif78.0
428:        613          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
429:        274          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
430:        200          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
431:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif77.1
432:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif77.0
433:        603          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
434:        283          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
435:        209          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
436:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif76.1
437:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif76.0
438:        584          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
439:        276          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
440:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
441:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif75.1
442:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif75.0
443:        604          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
444:        274          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
445:        202          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
446:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif74.1
447:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif74.0
448:        613          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
449:        278          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
450:        218          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
451:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif73.1
452:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif73.0
453:        591          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
454:        283          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
455:        212          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
456:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif72.1
457:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif72.0
458:        604          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
459:        283          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
460:        222          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
461:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif71.1
462:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif71.0
463:        587          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
464:        279          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
465:        185          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
466:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif70.1
467:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif70.0
468:        581          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
469:        282          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
470:        192          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
471:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif69.1
472:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif69.0
473:        593          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
474:        281          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
475:        220          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
476:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif68.1
477:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif68.0
478:        618          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
479:        279          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
480:        197          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
481:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif67.1
482:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif67.0
483:        610          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
484:        280          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
485:        200          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
486:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif66.1
487:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif66.0
488:        592          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
489:        272          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
490:        196          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
491:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif65.1
492:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif65.0
493:        597          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
494:        285          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
495:        202          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
496:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif64.1
497:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif64.0
498:        615          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
499:        281          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
500:        209          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
501:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif63.1
502:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif63.0
503:        593          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
504:        291          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
505:        189          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
506:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif62.1
507:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif62.0
508:        604          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
509:        283          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
510:        188          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
511:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif61.1
512:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif61.0
513:        608          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
514:        283          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
515:        196          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
516:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif60.1
517:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif60.0
518:        612          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
519:        282          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
520:        189          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
521:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif59.1
522:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif59.0
523:        594          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
524:        291          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
525:        193          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
526:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif58.1
527:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif58.0
528:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
529:        290          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
530:        194          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
531:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif57.1
532:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif57.0
533:        607          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
534:        287          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
535:        210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
536:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif56.1
537:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif56.0
538:        585          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
539:        284          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
540:        183          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
541:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif55.1
542:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif55.0
543:        602          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
544:        285          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
545:        202          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
546:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif54.1
547:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif54.0
548:       2665          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
549:        342          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
550:        198          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
551:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif53.1
552:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif53.0
553:        606          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
554:        285          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
555:        198          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
556:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif52.1
557:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif52.0
558:        592          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
559:        283          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
560:        190          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
561:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif51.1
562:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif51.0
563:        604          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
564:        279          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
565:        188          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
566:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif50.1
567:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif50.0
568:        612          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
569:        280          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
570:        192          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
571:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif49.1
572:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif49.0
573:        576          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
574:        281          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
575:        192          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
576:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif48.1
577:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif48.0
578:        596          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
579:        285          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
580:        198          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
581:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif47.1
582:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif47.0
583:        578          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
584:        286          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
585:        228          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
586:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif46.1
587:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif46.0
588:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
589:        287          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
590:        196          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
591:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif45.1
592:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif45.0
593:        584          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
594:        287          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
595:        209          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
596:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif44.1
597:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif44.0
598:        602          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
599:        290          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
600:        189          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
601:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif43.1
602:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif43.0
603:        591          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
604:        290          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
605:        185          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
606:          9          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif42.1
607:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif42.0
608:        610          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
609:        283          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
610:        193          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
611:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif41.1
612:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif41.0
613:        587          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
614:        294          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
615:        210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
616:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif40.1
617:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif40.0
618:        582          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
619:        293          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
620:        196          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
621:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif39.1
622:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif39.0
623:        609          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
624:        290          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
625:        212          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
626:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif38.1
627:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif38.0
628:        576          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
629:        298          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
630:        212          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
631:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif37.1
632:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif37.0
633:        596          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
634:        303          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
635:        212          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
636:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif36.1
637:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif36.0
638:        611          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
639:        305          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
640:        199          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
641:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif35.1
642:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif35.0
643:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
644:        298          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
645:        217          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
646:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif34.1
647:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif34.0
648:        604          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
649:        295          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
650:        188          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
651:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif33.1
652:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif33.0
653:        619          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
654:        309          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
655:        216          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
656:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif32.1
657:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif32.0
658:        578          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
659:        310          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
660:        194          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
661:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif31.1
662:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif31.0
663:        611          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
664:        301          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
665:        189          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
666:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif30.1
667:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif30.0
668:        605          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
669:        306          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
670:        186          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
671:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif29.1
672:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif29.0
673:        597          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
674:        298          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
675:        209          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
676:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif28.1
677:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif28.0
678:        609          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
679:        314          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
680:        206          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
681:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif27.1
682:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif27.0
683:        606          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
684:        317          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
685:        188          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
686:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif26.1
687:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif26.0
688:        617          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
689:        313          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
690:        233          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
691:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif25.1
692:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif25.0
693:        614          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
694:        315          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
695:        185          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
696:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif24.1
697:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif24.0
698:        611          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
699:        310          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
700:        178          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
701:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif23.1
702:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif23.0
703:        587          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
704:        318          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
705:        203          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
706:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif22.1
707:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif22.0
708:        599          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
709:        315          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
710:        191          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
711:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif21.1
712:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif21.0
713:        596          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
714:        299          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
715:        212          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
716:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif20.1
717:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif20.0
718:        607          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
719:        321          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
720:        219          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
721:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif19.1
722:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif19.0
723:        611          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
724:        273          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
725:        220          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
726:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif18.1
727:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif18.0
728:       2278          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
729:        354          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
730:        227          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
731:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif17.1
732:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif17.0
733:        624          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
734:        309          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
735:        193          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
736:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif16.1
737:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif16.0
738:        598          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
739:        319          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
740:        215          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
741:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif15.1
742:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif15.0
743:       2696          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
744:        394          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
745:        211          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
746:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif14.1
747:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif14.0
748:        580          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
749:        314          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
750:        203          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
751:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif13.1
752:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif13.0
753:        602          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
754:        321          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
755:        192          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
756:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif12.1
757:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif12.0
758:        606          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
759:        337          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
760:        193          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
761:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif11.1
762:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif11.0
763:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
764:        327          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
765:        213          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
766:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif10.1
767:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif10.0
768:        597          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
769:        323          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
770:        213          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
771:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif9.1
772:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif9.0
773:        627          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
774:        317          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
775:        210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
776:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif8.1
777:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif8.0
778:        604          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
779:        328          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
780:        223          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
781:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif7.1
782:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif7.0
783:        616          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
784:        137          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
785:        217          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
786:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif6.1
787:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif6.0
788:        607          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
789:        334          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
790:        214          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
791:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif5.1
792:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif5.0
793:        600          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
794:        327          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
795:        224          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
796:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif4.1
797:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif4.0
798:        618          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
799:        350          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
800:        229          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
801:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif3.1
802:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif3.0
803:        624          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
804:        347          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
805:        210          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
806:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif2.1
807:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif2.0
808:        597          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
809:        319          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
810:        228          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
811:         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif1.1
812:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     vif1.0
813:        627          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     blkif-backend
814:        328          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenconsoled
815:        219          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
816:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
817:     191818          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
818:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1
819:     413474          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-3
820:     568480          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-2
821:     617716          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-1
822:    4574526          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-0
823:       1853          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-3
824:       2261          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-2
825:       1099          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-1
826:       1191          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-0
840:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
841:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
842:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
843:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
844:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
845:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
846:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      pcpu
847:     276576          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     xenbus
848:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0       2954   xen-dyn-ipi       callfuncsingle15
849:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug15
850:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0        303   xen-dyn-ipi       callfunc15
851:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0    2075595   xen-dyn-ipi       resched15
852:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0        895   xen-dyn-ipi       spinlock15
853:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   16855096   xen-dyn-virq      timer15
854:          0          0          0          0          0          0          0          0          0          0          0          0          0          0       2500          0   xen-dyn-ipi       callfuncsingle14
855:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug14
856:          0          0          0          0          0          0          0          0          0          0          0          0          0          0        317          0   xen-dyn-ipi       callfunc14
857:          0          0          0          0          0          0          0          0          0          0          0          0          0          0    2062301          0   xen-dyn-ipi       resched14
858:          0          0          0          0          0          0          0          0          0          0          0          0          0          0        411          0   xen-dyn-ipi       spinlock14
859:          0          0          0          0          0          0          0          0          0          0          0          0          0          0     813954          0   xen-dyn-virq      timer14
860:          0          0          0          0          0          0          0          0          0          0          0          0          0       3584          0          0   xen-dyn-ipi       callfuncsingle13
861:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug13
862:          0          0          0          0          0          0          0          0          0          0          0          0          0        303          0          0   xen-dyn-ipi       callfunc13
863:          0          0          0          0          0          0          0          0          0          0          0          0          0    2085386          0          0   xen-dyn-ipi       resched13
864:          0          0          0          0          0          0          0          0          0          0          0          0          0        700          0          0   xen-dyn-ipi       spinlock13
865:          0          0          0          0          0          0          0          0          0          0          0          0          0    2665877          0          0   xen-dyn-virq      timer13
866:          0          0          0          0          0          0          0          0          0          0          0          0       5424          0          0          0   xen-dyn-ipi       callfuncsingle12
867:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug12
868:          0          0          0          0          0          0          0          0          0          0          0          0        342          0          0          0   xen-dyn-ipi       callfunc12
869:          0          0          0          0          0          0          0          0          0          0          0          0    1526363          0          0          0   xen-dyn-ipi       resched12
870:          0          0          0          0          0          0          0          0          0          0          0          0       3108          0          0          0   xen-dyn-ipi       spinlock12
871:          0          0          0          0          0          0          0          0          0          0          0          0     867572          0          0          0   xen-dyn-virq      timer12
872:          0          0          0          0          0          0          0          0          0          0          0       7519          0          0          0          0   xen-dyn-ipi       callfuncsingle11
873:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug11
874:          0          0          0          0          0          0          0          0          0          0          0        359          0          0          0          0   xen-dyn-ipi       callfunc11
875:          0          0          0          0          0          0          0          0          0          0          0    2100465          0          0          0          0   xen-dyn-ipi       resched11
876:          0          0          0          0          0          0          0          0          0          0          0        870          0          0          0          0   xen-dyn-ipi       spinlock11
877:          0          0          0          0          0          0          0          0          0          0          0    7609746          0          0          0          0   xen-dyn-virq      timer11
878:          0          0          0          0          0          0          0          0          0          0       8067          0          0          0          0          0   xen-dyn-ipi       callfuncsingle10
879:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug10
880:          0          0          0          0          0          0          0          0          0          0        378          0          0          0          0          0   xen-dyn-ipi       callfunc10
881:          0          0          0          0          0          0          0          0          0          0    2097167          0          0          0          0          0   xen-dyn-ipi       resched10
882:          0          0          0          0          0          0          0          0          0          0        971          0          0          0          0          0   xen-dyn-ipi       spinlock10
883:          0          0          0          0          0          0          0          0          0          0    6046500          0          0          0          0          0   xen-dyn-virq      timer10
884:          0          0          0          0          0          0          0          0          0      10587          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle9
885:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug9
886:          0          0          0          0          0          0          0          0          0        413          0          0          0          0          0          0   xen-dyn-ipi       callfunc9
887:          0          0          0          0          0          0          0          0          0    2085604          0          0          0          0          0          0   xen-dyn-ipi       resched9
888:          0          0          0          0          0          0          0          0          0        778          0          0          0          0          0          0   xen-dyn-ipi       spinlock9
889:          0          0          0          0          0          0          0          0          0    1651038          0          0          0          0          0          0   xen-dyn-virq      timer9
890:          0          0          0          0          0          0          0          0       6936          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle8
891:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug8
892:          0          0          0          0          0          0          0          0        360          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc8
893:          0          0          0          0          0          0          0          0    1916700          0          0          0          0          0          0          0   xen-dyn-ipi       resched8
894:          0          0          0          0          0          0          0          0       3587          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock8
895:          0          0          0          0          0          0          0          0   20032812          0          0          0          0          0          0          0   xen-dyn-virq      timer8
896:          0          0          0          0          0          0          0        296          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle7
897:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug7
898:          0          0          0          0          0          0          0        203          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc7
899:          0          0          0          0          0          0          0    2058143          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched7
900:          0          0          0          0          0          0          0        542          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock7
901:          0          0          0          0          0          0          0   17982743          0          0          0          0          0          0          0          0   xen-dyn-virq      timer7
902:          0          0          0          0          0          0        265          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle6
903:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug6
904:          0          0          0          0          0          0        201          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc6
905:          0          0          0          0          0          0    2051136          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched6
906:          0          0          0          0          0          0        514          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock6
907:          0          0          0          0          0          0   17986644          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer6
908:          0          0          0          0          0        268          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle5
909:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug5
910:          0          0          0          0          0        203          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc5
911:          0          0          0          0          0    2041173          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched5
912:          0          0          0          0          0        655          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock5
913:          0          0          0          0          0   18078108          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer5
914:          0          0          0          0        401          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle4
915:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug4
916:          0          0          0          0        224          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc4
917:          0          0          0          0    1981371          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched4
918:          0          0          0          0        724          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock4
919:          0          0          0          0   18310932          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer4
920:          0          0          0        637          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle3
921:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug3
922:          0          0          0        210          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc3
923:          0          0          0    2047278          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched3
924:          0          0          0        554          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock3
925:          0          0          0   18572009          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer3
926:          0          0        652          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle2
927:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug2
928:          0          0        248          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc2
929:          0          0    2046147          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched2
930:          0          0        528          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock2
931:          0          0   19548306          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer2
932:          0       1585          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle1
933:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug1
934:          0        259          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc1
935:          0    2021949          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched1
936:          0        419          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock1
937:          0   19782004          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer1
938:       1266          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle0
939:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug0
940:        209          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc0
941:    1185473          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched0
942:       1842          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock0
943:   21005497          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer0
NMI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
CNT:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance counter interrupts
PND:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance pending work
RES:    1185473    2021949    2046147    2047278    1981371    2041173    2051136    2058143    1916700    2085604    2097167    2100465    1526363    2085386    2062301    2075595   Rescheduling interrupts
CAL:       1475       1844        900        847        625        471        466        499       7296      11000       8445       7878       5766       3887       2817       3257   Function call interrupts
TLB:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:        284        284        284        284        284        284        284        284        284        284        284        284        284        284        284        284   Machine check polls
ERR:          0
MIS:          0

[-- Attachment #4: interrupts-beforestarting-domUs --]
[-- Type: application/octet-stream, Size: 28676 bytes --]

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      
  3:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge
  4:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge
  8:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-edge  rtc0
  9:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  acpi
 16:        103          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb2, uhci_hcd:usb6, uhci_hcd:usb7, uhci_hcd:usb8
 18:      97643          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ata_piix
 19:        188          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ehci_hcd:usb1, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
 21:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-ioapic-level  ata_piix
816:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
817:        144          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     evtchn:xenstored
818:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1
819:     306177          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-3
820:     416805          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-2
821:     450647          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-1
822:    3353043          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-rx-0
823:       1389          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-3
824:        198          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-2
825:        450          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-1
826:        547          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi-x     eth1-tx-0
840:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
841:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
842:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
843:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
844:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
845:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  xen-pirq-msi       aerdrv
846:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      pcpu
847:        242          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-event     xenbus
848:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         13   xen-dyn-ipi       callfuncsingle15
849:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug15
850:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         33   xen-dyn-ipi       callfunc15
851:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0     514766   xen-dyn-ipi       resched15
852:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0         13   xen-dyn-ipi       spinlock15
853:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0       7331   xen-dyn-virq      timer15
854:          0          0          0          0          0          0          0          0          0          0          0          0          0          0         37          0   xen-dyn-ipi       callfuncsingle14
855:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug14
856:          0          0          0          0          0          0          0          0          0          0          0          0          0          0         37          0   xen-dyn-ipi       callfunc14
857:          0          0          0          0          0          0          0          0          0          0          0          0          0          0     514809          0   xen-dyn-ipi       resched14
858:          0          0          0          0          0          0          0          0          0          0          0          0          0          0         14          0   xen-dyn-ipi       spinlock14
859:          0          0          0          0          0          0          0          0          0          0          0          0          0          0       4122          0   xen-dyn-virq      timer14
860:          0          0          0          0          0          0          0          0          0          0          0          0          0         40          0          0   xen-dyn-ipi       callfuncsingle13
861:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug13
862:          0          0          0          0          0          0          0          0          0          0          0          0          0         38          0          0   xen-dyn-ipi       callfunc13
863:          0          0          0          0          0          0          0          0          0          0          0          0          0     514822          0          0   xen-dyn-ipi       resched13
864:          0          0          0          0          0          0          0          0          0          0          0          0          0         17          0          0   xen-dyn-ipi       spinlock13
865:          0          0          0          0          0          0          0          0          0          0          0          0          0      25341          0          0   xen-dyn-virq      timer13
866:          0          0          0          0          0          0          0          0          0          0          0          0         55          0          0          0   xen-dyn-ipi       callfuncsingle12
867:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug12
868:          0          0          0          0          0          0          0          0          0          0          0          0         36          0          0          0   xen-dyn-ipi       callfunc12
869:          0          0          0          0          0          0          0          0          0          0          0          0     515175          0          0          0   xen-dyn-ipi       resched12
870:          0          0          0          0          0          0          0          0          0          0          0          0         10          0          0          0   xen-dyn-ipi       spinlock12
871:          0          0          0          0          0          0          0          0          0          0          0          0       2871          0          0          0   xen-dyn-virq      timer12
872:          0          0          0          0          0          0          0          0          0          0          0         25          0          0          0          0   xen-dyn-ipi       callfuncsingle11
873:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug11
874:          0          0          0          0          0          0          0          0          0          0          0         33          0          0          0          0   xen-dyn-ipi       callfunc11
875:          0          0          0          0          0          0          0          0          0          0          0     515936          0          0          0          0   xen-dyn-ipi       resched11
876:          0          0          0          0          0          0          0          0          0          0          0         12          0          0          0          0   xen-dyn-ipi       spinlock11
877:          0          0          0          0          0          0          0          0          0          0          0       1830          0          0          0          0   xen-dyn-virq      timer11
878:          0          0          0          0          0          0          0          0          0          0         77          0          0          0          0          0   xen-dyn-ipi       callfuncsingle10
879:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug10
880:          0          0          0          0          0          0          0          0          0          0         53          0          0          0          0          0   xen-dyn-ipi       callfunc10
881:          0          0          0          0          0          0          0          0          0          0     517315          0          0          0          0          0   xen-dyn-ipi       resched10
882:          0          0          0          0          0          0          0          0          0          0          9          0          0          0          0          0   xen-dyn-ipi       spinlock10
883:          0          0          0          0          0          0          0          0          0          0       2390          0          0          0          0          0   xen-dyn-virq      timer10
884:          0          0          0          0          0          0          0          0          0        208          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle9
885:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug9
886:          0          0          0          0          0          0          0          0          0         57          0          0          0          0          0          0   xen-dyn-ipi       callfunc9
887:          0          0          0          0          0          0          0          0          0     506598          0          0          0          0          0          0   xen-dyn-ipi       resched9
888:          0          0          0          0          0          0          0          0          0          4          0          0          0          0          0          0   xen-dyn-ipi       spinlock9
889:          0          0          0          0          0          0          0          0          0      13007          0          0          0          0          0          0   xen-dyn-virq      timer9
890:          0          0          0          0          0          0          0          0        375          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle8
891:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug8
892:          0          0          0          0          0          0          0          0         43          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc8
893:          0          0          0          0          0          0          0          0     251225          0          0          0          0          0          0          0   xen-dyn-ipi       resched8
894:          0          0          0          0          0          0          0          0         78          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock8
895:          0          0          0          0          0          0          0          0     737336          0          0          0          0          0          0          0   xen-dyn-virq      timer8
896:          0          0          0          0          0          0          0          5          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle7
897:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug7
898:          0          0          0          0          0          0          0         31          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc7
899:          0          0          0          0          0          0          0     514632          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched7
900:          0          0          0          0          0          0          0          2          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock7
901:          0          0          0          0          0          0          0       1852          0          0          0          0          0          0          0          0   xen-dyn-virq      timer7
902:          0          0          0          0          0          0         11          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle6
903:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug6
904:          0          0          0          0          0          0         33          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc6
905:          0          0          0          0          0          0     514635          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched6
906:          0          0          0          0          0          0          5          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock6
907:          0          0          0          0          0          0       1023          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer6
908:          0          0          0          0          0          5          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle5
909:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug5
910:          0          0          0          0          0         37          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc5
911:          0          0          0          0          0     514638          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched5
912:          0          0          0          0          0          5          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock5
913:          0          0          0          0          0       1100          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer5
914:          0          0          0          0          7          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle4
915:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug4
916:          0          0          0          0         32          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc4
917:          0          0          0          0     514850          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched4
918:          0          0          0          0         10          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock4
919:          0          0          0          0        965          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer4
920:          0          0          0          5          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle3
921:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug3
922:          0          0          0         32          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc3
923:          0          0          0     514732          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched3
924:          0          0          0         13          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock3
925:          0          0          0       9946          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer3
926:          0          0         20          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle2
927:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug2
928:          0          0         43          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc2
929:          0          0     515853          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched2
930:          0          0         10          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock2
931:          0          0     161964          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer2
932:          0         57          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle1
933:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug1
934:          0         47          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc1
935:          0     511415          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched1
936:          0         10          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock1
937:          0     569729          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer1
938:        142          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfuncsingle0
939:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      debug0
940:         30          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       callfunc0
941:     317658          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       resched0
942:         35          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-ipi       spinlock0
943:     480258          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   xen-dyn-virq      timer0
NMI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
CNT:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance counter interrupts
PND:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance pending work
RES:     317658     511415     515853     514732     514850     514638     514635     514632     251225     506598     517315     515936     515175     514822     514809     514766   Rescheduling interrupts
CAL:        172        104         63         37         39         42         44         36        418        265        130         58         91         78         74         46   Function call interrupts
TLB:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:        215        215        215        215        215        215        215        215        215        215        215        215        215        215        215        215   Machine check polls
ERR:          0
MIS:          0

[-- Attachment #5: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel