* VM start time slows with large number of VMs
@ 2007-09-04 16:51 Carb, Brian A
2007-09-05 17:11 ` Keir Fraser
0 siblings, 1 reply; 13+ messages in thread
From: Carb, Brian A @ 2007-09-04 16:51 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 19286 bytes --]
Hello,
We're running Xen Unstable changeset 15468 with SLES10 on a Unisys
ES7000/one 56x cpu and 160GB memory. Xen is built with NR_DYNIRQS=1024
and max_phys_cpus=64, and booted with dom0_mem=1024 and
xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on
external storage.
Our script starts a VM, waits for the start to complete, then starts the
next: "for i in *.cfg; do xm create $i; done". As more VMs start, the
time to start each one increases. I've attached a config file for one of
the VMs, as well as a portion of the xend.log which shows timestamps of
the events for 2 VMS, vm002 and vm263.
The total elapsed time of the start from "create" to "unpaused" for
vm002 is 13 seconds, but by the time 262 VMs are started, the start of
vm263 has slowed to 238 seconds. The host itself is not sluggish, as we
can open putty sessions to and interact with dom0. However, if we run
"xm top", its interface is very unresponsive. Could writing to the
xenstore be taking longer and longer? Could there be possible contention
for vmlinuz and initrd since they are being used by every VM to boot?
Would it be better to use bootloader/bootentry and store the vmlinuz and
initrd within each VM?
Any insight would be appreciated.
brian carb
unisys corporation, malvern pa
----- domU config file 002.cfg -----
kernel = "/boot/vmlinuz-xen"
ramdisk = "/boot/initrd-xen"
root = "/dev/sda2 ro"
disk = [ 'phy:/tmp/xendevs/xen002,sda,w' ]
memory = 512
vcpus = 4
name = 'xen002'
vif = [ 'mac=00:16:3e:0d:a0:01' ]
ip = '10.0.27.2'
netmask = '255.254.0.0'
extra = ' xencons=tty1 3'
----- xend.log -----
[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78)
XendDomainInfo.create(['vm', ['name', 'xen002'], ['memory', 512],
['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], ['ramdisk',
'/boot/initrd-xen'], ['ip', '10.0.27.2:1.2.3.4::255.254.0.0::eth0:off'],
['root', '/dev/sda2 ro'], ['args', ' xencons=tty1 3']]], ['device',
['vbd', ['uname', 'phy:/tmp/xendevs/xen002'], ['dev', 'sda'], ['mode',
'w']]], ['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]])
[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418)
XendDomainInfo.constructDomain
[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 KiB
free; need 2048; done.
[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16
[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472)
XendDomainInfo.initDomain: 16 256
[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503)
_initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
memory_static_min=0x0.
[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 KiB
free; need 524288; done.
[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux dom=16
vcpus=4
[2007-09-02 05:26:43 8597] DEBUG (image:197) domid = 16
[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize = 512
[2007-09-02 05:26:43 8597] DEBUG (image:199) image =
/boot/vmlinuz-xen
[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn = 1
[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2
[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline =
root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off
xencons=tty1 3
[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk =
/boot/initrd-xen
[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus = 4
[2007-09-02 05:26:43 8597] DEBUG (image:205) features =
[2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice: vbd
: {'uuid': '8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'bootable': 1,
'driver': 'paravirtualised', 'dev': 'sda', 'uname':
'phy:/tmp/xendevs/xen002', 'mode': 'w'}
[2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController:
writing {'virtual-device': '2048', 'device-type': 'disk', 'protocol':
'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vbd/16/2048'} to
/local/domain/16/device/vbd/2048.
[2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController:
writing {'domain': 'xen002', 'frontend':
'/local/domain/16/device/vbd/2048', 'uuid':
'8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'dev': 'sda', 'state': '1',
'params': '/tmp/xendevs/xen002', 'mode': 'w', 'online': '1',
'frontend-id': '16', 'type': 'phy'} to
/local/domain/0/backend/vbd/16/2048.
[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: vif
: {'mac': '00:16:3e:0d:a0:01', 'uuid':
'bb43e84a-a9a1-7488-1ba4-3b713eff82aa'}
[2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController:
writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol':
'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vif/16/0'} to /local/domain/16/device/vif/0.
[2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController:
writing {'domain': 'xen002', 'handle': '0', 'uuid':
'bb43e84a-a9a1-7488-1ba4-3b713eff82aa', 'script':
'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend':
'/local/domain/16/device/vif/0', 'mac': '00:16:3e:0d:a0:01', 'online':
'1', 'frontend-id': '16', 'type': 'netfront'} to
/local/domain/0/backend/vif/16/0.
[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM
details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid':
'f146fdfc-e624-f7cb-4baf-0471f01f6014', 'on_reboot': 'restart',
'start_time': '1188725206.15', 'on_poweroff': 'destroy',
'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count':
'0', 'vcpus': '4', 'vcpu_avail': '15', 'image': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (args 'root=/dev/sda2 ro
ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') (notes
(FEATURES
'writable_page_tables|writable_descriptor_tables|auto_translated_physmap
|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE -2147483648)
(GUEST_VERSION 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux)
(HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY
-2145386496) (XEN_VERSION xen-3.0)))", 'name': 'xen002'}
[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain
details: {'console/ring-ref': '34194913', 'image/entry': '-2145386496',
'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref':
'34194914', 'image/loader': 'generic', 'vm':
'/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014',
'control/platform-feature-multiprocessor-suspend': '1',
'cpu/2/availability': 'online', 'cpu/1/availability': 'online',
'image/features/writable-descriptor-tables': '1', 'image/virt-base':
'-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6',
'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux',
'console/limit': '1048576', 'image/paddr-offset': '-2147483648',
'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1',
'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb':
'1', 'image/features/writable-page-tables': '1',
'image/features/auto-translated-physmap': '1', 'name': 'xen002',
'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController:
writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0',
'backend': '/local/domain/0/backend/console/16/0'} to
/local/domain/16/device/console/0.
[2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController:
writing {'domain': 'xen002', 'protocol': 'vt100', 'uuid':
'8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e', 'frontend':
'/local/domain/16/device/console/0', 'state': '1', 'location': '2',
'online': '1', 'frontend-id': '16'} to
/local/domain/0/backend/console/16/0.
[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for devices
vif.
[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0.
[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain
details: {'console/ring-ref': '34194913', 'image/entry': '-2145386496',
'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref':
'34194914', 'image/loader': 'generic', 'vm':
'/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014',
'control/platform-feature-multiprocessor-suspend': '1',
'cpu/2/availability': 'online', 'cpu/1/availability': 'online',
'image/features/writable-descriptor-tables': '1', 'image/virt-base':
'-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6',
'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux',
'console/limit': '1048576', 'image/paddr-offset': '-2147483648',
'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1',
'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb':
'1', 'image/features/writable-page-tables': '1',
'image/features/auto-translated-physmap': '1', 'name': 'xen002',
'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907)
XendDomainInfo.handleShutdownWatch
[2007-09-02 05:26:53 8597] DEBUG (DevController:541)
hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status.
[2007-09-02 05:26:53 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
usb.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vbd.
[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048.
[2007-09-02 05:26:54 8597] DEBUG (DevController:541)
hotplugStatusCallback
/local/domain/0/backend/vbd/16/2048/hotplug-status.
[2007-09-02 05:26:54 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
irq.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vkbd.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vfb.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
console.
[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
pci.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
ioports.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
tap.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
vtpm.
[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16)
unpaused.
...
... [vms xen003 to xen262 deleted]
...
[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78)
XendDomainInfo.create(['vm', ['name', 'xen263'], ['memory', 512],
['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], ['ramdisk',
'/boot/initrd-xen'], ['ip',
'10.0.29.63:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 ro'],
['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname',
'phy:/tmp/xendevs/xen263'], ['dev', 'sda'], ['mode', 'w']]], ['device',
['vif', ['mac', '00:16:3e:0d:a0:01']]]])
[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418)
XendDomainInfo.constructDomain
[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB
free; need 2048; done.
[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277
[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472)
XendDomainInfo.initDomain: 277 256
[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503)
_initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
memory_static_min=0x0.
[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB
free; need 524288; done.
[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux dom=277
vcpus=4
[2007-09-02 17:44:29 8597] DEBUG (image:197) domid = 277
[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize = 512
[2007-09-02 17:44:29 8597] DEBUG (image:199) image =
/boot/vmlinuz-xen
[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn = 1
[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2
[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline =
root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off
xencons=tty1 3
[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk =
/boot/initrd-xen
[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus = 4
[2007-09-02 17:44:29 8597] DEBUG (image:205) features =
[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vbd
: {'uuid': '9d639202-73af-64ec-451d-3abe0fc57bc7', 'bootable': 1,
'driver': 'paravirtualised', 'dev': 'sda', 'uname':
'phy:/tmp/xendevs/xen263', 'mode': 'w'}
[2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController:
writing {'virtual-device': '2048', 'device-type': 'disk', 'protocol':
'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vbd/277/2048'} to
/local/domain/277/device/vbd/2048.
[2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController:
writing {'domain': 'xen263', 'frontend':
'/local/domain/277/device/vbd/2048', 'uuid':
'9d639202-73af-64ec-451d-3abe0fc57bc7', 'dev': 'sda', 'state': '1',
'params': '/tmp/xendevs/xen263', 'mode': 'w', 'online': '1',
'frontend-id': '277', 'type': 'phy'} to
/local/domain/0/backend/vbd/277/2048.
[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vif
: {'mac': '00:16:3e:0d:a0:01', 'uuid':
'a126d815-fd4a-332b-c881-235a7e003cfb'}
[2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController:
writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol':
'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vif/277/0'} to /local/domain/277/device/vif/0.
[2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController:
writing {'domain': 'xen263', 'handle': '0', 'uuid':
'a126d815-fd4a-332b-c881-235a7e003cfb', 'script':
'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend':
'/local/domain/277/device/vif/0', 'mac': '00:16:3e:0d:a0:01', 'online':
'1', 'frontend-id': '277', 'type': 'netfront'} to
/local/domain/0/backend/vif/277/0.
[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM
details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid':
'8f8f9839-dd61-b9cd-89a6-f66bb3d41726', 'on_reboot': 'restart',
'start_time': '1188769556.11', 'on_poweroff': 'destroy',
'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count':
'0', 'vcpus': '4', 'vcpu_avail': '15', 'image': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (args 'root=/dev/sda2 ro
ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') (notes
(FEATURES
'writable_page_tables|writable_descriptor_tables|auto_translated_physmap
|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE -2147483648)
(GUEST_VERSION 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux)
(HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY
-2145386496) (XEN_VERSION xen-3.0)))", 'name': 'xen263'}
[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain
details: {'console/ring-ref': '5227479', 'image/entry': '-2145386496',
'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref':
'5227480', 'image/loader': 'generic', 'vm':
'/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726',
'control/platform-feature-multiprocessor-suspend': '1',
'cpu/2/availability': 'online', 'cpu/1/availability': 'online',
'image/features/writable-descriptor-tables': '1', 'image/virt-base':
'-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6',
'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux',
'console/limit': '1048576', 'image/paddr-offset': '-2147483648',
'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1',
'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb':
'1', 'image/features/writable-page-tables': '1',
'image/features/auto-translated-physmap': '1', 'name': 'xen263',
'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController:
writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0',
'backend': '/local/domain/0/backend/console/277/0'} to
/local/domain/277/device/console/0.
[2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController:
writing {'domain': 'xen263', 'protocol': 'vt100', 'uuid':
'd0da45dd-1a00-da65-61dd-66dadd6e9897', 'frontend':
'/local/domain/277/device/console/0', 'state': '1', 'location': '2',
'online': '1', 'frontend-id': '277'} to
/local/domain/0/backend/console/277/0.
[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain
details: {'console/ring-ref': '5227479', 'image/entry': '-2145386496',
'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref':
'5227480', 'image/loader': 'generic', 'vm':
'/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726',
'control/platform-feature-multiprocessor-suspend': '1',
'cpu/2/availability': 'online', 'cpu/1/availability': 'online',
'image/features/writable-descriptor-tables': '1', 'image/virt-base':
'-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6',
'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux',
'console/limit': '1048576', 'image/paddr-offset': '-2147483648',
'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1',
'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb':
'1', 'image/features/writable-page-tables': '1',
'image/features/auto-translated-physmap': '1', 'name': 'xen263',
'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907)
XendDomainInfo.handleShutdownWatch
[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for devices
vif.
[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0.
[2007-09-02 17:48:23 8597] DEBUG (DevController:541)
hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status.
[2007-09-02 17:48:23 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices
usb.
[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices
vbd.
[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048.
[2007-09-02 17:48:25 8597] DEBUG (DevController:541)
hotplugStatusCallback
/local/domain/0/backend/vbd/277/2048/hotplug-status.
[2007-09-02 17:48:25 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
irq.
[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
vkbd.
[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
vfb.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
console.
[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
pci.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
ioports.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
tap.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
vtpm.
[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 (277)
unpaused.
[-- Attachment #1.2: Type: text/html, Size: 22673 bytes --]
[-- Attachment #2: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: VM start time slows with large number of VMs 2007-09-04 16:51 VM start time slows with large number of VMs Carb, Brian A @ 2007-09-05 17:11 ` Keir Fraser 2007-09-05 17:37 ` Daniel P. Berrange 2007-09-06 16:24 ` Carb, Brian A 0 siblings, 2 replies; 13+ messages in thread From: Keir Fraser @ 2007-09-05 17:11 UTC (permalink / raw) To: Carb, Brian A, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 20251 bytes --] xenstore slowness seems a reasonable hypothesis. How does performance of Œtime xenstore-ls >/dev/null¹ change as you increase the number of guests? -- Keir On 4/9/07 17:51, "Carb, Brian A" <Brian.Carb@unisys.com> wrote: > Hello, > > We're running Xen Unstable changeset 15468 with SLES10 on a Unisys ES7000/one > 56x cpu and 160GB memory. Xen is built with NR_DYNIRQS=1024 and > max_phys_cpus=64, and booted with dom0_mem=1024 and xenheap_megabytes=64. We > have 320 para-virtualized SLES10 VMs on external storage. > > Our script starts a VM, waits for the start to complete, then starts the next: > "for i in *.cfg; do xm create $i; done". As more VMs start, the time to start > each one increases. I've attached a config file for one of the VMs, as well as > a portion of the xend.log which shows timestamps of the events for 2 VMS, > vm002 and vm263. > > The total elapsed time of the start from "create" to "unpaused" for vm002 is > 13 seconds, but by the time 262 VMs are started, the start of vm263 has slowed > to 238 seconds. The host itself is not sluggish, as we can open putty sessions > to and interact with dom0. However, if we run "xm top", its interface is very > unresponsive. Could writing to the xenstore be taking longer and longer? > Could there be possible contention for vmlinuz and initrd since they are being > used by every VM to boot? Would it be better to use bootloader/bootentry and > store the vmlinuz and initrd within each VM? > > Any insight would be appreciated. > > brian carb > unisys corporation, malvern pa > > ----- domU config file 002.cfg ----- > kernel = "/boot/vmlinuz-xen" > ramdisk = "/boot/initrd-xen" > root = "/dev/sda2 ro" > disk = [ 'phy:/tmp/xendevs/xen002,sda,w' ] > memory = 512 > vcpus = 4 > name = 'xen002' > vif = [ 'mac=00:16:3e:0d:a0:01' ] > ip = '10.0.27.2' > netmask = '255.254.0.0' > extra = ' xencons=tty1 3' > ----- xend.log ----- > [2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78) > XendDomainInfo.create(['vm', ['name', 'xen002'], ['memory', 512], ['vcpus', > 4], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', > ['linux', ['kernel', '/boot/vmlinuz-xen'], ['ramdisk', '/boot/initrd-xen'], > ['ip', '10.0.27.2:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 ro'], > ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', > 'phy:/tmp/xendevs/xen002'], ['dev', 'sda'], ['mode', 'w']]], ['device', > ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) > [2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418) > XendDomainInfo.constructDomain > [2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 KiB free; > need 2048; done. > [2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16 > [2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472) > XendDomainInfo.initDomain: 16 256 > [2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503) > _initDomain:shadow_memory=0x0, memory_static_max=0x20000000, > memory_static_min=0x0. > [2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 KiB free; > need 524288; done. > [2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux dom=16 > vcpus=4 > [2007-09-02 05:26:43 8597] DEBUG (image:197) domid = 16 > [2007-09-02 05:26:43 8597] DEBUG (image:198) memsize = 512 > [2007-09-02 05:26:43 8597] DEBUG (image:199) image = > /boot/vmlinuz-xen > [2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn = 1 > [2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2 > [2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline = root=/dev/sda2 > ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3 > [2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk = /boot/initrd-xen > [2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus = 4 > [2007-09-02 05:26:43 8597] DEBUG (image:205) features = > [2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice: vbd : > {'uuid': '8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'bootable': 1, 'driver': > 'paravirtualised', 'dev': 'sda', 'uname': 'phy:/tmp/xendevs/xen002', 'mode': > 'w'} > [2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController: writing > {'virtual-device': '2048', 'device-type': 'disk', 'protocol': 'x86_64-abi', > 'backend-id': '0', 'state': '1', 'backend': > '/local/domain/0/backend/vbd/16/2048'} to /local/domain/16/device/vbd/2048. > [2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController: writing > {'domain': 'xen002', 'frontend': '/local/domain/16/device/vbd/2048', 'uuid': > '8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'dev': 'sda', 'state': '1', 'params': > '/tmp/xendevs/xen002', 'mode': 'w', 'online': '1', 'frontend-id': '16', > 'type': 'phy'} to /local/domain/0/backend/vbd/16/2048. > [2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: vif : > {'mac': '00:16:3e:0d:a0:01', 'uuid': 'bb43e84a-a9a1-7488-1ba4-3b713eff82aa'} > [2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController: writing > {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': 'x86_64-abi', > 'backend-id': '0', 'state': '1', 'backend': > '/local/domain/0/backend/vif/16/0'} to /local/domain/16/device/vif/0. > [2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController: writing > {'domain': 'xen002', 'handle': '0', 'uuid': > 'bb43e84a-a9a1-7488-1ba4-3b713eff82aa', 'script': > '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': > '/local/domain/16/device/vif/0', 'mac': '00:16:3e:0d:a0:01', 'online': '1', > 'frontend-id': '16', 'type': 'netfront'} to /local/domain/0/backend/vif/16/0. > [2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM details: > {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': > 'f146fdfc-e624-f7cb-4baf-0471f01f6014', 'on_reboot': 'restart', 'start_time': > '1188725206.15', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', > 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': > '15', 'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) > (args 'root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off > xencons=tty1 3') (notes (FEATURES > 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_p > gdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE -2147483648) (GUEST_VERSION > 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) > (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION > xen-3.0)))", 'name': 'xen002'} > [2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain details: > {'console/ring-ref': '34194913', 'image/entry': '-2145386496', 'console/port': > '2', 'cpu/3/availability': 'online', 'store/ring-ref': '34194914', > 'image/loader': 'generic', 'vm': '/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', > 'control/platform-feature-multiprocessor-suspend': '1', 'cpu/2/availability': > 'online', 'cpu/1/availability': 'online', > 'image/features/writable-descriptor-tables': '1', 'image/virt-base': > '-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6', > 'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux', > 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', > 'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1', > 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', > 'image/features/writable-page-tables': '1', > 'image/features/auto-translated-physmap': '1', 'name': 'xen002', 'domid': > '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} > [2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController: writing > {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': > '/local/domain/0/backend/console/16/0'} to /local/domain/16/device/console/0. > [2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController: writing > {'domain': 'xen002', 'protocol': 'vt100', 'uuid': > '8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e', 'frontend': > '/local/domain/16/device/console/0', 'state': '1', 'location': '2', 'online': > '1', 'frontend-id': '16'} to /local/domain/0/backend/console/16/0. > [2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for devices vif. > [2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0. > [2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain details: > {'console/ring-ref': '34194913', 'image/entry': '-2145386496', 'console/port': > '2', 'cpu/3/availability': 'online', 'store/ring-ref': '34194914', > 'image/loader': 'generic', 'vm': '/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', > 'control/platform-feature-multiprocessor-suspend': '1', 'cpu/2/availability': > 'online', 'cpu/1/availability': 'online', > 'image/features/writable-descriptor-tables': '1', 'image/virt-base': > '-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6', > 'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux', > 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', > 'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1', > 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', > 'image/features/writable-page-tables': '1', > 'image/features/auto-translated-physmap': '1', 'name': 'xen002', 'domid': > '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} > [2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907) > XendDomainInfo.handleShutdownWatch > [2007-09-02 05:26:53 8597] DEBUG (DevController:541) hotplugStatusCallback > /local/domain/0/backend/vif/16/0/hotplug-status. > [2007-09-02 05:26:53 8597] DEBUG (DevController:555) hotplugStatusCallback 1. > [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices usb. > [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vbd. > [2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048. > [2007-09-02 05:26:54 8597] DEBUG (DevController:541) hotplugStatusCallback > /local/domain/0/backend/vbd/16/2048/hotplug-status. > [2007-09-02 05:26:54 8597] DEBUG (DevController:555) hotplugStatusCallback 1. > [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices irq. > [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vkbd. > [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vfb. > [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices > console. > [2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0. > [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices pci. > [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices > ioports. > [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices tap. > [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices vtpm. > [2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16) unpaused. > ... > ... [vms xen003 to xen262 deleted] > ... > [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78) > XendDomainInfo.create(['vm', ['name', 'xen263'], ['memory', 512], ['vcpus', > 4], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', > ['linux', ['kernel', '/boot/vmlinuz-xen'], ['ramdisk', '/boot/initrd-xen'], > ['ip', '10.0.29.63:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 ro'], > ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', > 'phy:/tmp/xendevs/xen263'], ['dev', 'sda'], ['mode', 'w']]], ['device', > ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) > [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418) > XendDomainInfo.constructDomain > [2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB free; > need 2048; done. > [2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277 > [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472) > XendDomainInfo.initDomain: 277 256 > [2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503) > _initDomain:shadow_memory=0x0, memory_static_max=0x20000000, > memory_static_min=0x0. > [2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB free; > need 524288; done. > [2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux dom=277 > vcpus=4 > [2007-09-02 17:44:29 8597] DEBUG (image:197) domid = 277 > [2007-09-02 17:44:29 8597] DEBUG (image:198) memsize = 512 > [2007-09-02 17:44:29 8597] DEBUG (image:199) image = > /boot/vmlinuz-xen > [2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn = 1 > [2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2 > [2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline = root=/dev/sda2 > ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3 > [2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk = /boot/initrd-xen > [2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus = 4 > [2007-09-02 17:44:29 8597] DEBUG (image:205) features = > [2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vbd : > {'uuid': '9d639202-73af-64ec-451d-3abe0fc57bc7', 'bootable': 1, 'driver': > 'paravirtualised', 'dev': 'sda', 'uname': 'phy:/tmp/xendevs/xen263', 'mode': > 'w'} > [2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController: writing > {'virtual-device': '2048', 'device-type': 'disk', 'protocol': 'x86_64-abi', > 'backend-id': '0', 'state': '1', 'backend': > '/local/domain/0/backend/vbd/277/2048'} to /local/domain/277/device/vbd/2048. > [2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController: writing > {'domain': 'xen263', 'frontend': '/local/domain/277/device/vbd/2048', 'uuid': > '9d639202-73af-64ec-451d-3abe0fc57bc7', 'dev': 'sda', 'state': '1', 'params': > '/tmp/xendevs/xen263', 'mode': 'w', 'online': '1', 'frontend-id': '277', > 'type': 'phy'} to /local/domain/0/backend/vbd/277/2048. > [2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vif : > {'mac': '00:16:3e:0d:a0:01', 'uuid': 'a126d815-fd4a-332b-c881-235a7e003cfb'} > [2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController: writing > {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': 'x86_64-abi', > 'backend-id': '0', 'state': '1', 'backend': > '/local/domain/0/backend/vif/277/0'} to /local/domain/277/device/vif/0. > [2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController: writing > {'domain': 'xen263', 'handle': '0', 'uuid': > 'a126d815-fd4a-332b-c881-235a7e003cfb', 'script': > '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': > '/local/domain/277/device/vif/0', 'mac': '00:16:3e:0d:a0:01', 'online': '1', > 'frontend-id': '277', 'type': 'netfront'} to > /local/domain/0/backend/vif/277/0. > [2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM details: > {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': > '8f8f9839-dd61-b9cd-89a6-f66bb3d41726', 'on_reboot': 'restart', 'start_time': > '1188769556.11', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', > 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': > '15', 'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) > (args 'root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off > xencons=tty1 3') (notes (FEATURES > 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_p > gdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE -2147483648) (GUEST_VERSION > 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) > (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION > xen-3.0)))", 'name': 'xen263'} > [2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain details: > {'console/ring-ref': '5227479', 'image/entry': '-2145386496', 'console/port': > '2', 'cpu/3/availability': 'online', 'store/ring-ref': '5227480', > 'image/loader': 'generic', 'vm': '/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', > 'control/platform-feature-multiprocessor-suspend': '1', 'cpu/2/availability': > 'online', 'cpu/1/availability': 'online', > 'image/features/writable-descriptor-tables': '1', 'image/virt-base': > '-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6', > 'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux', > 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', > 'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1', > 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', > 'image/features/writable-page-tables': '1', > 'image/features/auto-translated-physmap': '1', 'name': 'xen263', 'domid': > '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} > [2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController: writing > {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': > '/local/domain/0/backend/console/277/0'} to > /local/domain/277/device/console/0. > [2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController: writing > {'domain': 'xen263', 'protocol': 'vt100', 'uuid': > 'd0da45dd-1a00-da65-61dd-66dadd6e9897', 'frontend': > '/local/domain/277/device/console/0', 'state': '1', 'location': '2', 'online': > '1', 'frontend-id': '277'} to /local/domain/0/backend/console/277/0. > [2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain details: > {'console/ring-ref': '5227479', 'image/entry': '-2145386496', 'console/port': > '2', 'cpu/3/availability': 'online', 'store/ring-ref': '5227480', > 'image/loader': 'generic', 'vm': '/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', > 'control/platform-feature-multiprocessor-suspend': '1', 'cpu/2/availability': > 'online', 'cpu/1/availability': 'online', > 'image/features/writable-descriptor-tables': '1', 'image/virt-base': > '-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6', > 'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux', > 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', > 'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1', > 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', > 'image/features/writable-page-tables': '1', > 'image/features/auto-translated-physmap': '1', 'name': 'xen263', 'domid': > '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} > [2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907) > XendDomainInfo.handleShutdownWatch > [2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for devices vif. > [2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0. > [2007-09-02 17:48:23 8597] DEBUG (DevController:541) hotplugStatusCallback > /local/domain/0/backend/vif/277/0/hotplug-status. > [2007-09-02 17:48:23 8597] DEBUG (DevController:555) hotplugStatusCallback 1. > [2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices usb. > [2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices vbd. > [2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048. > [2007-09-02 17:48:25 8597] DEBUG (DevController:541) hotplugStatusCallback > /local/domain/0/backend/vbd/277/2048/hotplug-status. > [2007-09-02 17:48:25 8597] DEBUG (DevController:555) hotplugStatusCallback 1. > [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices irq. > [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices vkbd. > [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices vfb. > [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices > console. > [2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0. > [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices pci. > [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices > ioports. > [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices tap. > [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices vtpm. > [2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 (277) > unpaused. > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel [-- Attachment #1.2: Type: text/html, Size: 22005 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: VM start time slows with large number of VMs 2007-09-05 17:11 ` Keir Fraser @ 2007-09-05 17:37 ` Daniel P. Berrange 2007-09-11 16:27 ` Carb, Brian A 2007-09-24 14:18 ` Carb, Brian A 2007-09-06 16:24 ` Carb, Brian A 1 sibling, 2 replies; 13+ messages in thread From: Daniel P. Berrange @ 2007-09-05 17:37 UTC (permalink / raw) To: Keir Fraser; +Cc: xen-devel, Carb, Brian A On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser wrote: > xenstore slowness seems a reasonable hypothesis. How does performance of > Œtime xenstore-ls >/dev/null¹ change as you increase the number of guests? There's another issue with the way hotplug scripts do locking which exhbits fairly pathelogical behaviour when a large number of guests are started concurrently. We've got a patch we're evaluating which changes the use of lock files + looping + sleeps, to instead use the 'flock' shell util. I suspect the performance issues will be a combination of both the locking stuff and the xenstore I/O. You might also want to try putting /var/lib/xenstored onto tmpfs eg in initscripts before xenstored starts ... mkdir /dev/shm/xenstored mount --bind /dev/shm/xenstored /var/lib/xenstored Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: VM start time slows with large number of VMs 2007-09-05 17:37 ` Daniel P. Berrange @ 2007-09-11 16:27 ` Carb, Brian A [not found] ` <089B0D75973E1241B941D0A9854F23FC08F4CE10@USEA-EXCH2.na.uis .unisys.com> 2007-09-24 14:18 ` Carb, Brian A 1 sibling, 1 reply; 13+ messages in thread From: Carb, Brian A @ 2007-09-11 16:27 UTC (permalink / raw) To: Daniel P. Berrange, Keir Fraser; +Cc: xen-devel, Carb, Brian A I tried timing xenstore-ls and then tried it again with the shared memory idea (adding the 2 lines to the /etc/init.d/rc3.0/S10xend script before it starts xend). Here are the average times for xenstore-ls: # vms std shm 001-020 .67 .59 021-040 1.70 1.54 041-060 2.41 2.36 061-080 3.26 3.14 081-100 4.06 3.95 101-120 4.96 4.90 121-140 6.56 6,02 141-160 7.33 7.21 161-180 8.00 8.62 181-200 9.90 9.61 Without using shared memory, the total elapsed time to start 200 VMs is about 95 minutes, vs 81 minutes with the shm statements. Time to start the 200th vm, from create to unpaused, is about 55 seconds, and with xenstore in shared memory that time is about 43 seconds. I'm not seeing the 200+ seconds I was seeing before, but that time was for vm263 and I've only been able to test with 200 VMs. This week I'll try to test with more. Since the shared and non-shared memory results are so close, how can I verify that the xenstore data is actually going in and out of shared memory? It seems as if the start times became reduced once I cleared out the old info from xenstore, regardless of shared memory. Is it worth trying bootloader/bootentry instead of kernel/ramdisk as well? brian carb unisys corporation - malvern, pa -----Original Message----- From: Daniel P. Berrange Sent: Wednesday, September 05, 2007 1:38 PM Subject: Re: [Xen-devel] VM start time slows with large number of VMs On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser wrote: > xenstore slowness seems a reasonable hypothesis. How does performance > of Œtime xenstore-ls >/dev/null¹ change as you increase the number of guests? There's another issue with the way hotplug scripts do locking which exhbits fairly pathelogical behaviour when a large number of guests are started concurrently. We've got a patch we're evaluating which changes the use of lock files + looping + sleeps, to instead use the 'flock' shell util. I suspect the performance issues will be a combination of both the locking stuff and the xenstore I/O. You might also want to try putting /var/lib/xenstored onto tmpfs eg in initscripts before xenstored starts ... mkdir /dev/shm/xenstored mount --bind /dev/shm/xenstored /var/lib/xenstored Dan. ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <089B0D75973E1241B941D0A9854F23FC08F4CE10@USEA-EXCH2.na.uis .unisys.com>]
* RE: VM start time slows with large number of VMs [not found] ` <089B0D75973E1241B941D0A9854F23FC08F4CE10@USEA-EXCH2.na.uis .unisys.com> @ 2007-09-11 17:42 ` Mats Petersson 0 siblings, 0 replies; 13+ messages in thread From: Mats Petersson @ 2007-09-11 17:42 UTC (permalink / raw) To: Daniel P. Berrange, Keir Fraser; +Cc: xen-devel, Carb, Brian A At 17:27 11/09/2007, Carb, Brian A wrote: > I tried timing xenstore-ls and then tried it again with the shared > memory idea (adding the 2 lines to the /etc/init.d/rc3.0/S10xend > script before it starts xend). Here are the average times for xenstore-ls: > > # vms std shm >001-020 .67 .59 >021-040 1.70 1.54 >041-060 2.41 2.36 >061-080 3.26 3.14 >081-100 4.06 3.95 >101-120 4.96 4.90 >121-140 6.56 6,02 >141-160 7.33 7.21 >161-180 8.00 8.62 >181-200 9.90 9.61 > >Without using shared memory, the total elapsed time to start 200 VMs >is about 95 minutes, vs 81 minutes with the shm statements. Time to >start the 200th vm, from create to unpaused, is about 55 seconds, >and with xenstore in shared memory that time is about 43 seconds. >I'm not seeing the 200+ seconds I was seeing before, but that time >was for vm263 and I've only been able to test with 200 VMs. This >week I'll try to test with more. > >Since the shared and non-shared memory results are so close, how can >I verify that the xenstore data is actually going in and out of >shared memory? It seems as if the start times became reduced once I >cleared out the old info from xenstore, regardless of shared memory. That would make sense. I'm not sure how well the xenstore structure is designed, with respect to holding hundred and thousands of different entries - I had a quick look at it, but not really looking at it deeply, just very briefly - the code I was looking at seemed to do a lot of small function calls to dig data out of the database a byte or some few bytes at a time, which didn't make me say "Wow, that's clever". -- Mats >Is it worth trying bootloader/bootentry instead of kernel/ramdisk as well? > >brian carb >unisys corporation - malvern, pa > >-----Original Message----- >From: Daniel P. Berrange >Sent: Wednesday, September 05, 2007 1:38 PM >Subject: Re: [Xen-devel] VM start time slows with large number of VMs > >On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser wrote: > > xenstore slowness seems a reasonable hypothesis. How does performance > > of Œtime xenstore-ls >/dev/null¹ change as you increase the > number of guests? > >There's another issue with the way hotplug scripts do locking which >exhbits fairly pathelogical behaviour when a large number of guests >are started concurrently. We've got a patch we're evaluating which >changes the use of lock files + looping + sleeps, to instead use >the 'flock' shell util. > >I suspect the performance issues will be a combination of both the >locking stuff and the xenstore I/O. You might also want to try >putting /var/lib/xenstored onto tmpfs eg in initscripts before >xenstored starts ... > > mkdir /dev/shm/xenstored > mount --bind /dev/shm/xenstored /var/lib/xenstored > > >Dan. > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: VM start time slows with large number of VMs 2007-09-05 17:37 ` Daniel P. Berrange 2007-09-11 16:27 ` Carb, Brian A @ 2007-09-24 14:18 ` Carb, Brian A 1 sibling, 0 replies; 13+ messages in thread From: Carb, Brian A @ 2007-09-24 14:18 UTC (permalink / raw) To: Daniel P. Berrange, Keir Fraser; +Cc: xen-devel We compared total elapsed time to start 320 para VMs on ES7000 (56x 160gb) where xenstore was on physical disk vs ram disk: physical: 5 hours 19 minutes ram disk: 4 hours 2 minutes brian carb unisys corporation - malvern, pa -----Original Message----- From: Daniel P. Berrange [mailto:berrange@redhat.com] Sent: Wednesday, September 05, 2007 1:38 PM To: Keir Fraser Cc: Carb, Brian A; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] VM start time slows with large number of VMs On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser wrote: > xenstore slowness seems a reasonable hypothesis. How does performance > of Œtime xenstore-ls >/dev/null¹ change as you increase the number of guests? There's another issue with the way hotplug scripts do locking which exhbits fairly pathelogical behaviour when a large number of guests are started concurrently. We've got a patch we're evaluating which changes the use of lock files + looping + sleeps, to instead use the 'flock' shell util. I suspect the performance issues will be a combination of both the locking stuff and the xenstore I/O. You might also want to try putting /var/lib/xenstored onto tmpfs eg in initscripts before xenstored starts ... mkdir /dev/shm/xenstored mount --bind /dev/shm/xenstored /var/lib/xenstored Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: VM start time slows with large number of VMs 2007-09-05 17:11 ` Keir Fraser 2007-09-05 17:37 ` Daniel P. Berrange @ 2007-09-06 16:24 ` Carb, Brian A [not found] ` <089B0D75973E1241B941D0A9854F23FC08E92D5C@USEA-EXCH2.na.uis .unisys.com> 1 sibling, 1 reply; 13+ messages in thread From: Carb, Brian A @ 2007-09-06 16:24 UTC (permalink / raw) To: Keir Fraser, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 20575 bytes --] keir I'll try timing this as I load up... but when I restart my test system, with only dom0 running, xenstore-ls is already slow before any VMs are started: time xenstore-ls >/dev/null real 0m21.725s user 0m0.010s sys 0m0.090s If I save the output from the xenstore-ls, the file has 61,318 lines with information from all the VMs run yesterday. I conclude that xenstore does not get cleared on startup. Is there a way to clear it? brian carb unisys corporation - malvern, pa ________________________________ From: Keir Fraser Sent: Wednesday, September 05, 2007 1:11 PM To: Carb, Brian A; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] VM start time slows with large number of VMs xenstore slowness seems a reasonable hypothesis. How does performance of 'time xenstore-ls >/dev/null' change as you increase the number of guests? -- Keir On 4/9/07 17:51, "Carb, Brian A" wrote: Hello, We're running Xen Unstable changeset 15468 with SLES10 on a Unisys ES7000/one 56x cpu and 160GB memory. Xen is built with NR_DYNIRQS=1024 and max_phys_cpus=64, and booted with dom0_mem=1024 and xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on external storage. Our script starts a VM, waits for the start to complete, then starts the next: "for i in *.cfg; do xm create $i; done". As more VMs start, the time to start each one increases. I've attached a config file for one of the VMs, as well as a portion of the xend.log which shows timestamps of the events for 2 VMS, vm002 and vm263. The total elapsed time of the start from "create" to "unpaused" for vm002 is 13 seconds, but by the time 262 VMs are started, the start of vm263 has slowed to 238 seconds. The host itself is not sluggish, as we can open putty sessions to and interact with dom0. However, if we run "xm top", its interface is very unresponsive. Could writing to the xenstore be taking longer and longer? Could there be possible contention for vmlinuz and initrd since they are being used by every VM to boot? Would it be better to use bootloader/bootentry and store the vmlinuz and initrd within each VM? Any insight would be appreciated. brian carb unisys corporation, malvern pa ----- domU config file 002.cfg ----- kernel = "/boot/vmlinuz-xen" ramdisk = "/boot/initrd-xen" root = "/dev/sda2 ro" disk = [ 'phy:/tmp/xendevs/xen002,sda,w' ] memory = 512 vcpus = 4 name = 'xen002' vif = [ 'mac=00:16:3e:0d:a0:01' ] ip = '10.0.27.2' netmask = '255.254.0.0' extra = ' xencons=tty1 3' ----- xend.log ----- [2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78) XendDomainInfo.create(['vm', ['name', 'xen002'], ['memory', 512], ['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], ['ramdisk', '/boot/initrd-xen'], ['ip', '10.0.27.2:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 ro'], ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', 'phy:/tmp/xendevs/xen002'], ['dev', 'sda'], ['mode', 'w']]], ['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) [2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418) XendDomainInfo.constructDomain [2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 KiB free; need 2048; done. [2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16 [2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472) XendDomainInfo.initDomain: 16 256 [2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503) _initDomain:shadow_memory=0x0, memory_static_max=0x20000000, memory_static_min=0x0. [2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 KiB free; need 524288; done. [2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux dom=16 vcpus=4 [2007-09-02 05:26:43 8597] DEBUG (image:197) domid = 16 [2007-09-02 05:26:43 8597] DEBUG (image:198) memsize = 512 [2007-09-02 05:26:43 8597] DEBUG (image:199) image = /boot/vmlinuz-xen [2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn = 1 [2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2 [2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline = root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3 [2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk = /boot/initrd-xen [2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus = 4 [2007-09-02 05:26:43 8597] DEBUG (image:205) features = [2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice: vbd : {'uuid': '8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'sda', 'uname': 'phy:/tmp/xendevs/xen002', 'mode': 'w'} [2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController: writing {'virtual-device': '2048', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/16/2048'} to /local/domain/16/device/vbd/2048. [2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController: writing {'domain': 'xen002', 'frontend': '/local/domain/16/device/vbd/2048', 'uuid': '8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'dev': 'sda', 'state': '1', 'params': '/tmp/xendevs/xen002', 'mode': 'w', 'online': '1', 'frontend-id': '16', 'type': 'phy'} to /local/domain/0/backend/vbd/16/2048. [2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: vif : {'mac': '00:16:3e:0d:a0:01', 'uuid': 'bb43e84a-a9a1-7488-1ba4-3b713eff82aa'} [2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController: writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/16/0'} to /local/domain/16/device/vif/0. [2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController: writing {'domain': 'xen002', 'handle': '0', 'uuid': 'bb43e84a-a9a1-7488-1ba4-3b713eff82aa', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': '/local/domain/16/device/vif/0', 'mac': '00:16:3e:0d:a0:01', 'online': '1', 'frontend-id': '16', 'type': 'netfront'} to /local/domain/0/backend/vif/16/0. [2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'f146fdfc-e624-f7cb-4baf-0471f01f6014', 'on_reboot': 'restart', 'start_time': '1188725206.15', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (args 'root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') (notes (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap |pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION xen-3.0)))", 'name': 'xen002'} [2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain details: {'console/ring-ref': '34194913', 'image/entry': '-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref': '34194914', 'image/loader': 'generic', 'vm': '/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', 'control/platform-feature-multiprocessor-suspend': '1', 'cpu/2/availability': 'online', 'cpu/1/availability': 'online', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux', 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', 'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'image/features/auto-translated-physmap': '1', 'name': 'xen002', 'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController: writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/16/0'} to /local/domain/16/device/console/0. [2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController: writing {'domain': 'xen002', 'protocol': 'vt100', 'uuid': '8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e', 'frontend': '/local/domain/16/device/console/0', 'state': '1', 'location': '2', 'online': '1', 'frontend-id': '16'} to /local/domain/0/backend/console/16/0. [2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for devices vif. [2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0. [2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain details: {'console/ring-ref': '34194913', 'image/entry': '-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref': '34194914', 'image/loader': 'generic', 'vm': '/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', 'control/platform-feature-multiprocessor-suspend': '1', 'cpu/2/availability': 'online', 'cpu/1/availability': 'online', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux', 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', 'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'image/features/auto-translated-physmap': '1', 'name': 'xen002', 'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907) XendDomainInfo.handleShutdownWatch [2007-09-02 05:26:53 8597] DEBUG (DevController:541) hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status. [2007-09-02 05:26:53 8597] DEBUG (DevController:555) hotplugStatusCallback 1. [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices usb. [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vbd. [2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048. [2007-09-02 05:26:54 8597] DEBUG (DevController:541) hotplugStatusCallback /local/domain/0/backend/vbd/16/2048/hotplug-status. [2007-09-02 05:26:54 8597] DEBUG (DevController:555) hotplugStatusCallback 1. [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices irq. [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vkbd. [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vfb. [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices console. [2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0. [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices pci. [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices ioports. [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices tap. [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices vtpm. [2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16) unpaused. ... ... [vms xen003 to xen262 deleted] ... [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78) XendDomainInfo.create(['vm', ['name', 'xen263'], ['memory', 512], ['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], ['ramdisk', '/boot/initrd-xen'], ['ip', '10.0.29.63:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 ro'], ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', 'phy:/tmp/xendevs/xen263'], ['dev', 'sda'], ['mode', 'w']]], ['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418) XendDomainInfo.constructDomain [2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB free; need 2048; done. [2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277 [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472) XendDomainInfo.initDomain: 277 256 [2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503) _initDomain:shadow_memory=0x0, memory_static_max=0x20000000, memory_static_min=0x0. [2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB free; need 524288; done. [2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux dom=277 vcpus=4 [2007-09-02 17:44:29 8597] DEBUG (image:197) domid = 277 [2007-09-02 17:44:29 8597] DEBUG (image:198) memsize = 512 [2007-09-02 17:44:29 8597] DEBUG (image:199) image = /boot/vmlinuz-xen [2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn = 1 [2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2 [2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline = root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3 [2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk = /boot/initrd-xen [2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus = 4 [2007-09-02 17:44:29 8597] DEBUG (image:205) features = [2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vbd : {'uuid': '9d639202-73af-64ec-451d-3abe0fc57bc7', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'sda', 'uname': 'phy:/tmp/xendevs/xen263', 'mode': 'w'} [2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController: writing {'virtual-device': '2048', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/277/2048'} to /local/domain/277/device/vbd/2048. [2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController: writing {'domain': 'xen263', 'frontend': '/local/domain/277/device/vbd/2048', 'uuid': '9d639202-73af-64ec-451d-3abe0fc57bc7', 'dev': 'sda', 'state': '1', 'params': '/tmp/xendevs/xen263', 'mode': 'w', 'online': '1', 'frontend-id': '277', 'type': 'phy'} to /local/domain/0/backend/vbd/277/2048. [2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vif : {'mac': '00:16:3e:0d:a0:01', 'uuid': 'a126d815-fd4a-332b-c881-235a7e003cfb'} [2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController: writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/277/0'} to /local/domain/277/device/vif/0. [2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController: writing {'domain': 'xen263', 'handle': '0', 'uuid': 'a126d815-fd4a-332b-c881-235a7e003cfb', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': '/local/domain/277/device/vif/0', 'mac': '00:16:3e:0d:a0:01', 'online': '1', 'frontend-id': '277', 'type': 'netfront'} to /local/domain/0/backend/vif/277/0. [2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': '8f8f9839-dd61-b9cd-89a6-f66bb3d41726', 'on_reboot': 'restart', 'start_time': '1188769556.11', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (args 'root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') (notes (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap |pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION xen-3.0)))", 'name': 'xen263'} [2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain details: {'console/ring-ref': '5227479', 'image/entry': '-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref': '5227480', 'image/loader': 'generic', 'vm': '/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', 'control/platform-feature-multiprocessor-suspend': '1', 'cpu/2/availability': 'online', 'cpu/1/availability': 'online', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux', 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', 'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'image/features/auto-translated-physmap': '1', 'name': 'xen263', 'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController: writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/277/0'} to /local/domain/277/device/console/0. [2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController: writing {'domain': 'xen263', 'protocol': 'vt100', 'uuid': 'd0da45dd-1a00-da65-61dd-66dadd6e9897', 'frontend': '/local/domain/277/device/console/0', 'state': '1', 'location': '2', 'online': '1', 'frontend-id': '277'} to /local/domain/0/backend/console/277/0. [2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain details: {'console/ring-ref': '5227479', 'image/entry': '-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', 'store/ring-ref': '5227480', 'image/loader': 'generic', 'vm': '/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', 'control/platform-feature-multiprocessor-suspend': '1', 'cpu/2/availability': 'online', 'cpu/1/availability': 'online', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '-2147483648', 'memory/target': '524288', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/guest-os': 'linux', 'console/limit': '1048576', 'image/paddr-offset': '-2147483648', 'image/hypercall-page': '-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'image/features/auto-translated-physmap': '1', 'name': 'xen263', 'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907) XendDomainInfo.handleShutdownWatch [2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for devices vif. [2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0. [2007-09-02 17:48:23 8597] DEBUG (DevController:541) hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status. [2007-09-02 17:48:23 8597] DEBUG (DevController:555) hotplugStatusCallback 1. [2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices usb. [2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices vbd. [2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048. [2007-09-02 17:48:25 8597] DEBUG (DevController:541) hotplugStatusCallback /local/domain/0/backend/vbd/277/2048/hotplug-status. [2007-09-02 17:48:25 8597] DEBUG (DevController:555) hotplugStatusCallback 1. [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices irq. [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices vkbd. [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices vfb. [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices console. [2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0. [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices pci. [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices ioports. [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices tap. [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices vtpm. [2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 (277) unpaused. ________________________________ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel [-- Attachment #1.2: Type: text/html, Size: 26534 bytes --] [-- Attachment #2: Type: text/plain, Size: 138 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <089B0D75973E1241B941D0A9854F23FC08E92D5C@USEA-EXCH2.na.uis .unisys.com>]
* RE: VM start time slows with large number of VMs [not found] ` <089B0D75973E1241B941D0A9854F23FC08E92D5C@USEA-EXCH2.na.uis .unisys.com> @ 2007-09-06 17:29 ` Mats Petersson 2007-09-06 17:31 ` Keir Fraser 2007-09-06 19:25 ` Carb, Brian A 0 siblings, 2 replies; 13+ messages in thread From: Mats Petersson @ 2007-09-06 17:29 UTC (permalink / raw) To: Carb, Brian A, Keir Fraser, xen-devel At 17:24 06/09/2007, Carb, Brian A wrote: >Content-class: urn:content-classes:message >Content-Type: multipart/alternative; > boundary="----_=_NextPart_001_01C7F0A2.55DB0EB0" > >keir > >I'll try timing this as I load up... > >but when I restart my test system, with only dom0 running, >xenstore-ls is already slow before any VMs are started: > > time xenstore-ls >/dev/null > real 0m21.725s > user 0m0.010s > sys 0m0.090s > >If I save the output from the xenstore-ls, the file has 61,318 lines >with information from all the VMs run yesterday. I conclude that >xenstore does not get cleared on startup. Is there a way to clear it? It is perfectly fine to remove the xenstore.tdb [filename from memory] in /var/xen/lib [again, from memory]. Unless of course you actually NEED some of your old Xenstrore entries for soemthing, but in general, they are fine to be removed completely. -- Mats > >brian carb >unisys corporation - malvern, pa > > > >---------- >From: Keir Fraser >Sent: Wednesday, September 05, 2007 1:11 PM >To: Carb, Brian A; xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] VM start time slows with large number of VMs > >xenstore slowness seems a reasonable hypothesis. How does >performance of 'time xenstore-ls >/dev/null' change as you increase >the number of guests? > > -- Keir > >On 4/9/07 17:51, "Carb, Brian A" wrote: > >Hello, > >We're running Xen Unstable changeset 15468 with SLES10 on a Unisys >ES7000/one 56x cpu and 160GB memory. Xen is built with >NR_DYNIRQS=1024 and max_phys_cpus=64, and booted with dom0_mem=1024 >and xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on >external storage. > >Our script starts a VM, waits for the start to complete, then starts >the next: "for i in *.cfg; do xm create $i; done". As more VMs >start, the time to start each one increases. I've attached a config >file for one of the VMs, as well as a portion of the xend.log which >shows timestamps of the events for 2 VMS, vm002 and vm263. > >The total elapsed time of the start from "create" to "unpaused" for >vm002 is 13 seconds, but by the time 262 VMs are started, the start >of vm263 has slowed to 238 seconds. The host itself is not sluggish, >as we can open putty sessions to and interact with dom0. However, >if we run "xm top", its interface is very unresponsive. Could >writing to the xenstore be taking longer and longer? Could there be >possible contention for vmlinuz and initrd since they are being used >by every VM to boot? Would it be better to use bootloader/bootentry >and store the vmlinuz and initrd within each VM? > >Any insight would be appreciated. > >brian carb >unisys corporation, malvern pa > >----- domU config file 002.cfg ----- >kernel = "/boot/vmlinuz-xen" >ramdisk = "/boot/initrd-xen" >root = "/dev/sda2 ro" >disk = [ 'phy:/tmp/xendevs/xen002,sda,w' ] >memory = 512 >vcpus = 4 >name = 'xen002' >vif = [ 'mac=00:16:3e:0d:a0:01' ] >ip = '10.0.27.2' >netmask = '255.254.0.0' >extra = ' xencons=tty1 3' >----- xend.log ----- >[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78) >XendDomainInfo.create(['vm', ['name', 'xen002'], ['memory', 512], >['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', >'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], >['ramdisk', '/boot/initrd-xen'], ['ip', >'10.0.27.2:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 >ro'], ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', >'phy:/tmp/xendevs/xen002'], ['dev', 'sda'], ['mode', 'w']]], >['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) >[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418) >XendDomainInfo.constructDomain >[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 >KiB free; need 2048; done. >[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16 >[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472) >XendDomainInfo.initDomain: 16 256 >[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503) >_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, >memory_static_min=0x0. >[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 >KiB free; need 524288; done. >[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux >dom=16 vcpus=4 >[2007-09-02 05:26:43 8597] DEBUG (image:197) domid = 16 >[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize = 512 >[2007-09-02 05:26:43 8597] DEBUG (image:199) image = >/boot/vmlinuz-xen >[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn = 1 >[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2 >[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline = >root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3 >[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk = /boot/initrd-xen >[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus = 4 >[2007-09-02 05:26:43 8597] DEBUG (image:205) features = >[2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice: >vbd : {'uuid': '8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'bootable': >1, 'driver': 'paravirtualised', 'dev': 'sda', 'uname': >'phy:/tmp/xendevs/xen002', 'mode': 'w'} >[2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController: >writing {'virtual-device': '2048', 'device-type': 'disk', >'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', >'backend': '/local/domain/0/backend/vbd/16/2048'} to >/local/domain/16/device/vbd/2048. >[2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen002', 'frontend': >'/local/domain/16/device/vbd/2048', 'uuid': >'8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'dev': 'sda', 'state': '1', >'params': '/tmp/xendevs/xen002', 'mode': 'w', 'online': '1', >'frontend-id': '16', 'type': 'phy'} to /local/domain/0/backend/vbd/16/2048. >[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: >vif : {'mac': '00:16:3e:0d:a0:01', 'uuid': >'bb43e84a-a9a1-7488-1ba4-3b713eff82aa'} >[2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController: >writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': >'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': >'/local/domain/0/backend/vif/16/0'} to /local/domain/16/device/vif/0. >[2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen002', 'handle': '0', 'uuid': >'bb43e84a-a9a1-7488-1ba4-3b713eff82aa', 'script': >'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': >'/local/domain/16/device/vif/0', 'mac': '00:16:3e:0d:a0:01', >'online': '1', 'frontend-id': '16', 'type': 'netfront'} to >/local/domain/0/backend/vif/16/0. >[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM >details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': >'f146fdfc-e624-f7cb-4baf-0471f01f6014', 'on_reboot': 'restart', >'start_time': '1188725206.15', 'on_poweroff': 'destroy', >'on_xend_start': 'ignore', 'on_crash': 'restart', >'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', >'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk >/boot/initrd-xen) (args 'root=/dev/sda2 ro >ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') (notes >(FEATURES >'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET >-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER >generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION >xen-3.0)))", 'name': 'xen002'} >[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain >details: {'console/ring-ref': '34194913', 'image/entry': >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', >'store/ring-ref': '34194914', 'image/loader': 'generic', 'vm': >'/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', >'control/platform-feature-multiprocessor-suspend': '1', >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', >'image/features/writable-descriptor-tables': '1', 'image/virt-base': >'-2147483648', 'memory/target': '524288', 'image/guest-version': >'2.6', 'image/features/supervisor-mode-kernel': '1', >'image/guest-os': 'linux', 'console/limit': '1048576', >'image/paddr-offset': '-2147483648', 'image/hypercall-page': >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': >'online', 'image/features/pae-pgdir-above-4gb': '1', >'image/features/writable-page-tables': '1', >'image/features/auto-translated-physmap': '1', 'name': 'xen002', >'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} >[2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController: >writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', >'backend': '/local/domain/0/backend/console/16/0'} to >/local/domain/16/device/console/0. >[2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen002', 'protocol': 'vt100', 'uuid': >'8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e', 'frontend': >'/local/domain/16/device/console/0', 'state': '1', 'location': '2', >'online': '1', 'frontend-id': '16'} to /local/domain/0/backend/console/16/0. >[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for devices vif. >[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0. >[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain >details: {'console/ring-ref': '34194913', 'image/entry': >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', >'store/ring-ref': '34194914', 'image/loader': 'generic', 'vm': >'/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', >'control/platform-feature-multiprocessor-suspend': '1', >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', >'image/features/writable-descriptor-tables': '1', 'image/virt-base': >'-2147483648', 'memory/target': '524288', 'image/guest-version': >'2.6', 'image/features/supervisor-mode-kernel': '1', >'image/guest-os': 'linux', 'console/limit': '1048576', >'image/paddr-offset': '-2147483648', 'image/hypercall-page': >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': >'online', 'image/features/pae-pgdir-above-4gb': '1', >'image/features/writable-page-tables': '1', >'image/features/auto-translated-physmap': '1', 'name': 'xen002', >'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} >[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907) >XendDomainInfo.handleShutdownWatch >[2007-09-02 05:26:53 8597] DEBUG (DevController:541) >hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status. >[2007-09-02 05:26:53 8597] DEBUG (DevController:555) hotplugStatusCallback 1. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices usb. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vbd. >[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048. >[2007-09-02 05:26:54 8597] DEBUG (DevController:541) >hotplugStatusCallback /local/domain/0/backend/vbd/16/2048/hotplug-status. >[2007-09-02 05:26:54 8597] DEBUG (DevController:555) hotplugStatusCallback 1. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices irq. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vkbd. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vfb. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for >devices console. >[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices pci. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for >devices ioports. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices tap. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices vtpm. >[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16) unpaused. >... >... [vms xen003 to xen262 deleted] >... >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78) >XendDomainInfo.create(['vm', ['name', 'xen263'], ['memory', 512], >['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', >'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], >['ramdisk', '/boot/initrd-xen'], ['ip', >'10.0.29.63:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 >ro'], ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', >'phy:/tmp/xendevs/xen263'], ['dev', 'sda'], ['mode', 'w']]], >['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418) >XendDomainInfo.constructDomain >[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB >free; need 2048; done. >[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277 >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472) >XendDomainInfo.initDomain: 277 256 >[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503) >_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, >memory_static_min=0x0. >[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB >free; need 524288; done. >[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux >dom=277 vcpus=4 >[2007-09-02 17:44:29 8597] DEBUG (image:197) domid = 277 >[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize = 512 >[2007-09-02 17:44:29 8597] DEBUG (image:199) image = >/boot/vmlinuz-xen >[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn = 1 >[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2 >[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline = >root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3 >[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk = /boot/initrd-xen >[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus = 4 >[2007-09-02 17:44:29 8597] DEBUG (image:205) features = >[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: >vbd : {'uuid': '9d639202-73af-64ec-451d-3abe0fc57bc7', 'bootable': >1, 'driver': 'paravirtualised', 'dev': 'sda', 'uname': >'phy:/tmp/xendevs/xen263', 'mode': 'w'} >[2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController: >writing {'virtual-device': '2048', 'device-type': 'disk', >'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', >'backend': '/local/domain/0/backend/vbd/277/2048'} to >/local/domain/277/device/vbd/2048. >[2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen263', 'frontend': >'/local/domain/277/device/vbd/2048', 'uuid': >'9d639202-73af-64ec-451d-3abe0fc57bc7', 'dev': 'sda', 'state': '1', >'params': '/tmp/xendevs/xen263', 'mode': 'w', 'online': '1', >'frontend-id': '277', 'type': 'phy'} to /local/domain/0/backend/vbd/277/2048. >[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: >vif : {'mac': '00:16:3e:0d:a0:01', 'uuid': >'a126d815-fd4a-332b-c881-235a7e003cfb'} >[2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController: >writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': >'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': >'/local/domain/0/backend/vif/277/0'} to /local/domain/277/device/vif/0. >[2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen263', 'handle': '0', 'uuid': >'a126d815-fd4a-332b-c881-235a7e003cfb', 'script': >'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': >'/local/domain/277/device/vif/0', 'mac': '00:16:3e:0d:a0:01', >'online': '1', 'frontend-id': '277', 'type': 'netfront'} to >/local/domain/0/backend/vif/277/0. >[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM >details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': >'8f8f9839-dd61-b9cd-89a6-f66bb3d41726', 'on_reboot': 'restart', >'start_time': '1188769556.11', 'on_poweroff': 'destroy', >'on_xend_start': 'ignore', 'on_crash': 'restart', >'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', >'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk >/boot/initrd-xen) (args 'root=/dev/sda2 ro >ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') >(notes (FEATURES >'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET >-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER >generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION >xen-3.0)))", 'name': 'xen263'} >[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain >details: {'console/ring-ref': '5227479', 'image/entry': >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', >'store/ring-ref': '5227480', 'image/loader': 'generic', 'vm': >'/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', >'control/platform-feature-multiprocessor-suspend': '1', >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', >'image/features/writable-descriptor-tables': '1', 'image/virt-base': >'-2147483648', 'memory/target': '524288', 'image/guest-version': >'2.6', 'image/features/supervisor-mode-kernel': '1', >'image/guest-os': 'linux', 'console/limit': '1048576', >'image/paddr-offset': '-2147483648', 'image/hypercall-page': >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': >'online', 'image/features/pae-pgdir-above-4gb': '1', >'image/features/writable-page-tables': '1', >'image/features/auto-translated-physmap': '1', 'name': 'xen263', >'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} >[2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController: >writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', >'backend': '/local/domain/0/backend/console/277/0'} to >/local/domain/277/device/console/0. >[2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen263', 'protocol': 'vt100', 'uuid': >'d0da45dd-1a00-da65-61dd-66dadd6e9897', 'frontend': >'/local/domain/277/device/console/0', 'state': '1', 'location': '2', >'online': '1', 'frontend-id': '277'} to /local/domain/0/backend/console/277/0. >[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain >details: {'console/ring-ref': '5227479', 'image/entry': >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', >'store/ring-ref': '5227480', 'image/loader': 'generic', 'vm': >'/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', >'control/platform-feature-multiprocessor-suspend': '1', >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', >'image/features/writable-descriptor-tables': '1', 'image/virt-base': >'-2147483648', 'memory/target': '524288', 'image/guest-version': >'2.6', 'image/features/supervisor-mode-kernel': '1', >'image/guest-os': 'linux', 'console/limit': '1048576', >'image/paddr-offset': '-2147483648', 'image/hypercall-page': >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': >'online', 'image/features/pae-pgdir-above-4gb': '1', >'image/features/writable-page-tables': '1', >'image/features/auto-translated-physmap': '1', 'name': 'xen263', >'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} >[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907) >XendDomainInfo.handleShutdownWatch >[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for devices vif. >[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0. >[2007-09-02 17:48:23 8597] DEBUG (DevController:541) >hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status. >[2007-09-02 17:48:23 8597] DEBUG (DevController:555) hotplugStatusCallback 1. >[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices usb. >[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices vbd. >[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048. >[2007-09-02 17:48:25 8597] DEBUG (DevController:541) >hotplugStatusCallback /local/domain/0/backend/vbd/277/2048/hotplug-status. >[2007-09-02 17:48:25 8597] DEBUG (DevController:555) hotplugStatusCallback 1. >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices irq. >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices vkbd. >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices vfb. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for >devices console. >[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices pci. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for >devices ioports. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices tap. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices vtpm. >[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 >(277) unpaused. > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com ><http://lists.xensource.com/xen-devel>http://lists.xensource.com/xen-devel > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: VM start time slows with large number of VMs 2007-09-06 17:29 ` Mats Petersson @ 2007-09-06 17:31 ` Keir Fraser 2007-09-06 17:44 ` Daniel P. Berrange 2007-09-06 19:25 ` Carb, Brian A 1 sibling, 1 reply; 13+ messages in thread From: Keir Fraser @ 2007-09-06 17:31 UTC (permalink / raw) To: Mats Petersson, Carb, Brian A, xen-devel On 6/9/07 18:29, "Mats Petersson" <mats@planetcatfish.com> wrote: >> If I save the output from the xenstore-ls, the file has 61,318 lines >> with information from all the VMs run yesterday. I conclude that >> xenstore does not get cleared on startup. Is there a way to clear it? > > > It is perfectly fine to remove the xenstore.tdb [filename from > memory] in /var/xen/lib [again, from memory]. Unless of course you > actually NEED some of your old Xenstrore entries for soemthing, but > in general, they are fine to be removed completely. Yes, actually this would be something perhaps we should do from the xend startup script, if we can determine that xenstored is not running. It'd be nice to get a patch. -- Keir ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: VM start time slows with large number of VMs 2007-09-06 17:31 ` Keir Fraser @ 2007-09-06 17:44 ` Daniel P. Berrange 0 siblings, 0 replies; 13+ messages in thread From: Daniel P. Berrange @ 2007-09-06 17:44 UTC (permalink / raw) To: Keir Fraser; +Cc: xen-devel, Mats Petersson, Carb, Brian A On Thu, Sep 06, 2007 at 06:31:28PM +0100, Keir Fraser wrote: > On 6/9/07 18:29, "Mats Petersson" <mats@planetcatfish.com> wrote: > > >> If I save the output from the xenstore-ls, the file has 61,318 lines > >> with information from all the VMs run yesterday. I conclude that > >> xenstore does not get cleared on startup. Is there a way to clear it? > > > > > > It is perfectly fine to remove the xenstore.tdb [filename from > > memory] in /var/xen/lib [again, from memory]. Unless of course you > > actually NEED some of your old Xenstrore entries for soemthing, but > > in general, they are fine to be removed completely. > > Yes, actually this would be something perhaps we should do from the xend > startup script, if we can determine that xenstored is not running. It'd be > nice to get a patch. Or just put the database on /dev/shm so its in RAM - that way it just goes away whenever you reboot & you reduce the disk I/O overhead xenstored has. IIRC last time I tested it doubled xenstored performance using /dev/shm Dan -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: VM start time slows with large number of VMs 2007-09-06 17:29 ` Mats Petersson 2007-09-06 17:31 ` Keir Fraser @ 2007-09-06 19:25 ` Carb, Brian A 2007-09-06 21:25 ` Keir Fraser [not found] ` <089B0D75973E1241B941D0A9854F23FC08E93337@USEA-EXCH2.na.uis .unisys.com> 1 sibling, 2 replies; 13+ messages in thread From: Carb, Brian A @ 2007-09-06 19:25 UTC (permalink / raw) To: Mats Petersson, Keir Fraser, xen-devel I can't seem to clear out this information. I tried /etc/init.d/xend stop killall xensored xenconsoled rm -rf /var/lib/xenstored <- has the tdb rm -rf /var/lib/xend <- has domains, state and storage dirs but when i startup xend again, xenstore-ls still shows all the old domains. I must be missing something. brian carb unisys corporation - malvern, pa -----Original Message----- From: mats petersson Sent: Thursday, September 06, 2007 1:30 PM To: Carb, Brian A; Keir Fraser; xen-devel@lists.xensource.com Subject: RE: [Xen-devel] VM start time slows with large number of VMs At 17:24 06/09/2007, Carb, Brian A wrote: >Content-class: urn:content-classes:message >Content-Type: multipart/alternative; > boundary="----_=_NextPart_001_01C7F0A2.55DB0EB0" > >keir > >I'll try timing this as I load up... > >but when I restart my test system, with only dom0 running, xenstore-ls >is already slow before any VMs are started: > > time xenstore-ls >/dev/null > real 0m21.725s > user 0m0.010s > sys 0m0.090s > >If I save the output from the xenstore-ls, the file has 61,318 lines >with information from all the VMs run yesterday. I conclude that >xenstore does not get cleared on startup. Is there a way to clear it? It is perfectly fine to remove the xenstore.tdb [filename from memory] in /var/xen/lib [again, from memory]. Unless of course you actually NEED some of your old Xenstrore entries for soemthing, but in general, they are fine to be removed completely. -- Mats > >brian carb >unisys corporation - malvern, pa > > > >---------- >From: Keir Fraser >Sent: Wednesday, September 05, 2007 1:11 PM >To: Carb, Brian A; xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] VM start time slows with large number of VMs > >xenstore slowness seems a reasonable hypothesis. How does >performance of 'time xenstore-ls >/dev/null' change as you increase >the number of guests? > > -- Keir > >On 4/9/07 17:51, "Carb, Brian A" wrote: > >Hello, > >We're running Xen Unstable changeset 15468 with SLES10 on a Unisys >ES7000/one 56x cpu and 160GB memory. Xen is built with >NR_DYNIRQS=1024 and max_phys_cpus=64, and booted with dom0_mem=1024 >and xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on >external storage. > >Our script starts a VM, waits for the start to complete, then starts >the next: "for i in *.cfg; do xm create $i; done". As more VMs >start, the time to start each one increases. I've attached a config >file for one of the VMs, as well as a portion of the xend.log which >shows timestamps of the events for 2 VMS, vm002 and vm263. > >The total elapsed time of the start from "create" to "unpaused" for >vm002 is 13 seconds, but by the time 262 VMs are started, the start >of vm263 has slowed to 238 seconds. The host itself is not sluggish, >as we can open putty sessions to and interact with dom0. However, >if we run "xm top", its interface is very unresponsive. Could >writing to the xenstore be taking longer and longer? Could there be >possible contention for vmlinuz and initrd since they are being used >by every VM to boot? Would it be better to use bootloader/bootentry >and store the vmlinuz and initrd within each VM? > >Any insight would be appreciated. > >brian carb >unisys corporation, malvern pa > >----- domU config file 002.cfg ----- >kernel = "/boot/vmlinuz-xen" >ramdisk = "/boot/initrd-xen" >root = "/dev/sda2 ro" >disk = [ 'phy:/tmp/xendevs/xen002,sda,w' ] >memory = 512 >vcpus = 4 >name = 'xen002' >vif = [ 'mac=00:16:3e:0d:a0:01' ] >ip = '10.0.27.2' >netmask = '255.254.0.0' >extra = ' xencons=tty1 3' >----- xend.log ----- >[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78) >XendDomainInfo.create(['vm', ['name', 'xen002'], ['memory', 512], >['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', >'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], >['ramdisk', '/boot/initrd-xen'], ['ip', >'10.0.27.2:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 >ro'], ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', >'phy:/tmp/xendevs/xen002'], ['dev', 'sda'], ['mode', 'w']]], >['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) >[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418) >XendDomainInfo.constructDomain >[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 >KiB free; need 2048; done. >[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16 >[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472) >XendDomainInfo.initDomain: 16 256 >[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503) >_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, >memory_static_min=0x0. >[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 >KiB free; need 524288; done. >[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux >dom=16 vcpus=4 >[2007-09-02 05:26:43 8597] DEBUG (image:197) domid = 16 >[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize = 512 >[2007-09-02 05:26:43 8597] DEBUG (image:199) image = >/boot/vmlinuz-xen >[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn = 1 >[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2 >[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline = >root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3 >[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk = /boot/initrd-xen >[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus = 4 >[2007-09-02 05:26:43 8597] DEBUG (image:205) features = >[2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice: >vbd : {'uuid': '8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'bootable': >1, 'driver': 'paravirtualised', 'dev': 'sda', 'uname': >'phy:/tmp/xendevs/xen002', 'mode': 'w'} >[2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController: >writing {'virtual-device': '2048', 'device-type': 'disk', >'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', >'backend': '/local/domain/0/backend/vbd/16/2048'} to >/local/domain/16/device/vbd/2048. >[2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen002', 'frontend': >'/local/domain/16/device/vbd/2048', 'uuid': >'8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'dev': 'sda', 'state': '1', >'params': '/tmp/xendevs/xen002', 'mode': 'w', 'online': '1', >'frontend-id': '16', 'type': 'phy'} to /local/domain/0/backend/vbd/16/2048. >[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: >vif : {'mac': '00:16:3e:0d:a0:01', 'uuid': >'bb43e84a-a9a1-7488-1ba4-3b713eff82aa'} >[2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController: >writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': >'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': >'/local/domain/0/backend/vif/16/0'} to /local/domain/16/device/vif/0. >[2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen002', 'handle': '0', 'uuid': >'bb43e84a-a9a1-7488-1ba4-3b713eff82aa', 'script': >'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': >'/local/domain/16/device/vif/0', 'mac': '00:16:3e:0d:a0:01', >'online': '1', 'frontend-id': '16', 'type': 'netfront'} to >/local/domain/0/backend/vif/16/0. >[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM >details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': >'f146fdfc-e624-f7cb-4baf-0471f01f6014', 'on_reboot': 'restart', >'start_time': '1188725206.15', 'on_poweroff': 'destroy', >'on_xend_start': 'ignore', 'on_crash': 'restart', >'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', >'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk >/boot/initrd-xen) (args 'root=/dev/sda2 ro >ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') (notes >(FEATURES >'writable_page_tables|writable_descriptor_tables|auto_translated_physma p|pae_pgdir_above_4gb|supervisor_mode_kernel') >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET >-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER >generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION >xen-3.0)))", 'name': 'xen002'} >[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain >details: {'console/ring-ref': '34194913', 'image/entry': >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', >'store/ring-ref': '34194914', 'image/loader': 'generic', 'vm': >'/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', >'control/platform-feature-multiprocessor-suspend': '1', >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', >'image/features/writable-descriptor-tables': '1', 'image/virt-base': >'-2147483648', 'memory/target': '524288', 'image/guest-version': >'2.6', 'image/features/supervisor-mode-kernel': '1', >'image/guest-os': 'linux', 'console/limit': '1048576', >'image/paddr-offset': '-2147483648', 'image/hypercall-page': >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': >'online', 'image/features/pae-pgdir-above-4gb': '1', >'image/features/writable-page-tables': '1', >'image/features/auto-translated-physmap': '1', 'name': 'xen002', >'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} >[2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController: >writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', >'backend': '/local/domain/0/backend/console/16/0'} to >/local/domain/16/device/console/0. >[2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen002', 'protocol': 'vt100', 'uuid': >'8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e', 'frontend': >'/local/domain/16/device/console/0', 'state': '1', 'location': '2', >'online': '1', 'frontend-id': '16'} to /local/domain/0/backend/console/16/0. >[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for devices vif. >[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0. >[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain >details: {'console/ring-ref': '34194913', 'image/entry': >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', >'store/ring-ref': '34194914', 'image/loader': 'generic', 'vm': >'/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', >'control/platform-feature-multiprocessor-suspend': '1', >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', >'image/features/writable-descriptor-tables': '1', 'image/virt-base': >'-2147483648', 'memory/target': '524288', 'image/guest-version': >'2.6', 'image/features/supervisor-mode-kernel': '1', >'image/guest-os': 'linux', 'console/limit': '1048576', >'image/paddr-offset': '-2147483648', 'image/hypercall-page': >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': >'online', 'image/features/pae-pgdir-above-4gb': '1', >'image/features/writable-page-tables': '1', >'image/features/auto-translated-physmap': '1', 'name': 'xen002', >'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} >[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907) >XendDomainInfo.handleShutdownWatch >[2007-09-02 05:26:53 8597] DEBUG (DevController:541) >hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status. >[2007-09-02 05:26:53 8597] DEBUG (DevController:555) hotplugStatusCallback 1. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices usb. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vbd. >[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048. >[2007-09-02 05:26:54 8597] DEBUG (DevController:541) >hotplugStatusCallback /local/domain/0/backend/vbd/16/2048/hotplug-status. >[2007-09-02 05:26:54 8597] DEBUG (DevController:555) hotplugStatusCallback 1. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices irq. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vkbd. >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices vfb. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for >devices console. >[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices pci. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for >devices ioports. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices tap. >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices vtpm. >[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16) unpaused. >... >... [vms xen003 to xen262 deleted] >... >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78) >XendDomainInfo.create(['vm', ['name', 'xen263'], ['memory', 512], >['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', >'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], >['ramdisk', '/boot/initrd-xen'], ['ip', >'10.0.29.63:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 >ro'], ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', >'phy:/tmp/xendevs/xen263'], ['dev', 'sda'], ['mode', 'w']]], >['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418) >XendDomainInfo.constructDomain >[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB >free; need 2048; done. >[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277 >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472) >XendDomainInfo.initDomain: 277 256 >[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503) >_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, >memory_static_min=0x0. >[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB >free; need 524288; done. >[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux >dom=277 vcpus=4 >[2007-09-02 17:44:29 8597] DEBUG (image:197) domid = 277 >[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize = 512 >[2007-09-02 17:44:29 8597] DEBUG (image:199) image = >/boot/vmlinuz-xen >[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn = 1 >[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2 >[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline = >root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3 >[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk = /boot/initrd-xen >[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus = 4 >[2007-09-02 17:44:29 8597] DEBUG (image:205) features = >[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: >vbd : {'uuid': '9d639202-73af-64ec-451d-3abe0fc57bc7', 'bootable': >1, 'driver': 'paravirtualised', 'dev': 'sda', 'uname': >'phy:/tmp/xendevs/xen263', 'mode': 'w'} >[2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController: >writing {'virtual-device': '2048', 'device-type': 'disk', >'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', >'backend': '/local/domain/0/backend/vbd/277/2048'} to >/local/domain/277/device/vbd/2048. >[2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen263', 'frontend': >'/local/domain/277/device/vbd/2048', 'uuid': >'9d639202-73af-64ec-451d-3abe0fc57bc7', 'dev': 'sda', 'state': '1', >'params': '/tmp/xendevs/xen263', 'mode': 'w', 'online': '1', >'frontend-id': '277', 'type': 'phy'} to /local/domain/0/backend/vbd/277/2048. >[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: >vif : {'mac': '00:16:3e:0d:a0:01', 'uuid': >'a126d815-fd4a-332b-c881-235a7e003cfb'} >[2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController: >writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': >'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': >'/local/domain/0/backend/vif/277/0'} to /local/domain/277/device/vif/0. >[2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen263', 'handle': '0', 'uuid': >'a126d815-fd4a-332b-c881-235a7e003cfb', 'script': >'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': >'/local/domain/277/device/vif/0', 'mac': '00:16:3e:0d:a0:01', >'online': '1', 'frontend-id': '277', 'type': 'netfront'} to >/local/domain/0/backend/vif/277/0. >[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM >details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': >'8f8f9839-dd61-b9cd-89a6-f66bb3d41726', 'on_reboot': 'restart', >'start_time': '1188769556.11', 'on_poweroff': 'destroy', >'on_xend_start': 'ignore', 'on_crash': 'restart', >'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', >'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk >/boot/initrd-xen) (args 'root=/dev/sda2 ro >ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') >(notes (FEATURES >'writable_page_tables|writable_descriptor_tables|auto_translated_physma p|pae_pgdir_above_4gb|supervisor_mode_kernel') >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET >-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER >generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION >xen-3.0)))", 'name': 'xen263'} >[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain >details: {'console/ring-ref': '5227479', 'image/entry': >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', >'store/ring-ref': '5227480', 'image/loader': 'generic', 'vm': >'/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', >'control/platform-feature-multiprocessor-suspend': '1', >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', >'image/features/writable-descriptor-tables': '1', 'image/virt-base': >'-2147483648', 'memory/target': '524288', 'image/guest-version': >'2.6', 'image/features/supervisor-mode-kernel': '1', >'image/guest-os': 'linux', 'console/limit': '1048576', >'image/paddr-offset': '-2147483648', 'image/hypercall-page': >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': >'online', 'image/features/pae-pgdir-above-4gb': '1', >'image/features/writable-page-tables': '1', >'image/features/auto-translated-physmap': '1', 'name': 'xen263', >'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} >[2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController: >writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', >'backend': '/local/domain/0/backend/console/277/0'} to >/local/domain/277/device/console/0. >[2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController: >writing {'domain': 'xen263', 'protocol': 'vt100', 'uuid': >'d0da45dd-1a00-da65-61dd-66dadd6e9897', 'frontend': >'/local/domain/277/device/console/0', 'state': '1', 'location': '2', >'online': '1', 'frontend-id': '277'} to /local/domain/0/backend/console/277/0. >[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain >details: {'console/ring-ref': '5227479', 'image/entry': >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', >'store/ring-ref': '5227480', 'image/loader': 'generic', 'vm': >'/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', >'control/platform-feature-multiprocessor-suspend': '1', >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', >'image/features/writable-descriptor-tables': '1', 'image/virt-base': >'-2147483648', 'memory/target': '524288', 'image/guest-version': >'2.6', 'image/features/supervisor-mode-kernel': '1', >'image/guest-os': 'linux', 'console/limit': '1048576', >'image/paddr-offset': '-2147483648', 'image/hypercall-page': >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': >'online', 'image/features/pae-pgdir-above-4gb': '1', >'image/features/writable-page-tables': '1', >'image/features/auto-translated-physmap': '1', 'name': 'xen263', >'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} >[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907) >XendDomainInfo.handleShutdownWatch >[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for devices vif. >[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0. >[2007-09-02 17:48:23 8597] DEBUG (DevController:541) >hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status. >[2007-09-02 17:48:23 8597] DEBUG (DevController:555) hotplugStatusCallback 1. >[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices usb. >[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices vbd. >[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048. >[2007-09-02 17:48:25 8597] DEBUG (DevController:541) >hotplugStatusCallback /local/domain/0/backend/vbd/277/2048/hotplug-status. >[2007-09-02 17:48:25 8597] DEBUG (DevController:555) hotplugStatusCallback 1. >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices irq. >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices vkbd. >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices vfb. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for >devices console. >[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices pci. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for >devices ioports. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices tap. >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices vtpm. >[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 >(277) unpaused. > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com ><http://lists.xensource.com/xen-devel>http://lists.xensource.com/xen-de vel > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: VM start time slows with large number of VMs 2007-09-06 19:25 ` Carb, Brian A @ 2007-09-06 21:25 ` Keir Fraser [not found] ` <089B0D75973E1241B941D0A9854F23FC08E93337@USEA-EXCH2.na.uis .unisys.com> 1 sibling, 0 replies; 13+ messages in thread From: Keir Fraser @ 2007-09-06 21:25 UTC (permalink / raw) To: Carb, Brian A, Mats Petersson, xen-devel On 6/9/07 20:25, "Carb, Brian A" <Brian.Carb@unisys.com> wrote: > killall xensored xenconsoled > > rm -rf /var/lib/xenstored <- has the tdb > rm -rf /var/lib/xend <- has domains, state and storage dirs > > > but when i startup xend again, xenstore-ls still shows all the old > domains. I must be missing something. Do you actually spell xenstored properly in your killall line and the typo is restricted to this email? Killing xenstored, deleting /var/lib/xenstored/*, and restarting xenstored should suffice. -- Keir ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <089B0D75973E1241B941D0A9854F23FC08E93337@USEA-EXCH2.na.uis .unisys.com>]
* RE: VM start time slows with large number of VMs [not found] ` <089B0D75973E1241B941D0A9854F23FC08E93337@USEA-EXCH2.na.uis .unisys.com> @ 2007-09-06 21:32 ` Mats Petersson 0 siblings, 0 replies; 13+ messages in thread From: Mats Petersson @ 2007-09-06 21:32 UTC (permalink / raw) To: Carb, Brian A, Keir Fraser, xen-devel At 20:25 06/09/2007, Carb, Brian A wrote: >I can't seem to clear out this information. I tried > > /etc/init.d/xend stop > killall xensored xenconsoled I take it that was with the CORRECT spelling of xenstore? Verify that there are no xen processes running. > rm -rf /var/lib/xenstored <- has the tdb > rm -rf /var/lib/xend <- has domains, state and storage dirs > > >but when i startup xend again, xenstore-ls still shows all the old >domains. I must be missing something. I have successfully removed this file, and the when starting xend afterwards, the system has no "history" of xenstore data. -- Mats >brian carb >unisys corporation - malvern, pa > >-----Original Message----- >From: mats petersson Sent: Thursday, September 06, 2007 1:30 PM >To: Carb, Brian A; Keir Fraser; xen-devel@lists.xensource.com >Subject: RE: [Xen-devel] VM start time slows with large number of VMs > >At 17:24 06/09/2007, Carb, Brian A wrote: > >Content-class: urn:content-classes:message > >Content-Type: multipart/alternative; > > boundary="----_=_NextPart_001_01C7F0A2.55DB0EB0" > > > >keir > > > >I'll try timing this as I load up... > > > >but when I restart my test system, with only dom0 running, xenstore-ls > >is already slow before any VMs are started: > > > > time xenstore-ls >/dev/null > > real 0m21.725s > > user 0m0.010s > > sys 0m0.090s > > > >If I save the output from the xenstore-ls, the file has 61,318 lines > >with information from all the VMs run yesterday. I conclude that > >xenstore does not get cleared on startup. Is there a way to clear it? > > >It is perfectly fine to remove the xenstore.tdb [filename from memory] >in /var/xen/lib [again, from memory]. Unless of course you actually NEED >some of your old Xenstrore entries for soemthing, but in general, they >are fine to be removed completely. > >-- >Mats > > > > >brian carb > >unisys corporation - malvern, pa > > > > > > > >---------- > >From: Keir Fraser > >Sent: Wednesday, September 05, 2007 1:11 PM > >To: Carb, Brian A; xen-devel@lists.xensource.com > >Subject: Re: [Xen-devel] VM start time slows with large number of VMs > > > >xenstore slowness seems a reasonable hypothesis. How does > >performance of 'time xenstore-ls >/dev/null' change as you increase > >the number of guests? > > > > -- Keir > > > >On 4/9/07 17:51, "Carb, Brian A" wrote: > > > >Hello, > > > >We're running Xen Unstable changeset 15468 with SLES10 on a Unisys > >ES7000/one 56x cpu and 160GB memory. Xen is built with > >NR_DYNIRQS=1024 and max_phys_cpus=64, and booted with dom0_mem=1024 > >and xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on > >external storage. > > > >Our script starts a VM, waits for the start to complete, then starts > >the next: "for i in *.cfg; do xm create $i; done". As more VMs > >start, the time to start each one increases. I've attached a config > >file for one of the VMs, as well as a portion of the xend.log which > >shows timestamps of the events for 2 VMS, vm002 and vm263. > > > >The total elapsed time of the start from "create" to "unpaused" for > >vm002 is 13 seconds, but by the time 262 VMs are started, the start > >of vm263 has slowed to 238 seconds. The host itself is not sluggish, > >as we can open putty sessions to and interact with dom0. However, > >if we run "xm top", its interface is very unresponsive. Could > >writing to the xenstore be taking longer and longer? Could there be > >possible contention for vmlinuz and initrd since they are being used > >by every VM to boot? Would it be better to use bootloader/bootentry > >and store the vmlinuz and initrd within each VM? > > > >Any insight would be appreciated. > > > >brian carb > >unisys corporation, malvern pa > > > >----- domU config file 002.cfg ----- > >kernel = "/boot/vmlinuz-xen" > >ramdisk = "/boot/initrd-xen" > >root = "/dev/sda2 ro" > >disk = [ 'phy:/tmp/xendevs/xen002,sda,w' ] > >memory = 512 > >vcpus = 4 > >name = 'xen002' > >vif = [ 'mac=00:16:3e:0d:a0:01' ] > >ip = '10.0.27.2' > >netmask = '255.254.0.0' > >extra = ' xencons=tty1 3' > >----- xend.log ----- > >[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78) > >XendDomainInfo.create(['vm', ['name', 'xen002'], ['memory', 512], > >['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', > >'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], > >['ramdisk', '/boot/initrd-xen'], ['ip', > >'10.0.27.2:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 > >ro'], ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', > >'phy:/tmp/xendevs/xen002'], ['dev', 'sda'], ['mode', 'w']]], > >['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) > >[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418) > >XendDomainInfo.constructDomain > >[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 > >KiB free; need 2048; done. > >[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16 > >[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472) > >XendDomainInfo.initDomain: 16 256 > >[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503) > >_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, > >memory_static_min=0x0. > >[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 > >KiB free; need 524288; done. > >[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux > >dom=16 vcpus=4 > >[2007-09-02 05:26:43 8597] DEBUG (image:197) domid = 16 > >[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize = 512 > >[2007-09-02 05:26:43 8597] DEBUG (image:199) image = > >/boot/vmlinuz-xen > >[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn = 1 > >[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2 > >[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline = > >root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off >xencons=tty1 3 > >[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk = >/boot/initrd-xen > >[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus = 4 > >[2007-09-02 05:26:43 8597] DEBUG (image:205) features = > >[2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice: > >vbd : {'uuid': '8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'bootable': > >1, 'driver': 'paravirtualised', 'dev': 'sda', 'uname': > >'phy:/tmp/xendevs/xen002', 'mode': 'w'} > >[2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController: > >writing {'virtual-device': '2048', 'device-type': 'disk', > >'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', > >'backend': '/local/domain/0/backend/vbd/16/2048'} to > >/local/domain/16/device/vbd/2048. > >[2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController: > >writing {'domain': 'xen002', 'frontend': > >'/local/domain/16/device/vbd/2048', 'uuid': > >'8d79c4ed-4ddc-afee-9725-f5c2d388a02e', 'dev': 'sda', 'state': '1', > >'params': '/tmp/xendevs/xen002', 'mode': 'w', 'online': '1', > >'frontend-id': '16', 'type': 'phy'} to >/local/domain/0/backend/vbd/16/2048. > >[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: > >vif : {'mac': '00:16:3e:0d:a0:01', 'uuid': > >'bb43e84a-a9a1-7488-1ba4-3b713eff82aa'} > >[2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController: > >writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': > >'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': > >'/local/domain/0/backend/vif/16/0'} to /local/domain/16/device/vif/0. > >[2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController: > >writing {'domain': 'xen002', 'handle': '0', 'uuid': > >'bb43e84a-a9a1-7488-1ba4-3b713eff82aa', 'script': > >'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': > >'/local/domain/16/device/vif/0', 'mac': '00:16:3e:0d:a0:01', > >'online': '1', 'frontend-id': '16', 'type': 'netfront'} to > >/local/domain/0/backend/vif/16/0. > >[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM > >details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': > >'f146fdfc-e624-f7cb-4baf-0471f01f6014', 'on_reboot': 'restart', > >'start_time': '1188725206.15', 'on_poweroff': 'destroy', > >'on_xend_start': 'ignore', 'on_crash': 'restart', > >'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', > >'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk > >/boot/initrd-xen) (args 'root=/dev/sda2 ro > >ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') (notes > >(FEATURES > >'writable_page_tables|writable_descriptor_tables|auto_translated_physma >p|pae_pgdir_above_4gb|supervisor_mode_kernel') > >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET > >-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER > >generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION > >xen-3.0)))", 'name': 'xen002'} > >[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain > >details: {'console/ring-ref': '34194913', 'image/entry': > >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', > >'store/ring-ref': '34194914', 'image/loader': 'generic', 'vm': > >'/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', > >'control/platform-feature-multiprocessor-suspend': '1', > >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', > >'image/features/writable-descriptor-tables': '1', 'image/virt-base': > >'-2147483648', 'memory/target': '524288', 'image/guest-version': > >'2.6', 'image/features/supervisor-mode-kernel': '1', > >'image/guest-os': 'linux', 'console/limit': '1048576', > >'image/paddr-offset': '-2147483648', 'image/hypercall-page': > >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': > >'online', 'image/features/pae-pgdir-above-4gb': '1', > >'image/features/writable-page-tables': '1', > >'image/features/auto-translated-physmap': '1', 'name': 'xen002', > >'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} > >[2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController: > >writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', > >'backend': '/local/domain/0/backend/console/16/0'} to > >/local/domain/16/device/console/0. > >[2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController: > >writing {'domain': 'xen002', 'protocol': 'vt100', 'uuid': > >'8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e', 'frontend': > >'/local/domain/16/device/console/0', 'state': '1', 'location': '2', > >'online': '1', 'frontend-id': '16'} to >/local/domain/0/backend/console/16/0. > >[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for >devices vif. > >[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0. > >[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain > >details: {'console/ring-ref': '34194913', 'image/entry': > >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', > >'store/ring-ref': '34194914', 'image/loader': 'generic', 'vm': > >'/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014', > >'control/platform-feature-multiprocessor-suspend': '1', > >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', > >'image/features/writable-descriptor-tables': '1', 'image/virt-base': > >'-2147483648', 'memory/target': '524288', 'image/guest-version': > >'2.6', 'image/features/supervisor-mode-kernel': '1', > >'image/guest-os': 'linux', 'console/limit': '1048576', > >'image/paddr-offset': '-2147483648', 'image/hypercall-page': > >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': > >'online', 'image/features/pae-pgdir-above-4gb': '1', > >'image/features/writable-page-tables': '1', > >'image/features/auto-translated-physmap': '1', 'name': 'xen002', > >'domid': '16', 'image/xen-version': 'xen-3.0', 'store/port': '1'} > >[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907) > >XendDomainInfo.handleShutdownWatch > >[2007-09-02 05:26:53 8597] DEBUG (DevController:541) > >hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status. > >[2007-09-02 05:26:53 8597] DEBUG (DevController:555) >hotplugStatusCallback 1. > >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for >devices usb. > >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for >devices vbd. > >[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048. > >[2007-09-02 05:26:54 8597] DEBUG (DevController:541) > >hotplugStatusCallback >/local/domain/0/backend/vbd/16/2048/hotplug-status. > >[2007-09-02 05:26:54 8597] DEBUG (DevController:555) >hotplugStatusCallback 1. > >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for >devices irq. > >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for >devices vkbd. > >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for >devices vfb. > >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for > >devices console. > >[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0. > >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for >devices pci. > >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for > >devices ioports. > >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for >devices tap. > >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for >devices vtpm. > >[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16) >unpaused. > >... > >... [vms xen003 to xen262 deleted] > >... > >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78) > >XendDomainInfo.create(['vm', ['name', 'xen263'], ['memory', 512], > >['vcpus', 4], ['on_xend_start', 'ignore'], ['on_xend_stop', > >'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-xen'], > >['ramdisk', '/boot/initrd-xen'], ['ip', > >'10.0.29.63:1.2.3.4::255.254.0.0::eth0:off'], ['root', '/dev/sda2 > >ro'], ['args', ' xencons=tty1 3']]], ['device', ['vbd', ['uname', > >'phy:/tmp/xendevs/xen263'], ['dev', 'sda'], ['mode', 'w']]], > >['device', ['vif', ['mac', '00:16:3e:0d:a0:01']]]]) > >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418) > >XendDomainInfo.constructDomain > >[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB > >free; need 2048; done. > >[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277 > >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472) > >XendDomainInfo.initDomain: 277 256 > >[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503) > >_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, > >memory_static_min=0x0. > >[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB > >free; need 524288; done. > >[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux > >dom=277 vcpus=4 > >[2007-09-02 17:44:29 8597] DEBUG (image:197) domid = 277 > >[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize = 512 > >[2007-09-02 17:44:29 8597] DEBUG (image:199) image = > >/boot/vmlinuz-xen > >[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn = 1 > >[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2 > >[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline = > >root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off >xencons=tty1 3 > >[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk = >/boot/initrd-xen > >[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus = 4 > >[2007-09-02 17:44:29 8597] DEBUG (image:205) features = > >[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: > >vbd : {'uuid': '9d639202-73af-64ec-451d-3abe0fc57bc7', 'bootable': > >1, 'driver': 'paravirtualised', 'dev': 'sda', 'uname': > >'phy:/tmp/xendevs/xen263', 'mode': 'w'} > >[2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController: > >writing {'virtual-device': '2048', 'device-type': 'disk', > >'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', > >'backend': '/local/domain/0/backend/vbd/277/2048'} to > >/local/domain/277/device/vbd/2048. > >[2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController: > >writing {'domain': 'xen263', 'frontend': > >'/local/domain/277/device/vbd/2048', 'uuid': > >'9d639202-73af-64ec-451d-3abe0fc57bc7', 'dev': 'sda', 'state': '1', > >'params': '/tmp/xendevs/xen263', 'mode': 'w', 'online': '1', > >'frontend-id': '277', 'type': 'phy'} to >/local/domain/0/backend/vbd/277/2048. > >[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: > >vif : {'mac': '00:16:3e:0d:a0:01', 'uuid': > >'a126d815-fd4a-332b-c881-235a7e003cfb'} > >[2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController: > >writing {'mac': '00:16:3e:0d:a0:01', 'handle': '0', 'protocol': > >'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': > >'/local/domain/0/backend/vif/277/0'} to /local/domain/277/device/vif/0. > >[2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController: > >writing {'domain': 'xen263', 'handle': '0', 'uuid': > >'a126d815-fd4a-332b-c881-235a7e003cfb', 'script': > >'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': > >'/local/domain/277/device/vif/0', 'mac': '00:16:3e:0d:a0:01', > >'online': '1', 'frontend-id': '277', 'type': 'netfront'} to > >/local/domain/0/backend/vif/277/0. > >[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM > >details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': > >'8f8f9839-dd61-b9cd-89a6-f66bb3d41726', 'on_reboot': 'restart', > >'start_time': '1188769556.11', 'on_poweroff': 'destroy', > >'on_xend_start': 'ignore', 'on_crash': 'restart', > >'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', > >'image': "(linux (kernel /boot/vmlinuz-xen) (ramdisk > >/boot/initrd-xen) (args 'root=/dev/sda2 ro > >ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off xencons=tty1 3') > >(notes (FEATURES > >'writable_page_tables|writable_descriptor_tables|auto_translated_physma >p|pae_pgdir_above_4gb|supervisor_mode_kernel') > >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET > >-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER > >generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION > >xen-3.0)))", 'name': 'xen263'} > >[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain > >details: {'console/ring-ref': '5227479', 'image/entry': > >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', > >'store/ring-ref': '5227480', 'image/loader': 'generic', 'vm': > >'/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', > >'control/platform-feature-multiprocessor-suspend': '1', > >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', > >'image/features/writable-descriptor-tables': '1', 'image/virt-base': > >'-2147483648', 'memory/target': '524288', 'image/guest-version': > >'2.6', 'image/features/supervisor-mode-kernel': '1', > >'image/guest-os': 'linux', 'console/limit': '1048576', > >'image/paddr-offset': '-2147483648', 'image/hypercall-page': > >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': > >'online', 'image/features/pae-pgdir-above-4gb': '1', > >'image/features/writable-page-tables': '1', > >'image/features/auto-translated-physmap': '1', 'name': 'xen263', > >'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} > >[2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController: > >writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', > >'backend': '/local/domain/0/backend/console/277/0'} to > >/local/domain/277/device/console/0. > >[2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController: > >writing {'domain': 'xen263', 'protocol': 'vt100', 'uuid': > >'d0da45dd-1a00-da65-61dd-66dadd6e9897', 'frontend': > >'/local/domain/277/device/console/0', 'state': '1', 'location': '2', > >'online': '1', 'frontend-id': '277'} to >/local/domain/0/backend/console/277/0. > >[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain > >details: {'console/ring-ref': '5227479', 'image/entry': > >'-2145386496', 'console/port': '2', 'cpu/3/availability': 'online', > >'store/ring-ref': '5227480', 'image/loader': 'generic', 'vm': > >'/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726', > >'control/platform-feature-multiprocessor-suspend': '1', > >'cpu/2/availability': 'online', 'cpu/1/availability': 'online', > >'image/features/writable-descriptor-tables': '1', 'image/virt-base': > >'-2147483648', 'memory/target': '524288', 'image/guest-version': > >'2.6', 'image/features/supervisor-mode-kernel': '1', > >'image/guest-os': 'linux', 'console/limit': '1048576', > >'image/paddr-offset': '-2147483648', 'image/hypercall-page': > >'-2145361920', 'image/suspend-cancel': '1', 'cpu/0/availability': > >'online', 'image/features/pae-pgdir-above-4gb': '1', > >'image/features/writable-page-tables': '1', > >'image/features/auto-translated-physmap': '1', 'name': 'xen263', > >'domid': '277', 'image/xen-version': 'xen-3.0', 'store/port': '1'} > >[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907) > >XendDomainInfo.handleShutdownWatch > >[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for >devices vif. > >[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0. > >[2007-09-02 17:48:23 8597] DEBUG (DevController:541) > >hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status. > >[2007-09-02 17:48:23 8597] DEBUG (DevController:555) >hotplugStatusCallback 1. > >[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for >devices usb. > >[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for >devices vbd. > >[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048. > >[2007-09-02 17:48:25 8597] DEBUG (DevController:541) > >hotplugStatusCallback >/local/domain/0/backend/vbd/277/2048/hotplug-status. > >[2007-09-02 17:48:25 8597] DEBUG (DevController:555) >hotplugStatusCallback 1. > >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for >devices irq. > >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for >devices vkbd. > >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for >devices vfb. > >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for > >devices console. > >[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0. > >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for >devices pci. > >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for > >devices ioports. > >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for >devices tap. > >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for >devices vtpm. > >[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 > >(277) unpaused. > > > > > >_______________________________________________ > >Xen-devel mailing list > >Xen-devel@lists.xensource.com > ><http://lists.xensource.com/xen-devel>http://lists.xensource.com/xen-de >vel > > > > > >_______________________________________________ > >Xen-devel mailing list > >Xen-devel@lists.xensource.com > >http://lists.xensource.com/xen-devel ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2007-09-24 14:18 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-09-04 16:51 VM start time slows with large number of VMs Carb, Brian A
2007-09-05 17:11 ` Keir Fraser
2007-09-05 17:37 ` Daniel P. Berrange
2007-09-11 16:27 ` Carb, Brian A
[not found] ` <089B0D75973E1241B941D0A9854F23FC08F4CE10@USEA-EXCH2.na.uis .unisys.com>
2007-09-11 17:42 ` Mats Petersson
2007-09-24 14:18 ` Carb, Brian A
2007-09-06 16:24 ` Carb, Brian A
[not found] ` <089B0D75973E1241B941D0A9854F23FC08E92D5C@USEA-EXCH2.na.uis .unisys.com>
2007-09-06 17:29 ` Mats Petersson
2007-09-06 17:31 ` Keir Fraser
2007-09-06 17:44 ` Daniel P. Berrange
2007-09-06 19:25 ` Carb, Brian A
2007-09-06 21:25 ` Keir Fraser
[not found] ` <089B0D75973E1241B941D0A9854F23FC08E93337@USEA-EXCH2.na.uis .unisys.com>
2007-09-06 21:32 ` Mats Petersson
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.