From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nikola Ciprich Subject: guests getting stuck under CPU load Date: Tue, 9 Sep 2008 14:17:43 +0200 Message-ID: <20080909121742.GC27882@develbox.linuxbox.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: nikola.ciprich@linuxbox.cz To: KVM list Return-path: Received: from gwu.lbox.cz ([62.245.111.132]:44660 "EHLO gwu.lbox.cz" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752212AbYIIMRq (ORCPT ); Tue, 9 Sep 2008 08:17:46 -0400 Content-Disposition: inline Sender: kvm-owner@vger.kernel.org List-ID: Hi, I'm experiencing problems with some KVM guests getting stuck for long time when all their "CPUs" are loaded. Host: 8xCPU intel, 2.6.26 x86_64, kvm-73, 12GB RAM Guest: 8XCPU, 2.6.24 x86_64 1GB RAM while doing kernel compilation on guest (using -j8), the guest sometimes hangs for long time, and on host I see: (sorry for long report, but I guess it might come useful) [1036764.088311] BUG: soft lockup - CPU#3 stuck for 511s! [qemu-system-x86:16404] [1036764.088311] Modules linked in: drbd reiserfs tun bitrev cn lock_dlm gfs2 crc32 dlm configfs ipmi_si ipmi_devintf ipmi_msghandler ipt_REJECT xt_tcpudp xt_multiport nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables kvm_intel kvm ipv6 nfs lockd nfs_acl sunrpc 8021q bridge llc dm_mirror dm_log dm_mod wmi sbs sbshc fan battery backlight acpi_memhotplug ac lp nvram e1000e sg piix pata_acpi container ide_pci_generic thermal processor thermal_sys button i5000_edac ata_piix ata_generic rng_core parport_pc parport pcspkr edac_core iTCO_wdt i2c_i801 i2c_core rtc_cmos rtc_core rtc_lib shpchp pci_hotplug ide_disk ide_core aic94xx libsas firmware_class scsi_transport_sas ahci libata dock sd_mod scsi_mod raid456 async_xor async_memcpy async_tx xor raid1 ext3 jbd uhci_hcd ohci _hcd ehci_hcd [last unloaded: drbd] [1036764.088311] CPU 3: [1036764.088311] Modules linked in: drbd ... stripped .. [1036764.088311] Pid: 16404, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1036764.088311] RIP: 0010:[] [] :kvm:pic_get_irq+0x15/0xa0 [1036764.088311] RSP: 0018:ffff8103020afca0 EFLAGS: 00000202 [1036764.088311] RAX: 00000000ffffff01 RBX: ffff810162195860 RCX: 0000000000000000 [1036764.088311] RDX: 0000000000000003 RSI: 0000000000000001 RDI: ffff81010906eec0 [1036764.088311] RBP: ffff8103020afc78 R08: 0000000000000001 R09: 0000000000000000 [1036764.088311] R10: 2222222222222222 R11: 0000000000000000 R12: ffff8101521962f8 [1036764.088311] R13: 2222222222222222 R14: 2222222222222222 R15: 2222222222222222 [1036764.088311] FS: 0000000042cda940(0063) GS:ffff81032fce65c0(0000) knlGS:0000000000000000 [1036764.088311] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [1036764.088311] CR2: 00002adf28fd56d0 CR3: 000000018f4d1000 CR4: 00000000000026e0 [1036764.088311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1036764.088311] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 [1036764.088311] [1036764.088311] Call Trace: [1036764.088311] [] ? :kvm:pic_update_irq+0x33/0x80 [1036764.088311] [] ? :kvm:kvm_inject_pit_timer_irqs+0x91/0xc0 [1036764.088311] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0x238/0x740 [1036764.088311] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1036764.088311] [] ? hrtimer_start+0xe0/0x180 [1036764.088311] [] ? avc_has_perm+0x5d/0x90 [1036764.088311] [] ? inode_has_perm+0x5b/0xa0 [1036764.088311] [] ? enqueue_hrtimer+0x6e/0xf0 [1036764.088311] [] ? __dequeue_signal+0x19/0x1c0 [1036764.088311] [] ? file_has_perm+0xe9/0xf0 [1036764.088311] [] ? vfs_ioctl+0x2f/0xa0 [1036764.088311] [] ? do_vfs_ioctl+0x74/0x2c0 [1036764.088311] [] ? sys_ioctl+0x91/0xb0 [1036764.088311] [] ? tracesys+0xd5/0xda [1036764.088677] BUG: soft lockup - CPU#5 stuck for 511s! [qemu-system-x86:16789] [1036764.088677] Modules linked in: drbd ...stripped... [1036764.088677] CPU 5: [1036764.088677] Modules linked in: drbd ...stripped... [1036764.088677] Pid: 16789, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1036764.088677] RIP: 0010:[] [] file_has_perm+0x82/0xf0 [1036764.088677] RSP: 0018:ffff8102674cbea8 EFLAGS: 00000206 [1036764.088677] RAX: ffff81020b526430 RBX: ffff8102674cbeb8 RCX: 0000000000000000 [1036764.088677] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 [1036764.088677] RBP: ffff8102ec0c6140 R08: 0000000000000000 R09: ffff8102674cbeb8 [1036764.088677] R10: 0000000000000008 R11: ffffffff80319be0 R12: 0000000000000001 [1036764.088677] R13: ffff81015dcf92c0 R14: ffff81032fc39138 R15: ffff81032f807c10 [1036764.088677] FS: 0000000041714940(0063) GS:ffff81032fd7fc40(0000) knlGS:0000000000000000 [1036764.088677] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [1036764.088677] CR2: 00000000080ddf5c CR3: 000000031a8f5000 CR4: 00000000000026e0 [1036764.088677] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1036764.088677] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1036764.088677] [1036764.088677] Call Trace: [1036764.088677] [] ? file_has_perm+0x66/0xf0 [1036764.088677] [] ? sys_ioctl+0x4d/0xb0 [1036764.088677] [] ? tracesys+0xd5/0xda [1036764.088993] BUG: soft lockup - CPU#7 stuck for 510s! [qemu-system-x86:14865] [1036764.088993] Modules linked in: drbd ...stripped... [1036764.088993] CPU 7: [1036764.088993] Modules linked in: drbd ...stripped... [1036764.088993] Pid: 14865, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1036764.088994] RIP: 0010:[] [] :kvm:rmap_remove+0xf0/0x1d0 [1036764.088994] RSP: 0018:ffff8101940f19f8 EFLAGS: 00000293 [1036764.088994] RAX: 0000000000000000 RBX: 000000000003871d RCX: ffff81022191f910 [1036764.088994] RDX: 0000000000000001 RSI: 00000000000336de RDI: ffff8101ad8f38f8 [1036764.088994] RBP: ffff8102a7227288 R08: ffff8100b95519c0 R09: ffffc20003204ef0 [1036764.088994] R10: 0000000000000001 R11: ffffffffa039b170 R12: ffffffff8028289f [1036764.088994] R13: ffffe200063fa038 R14: ffffffff80480377 R15: 00007f41f18a5000 [1036764.088994] FS: 000000005435b940(0000) GS:ffff81032fd8f340(0000) knlGS:0000000000000000 [1036764.088994] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [1036764.088994] CR2: 00002b917f7736d0 CR3: 0000000145609000 CR4: 00000000000026e0 [1036764.088994] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1036764.088994] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 [1036764.088994] Call Trace: [1036764.088994] [] ? :kvm:rmap_remove+0x95/0x1d0 [1036764.088994] [] ? :kvm:kvm_mmu_zap_page+0x214/0x2a0 [1036764.088994] [] ? :kvm:kvm_mmu_pte_write+0x35c/0x8c0 [1036764.088994] [] ? :kvm:paging64_gpte_access+0x1d/0x40 [1036764.088994] [] ? :kvm:paging64_walk_addr+0xc8/0x330 [1036764.088994] [] ? :kvm:emulator_write_phys+0x44/0x70 [1036764.088994] [] ? :kvm:emulator_write_emulated_onepage+0x96/0x120 [1036764.088994] [] ? :kvm:x86_emulate_insn+0x3e1/0x4a30 [1036764.088994] [] ? sys_mincore+0x14f/0x4d0 [1036764.088994] [] ? :kvm:x86_decode_insn+0xb3c/0xc10 [1036764.088994] [] ? :kvm:emulate_instruction+0x15f/0x300 [1036764.088994] [] ? :kvm:kvm_mmu_page_fault+0x5b/0x90 [1036764.088994] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0x163/0x740 [1036764.088994] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1036764.088994] [] ? _spin_lock_irqsave+0x37/0x50 [1036764.088994] [] ? avc_has_perm+0x5d/0x90 [1036764.088994] [] ? inode_has_perm+0x5b/0xa0 [1036764.088994] [] ? ktime_get_ts+0x25/0x60 [1036764.088994] [] ? ktime_get+0xc/0x50 [1036764.088994] [] ? file_has_perm+0xe9/0xf0 [1036764.088994] [] ? vfs_ioctl+0x2f/0xa0 [1036764.088994] [] ? do_vfs_ioctl+0x74/0x2c0 [1036764.088994] [] ? sys_ioctl+0x91/0xb0 [1036764.088994] [] ? tracesys+0xd5/0xda [1036764.092271] BUG: soft lockup - CPU#1 stuck for 510s! [qemu-system-x86:14858] [1036764.092271] Modules linked in: drbd ...stripped... [1036764.092271] CPU 1: [1036764.092271] Modules linked in: drbd ...stripped... [1036764.092271] Pid: 14858, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1036764.092271] RIP: 0010:[] [] :kvm:kvm_arch_vcpu_ioctl_run+0x111/0x740 [1036764.092271] RSP: 0018:ffff81030c8e9cd8 EFLAGS: 00000202 [1036764.092271] RAX: 0000000000000000 RBX: ffff810125f0a1c0 RCX: 0000000000000000 [1036764.092271] RDX: 0000000000000000 RSI: ffff8100175695a8 RDI: 0000000000004404 [1036764.092271] RBP: ffff8103205fc0b0 R08: 5010000000000000 R09: 0000000000000000 [1036764.092271] R10: 0000000000000000 R11: ffffffff802c56d0 R12: ffffffff803429d4 [1036764.092271] R13: ffff810125f0a1e8 R14: ffff810125f0a1e8 R15: ffff810125f0a1e8 [1036764.092271] FS: 0000000051355940(0000) GS:ffff81032fc74e40(0000) knlGS:0000000000000000 [1036764.092271] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [1036764.092271] CR2: 00002b612f9e5000 CR3: 0000000145609000 CR4: 00000000000026e0 [1036764.092271] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1036764.092271] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 [1036764.092271] Call Trace: [1036764.092271] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0xff/0x740 [1036764.092271] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1036764.092271] [] ? _spin_lock_irqsave+0x37/0x50 [1036764.092271] [] ? avc_has_perm+0x5d/0x90 [1036764.092271] [] ? inode_has_perm+0x5b/0xa0 [1036764.092271] [] ? file_has_perm+0xe9/0xf0 [1036764.092271] [] ? vfs_ioctl+0x2f/0xa0 [1036764.092271] [] ? do_vfs_ioctl+0x74/0x2c0 [1036764.092271] [] ? sys_ioctl+0x91/0xb0 [1036764.092271] [] ? tracesys+0xd5/0xda [1051599.426360] BUG: soft lockup - CPU#3 stuck for 94s! [swapper:0] [1051599.426360] Modules linked in: drbd ...stripped... [1051599.426360] CPU 3: [1051599.426360] Modules linked in: drbd ...stripped... [1051599.426360] Pid: 0, comm: swapper Not tainted 2.6.26lb.00_01_PRE14 #1 [1051599.426360] RIP: 0010:[] [] mwait_idle+0x41/0x50 [1051599.426360] RSP: 0018:ffff81032fd3ff30 EFLAGS: 00000246 [1051599.426360] RAX: 0000000000000000 RBX: ffffffff806855c0 RCX: 0000000000000000 [1051599.426360] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff805b4100 [1051599.426360] RBP: ffff8100010e1e00 R08: 0000000000000000 R09: ffffffff8024e820 [1051599.426360] R10: 0000000000100000 R11: 0000000000000002 R12: ffffffff8024d3c5 [1051599.426360] R13: ffff81032fd3fec8 R14: 000000000716d6a0 R15: ffff81032b0fe500 [1051599.426360] FS: 0000000000000000(0000) GS:ffff81032fce65c0(0000) knlGS:0000000000000000 [1051599.426360] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [1051599.426360] CR2: 0000000000630000 CR3: 00000001dd1d4000 CR4: 00000000000026e0 [1051599.426360] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1051599.426360] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 [1051599.426360] [] ? cpu_idle+0x59/0xb0 [1051599.426360] BUG: soft lockup - CPU#5 stuck for 94s! [qemu-system-x86:14862] [1051599.426360] Modules linked in: drbd ...stripped... [1051599.426360] CPU 5: [1051599.426360] Modules linked in: drbd ...stripped... [1051599.426360] Pid: 14862, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1051599.426360] RIP: 0010:[] [] native_read_tsc+0x8/0x20 [1051599.426360] RSP: 0018:ffff8101d4805b30 EFLAGS: 00000206 [1051599.426360] RAX: 000000007a0f0e56 RBX: 000000007a0f0dfc RCX: 000000007a0f0dfc [1051599.426360] RDX: 0000000000029c8b RSI: 0000000000000000 RDI: 0000000000000001 [1051599.426360] RBP: ffff8102a7227288 R08: 0000000000000001 R09: ffffffffa037cfc1 [1051599.426360] R10: 0000000000000001 R11: ffffffffa039b170 R12: 0000000000000007 [1051599.426360] R13: ffff8102a7227288 R14: ffffffff8028289f R15: ffffe200110f1130 [1051599.426360] FS: 0000000052b58940(0000) GS:ffff81032fd7fc40(0000) knlGS:0000000000000000 [1051599.426360] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [1051599.426360] CR2: 00002af8ae592a60 CR3: 0000000145609000 CR4: 00000000000026e0 [1051599.426360] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1051599.426360] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1051599.426360] Call Trace: [1051599.426360] [] ? __delay+0x34/0x60 [1051599.426360] [] ? _raw_spin_lock+0x10d/0x180 [1051599.426360] [] ? :kvm:paging64_page_fault+0x161/0x4b0 [1051599.426360] [] ? hrtimer_start+0xe0/0x180 [1051599.426360] [] ? thread_return+0x5e/0x600 [1051599.426360] [] ? :kvm:kvm_mmu_page_fault+0x19/0x90 [1051599.426360] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0x163/0x740 [1051599.426360] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1051599.426360] [] ? _spin_lock_irqsave+0x37/0x50 [1051599.426360] [] ? avc_has_perm+0x5d/0x90 [1051599.426360] [] ? inode_has_perm+0x5b/0xa0 [1051599.426360] [] ? autoremove_wake_function+0x0/0x30 [1051599.426360] [] ? file_has_perm+0xe9/0xf0 [1051599.426360] [] ? vfs_ioctl+0x2f/0xa0 [1051599.426360] [] ? do_vfs_ioctl+0x74/0x2c0 [1051599.426360] [] ? sys_ioctl+0x91/0xb0 [1051599.426360] [] ? tracesys+0xd5/0xda [1051599.426602] BUG: soft lockup - CPU#7 stuck for 94s! [syslog-ng:4445] [1051599.426602] Modules linked in: drbd ...stripped... [1051599.426602] CPU 7: [1051599.426602] Modules linked in: drbd ...stripped... [1051599.426602] Pid: 4445, comm: syslog-ng Not tainted 2.6.26lb.00_01_PRE14 #1 [1051599.426602] RIP: 0033:[<00007f983abaa303>] [<00007f983abaa303>] [1051599.426602] RSP: 002b:00007fff434aef40 EFLAGS: 00000202 [1051599.426602] RAX: 00000000000007d7 RBX: 0000000000000504 RCX: 00000000000007d0 [1051599.426602] RDX: 00000000000007d7 RSI: 0000000000000000 RDI: 0000000000000000 [1051599.426602] RBP: 0000000001811cc0 R08: 00000000000002dd R09: 00000000000007d7 [1051599.426602] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000002000 [1051599.426602] R13: 00000000017f4260 R14: 00000000017f4140 R15: 0000000000000000 [1051599.426602] FS: 00007f983b4a26e0(0000) GS:ffff81032fd8f340(0000) knlGS:0000000000000000 [1051599.426602] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [1051599.426602] CR2: 00000000b7ef9000 CR3: 000000032e164000 CR4: 00000000000026e0 [1051599.426602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1051599.426602] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 [1051599.426602] Call Trace: nik: NONE!? [1051599.426602] [1051599.426360] BUG: soft lockup - CPU#4 stuck for 94s! [qemu-system-x86:14863] [1051599.426360] Modules linked in: drbd ...stripped... [1051599.426360] CPU 4: [1051599.426360] Modules linked in: drbd ...stripped... [1051599.426360] Pid: 14863, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1051599.426360] RIP: 0010:[] [] :kvm:rmap_remove+0xf0/0x1d0 [1051599.426360] RSP: 0018:ffff8102fa71b9f8 EFLAGS: 00000246 [1051599.426360] RAX: ffff81015c1a99c1 RBX: 000000000028887b RCX: ffff8102e8df84a8 [1051599.426360] RDX: 0000000000000000 RSI: 0000000000007495 RDI: ffff81015c1a99c0 [1051599.426360] RBP: ffff8102fa71b998 R08: 0000000000000000 R09: ffffc200030a3ca8 [1051599.426360] R10: 0000000000000001 R11: ffffffffa039b170 R12: 000000000716d69e [1051599.426360] R13: ffffc2000322d580 R14: ffffffff8024fd51 R15: ffffffffa039b170 [1051599.426360] FS: 0000000053359940(0000) GS:ffff81032fcf28c0(0000) knlGS:0000000000000000 [1051599.426360] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [1051599.426360] CR2: 00002b93a50786d0 CR3: 0000000145609000 CR4: 00000000000026e0 [1051599.426360] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1051599.426360] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 [1051599.426360] Call Trace: [1051599.426360] [] ? :kvm:rmap_remove+0x95/0x1d0 [1051599.426360] [] ? :kvm:kvm_mmu_zap_page+0x214/0x2a0 [1051599.426360] [] ? :kvm:kvm_mmu_pte_write+0x35c/0x8c0 [1051599.426360] [] ? :kvm:paging64_gpte_access+0x1d/0x40 [1051599.426360] [] ? :kvm:paging64_walk_addr+0xc8/0x330 [1051599.426360] [] ? :kvm:emulator_write_phys+0x44/0x70 [1051599.426360] [] ? :kvm:emulator_write_emulated_onepage+0x96/0x120 [1051599.426360] [] ? :kvm:x86_emulate_insn+0x3e1/0x4a30 [1051599.426360] [] ? sys_mincore+0x14f/0x4d0 [1051599.426360] [] ? :kvm:x86_decode_insn+0xb3c/0xc10 [1051599.426360] [] ? thread_return+0x5e/0x600 [1051599.426360] [] ? :kvm:emulate_instruction+0x15f/0x300 [1051599.426360] [] ? :kvm:kvm_mmu_page_fault+0x5b/0x90 [1051599.426360] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0x163/0x740 [1051599.426360] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1051599.426360] [] ? _spin_lock_irqsave+0x37/0x50 [1051599.426360] [] ? avc_has_perm+0x5d/0x90 [1051599.426360] [] ? inode_has_perm+0x5b/0xa0 [1051599.426360] [] ? selinux_file_ioctl+0x0/0xf0 [1051599.426360] [] ? file_has_perm+0xe9/0xf0 [1051599.426360] [] ? vfs_ioctl+0x2f/0xa0 [1051599.426360] [] ? do_vfs_ioctl+0x74/0x2c0 [1051599.426360] [] ? sys_ioctl+0x91/0xb0 [1051599.426360] [] ? tracesys+0xd5/0xda [1051599.426602] BUG: soft lockup - CPU#6 stuck for 94s! [qemu-system-x86:14864] [1051599.426602] Modules linked in: drbd ...stripped... [1051599.426602] CPU 6: [1051599.426602] Modules linked in: drbd ...stripped... [1051599.426602] Pid: 14864, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1051599.426602] RIP: 0010:[] [] native_read_tsc+0x8/0x20 [1051599.426602] RSP: 0018:ffff8101adcddb30 EFLAGS: 00000202 [1051599.426602] RAX: 000000007a1528cc RBX: 0000000000000aa1 RCX: 000000007a152860 [1051599.426602] RDX: 0000000000029c8b RSI: 0000000000000000 RDI: 0000000000000001 [1051599.426602] RBP: ffff8102a7227288 R08: 0000000000000001 R09: ffffffffa037cfc1 [1051599.426602] R10: 0000000000000002 R11: ffffffffa039b170 R12: 0000000000000007 [1051599.426602] R13: ffff8102a7227288 R14: ffffffff8028289f R15: ffffe2001106bb88 [1051599.426602] FS: 0000000053b5a940(0000) GS:ffff81032fd8ff40(0000) knlGS:0000000000000000 [1051599.426602] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [1051599.426602] CR2: 00002ab6259576d0 CR3: 0000000145609000 CR4: 00000000000026e0 [1051599.426602] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1051599.426602] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 [1051599.426602] Call Trace: [1051599.426602] [] ? __delay+0x15/0x60 [1051599.426602] [] ? _raw_spin_lock+0x10d/0x180 [1051599.426602] [] ? :kvm:paging64_page_fault+0x161/0x4b0 [1051599.426602] [] ? ktime_get+0xc/0x50 [1051599.426602] [] ? :kvm_intel:vmx_vcpu_load+0x17/0x190 [1051599.426602] [] ? thread_return+0x5e/0x600 [1051599.426602] [] ? :kvm:kvm_mmu_page_fault+0x19/0x90 [1051599.426602] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0x163/0x740 [1051599.426602] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1051599.426602] [] ? _spin_lock_irqsave+0x37/0x50 [1051599.426602] [] ? avc_has_perm+0x5d/0x90 [1051599.426602] [] ? inode_has_perm+0x5b/0xa0 [1051599.426602] [] ? file_has_perm+0xe9/0xf0 [1051599.426602] [] ? vfs_ioctl+0x2f/0xa0 [1051599.426602] [] ? do_vfs_ioctl+0x74/0x2c0 [1051599.426602] [] ? sys_ioctl+0x91/0xb0 [1051599.426602] [] ? tracesys+0xd5/0xda [1051599.428703] BUG: soft lockup - CPU#0 stuck for 94s! [qemu-system-x86:14859] [1051599.428703] Modules linked in: drbd ...stripped... [1051599.428703] CPU 0: [1051599.428703] Modules linked in: drbd ...stripped... [1051599.428703] Pid: 14859, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1051599.428703] RIP: 0010:[] [] :kvm:kvm_arch_vcpu_ioctl_run+0x111/0x740 [1051599.428703] RSP: 0018:ffff8101f983dcd8 EFLAGS: 00000206 [1051599.428703] RAX: 0000000000000000 RBX: ffff81012a808200 RCX: 0000000000000000 [1051599.428703] RDX: 0000000000000000 RSI: 8000000018c0e067 RDI: 0000000000004404 [1051599.428703] RBP: ffff8103205fc0b0 R08: ffffffff80537e44 R09: 0000000000000001 [1051599.428703] R10: 0000000000000002 R11: 00000000000001a8 R12: ffffffff803429d4 [1051599.428703] R13: 0000000000201810 R14: 0000000000008000 R15: 0000000000009908 [1051599.428703] FS: 0000000051b56940(0000) GS:ffffffff805e7000(0000) knlGS:0000000000000000 [1051599.428703] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [1051599.428703] CR2: 00002b9ec56c36d0 CR3: 0000000145609000 CR4: 00000000000026e0 [1051599.428703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1051599.428703] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1051599.428703] Call Trace: [1051599.428703] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0xff/0x740 [1051599.428703] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1051599.428703] [] ? _spin_lock_irqsave+0x37/0x50 [1051599.428703] [] ? avc_has_perm+0x5d/0x90 [1051599.428703] [] ? inode_has_perm+0x5b/0xa0 [1051599.428703] [] ? ktime_get_ts+0x25/0x60 [1051599.428703] [] ? __dequeue_signal+0x19/0x1c0 [1051599.428703] [] ? ktime_get+0xc/0x50 [1051599.428703] [] ? file_has_perm+0xe9/0xf0 [1051599.428703] [] ? vfs_ioctl+0x2f/0xa0 [1051599.428703] [] ? do_vfs_ioctl+0x74/0x2c0 [1051599.428703] [] ? sys_ioctl+0x91/0xb0 [1051599.428703] [] ? tracesys+0xd5/0xda [1051599.428703] BUG: soft lockup - CPU#1 stuck for 94s! [qemu-system-x86:14858] [1051599.428703] Modules linked in: drbd ...stripped... [1051599.428703] CPU 1: [1051599.428703] Modules linked in: drbd ...stripped... [1051599.428703] Pid: 14858, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1051599.428703] RIP: 0010:[] [] native_read_tsc+0x8/0x20 [1051599.428703] RSP: 0018:ffff81030c8e9b30 EFLAGS: 00000206 [1051599.428703] RAX: 000000007a584bae RBX: 00000000000003f6 RCX: 000000007a584b42 [1051599.428703] RDX: 0000000000029c8b RSI: 0000000000000000 RDI: 0000000000000001 [1051599.428703] RBP: ffff8102a7227288 R08: 0000000000000001 R09: ffffffffa037cfc1 [1051599.428703] R10: 00007fff609fa820 R11: ffffffffa039b170 R12: 0000000000000007 [1051599.428703] R13: ffff8102a7227288 R14: ffffffff8028289f R15: ffffe2000f01f650 [1051599.428703] FS: 0000000051355940(0000) GS:ffff81032fc74e40(0000) knlGS:ffffffff80562000 [1051599.428703] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [1051599.428703] CR2: 00002b8b4dd20240 CR3: 0000000145609000 CR4: 00000000000026e0 [1051599.428703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1051599.428703] DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 [1051599.428703] Call Trace: [1051599.428703] [] ? __delay+0x15/0x60 [1051599.428703] [] ? _raw_spin_lock+0x10d/0x180 [1051599.428703] [] ? :kvm:paging64_page_fault+0x161/0x4b0 [1051599.428703] [] ? :kvm:kvm_mmu_page_fault+0x19/0x90 [1051599.428703] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0x163/0x740 [1051599.428703] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1051599.428703] [] ? _spin_lock_irqsave+0x37/0x50 [1051599.428703] [] ? avc_has_perm+0x5d/0x90 [1051599.428703] [] ? inode_has_perm+0x5b/0xa0 [1051599.428703] [] ? autoremove_wake_function+0x0/0x30 [1051599.428703] [] ? file_has_perm+0xe9/0xf0 [1051599.428703] [] ? vfs_ioctl+0x2f/0xa0 [1051599.428703] [] ? do_vfs_ioctl+0x74/0x2c0 [1051599.428703] [] ? sys_ioctl+0x91/0xb0 [1051599.428703] [] ? tracesys+0xd5/0xda [1051599.428703] BUG: soft lockup - CPU#2 stuck for 94s! [qemu-system-x86:14865] [1051599.428703] Modules linked in: drbd ...stripped... [1051599.428703] CPU 2: [1051599.428703] Modules linked in: drbd ...stripped... [1051599.428703] Pid: 14865, comm: qemu-system-x86 Not tainted 2.6.26lb.00_01_PRE14 #1 [1051599.428703] RIP: 0010:[] [] native_read_tsc+0xa/0x20 [1051599.428703] RSP: 0018:ffff8101940f1b30 EFLAGS: 00000206 [1051599.428703] RAX: 000000007a5e6696 RBX: 0000000000000312 RCX: 000000007a5e6696 [1051599.428703] RDX: 0000000000029c8b RSI: 0000000000000000 RDI: 0000000000000001 [1051599.428703] RBP: ffff8102a7227288 R08: 0000000000000001 R09: ffffffffa037cfc1 [1051599.428703] R10: 0000000000007fff R11: ffffffffa039b170 R12: 0000000000000007 [1051599.428703] R13: ffff8102a7227288 R14: ffffffff8028289f R15: ffffe2000b5ee5d8 [1051599.428703] FS: 000000005435b940(0000) GS:ffff81032fc742c0(0000) knlGS:ffff81003f8e0240 [1051599.428703] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [1051599.428703] CR2: 00002b29cc972018 CR3: 0000000145609000 CR4: 00000000000026e0 [1051599.428703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [1051599.428703] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [1051599.428703] Call Trace: [1051599.428703] [] ? __delay+0x15/0x60 [1051599.428703] [] ? _raw_spin_lock+0x10d/0x180 [1051599.428703] [] ? :kvm:paging64_page_fault+0x161/0x4b0 [1051599.428703] [] ? _spin_unlock_irq+0x21/0x30 [1051599.428703] [] ? :kvm:kvm_mmu_page_fault+0x19/0x90 [1051599.428703] [] ? :kvm:kvm_arch_vcpu_ioctl_run+0x163/0x740 [1051599.428703] [] ? :kvm:kvm_vcpu_ioctl+0x3fd/0x470 [1051599.428703] [] ? _spin_lock_irqsave+0x37/0x50 [1051599.428703] [] ? avc_has_perm+0x5d/0x90 [1051599.428703] [] ? inode_has_perm+0x5b/0xa0 [1051599.428703] [] ? file_has_perm+0xe9/0xf0 [1051599.428703] [] ? vfs_ioctl+0x2f/0xa0 [1051599.428703] [] ? do_vfs_ioctl+0x74/0x2c0 [1051599.428703] [] ? sys_ioctl+0x91/0xb0 [1051599.428703] [] ? tracesys+0xd5/0xda Any ideas? It is important to note that machines I'm having this problem are running large amount of guests (~40), but (apart for guest doing compilation) they are not that busy, usual loadavg is around 1. Thanks a lot in advance! best regards nik -- ------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@linuxbox.cz -------------------------------------