From mboxrd@z Thu Jan 1 00:00:00 1970 From: Artur Samborski Subject: Re: FreeBSD-amd64 fails to start with SMP on quemu-kvm Date: Thu, 17 Jan 2013 08:53:43 +0100 Message-ID: <50F7AE07.9040703@semihalf.com> References: <50EB0232.4090003@semihalf.com> <20130107230028.GE1139@amt.cnet> <50EC856F.7030905@semihalf.com> <20130109101540.GC700@redhat.com> <50F02841.9040203@semihalf.com> <20130113092342.GE11529@redhat.com> <50F53B23.7090003@semihalf.com> <20130116130550.GV11529@redhat.com> <20130116144237.GA3339@redhat.com> <20130117063751.GB11529@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "Michael S. Tsirkin" , Marcelo Tosatti , kvm@vger.kernel.org To: Gleb Natapov Return-path: Received: from smtp.semihalf.com ([213.17.239.109]:53353 "EHLO smtp.semihalf.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755444Ab3AQHxr (ORCPT ); Thu, 17 Jan 2013 02:53:47 -0500 In-Reply-To: <20130117063751.GB11529@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: W dniu 17.01.2013 07:37, Gleb Natapov pisze: > On Wed, Jan 16, 2013 at 04:42:38PM +0200, Michael S. Tsirkin wrote: >>>> First test: >>>> >>>> - kvm.git kernel >>>> - 2 kvm guest running: >>>> - Linux (in idle) >>>> - Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world >>>> and kernel build) >>>> - KVM host hangs after about 5 hours >>>> - nothing special in system logs >>>> - message caught on one of the active SSH session: >>>> >>>> kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s! >>>> [vhost-3686:3700] >>>> >>>> Second test: >>>> >>>> - kvm.git kernel >>>> - 1 kvm guest running: >>>> - Linux (at the time of hang -- in idle) >>>> - about 10 minutes before KVM host hangs -- load about 6 (kernel build) >>>> - in system logs: >>>> >>>> BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800] >>>> Modules linked in: binfmt_misc ip6table_filter ip6_tables >>>> ebtable_nat ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4 >>>> xt_state nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc >>>> be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 >>>> mdio libcxgbi ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm >>>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi >>>> scsi_transport_iscsi ioatdma vhost_net iTCO_wdt iTCO_vendor_support >>>> ses lpc_ich tun macvtap macvlan mfd_core enclosure bnx2 joydev >>>> i7core_edac dca edac_core wmi coretemp dcdbas kvm_intel pcspkr >>>> crc32c_intel kvm serio_raw acpi_power_meter microcode uinput >>>> ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas >>>> CPU 0 >>>> Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc. >>>> PowerEdge R610/086HF8 >>>> RIP: 0010:[] [] >>>> skb_flow_dissect+0xbf/0x3e0 >>>> RSP: 0018:ffff88042145dbd8 EFLAGS: 00000246 >>>> RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd >>>> RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00 >>>> RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412 >>>> R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57 >>>> R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78 >>>> FS: 0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000 >>>> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >>>> CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0 >>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >>>> Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task >>>> ffff880421151740) >>>> Stack: >>>> ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412 >>>> 000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881 >>>> 0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412 >>>> Call Trace: >>>> [] ? skb_copy_datagram_from_iovec+0x61/0x280 >>>> [] __skb_get_rxhash+0x1a/0xd0 >>>> [] tun_get_user+0x3e0/0x760 [tun] >>>> [] tun_sendmsg+0x5a/0x80 [tun] >>>> [] handle_tx+0x28a/0x680 [vhost_net] >>>> [] handle_tx_kick+0x15/0x20 [vhost_net] >>>> [] vhost_worker+0xed/0x190 [vhost_net] >>>> [] ? vhost_work_flush+0x110/0x110 [vhost_net] >>>> [] kthread+0xc0/0xd0 >>>> [] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0 >>>> [] ? kthread_create_on_node+0x120/0x120 >>>> [] ret_from_fork+0x7c/0xb0 >>>> [] ? kthread_create_on_node+0x120/0x120 >>>> Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49 >>>> 03 94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04 >>>> <0f> b6 42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb >>>> >>>> repeated several times with similar content >>>> >>>> -- >>>> Artur >> >> >> Looks like this? >> https://git.kernel.org/?p=linux/kernel/git/davem/net.git;a=commit;h=76fe45812a3b134c39170ca32dfd4b7217d33145 >> > Artur, > > Can you please apply this fix on top of kvm.git/queue and try again. > Hi, thanks for the advice. I applied the patch created from commit 76fe45812a3b134c39170ca32dfd4b7217d33145 to branch queue previous evening. Fortunately, it seems that it works now. During the whole night (about 10 hours) I tested two virtual machines (Linux and FreeBSD-amd64) under high load and nothing bad happened. From the first observation it seems to me that the performance of the VM with FreeBSD-amd64 is much worse than the VM with FreeBSD-i386 -- but, it requires further testing. -- Artur