Re: FreeBSD-amd64 fails to start with SMP on quemu-kvm

[Artur Samborski <sam@semihalf.com>]

W dniu 17.01.2013 07:37, Gleb Natapov pisze:
> On Wed, Jan 16, 2013 at 04:42:38PM +0200, Michael S. Tsirkin wrote:
>>>> First test:
>>>>
>>>>   - kvm.git kernel
>>>>   - 2 kvm guest running:
>>>>     - Linux (in idle)
>>>>     - Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world
>>>> and kernel build)
>>>>   - KVM host hangs after about 5 hours
>>>>   - nothing special in system logs
>>>>   - message caught on one of the active SSH session:
>>>>
>>>> kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s!
>>>> [vhost-3686:3700]
>>>>
>>>> Second test:
>>>>
>>>>   - kvm.git kernel
>>>>   - 1 kvm guest running:
>>>>     - Linux (at the time of hang -- in idle)
>>>>     - about 10 minutes before KVM host hangs -- load about 6 (kernel build)
>>>>   - in system logs:
>>>>
>>>> BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800]
>>>> Modules linked in: binfmt_misc ip6table_filter ip6_tables
>>>> ebtable_nat ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4
>>>> xt_state nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc
>>>> be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3
>>>> mdio libcxgbi ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm
>>>> ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi
>>>> scsi_transport_iscsi ioatdma vhost_net iTCO_wdt iTCO_vendor_support
>>>> ses lpc_ich tun macvtap macvlan mfd_core enclosure bnx2 joydev
>>>> i7core_edac dca edac_core wmi coretemp dcdbas kvm_intel pcspkr
>>>> crc32c_intel kvm serio_raw acpi_power_meter microcode uinput
>>>> ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas
>>>> CPU 0
>>>> Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc.
>>>> PowerEdge R610/086HF8
>>>> RIP: 0010:[<ffffffff8152876f>]  [<ffffffff8152876f>]
>>>> skb_flow_dissect+0xbf/0x3e0
>>>> RSP: 0018:ffff88042145dbd8  EFLAGS: 00000246
>>>> RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd
>>>> RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00
>>>> RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412
>>>> R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57
>>>> R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78
>>>> FS:  0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000
>>>> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>> CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0
>>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>> Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task
>>>> ffff880421151740)
>>>> Stack:
>>>>   ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412
>>>>   000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881
>>>>   0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412
>>>> Call Trace:
>>>>   [<ffffffff81525881>] ? skb_copy_datagram_from_iovec+0x61/0x280
>>>>   [<ffffffff8152a06a>] __skb_get_rxhash+0x1a/0xd0
>>>>   [<ffffffffa01109e0>] tun_get_user+0x3e0/0x760 [tun]
>>>>   [<ffffffffa0110dba>] tun_sendmsg+0x5a/0x80 [tun]
>>>>   [<ffffffffa013cd2a>] handle_tx+0x28a/0x680 [vhost_net]
>>>>   [<ffffffffa013d155>] handle_tx_kick+0x15/0x20 [vhost_net]
>>>>   [<ffffffffa013995d>] vhost_worker+0xed/0x190 [vhost_net]
>>>>   [<ffffffffa0139870>] ? vhost_work_flush+0x110/0x110 [vhost_net]
>>>>   [<ffffffff81081750>] kthread+0xc0/0xd0
>>>>   [<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0
>>>>   [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
>>>>   [<ffffffff8163feac>] ret_from_fork+0x7c/0xb0
>>>>   [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
>>>> Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49
>>>> 03 94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04
>>>> <0f> b6 42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb
>>>>
>>>> repeated several times with similar content
>>>>
>>>> --
>>>> Artur
>>
>>
>> Looks like this?
>> https://git.kernel.org/?p=linux/kernel/git/davem/net.git;a=commit;h=76fe45812a3b134c39170ca32dfd4b7217d33145
>>
> Artur,
>
> Can you please apply this fix on top of kvm.git/queue and try again.
>

Hi, thanks for the advice. I applied the patch created from commit 
76fe45812a3b134c39170ca32dfd4b7217d33145 to branch  queue previous 
evening. Fortunately, it seems that it works now. During the whole night 
(about 10 hours) I tested two virtual machines (Linux and FreeBSD-amd64) 
under high load and nothing bad happened.

 From the first observation it seems to me that the performance of the 
VM with FreeBSD-amd64 is much worse than the VM with FreeBSD-i386 -- 
but, it requires further testing.

-- 
Artur