Re: FreeBSD-amd64 fails to start with SMP on quemu-kvm

[Artur Samborski <sam@semihalf.com>]

W dniu 13.01.2013 10:23, Gleb Natapov pisze:
> On Fri, Jan 11, 2013 at 03:57:05PM +0100, Artur Samborski wrote:
>> W dniu 09.01.2013 11:15, Gleb Natapov pisze:
>>> On Tue, Jan 08, 2013 at 09:45:35PM +0100, Artur Samborski wrote:
>>>> W dniu 08.01.2013 00:00, Marcelo Tosatti pisze:
>>>>> On Mon, Jan 07, 2013 at 06:13:22PM +0100, Artur Samborski wrote:
>>>>>> Hello,
>>>>>>
>>>>>> When i try to run FreeBSD-amd64 on more than 1 vcpu in quemu-kvm
>>>>>> (Fedora Core 17) eg. to run FreeBSD-9.0-RELEASE-amd64 with:
>>>>>>
>>>>>> qemu-kvm -m 1024m -cpu host -smp 2 -cdrom
>>>>>> /storage/iso/FreeBSD-9.0-RELEASE-amd64-dvd1.iso
>>>>>>
>>>>>> it freezes KVM with:
>>>>>>
>>>>>> KVM internal error. Suberror: 1
>>>>>> emulation failure
>>>>>> RAX=ffffffff80b0d4c0 RBX=000000000009f000 RCX=00000000c0000080
>>>>>> RDX=0000000000000000
>>>>>> RSI=000000000000d238 RDI=0000000000000000 RBP=0000000000000000
>>>>>> RSP=0000000000000000
>>>>>> R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000
>>>>>> R11=0000000000000000
>>>>>> R12=0000000000000000 R13=0000000000000000 R14=0000000000000000
>>>>>> R15=0000000000000000
>>>>>> RIP=000000000009f076 RFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>>>>>> ES =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
>>>>>> CS =0008 0000000000000000 00000000 00209900 DPL=0 CS64 [--A]
>>>>>> SS =9f00 000000000009f000 0000ffff 0000f300 DPL=3 DS16 [-WA]
>>>>>> DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>>>>>> FS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
>>>>>> GS =0000 0000000000000000 0000ffff 0000f300 DPL=3 DS16 [-WA]
>>>>>> LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
>>>>>> TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
>>>>>> GDT=     000000000009f080 00000020
>>>>>> IDT=     0000000000000000 0000ffff
>>>>>> CR0=80000011 CR2=0000000000000000 CR3=000000000009c000 CR4=00000030
>>>>>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
>>>>>> DR3=0000000000000000
>>>>>> DR6=00000000ffff0ff0 DR7=0000000000000400
>>>>>> EFER=0000000000000501
>>>>>> Code=00 00 00 80 0f 22 c0 ea 70 f0 09 00 08 00 48 b8 c0 d4 b0 80
>>>>>> <ff> ff ff ff ff e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>>>>> 00 99 20 00 ff ff 00 00
>>>>>
>>>>> Artur,
>>>>>
>>>>> Can you check whether
>>>>>
>>>>> https://patchwork-mail.kernel.org/patch/1942681/
>>>>>
>>>>> fixes your problem
>>>>>
>>>>
>>>> Hi, thanks for the reply.
>>>>
>>>> Unfortunately, the patch does not help. Attempt to start FreeBSD
>>>> amd64 via quemu-kvm with -smp parameter greater than 1 fails in
>>>> exactly the same way as before.
>>>>
>>>> The patch was applied to the kernel from the 3.6.11-1.fc17.src.rpm package.
>>>>
>>>> Do I need some additional patches?
>>>>
>>>>
>>> Can you try queue branch from kvm.git?
>>>
>>
>> Thanks for the advice - I used kernel from kvm.git(queue) and was
>> able to run FreeBSD-amd guest with SMP on my kvm-host server without
>> previous problems. Unfortunately, after few hours server was hung
>> up. Case requires further investigations, but generally speaking we
>> went forward. Unfortunately, experiments on the server are
>> difficult, because it is used for everyday work.
>>
> Thanks for testing. This hang indeed looks like completely different
> problem. What workload the VM was running?
>

First test:

  - kvm.git kernel
  - 2 kvm guest running:
    - Linux (in idle)
    - Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world and 
kernel build)
  - KVM host hangs after about 5 hours
  - nothing special in system logs
  - message caught on one of the active SSH session:

kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s! 
[vhost-3686:3700]

Second test:

  - kvm.git kernel
  - 1 kvm guest running:
    - Linux (at the time of hang -- in idle)
    - about 10 minutes before KVM host hangs -- load about 6 (kernel build)
  - in system logs:

BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800]
Modules linked in: binfmt_misc ip6table_filter ip6_tables ebtable_nat 
ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4 xt_state 
nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc be2iscsi 
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi 
ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad 
ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ioatdma 
vhost_net iTCO_wdt iTCO_vendor_support ses lpc_ich tun macvtap macvlan 
mfd_core enclosure bnx2 joydev i7core_edac dca edac_core wmi coretemp 
dcdbas kvm_intel pcspkr crc32c_intel kvm serio_raw acpi_power_meter 
microcode uinput ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas
CPU 0
Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc. PowerEdge 
R610/086HF8
RIP: 0010:[<ffffffff8152876f>]  [<ffffffff8152876f>] 
skb_flow_dissect+0xbf/0x3e0
RSP: 0018:ffff88042145dbd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd
RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00
RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412
R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57
R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78
FS:  0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task 
ffff880421151740)
Stack:
  ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412
  000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881
  0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412
Call Trace:
  [<ffffffff81525881>] ? skb_copy_datagram_from_iovec+0x61/0x280
  [<ffffffff8152a06a>] __skb_get_rxhash+0x1a/0xd0
  [<ffffffffa01109e0>] tun_get_user+0x3e0/0x760 [tun]
  [<ffffffffa0110dba>] tun_sendmsg+0x5a/0x80 [tun]
  [<ffffffffa013cd2a>] handle_tx+0x28a/0x680 [vhost_net]
  [<ffffffffa013d155>] handle_tx_kick+0x15/0x20 [vhost_net]
  [<ffffffffa013995d>] vhost_worker+0xed/0x190 [vhost_net]
  [<ffffffffa0139870>] ? vhost_work_flush+0x110/0x110 [vhost_net]
  [<ffffffff81081750>] kthread+0xc0/0xd0
  [<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0
  [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
  [<ffffffff8163feac>] ret_from_fork+0x7c/0xb0
  [<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49 03 
94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04 <0f> b6 
42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb

repeated several times with similar content

-- 
Artur