[Bug 40542] New: overflow/panic on KVM hipervizor

[Bug 40542] New: overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542

           Summary: overflow/panic on KVM hipervizor
           Product: Virtualization
           Version: unspecified
    Kernel Version: 3.0.0
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: kvm
        AssignedTo: virtualization_kvm@kernel-bugs.osdl.org
        ReportedBy: slawek@rozbicki.eu
        Regression: No


Created an attachment (id=67522)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=67522)
some debug info like /proc/cpuinfo and .config from kernel

I have 2 machines virtualized with KVM on Gentoo host. There is a gentoo and
win2k8r2 guests (there is apache2 on gentoo which connects with php to mssql
server on the second guest). 

During the apache benchmark (ab -c 50 -n 100000 http://address/), hipervisor
TCP stack crashes (not entirely, but random anomalies are noticed for example
iptables -Lnv makes ssh session crashed). Here is kern.log snap:

---
 Aug  4 18:54:46 flu kernel: general protection fault: 0000 [#1] SMP
Aug  4 18:54:46 flu kernel: CPU 0
Aug  4 18:54:46 flu kernel: Modules linked in: bridge stp llc kvm_intel kvm
Aug  4 18:54:46 flu kernel:
Aug  4 18:54:46 flu kernel: Pid: 4327, comm: bash Not tainted
3.0.0-xxxx-std-ipv4-64 #7 Supermicro X8STi/X8STi
Aug  4 18:54:46 flu kernel: RIP: 0010:[<ffffffff810f8aaa>] 
[<ffffffff810f8aaa>] fget_light+0x6a/0xc0
Aug  4 18:54:46 flu kernel: RSP: 0018:ffff880579015f38  EFLAGS: 00010202
Aug  4 18:54:46 flu kernel: RAX: ffff8806196fa180 RBX: 00007fffdade98f0 RCX:
0000000000000008
Aug  4 18:54:46 flu kernel: RDX: 0000880618ffac80 RSI: ffff880579015f54 RDI:
ffff880618fc77f8
Aug  4 18:54:46 flu kernel: RBP: ffff880579015f38 R08: 0000000000000000 R09:
0000000000000000
Aug  4 18:54:46 flu kernel: R10: 0000000000000008 R11: 0000000000000246 R12:
00000000000000ff
Aug  4 18:54:46 flu kernel: R13: 0000000000005410 R14: 00007fffdade985c R15:
00000000007b6ae0
Aug  4 18:54:46 flu kernel: FS:  00007f495700d700(0000)
GS:ffff88063fc00000(0000) knlGS:0000000000000000
Aug  4 18:54:46 flu kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug  4 18:54:46 flu kernel: CR2: 00000000004620d0 CR3: 000000057927b000 CR4:
00000000000026e0
Aug  4 18:54:46 flu kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Aug  4 18:54:46 flu kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Aug  4 18:54:46 flu kernel: Process bash (pid: 4327, threadinfo
ffff880579014000, task ffff88061aacc750)
Aug  4 18:54:46 flu kernel: Stack:
Aug  4 18:54:46 flu kernel: ffff880579015f78 ffffffff8110786a 0000000000010000
0000000000000000
Aug  4 18:54:46 flu kernel: 00007fffdade98f0 0000000000000082 00007fffdade9990
0000000000000000
Aug  4 18:54:46 flu kernel: 00000000000010e7 ffffffff818fe2fb 0000000000000246
0000000000000008
Aug  4 18:54:46 flu kernel: Call Trace:
Aug  4 18:54:46 flu kernel: [<ffffffff8110786a>] sys_ioctl+0x2a/0x80
Aug  4 18:54:46 flu kernel: [<ffffffff818fe2fb>] system_call_fastpath+0x16/0x1b
Aug  4 18:54:46 flu kernel: Code: 74 37 31 d2 48 89 d0 c9 c3 0f 1f 84 00 00 00
00 00 48 8b 40 08 3b 38 73 e9 89 ff 48 c1 e7 03 48 03 78 08 48 8b 17 48 85 d2
74 d9 <f6> 42 3d 40 75 d1 48 89 d0 c9 c3 0f 1f 00 48 8b 7a 30 4c 8d 42
Aug  4 18:54:46 flu kernel: RIP  [<ffffffff810f8aaa>] fget_light+0x6a/0xc0
Aug  4 18:54:46 flu kernel: RSP <ffff880579015f38>
Aug  4 18:54:46 flu kernel: ---[ end trace 9de898b3d5b9d4c6 ]---
Aug  4 18:54:46 flu kernel: general protection fault: 0000 [#2] SMP
Aug  4 18:54:46 flu kernel: CPU 0
Aug  4 18:54:46 flu kernel: Modules linked in: bridge stp llc kvm_intel kvm
Aug  4 18:54:46 flu kernel:
Aug  4 18:54:46 flu kernel: Pid: 4327, comm: bash Tainted: G      D    
3.0.0-xxxx-std-ipv4-64 #7 Supermicro X8STi/X8STi
Aug  4 18:54:46 flu kernel: RIP: 0010:[<ffffffff810f5567>] 
[<ffffffff810f5567>] filp_close+0x17/0x90
Aug  4 18:54:46 flu kernel: RSP: 0018:ffff880579015cc8  EFLAGS: 00010282
Aug  4 18:54:46 flu kernel: RAX: ffff880618fc77f8 RBX: 0000880618ffac80 RCX:
ffff8806196fa240
Aug  4 18:54:46 flu kernel: RDX: 0000000000000000 RSI: ffff880619648140 RDI:
0000880618ffac80
Aug  4 18:54:46 flu kernel: RBP: ffff880579015ce8 R08: 0000000000000000 R09:
0000000000000000
Aug  4 18:54:46 flu kernel: R10: ffff88061875c090 R11: 0000000000000446 R12:
0000000000000001
Aug  4 18:54:46 flu kernel: R13: ffff8806196fa180 R14: ffff880619648140 R15:
00000000000000ff
Aug  4 18:54:46 flu kernel: FS:  0000000000000000(0000)
GS:ffff88063fc00000(0000) knlGS:0000000000000000
Aug  4 18:54:46 flu kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug  4 18:54:46 flu kernel: CR2: 00000000004620d0 CR3: 0000000001b93000 CR4:
00000000000026e0
Aug  4 18:54:46 flu kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Aug  4 18:54:46 flu kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Aug  4 18:54:46 flu kernel: Process bash (pid: 4327, threadinfo
ffff880579014000, task ffff88061aacc750)
Aug  4 18:54:46 flu kernel: Stack:
Aug  4 18:54:46 flu kernel: ffff880617983840 0000000000000003 0000000000000001
ffff8806196fa180
Aug  4 18:54:46 flu kernel: ffff880579015d38 ffffffff8104feb0 ffff8806179838a0
0000000000000000
Aug  4 18:54:46 flu kernel: ffff880579015d58 ffff88061aacc750 ffff880619648140
ffff880617983840
Aug  4 18:54:46 flu kernel: Call Trace:
Aug  4 18:54:46 flu kernel: [<ffffffff8104feb0>] put_files_struct+0xd0/0xf0
Aug  4 18:54:46 flu kernel: [<ffffffff8104ff15>] exit_files+0x45/0x50
Aug  4 18:54:46 flu kernel: [<ffffffff81051da8>] do_exit+0x718/0x830
Aug  4 18:54:46 flu kernel: [<ffffffff81005bfc>] oops_end+0x9c/0xe0
Aug  4 18:54:46 flu kernel: [<ffffffff81005d36>] die+0x56/0x90
Aug  4 18:54:46 flu kernel: [<ffffffff81003942>]
do_general_protection+0x152/0x160
Aug  4 18:54:46 flu kernel: [<ffffffff818fdeaf>] general_protection+0x1f/0x30
Aug  4 18:54:46 flu kernel: [<ffffffff810f8aaa>] ? fget_light+0x6a/0xc0
Aug  4 18:54:46 flu kernel: [<ffffffff8105c4d1>] ? sys_rt_sigprocmask+0x71/0xc0
Aug  4 18:54:46 flu kernel: [<ffffffff8110786a>] sys_ioctl+0x2a/0x80
Aug  4 18:54:46 flu kernel: [<ffffffff818fe2fb>] system_call_fastpath+0x16/0x1b
Aug  4 18:54:46 flu kernel: Code: 24 4c 8b 64 24 08 c9 c3 66 66 66 2e 0f 1f 84
00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 48
89 fb
Aug  4 18:54:46 flu kernel: RIP  [<ffffffff810f5567>] filp_close+0x17/0x90
Aug  4 18:54:46 flu kernel: RSP <ffff880579015cc8>
Aug  4 18:54:46 flu kernel: ---[ end trace 9de898b3d5b9d4c7 ]---
Aug  4 18:54:46 flu kernel: Fixing recursive fault but reboot is needed!
---

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #1 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-04 17:24:03 ---
moreover, hardware tests were done. Looks like cpu, hdd and ram aren't faulty.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #2 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-04 17:33:52 ---
http port is DNATed by iptables to gentoo guest on tap virtio adapter.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #3 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-04 17:58:18 ---
Created an attachment (id=67532)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=67532)
/etc/conf.d/net (network configuration)

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542


Marcelo Tosatti <mtosatti@redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mtosatti@redhat.com




--- Comment #4 from Marcelo Tosatti <mtosatti@redhat.com>  2011-08-05 17:17:29 ---
(In reply to comment #3)
> Created an attachment (id=67532)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=67532) [details]
> /etc/conf.d/net (network configuration)

Can you please enable SLUB debugging, with the "slub_debug=ZFPU" kernel
parameter.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #5 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-06 13:09:43 ---
I've tested if this problem exists even if no TLB and KSM is compiled in
kernel. Moreover the kernel is now very small (i've removed lots of unused
kernel features). Unfortunately, the problem is still present. slub debug is on
its way:


CONFIG_SLUB_DEBUG=y
CONFIG_SLUB=y
CONFIG_SLUB_DEBUG_ON=y
# CONFIG_SLUB_STATS is not set

but there is no ZFPU option. I'll provide updated .config and kern.log when
it's crashed again.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #6 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-06 16:21:48 ---
Created an attachment (id=67842)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=67842)
kern.log with SLAB DEBUG = ON

Aug 6 18:10 - first occurance of SLAB debugged exception/crash. Please
investigate. I'll provide updated .config of kernel, because ive recompiled it
with less features.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #7 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-06 16:31:15 ---
its probably irrelevant but after last crash, now my ext4 root partition is
damaged and /forcefsck is not fixing it (in spite tune2fs shows me that the
check was made). Something tells me its a netfilter bug. I am going to turn the
virtio net adapter off, find out if this helps and if not switch to VDE just in
order to avoid iptables dnat.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #8 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-06 16:33:22 ---
Created an attachment (id=67852)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=67852)
/proc/config.gz

updated .config from /proc

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

Re: [Bug 40542] overflow/panic on KVM hipervizor

[Brad Campbell]

On 05/08/11 01:33, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=40542
>
>
>
>
>
> --- Comment #2 from Slawek Rozbicki<slawek@rozbicki.eu>   2011-08-04 17:33:52 ---
> http port is DNATed by iptables to gentoo guest on tap virtio adapter.
>

This looks like the bug I've been fighting with on and off.

I'll wager if you can run your tests without using any form of NAT (and 
preferably with netfilter compiled out) you won't see the problem occur.

http://permalink.gmane.org/gmane.linux.kernel.mm/64047

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #9 from Anonymous Emailer <anonymous@kernel-bugs.osdl.org>  2011-08-07 14:55:13 ---
Reply-To: brad@fnarfbargle.com

On 05/08/11 01:33, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=40542
>
>
>
>
>
> --- Comment #2 from Slawek Rozbicki<slawek@rozbicki.eu>   2011-08-04 17:33:52 ---
> http port is DNATed by iptables to gentoo guest on tap virtio adapter.
>

This looks like the bug I've been fighting with on and off.

I'll wager if you can run your tests without using any form of NAT (and 
preferably with netfilter compiled out) you won't see the problem occur.

http://permalink.gmane.org/gmane.linux.kernel.mm/64047

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

Re: [Bug 40542] overflow/panic on KVM hipervizor

[Avi Kivity]

On 08/07/2011 04:39 PM, Brad Campbell wrote:
>
> This looks like the bug I've been fighting with on and off.

What's the bugzilla number for that?

(unfortunately, no great insight except for "CLOSED DUPLICATE")

hopefully someone from netdev can take a look, DNAT is seriously broken.

-- 
error compiling committee.c: too many arguments to function

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #10 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-07 15:10:09 ---
My environment needs apache port mapping from kvm guest to public IP address. I
am able to run apache benchmark from KVM host just to investigate further this
bug, but eventualy i need to have DNAT and masquerade. 

Is there anything I can do to help you guys address this issue? At the end of
the day, it is going to be production environment.

I suppose if everything fails, I'll buy another IP address and make bridged
network.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #11 from Avi Kivity <avi@redhat.com>  2011-08-07 16:01:51 ---
On 08/07/2011 04:39 PM, Brad Campbell wrote:
>
> This looks like the bug I've been fighting with on and off.

What's the bugzilla number for that?

(unfortunately, no great insight except for "CLOSED DUPLICATE")

hopefully someone from netdev can take a look, DNAT is seriously broken.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

Re: [Bug 40542] overflow/panic on KVM hipervizor

[Brad Campbell]

On 07/08/11 23:08, Avi Kivity wrote:
> On 08/07/2011 04:39 PM, Brad Campbell wrote:
>>
>> This looks like the bug I've been fighting with on and off.
>
> What's the bugzilla number for that?
>
> (unfortunately, no great insight except for "CLOSED DUPLICATE")
>
> hopefully someone from netdev can take a look, DNAT is seriously broken.
>
I can reproduce it at will, but it's on a live production machine. I've just ordered a second 
machine which I can use to reproduce and test against. From a bisection standpoint I'm about half 
way between 2.6.35 & 2.6.36, but until the second machine arrives I'm just unable to chase it any 
further.

Brad

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #12 from Anonymous Emailer <anonymous@kernel-bugs.osdl.org>  2011-08-08 01:41:09 ---
Reply-To: brad@fnarfbargle.com

On 07/08/11 23:08, Avi Kivity wrote:
> On 08/07/2011 04:39 PM, Brad Campbell wrote:
>>
>> This looks like the bug I've been fighting with on and off.
>
> What's the bugzilla number for that?
>
> (unfortunately, no great insight except for "CLOSED DUPLICATE")
>
> hopefully someone from netdev can take a look, DNAT is seriously broken.
>
I can reproduce it at will, but it's on a live production machine. I've just
ordered a second 
machine which I can use to reproduce and test against. From a bisection
standpoint I'm about half 
way between 2.6.35 & 2.6.36, but until the second machine arrives I'm just
unable to chase it any 
further.

Brad

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542





--- Comment #13 from Slawek Rozbicki <slawek@rozbicki.eu>  2011-08-19 10:37:39 ---
i've changed my network layout to provide SNAT to the guests via iptables on 2
additional IPs. It seems that this bug doesn't exist anymore. Looks like static
nat is a walkaround. It's stable after an hour of testing. I've updated qemu as
well.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 40542] overflow/panic on KVM hipervizor

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=40542

xerofoify@gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |xerofoify@gmail.com

--- Comment #14 from xerofoify@gmail.com ---
This bug is outdated, please test against a newer kernel.
Cheers Nick

-- 
You are receiving this mail because:
You are watching the assignee of the bug.