Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/01/2011 01:53 PM, Brad Campbell wrote:
> On 01/06/11 17:41, Avi Kivity wrote:
>> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>>
>>> bridge and netfilter, IIRC this was also the problem last time.
>>>
>>> Do you have any ebtables loaded?
>
> Never heard of them, but making a cursory check just in case..
>
> brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
>>> Can you try building a kernel without ebtables? Without netfilter at 
>>> all?
>
> Well, without netfilter I can't get it to crash. The problem is 
> without netfilter I can't actually use it the way I use it to get it 
> to crash.
>
> I rebooted into a netfilter kernel, and did all the steps I'd used on 
> the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that, the 
> backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the 
> public address of the machine, which is then redirected by netfilter 
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not 
> cause it to crash, however without netfilter loaded I can't bounce off 
> the public IP. It's all rather confusing really.
>
> What next Sherlock?
>

Maybe the Sherlocks at netdev@ can tell.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[CaT]

On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
> I rebooted into a netfilter kernel, and did all the steps I'd used
> on the no-netfilter kernel and it ticked along happily.
> 
> So the result of the experiment is inconclusive. Having said that,
> the backtraces certainly smell networky.
> 
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
> 
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce
> off the public IP. It's all rather confusing really.
> 
> What next Sherlock?

I think you're hitting something I've seen. Can you try rewriting
your firewall rules so that it does not reference any bridge
interfaces at all. Instead, reference the real interface names
in their place. I'm betting it wont crash.

(netdev added to CC since we're aleady bouncing there)

-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 01/06/11 19:18, CaT wrote:
> On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
>> I rebooted into a netfilter kernel, and did all the steps I'd used
>> on the no-netfilter kernel and it ticked along happily.
>>
>> So the result of the experiment is inconclusive. Having said that,
>> the backtraces certainly smell networky.
>>
>> To get it to crash, I have to start IE in the VM and https to the
>> public address of the machine, which is then redirected by netfilter
>> back into another of the VM's.
>>
>> I can https directly to the other VM's address, but that does not
>> cause it to crash, however without netfilter loaded I can't bounce
>> off the public IP. It's all rather confusing really.
>>
>> What next Sherlock?
>
> I think you're hitting something I've seen. Can you try rewriting
> your firewall rules so that it does not reference any bridge
> interfaces at all. Instead, reference the real interface names
> in their place. I'm betting it wont crash.
>

Unfortunately the only interface that is mentioned by name anywhere in 
my firewall is $DMZ (which is ppp0 and not part of any bridge).

All of the nat/dnat and other horrible hacks are based on IP addresses.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[CaT]

On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
> Unfortunately the only interface that is mentioned by name anywhere
> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
> 
> All of the nat/dnat and other horrible hacks are based on IP addresses.

Damn. Not referencing the bridge interfaces at all stopped our host from
going down in flames when we passed it a few packets. These are two
of the oopses we got from it. Whilst the kernel here is .35 we got the
same issue from a range of kernels. Seems related.

The oopses may be a bit weird. Copy and paste from an ipmi terminal.


slab error in cache_alloc_debugcheck_after(): cache `size-64': double 
free, or n
Pid: 2431, comm: kvm Tainted: G      D     
2.6.35.9-local.20110314-141930 #1
Call Trace:
<IRQ>  [<ffffffff810fb8bf>] ? __slab_error+0x1f/0x30
  [<ffffffff810fc22b>] ? cache_alloc_debugcheck_after+0x6b/0x1f0
  [<ffffffff81530a00>] ? br_nf_pre_routing_finish+0x0/0x370
  [<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
  [<ffffffff810fdd3d>] ? kmem_cache_alloc_notrace+0x7d/0xf0

  [<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
  [<ffffffff81466e66>] ? nf_iterate+0x66/0xb0
  [<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
  [<ffffffff81466f14>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
  [<ffffffff8152bd3c>] ? br_handle_frame+0x18c/0x250
  [<ffffffff81445459>] ? __netif_receive_skb+0x169/0x2a0
  [<ffffffff81445673>] ? process_backlog+0xe3/0x1d0
  [<ffffffff81446347>] ? net_rx_action+0x87/0x1c0
  [<ffffffff810793f7>] ? __do_softirq+0xa7/0x1d0
  [<ffffffff81035b8c>] ? call_softirq+0x1c/0x30
<EOI>  [<ffffffff81037c6d>] ? do_softirq+0x4d/0x80
  [<ffffffff81446b4e>] ? netif_rx_ni+0x1e/0x30
  [<ffffffff8139541a>] ? tun_chr_aio_write+0x36a/0x510
  [<ffffffff813950b0>] ? tun_chr_aio_write+0x0/0x510
  [<ffffffff81102859>] ? do_sync_readv_writev+0xa9/0xf0
  [<ffffffff810973fb>] ? ktime_get+0x5b/0xe0
  [<ffffffff8104f958>] ? lapic_next_event+0x18/0x20
  [<ffffffff8109be18>] ? tick_dev_program_event+0x38/0x100
  [<ffffffff81102697>] ? rw_copy_check_uvector+0x77/0x130
  [<ffffffff81102f0c>] ? do_readv_writev+0xdc/0x200
  [<ffffffff8108dfec>] ? sys_timer_settime+0x13c/0x2e0
  [<ffffffff8110317e>] ? sys_writev+0x4e/0x90
  [<ffffffff81034d6b>] ? system_call_fastpath+0x16/0x1b
ffff8801e7621500: redzone 1:0xbf05bd0100000006, redzone 2:0x9f911029d74e35b

----------

Code: 40 01 00 00 4c 8b a4 24 48 01 00 00 4c 8b ac 24 50 01 00 00 4c 8b 
b4 24 5
RIP  [<ffffffff81652c67>] icmp_send+0x297/0x650
  RSP <ffff880001c036b8>
---[ end trace 9d3f7be7684ac91e ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G      D     
2.6.35.9-local.20110314-144920 #2
Call Trace:
<IRQ>  [<ffffffff8170eada>] ? panic+0x94/0x116
  [<ffffffff81711326>] ? _raw_spin_lock_irqsave+0x26/0x40
  [<ffffffff8103a05f>] ? oops_end+0xef/0xf0
  [<ffffffff81711a15>] ? general_protection+0x25/0x30
  [<ffffffff81652c2f>] ? icmp_send+0x25f/0x650
  [<ffffffff81652c67>] ? icmp_send+0x297/0x650
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
  [<ffffffff816dc461>] ? br_nf_forward_ip+0x201/0x3e0
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
  [<ffffffff816d66e9>] ? __br_forward+0x69/0xb0
  [<ffffffff816d741a>] ? br_handle_frame_finish+0x12a/0x280
  [<ffffffff816dcac8>] ? br_nf_pre_routing_finish+0x208/0x370
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816dc8c0>] ? br_nf_pre_routing_finish+0x0/0x370
  [<ffffffff816dc538>] ? br_nf_forward_ip+0x2d8/0x3e0
  [<ffffffff816dd3b5>] ? br_nf_pre_routing+0x785/0x980
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
  [<ffffffff816d76fc>] ? br_handle_frame+0x18c/0x250
  [<ffffffff815dec5b>] ? __netif_receive_skb+0x1cb/0x350
  [<ffffffff8103d115>] ? read_tsc+0x5/0x20
  [<ffffffff815dfa18>] ? netif_receive_skb+0x78/0x80
  [<ffffffff815e0217>] ? napi_gro_receive+0x27/0x40
  [<ffffffff815e01d8>] ? napi_skb_finish+0x38/0x50
  [<ffffffff8152586d>] ? bnx2_poll_work+0xd0d/0x13d0
  [<ffffffff8160c950>] ? ctnetlink_conntrack_event+0x210/0x7d0
  [<ffffffff81092029>] ? autoremove_wake_function+0x9/0x30
  [<ffffffff8109a71b>] ? ktime_get+0x5b/0xe0
  [<ffffffff81526051>] ? bnx2_poll+0x61/0x230
  [<ffffffff81051db8>] ? lapic_next_event+0x18/0x20
  [<ffffffff815dfbef>] ? net_rx_action+0x9f/0x200
  [<ffffffff8109636f>] ? __hrtimer_start_range_ns+0x22f/0x410
  [<ffffffff8107c35f>] ? __do_softirq+0xaf/0x1e0
  [<ffffffff810ab547>] ? handle_IRQ_event+0x47/0x160
  [<ffffffff81036d5c>] ? call_softirq+0x1c/0x30
  [<ffffffff81038c85>] ? do_softirq+0x65/0xa0
  [<ffffffff8107c235>] ? irq_exit+0x85/0x90
  [<ffffffff8103820b>] ? do_IRQ+0x6b/0xe0
  [<ffffffff817117d3>] ? ret_from_intr+0x0/0x11
<EOI>  [<ffffffff81269340>] ? intel_idle+0xf0/0x180
  [<ffffffff81269320>] ? intel_idle+0xd0/0x180
  [<ffffffff815b0b0f>] ? cpuidle_idle_call+0x9f/0x140
  [<ffffffff81035032>] ? cpu_idle+0x62/0xb0
  [<ffffffff81a40c77>] ? start_kernel+0x2ef/0x2fa
  [<ffffffff81a403e3>] ? x86_64_start_kernel+0xfb/0x10a



-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 02/06/11 07:03, CaT wrote:
> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>> Unfortunately the only interface that is mentioned by name anywhere
>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>
>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>
> Damn. Not referencing the bridge interfaces at all stopped our host from
> going down in flames when we passed it a few packets. These are two
> of the oopses we got from it. Whilst the kernel here is .35 we got the
> same issue from a range of kernels. Seems related.

Well, I tried sending an explanatory message to netdev, netfilter & cc'd 
to kvm, but it appears not to have made it to kvm or netfilter, and the 
cc to netdev has not elicited a response. My resend to netfilter seems 
to have dropped into the bit bucket also.

Is there anyone who can point me at the appropriate cage to rattle? I 
know it appears to be a netfilter issue, but I don't seem to be able to 
get a message to the list (and I am subscribed to it and have been 
getting mail for months) and I'm not sure who to pester. The other 
alternative is I just stop doing "that" and wait for it to bite someone 
else.

Cheers.
Brad

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Bernhard Held]

Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html

Bernhard

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 03/06/11 23:50, Bernhard Held wrote:
> Am 03.06.2011 15:38, schrieb Brad Campbell:
>> On 02/06/11 07:03, CaT wrote:
>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>
>>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>>
>>> Damn. Not referencing the bridge interfaces at all stopped our host from
>>> going down in flames when we passed it a few packets. These are two
>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>> same issue from a range of kernels. Seems related.
>>
>> Well, I tried sending an explanatory message to netdev, netfilter &
>> cc'd to kvm,
>> but it appears not to have made it to kvm or netfilter, and the cc to
>> netdev has
>> not elicited a response. My resend to netfilter seems to have dropped
>> into the
>> bit bucket also.
> Just another reference 3.5 months ago:
> http://www.spinics.net/lists/netfilter-devel/msg17239.html

<waves hands around shouting "I have a reproducible test case for this 
and don't mind patching and crashing the machine to get it fixed">

Attempted to add netfilter-devel to the cc this time.

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Bart De Schuymer]

Hi Brad,

This has probably nothing to do with ebtables, so please rmmod in case 
it's loaded.
A few questions I didn't directly see an answer to in the threads I 
scanned...
I'm assuming you actually use the bridging firewall functionality. So, 
what iptables modules do you use? Can you reduce your iptables rules to 
a core that triggers the bug?
Or does it get triggered even with an empty set of firewall rules?
Are you using a stock .35 kernel or is it patched?
Is this something I can trigger on a poor guy's laptop or does it 
require specialized hardware (I'm catching up on qemu/kvm...)?

cheers,
Bart

PS: I'm not sure if we should keep CC-ing everybody, netfilter-devel 
together with kvm should probably do fine.

Op 3/06/2011 18:07, Brad Campbell schreef:
> On 03/06/11 23:50, Bernhard Held wrote:
>> Am 03.06.2011 15:38, schrieb Brad Campbell:
>>> On 02/06/11 07:03, CaT wrote:
>>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>>
>>>>> All of the nat/dnat and other horrible hacks are based on IP 
>>>>> addresses.
>>>>
>>>> Damn. Not referencing the bridge interfaces at all stopped our host 
>>>> from
>>>> going down in flames when we passed it a few packets. These are two
>>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>> same issue from a range of kernels. Seems related.
>>>
>>> Well, I tried sending an explanatory message to netdev, netfilter &
>>> cc'd to kvm,
>>> but it appears not to have made it to kvm or netfilter, and the cc to
>>> netdev has
>>> not elicited a response. My resend to netfilter seems to have dropped
>>> into the
>>> bit bucket also.
>> Just another reference 3.5 months ago:
>> http://www.spinics.net/lists/netfilter-devel/msg17239.html
>
> <waves hands around shouting "I have a reproducible test case for this 
> and don't mind patching and crashing the machine to get it fixed">
>
> Attempted to add netfilter-devel to the cc this time.
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>


-- 
Bart De Schuymer
www.artinalgorithms.be

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le lundi 06 juin 2011 à 22:10 +0200, Bart De Schuymer a écrit :
> Hi Brad,
> 
> This has probably nothing to do with ebtables, so please rmmod in case 
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I 
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So, 
> what iptables modules do you use? Can you reduce your iptables rules to 
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it 
> require specialized hardware (I'm catching up on qemu/kvm...)?
> 

Keep netdev, as this most probably is a networking bug.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 04:10, Bart De Schuymer wrote:
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?

Not specialised hardware as such, I've just not been able to reproduce 
it outside of this specific operating scenario.

I can't trigger it with empty firewall rules as it relies on a DNAT to 
occur. If I try it directly to the internal IP address (as I have to 
without netfilter loaded) then of course nothing fails.

It's a pain in the bum as a fault, but it's one I can easily reproduce 
as long as I use the same set of circumstances.

I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it 
on that then I'll attempt to pare down the IPTABLES rules to a bare minimum.

It is nothing to do with ebtables as I don't compile it. I'm not really 
sure about "bridging firewall" functionality. I just use a couple of 
hand coded bash scripts to set the tables up.

brad@srv:~$ lsmod
Module                  Size  Used by
xt_iprange              1637  1
xt_DSCP                 2077  2
xt_length               1216  1
xt_CLASSIFY             1091  26
sch_sfq                 6681  4
xt_CHECKSUM             1229  2 brad@srv:~$ lsmod
Module                  Size  Used by
xt_iprange              1637  1
xt_DSCP                 2077  2
xt_length               1216  1
xt_CLASSIFY             1091  26
sch_sfq                 6681  4
xt_CHECKSUM             1229  2
ipt_REJECT              2277  1
ipt_MASQUERADE          1759  7
ipt_REDIRECT            1133  1
xt_recent               8223  2
xt_state                1226  5
iptable_nat             3993  1
nf_nat                 16773  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11868  8 iptable_nat,nf_nat
nf_conntrack           60962  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1417  1 nf_conntrack_ipv4
xt_TCPMSS               2567  2
xt_tcpmss               1469  0
xt_tcpudp               2467  56
iptable_mangle          1487  1
pppoe                   9574  2
pppox                   2188  1 pppoe
iptable_filter          1442  1
ip_tables              16762  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               20462  17 
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic            24243  6 pppoe,pppox
slhc                    5293  1 ppp_generic
cls_u32                 6468  6
sch_htb                14432  2
deflate                 1937  0
zlib_deflate           21228  1 deflate
des_generic            16135  0
cbc                     2721  0
ecb                     1975  0
crypto_blkcipher       13645  2 cbc,ecb
sha1_generic            2095  0
md5                     4001  0
hmac                    2977  0
crypto_hash            14519  3 sha1_generic,md5,hmac
cryptomgr               2636  0
aead                    6137  1 cryptomgr
crypto_algapi          15289  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27372  0
fuse                   66747  1
w83627ehf              32052  0
hwmon_vid               2867  1 w83627ehf
vhost_net              16802  6
powernow_k8            12932  0
mperf                   1263  1 powernow_k8
kvm_amd                53431  24
kvm                   235155  1 kvm_amd
pl2303                 12732  1
xhci_hcd               62865  0
i2c_piix4               8391  0
k10temp                 3183  0
usbserial              34452  3 pl2303
usb_storage            37887  1
usb_libusual           10999  1 usb_storage
ohci_hcd               18105  0
ehci_hcd               33641  0
ahci                   20748  4
usbcore               130936  7 
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci                21202  1 ahci
sata_mv                26939  0
megaraid_sas           71659  14

Nat Table (external ip substituted for xxx.xxx.xxx.xxx)

Chain PREROUTING (policy ACCEPT 1761K packets, 152M bytes)
  pkts bytes target     prot opt in     out     source 
destination
     5   210 DNAT       udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1195 to:192.168.253.199
     6   252 DNAT       udp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       udp dpt:1195 to:192.168.253.199
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25001 to:192.168.253.199:465
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25000 to:192.168.253.199:993
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:25001 to:192.168.253.199:465
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:25000 to:192.168.253.199:993
     2   142 DNAT       47   --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           to:192.168.253.199
    18   880 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:1723 to:192.168.253.199
     0     0 DNAT       47   --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       to:192.168.253.199
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:1723 to:192.168.253.199
  2969  149K DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:443 to:192.168.253.198
    20  1280 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:443 to:192.168.253.198
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3101 to:192.168.253.197
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:3101 to:192.168.253.197
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:4101 to:192.168.253.197
44528 2718K REDIRECT   tcp  --  !ppp0  *       0.0.0.0/0 
!192.168.0.0/16      tcp dpt:80 redir ports 8080
     0     0 DNAT       tcp  --  *      *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3724 to:192.168.2.107
  596K   33M DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpts:2001:2030 to:10.99.99.2
1420K  119M DNAT       udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpts:2001:2030 to:10.99.99.2
  7483  449K DNAT       all  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       to:192.168.2.1


Mangle Table

Chain INPUT (policy ACCEPT 270K packets, 17M bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain OUTPUT (policy ACCEPT 170K packets, 12M bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain POSTROUTING (policy ACCEPT 2205K packets, 166M bytes)
  pkts bytes target     prot opt in     out     source 
destination
     0     0 MASQUERADE  all  --  *      *       0.0.0.0/0 
192.168.254.3
     6   360 ACCEPT     all  --  *      *       0.0.0.0/0 
xxx.xxx.xxx.xxx
20424 2120K MASQUERADE  all  --  *      ppp0    192.168.0.0/16 
!192.168.0.0/16
     0     0 MASQUERADE  all  --  *      ppp0    10.0.0.0/24 
0.0.0.0/0
     3   204 MASQUERADE  all  --  *      *       192.168.2.0/24 
10.8.0.0/24
1418K  128M MASQUERADE  all  --  *      *       10.99.99.0/24 
0.0.0.0/0
68248 4095K MASQUERADE  all  --  *      *       192.168.253.0/24 
10.8.0.0/16
13305 2405K MASQUERADE  all  --  *      *       192.168.253.0/24 
!192.168.0.0/16

Chain PREROUTING (policy ACCEPT 278M packets, 293G bytes)
  pkts bytes target     prot opt in     out     source 
destination
   169 55528 CHECKSUM   udp  --  br1    *       0.0.0.0/0 
0.0.0.0/0           udp dpt:67 CHECKSUM fill

Chain INPUT (policy ACCEPT 180M packets, 250G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain POSTROUTING (policy ACCEPT 253M packets, 223G bytes)
  pkts bytes target     prot opt in     out     source 
destination
   165 54182 CHECKSUM   udp  --  *      br1     0.0.0.0/0 
0.0.0.0/0           udp spt:67 CHECKSUM fill
    51  3712 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:53 CLASSIFY set 1:20
85274 6454K CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp dpt:53 CLASSIFY set 1:20
   187  257K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:81 CLASSIFY set 1:20
   25M 1180M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x3F/0x10 state ESTABLISHED length 40:100 
CLASSIFY set 1:15
  728K   67M CLASSIFY   icmp --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:15
   231 23484 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:2401 CLASSIFY set 1:15
65636 5610K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 CLASSIFY set 1:10
  2018  315K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:22 CLASSIFY set 1:10
    80 10092 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:3389 CLASSIFY set 1:10
26063 8910K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:8080 CLASSIFY set 1:15
  932K  131M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:80 CLASSIFY set 1:15
  3511  267K CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp dpt:123 CLASSIFY set 1:10
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:20 CLASSIFY set 1:15
     3   180 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:20 CLASSIFY set 1:15
94058   38M CLASSIFY   47   --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:10
1086K  183M CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1194 CLASSIFY set 1:10
1086K  183M TOS        udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1194 TOS set 0x10/0x3f
48817   10M CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1195 CLASSIFY set 1:10
48817   10M TOS        udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1195 TOS set 0x10/0x3f
94058   38M CLASSIFY   47   --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:15
   106  7207 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:1863 CLASSIFY set 1:15
  188K   34M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:443 CLASSIFY set 1:15
51541 3327K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpts:6660:6669 CLASSIFY set 1:15
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spts:2021:2030 CLASSIFY set 1:15
    85  4944 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:19999 CLASSIFY set 1:15
  208K   86M CLASSIFY   udp  --  *      *       0.0.0.0/0 
0.0.0.0/0           source IP range 192.168.2.80-192.168.2.120 CLASSIFY 
set 1:10
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:12345 CLASSIFY set 1:15
     1    80 CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:12345 CLASSIFY set 1:15


Default table

Chain INPUT (policy ACCEPT 176M packets, 247G bytes)
  pkts bytes target     prot opt in     out     source 
destination
     0     0 ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:4569
1187K  582M ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1194
     2   577 ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1195
    28  1224 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3389
   230 12372            tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 state NEW recent: SET name: DEFAULT side: 
source
     3   180 DROP       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 state NEW recent: UPDATE seconds: 300 
hit_count: 4 name: DEFAULT side: source
  1750  143K ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22
     3   144 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:113
   120  6090 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:81
36094   29M ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25
1456K 1706M ACCEPT     all  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           state RELATED,ESTABLISHED
31047 2334K REJECT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp option=!2 reject-with tcp-reset
  552K   60M ACCEPT     all  --  !ppp0  *       0.0.0.0/0 
0.0.0.0/0           state NEW
13552 1207K ACCEPT     icmp --  ppp0   *       0.0.0.0/0 
0.0.0.0/0
  5712  392K DROP       all  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
  pkts bytes target     prot opt in     out     source 
destination
1207K   68M TCPMSS     tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
  pkts bytes target     prot opt in     out     source 
destination
31675 1895K TCPMSS     tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

lsmod

ipt_REJECT              2277  1
ipt_MASQUERADE          1759  7
ipt_REDIRECT            1133  1
xt_recent               8223  2
xt_state                1226  5
iptable_nat             3993  1
nf_nat                 16773  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11868  8 iptable_nat,nf_nat
nf_conntrack           60962  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1417  1 nf_conntrack_ipv4
xt_TCPMSS               2567  2
xt_tcpmss               1469  0
xt_tcpudp               2467  56
iptable_mangle          1487  1
pppoe                   9574  2
pppox                   2188  1 pppoe
iptable_filter          1442  1
ip_tables              16762  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               20462  17 
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic            24243  6 pppoe,pppox
slhc                    5293  1 ppp_generic
cls_u32                 6468  6
sch_htb                14432  2
deflate                 1937  0
zlib_deflate           21228  1 deflate
des_generic            16135  0
cbc                     2721  0
ecb                     1975  0
crypto_blkcipher       13645  2 cbc,ecb
sha1_generic            2095  0
md5                     4001  0
hmac                    2977  0
crypto_hash            14519  3 sha1_generic,md5,hmac
cryptomgr               2636  0
aead                    6137  1 cryptomgr
crypto_algapi          15289  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27372  0
fuse                   66747  1
w83627ehf              32052  0
hwmon_vid               2867  1 w83627ehf
vhost_net              16802  6
powernow_k8            12932  0
mperf                   1263  1 powernow_k8
kvm_amd                53431  24
kvm                   235155  1 kvm_amd
pl2303                 12732  1
xhci_hcd               62865  0
i2c_piix4               8391  0
k10temp                 3183  0
usbserial              34452  3 pl2303
usb_storage            37887  1
usb_libusual           10999  1 usb_storage
ohci_hcd               18105  0
ehci_hcd               33641  0
ahci                   20748  4
usbcore               130936  7 
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci                21202  1 ahci
sata_mv                26939  0
megaraid_sas           71659  14

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Patrick McHardy]

On 07.06.2011 05:33, Brad Campbell wrote:
> On 07/06/11 04:10, Bart De Schuymer wrote:
>> Hi Brad,
>>
>> This has probably nothing to do with ebtables, so please rmmod in case
>> it's loaded.
>> A few questions I didn't directly see an answer to in the threads I
>> scanned...
>> I'm assuming you actually use the bridging firewall functionality. So,
>> what iptables modules do you use? Can you reduce your iptables rules to
>> a core that triggers the bug?
>> Or does it get triggered even with an empty set of firewall rules?
>> Are you using a stock .35 kernel or is it patched?
>> Is this something I can trigger on a poor guy's laptop or does it
>> require specialized hardware (I'm catching up on qemu/kvm...)?
> 
> Not specialised hardware as such, I've just not been able to reproduce
> it outside of this specific operating scenario.

The last similar problem we've had was related to the 32/64 bit compat
code. Are you running 32 bit userspace on a 64 bit kernel?

> I can't trigger it with empty firewall rules as it relies on a DNAT to
> occur. If I try it directly to the internal IP address (as I have to
> without netfilter loaded) then of course nothing fails.
> 
> It's a pain in the bum as a fault, but it's one I can easily reproduce
> as long as I use the same set of circumstances.
> 
> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
> on that then I'll attempt to pare down the IPTABLES rules to a bare
> minimum.
> 
> It is nothing to do with ebtables as I don't compile it. I'm not really
> sure about "bridging firewall" functionality. I just use a couple of
> hand coded bash scripts to set the tables up.

>From one of your previous mails:

> # CONFIG_BRIDGE_NF_EBTABLES is not set

How about CONFIG_BRIDGE_NETFILTER?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 21:30, Patrick McHardy wrote:
> On 07.06.2011 05:33, Brad Campbell wrote:
>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>> Hi Brad,
>>>
>>> This has probably nothing to do with ebtables, so please rmmod in case
>>> it's loaded.
>>> A few questions I didn't directly see an answer to in the threads I
>>> scanned...
>>> I'm assuming you actually use the bridging firewall functionality. So,
>>> what iptables modules do you use? Can you reduce your iptables rules to
>>> a core that triggers the bug?
>>> Or does it get triggered even with an empty set of firewall rules?
>>> Are you using a stock .35 kernel or is it patched?
>>> Is this something I can trigger on a poor guy's laptop or does it
>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>
>> Not specialised hardware as such, I've just not been able to reproduce
>> it outside of this specific operating scenario.
>
> The last similar problem we've had was related to the 32/64 bit compat
> code. Are you running 32 bit userspace on a 64 bit kernel?

No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.

Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is 
current git


>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>> occur. If I try it directly to the internal IP address (as I have to
>> without netfilter loaded) then of course nothing fails.
>>
>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>> as long as I use the same set of circumstances.
>>
>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>> minimum.
>>
>> It is nothing to do with ebtables as I don't compile it. I'm not really
>> sure about "bridging firewall" functionality. I just use a couple of
>> hand coded bash scripts to set the tables up.
>
>  From one of your previous mails:
>
>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
> How about CONFIG_BRIDGE_NETFILTER?
>

It was compiled in.

With the following table set I was able to reproduce the problem on 
3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

root@srv:~# iptables-save
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*filter
:INPUT ACCEPT [978:107619]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1659:291870]
-A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
-A INPUT -i ppp0 -j DROP
COMMIT
# Completed on Tue Jun  7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*nat
:PREROUTING ACCEPT [813:49170]
:INPUT ACCEPT [91:7090]
:OUTPUT ACCEPT [267:20731]
:POSTROUTING ACCEPT [296:22281]
-A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443 
-j DNAT --to-destination 192.168.253.198
COMMIT
# Completed on Tue Jun  7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*mangle
:PREROUTING ACCEPT [2729:274392]
:INPUT ACCEPT [2508:262976]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1674:293701]
:POSTROUTING ACCEPT [2131:346411]
-A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 
1400:1536 -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Tue Jun  7 22:11:30 2011

I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access 
the address the way I was doing it, so that's a no-go for me.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Patrick McHardy]

On 07.06.2011 16:40, Brad Campbell wrote:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
> 
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
> 
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
> 
> 
>>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>>> occur. If I try it directly to the internal IP address (as I have to
>>> without netfilter loaded) then of course nothing fails.
>>>
>>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>>> as long as I use the same set of circumstances.
>>>
>>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>>> minimum.
>>>
>>> It is nothing to do with ebtables as I don't compile it. I'm not really
>>> sure about "bridging firewall" functionality. I just use a couple of
>>> hand coded bash scripts to set the tables up.
>>
>>  From one of your previous mails:
>>
>>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>>
>> How about CONFIG_BRIDGE_NETFILTER?
>>
> 
> It was compiled in.
> 
> With the following table set I was able to reproduce the problem on
> 3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

Which kernel was the last version without this problem?

> root@srv:~# iptables-save
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *filter
> :INPUT ACCEPT [978:107619]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1659:291870]
> -A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
> -A INPUT -i ppp0 -j DROP
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *nat
> :PREROUTING ACCEPT [813:49170]
> :INPUT ACCEPT [91:7090]
> :OUTPUT ACCEPT [267:20731]
> :POSTROUTING ACCEPT [296:22281]
> -A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
> -j DNAT --to-destination 192.168.253.198
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *mangle
> :PREROUTING ACCEPT [2729:274392]
> :INPUT ACCEPT [2508:262976]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1674:293701]
> :POSTROUTING ACCEPT [2131:346411]
> -A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
> 1400:1536 -j TCPMSS --clamp-mss-to-pmtu
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011

The main suspects would be NAT and TCPMSS. Did you also try whether
the crash occurs with only one of these these rules?

> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> the address the way I was doing it, so that's a no-go for me.

That's really weird since you're apparently not using any bridge
netfilter features. It shouldn't have any effect besides changing
at which point ip_tables is invoked. How are your network devices
configured (specifically any bridges)?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
> 
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
> 
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?

Something in the kernel does 

u16 *ptr = addr (given by kmalloc())

ptr[-1] = 0;

Could be an off-one error in a memmove()/memcopy() or loop...

I cant see a network issue here.

I checked arch/x86/lib/memmove_64.S and it seems fine.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Patrick McHardy]

On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> 
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
> 
> Something in the kernel does 
> 
> u16 *ptr = addr (given by kmalloc())
> 
> ptr[-1] = 0;
> 
> Could be an off-one error in a memmove()/memcopy() or loop...
> 
> I cant see a network issue here.

So far me neither, but netfilter appears to trigger the bug.

> I checked arch/x86/lib/memmove_64.S and it seems fine.

I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.

Would it help if I tried some older kernels? This issue only surfaced 
for me recently as I only installed the VM's in question about 12 weeks 
ago and have only just started really using them in anger. I could try 
reproducing it on progressively older kernels to see if I can find one 
that works and then bisect from there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le mercredi 08 juin 2011 à 08:18 +0800, Brad Campbell a écrit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 à 17:35 +0200, Patrick McHardy a écrit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
> 
> Would it help if I tried some older kernels? This issue only surfaced 
> for me recently as I only installed the VM's in question about 12 weeks 
> ago and have only just started really using them in anger. I could try 
> reproducing it on progressively older kernels to see if I can find one 
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
 	}
 
+#if 0
 	if (fastpath &&
 	    size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
 		memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		off = nhead;
 		goto adjust_others;
 	}
-
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (!data)
 		goto nodata;


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 08/06/11 11:59, Eric Dumazet wrote:

> Well, a bisection definitely should help, but needs a lot of time in
> your case.

Yes. compile, test, crash, walk out to the other building to press 
reset, lather, rinse, repeat.

I need a reset button on the end of a 50M wire, or a hardware watchdog!

Actually it's not so bad. If I turn off slub debugging the kernel panics 
and reboots itself.

This.. :
[    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[    2.913066] netconsole: device eth0 not up yet, forcing it
[    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[    3.660118] Switching to clocksource tsc
[   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
rtl_nic/rtl8168e-1.fw (-2)
[   63.223513] r8169 0000:03:00.0: eth0: link down
[   63.223556] r8169 0000:03:00.0: eth0: link down

..is slowing down reboots considerably. 3.0-rc does _not_ like some 
timing hardware in my machine. Having said that, at least it does not 
randomly panic on SCSI like 2.6.39 does.

Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
it. I've replicated it with only the single DNAT rule.


> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
>   	}
>
> +#if 0
>   	if (fastpath&&
>   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
>   		memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		off = nhead;
>   		goto adjust_others;
>   	}
> -
> +#endif
>   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
>   	if (!data)
>   		goto nodata;
>
>
>

Nope.. that's not it. <sigh> That might have changed the characteristic 
of the fault slightly, but unfortunately I got caught with a couple of 
fsck's, so I only got to test it 3 times tonight.

It's unfortunate that this is a production system, so I can only take it 
down between about 9pm and 1am. That would normally be pretty 
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
panics at the wrong time.

I'm out of time tonight, but I'll have a crack at some bisection 
tomorrow night. Now I just have to go back far enough that it works, and 
be near enough not to have to futz around with /proc /sys or drivers.

I really, really, really appreciate you guys helping me with this. It 
has been driving me absolutely bonkers. If I'm ever in the same town as 
any of you, dinner and drinks are on me.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le jeudi 09 juin 2011 à 01:02 +0800, Brad Campbell a écrit :
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
> 
> Yes. compile, test, crash, walk out to the other building to press 
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
> 
> Actually it's not so bad. If I turn off slub debugging the kernel panics 
> and reboots itself.
> 
> This.. :
> [    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [    2.913066] netconsole: device eth0 not up yet, forcing it
> [    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [    3.660118] Switching to clocksource tsc
> [   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
> rtl_nic/rtl8168e-1.fw (-2)
> [   63.223513] r8169 0000:03:00.0: eth0: link down
> [   63.223556] r8169 0000:03:00.0: eth0: link down
> 
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some 
> timing hardware in my machine. Having said that, at least it does not 
> randomly panic on SCSI like 2.6.39 does.
> 
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
> it. I've replicated it with only the single DNAT rule.
> 
> 
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> >   	}
> >
> > +#if 0
> >   	if (fastpath&&
> >   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> >   		memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		off = nhead;
> >   		goto adjust_others;
> >   	}
> > -
> > +#endif
> >   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> >   	if (!data)
> >   		goto nodata;
> >
> >
> >
> 
> Nope.. that's not it. <sigh> That might have changed the characteristic 
> of the fault slightly, but unfortunately I got caught with a couple of 
> fsck's, so I only got to test it 3 times tonight.
> 
> It's unfortunate that this is a production system, so I can only take it 
> down between about 9pm and 1am. That would normally be pretty 
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
> panics at the wrong time.
> 
> I'm out of time tonight, but I'll have a crack at some bisection 
> tomorrow night. Now I just have to go back far enough that it works, and 
> be near enough not to have to futz around with /proc /sys or drivers.
> 
> I really, really, really appreciate you guys helping me with this. It 
> has been driving me absolutely bonkers. If I'm ever in the same town as 
> any of you, dinner and drinks are on me.

Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Simon Horman]

On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> >Well, a bisection definitely should help, but needs a lot of time in
> >your case.
> 
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!

Not strictly on-topic, but in situations where I have machines
that either don't have lights-out facilities or have broken ones
I find that network controlled power switches to be very useful.

At one point I would have need an 8000km long wire to the reset switch :-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Mark Lord]

On 11-06-09 10:52 PM, Simon Horman wrote:
> On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
>> On 08/06/11 11:59, Eric Dumazet wrote:
>>
>>> Well, a bisection definitely should help, but needs a lot of time in
>>> your case.
>>
>> Yes. compile, test, crash, walk out to the other building to press
>> reset, lather, rinse, repeat.
>>
>> I need a reset button on the end of a 50M wire, or a hardware watchdog!


Something many of us don't realize is that nearly all Intel chipsets
have a built-in hardware watchdog timer.  This includes chipset for
consumer desktop boards as well as the big iron server stuff.

It's the "i8xx_tco" driver in the kernel enables use of them:

    modprobe i8xx_tco

Cheers

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Henrique de Moraes Holschuh]

On Fri, 10 Jun 2011, Mark Lord wrote:
> Something many of us don't realize is that nearly all Intel chipsets
> have a built-in hardware watchdog timer.  This includes chipset for
> consumer desktop boards as well as the big iron server stuff.
> 
> It's the "i8xx_tco" driver in the kernel enables use of them:

That's the old module name, but yes, it is very useful in desktops and
laptops (when it works).   Server-class hardware will have a baseboard
management unit that can really power-cycle the system instead of just
rebooting.

And test it first before you depend on it triggering at a remote location,
as the firmware might cause the Intel chipset watchdog to actually hang the
box instead of causing a proper reboot (happens on the IBM thinkpad T43, for
example).

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/10/2011 05:52 AM, Simon Horman wrote:
> At one point I would have need an 8000km long wire to the reset switch :-)

Even more off-topic, there has been a case when a 200,000,000 km long 
wire to the reset button was needed.  IIRC they got away with a watchdog.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 23:35, Patrick McHardy wrote:

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?

To be honest I'm actually having trouble finding where TCPMSS is 
actually set in that ruleset. This is a production machine so I can only 
take it down after about 9PM at night. I'll have another crack at it 
tonight.

>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>> the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
>

I have one bridge with all my virtual machines on it.

In this particular instance the packets leave VM A destined for the IP 
address of ppp0 (the external interface). This is intercepted by the 
DNAT PREROUTING rule above and shunted back to VM B.

The VM's are on br1 and the external address is ppp0. Without 
CONFIG_BRIDGE_NETFILTER compiled in I can see the traffic entering and 
leaving VM B with tcpdump, but the packets never seem to get back to VM A.

VM A is XP 32 bit, VM B is Linux. I have some other Linux VM's, so I'll 
do some more testing tonight between those to see where the packets are 
going without CONFIG_BRIDGE_NETFILTER set.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Bart De Schuymer]

Op 7/06/2011 16:40, Brad Campbell schreef:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables 
>>>> rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit 
> kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is 
> current git
>
If the bug is easily triggered with your guest os, then you could try to 
capture the traffic with wireshark (or something else) in a 
configuration that doesn't crash your system. Save the traffic in a pcap 
file. Then you can see if resending that traffic in the vulnerable 
configuration triggers the bug (I don't know if something in Windows 
exists, but tcpreplay should work for Linux). Once you have such a 
capture , chances are the bug is even easily reproducible by us (unless 
it's hardware-specific). Success isn't guaranteed, but I think it's 
worth a shot...

cheers,
Bart


-- 
Bart De Schuymer
www.artinalgorithms.be

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 08/06/11 02:04, Bart De Schuymer wrote:

> If the bug is easily triggered with your guest os, then you could try to
> capture the traffic with wireshark (or something else) in a
> configuration that doesn't crash your system. Save the traffic in a pcap
> file. Then you can see if resending that traffic in the vulnerable
> configuration triggers the bug (I don't know if something in Windows
> exists, but tcpreplay should work for Linux). Once you have such a
> capture , chances are the bug is even easily reproducible by us (unless
> it's hardware-specific). Success isn't guaranteed, but I think it's
> worth a shot...

The issue with this is I don't have a configuration that does not crash 
the system. This only happens under the specific circumstance that 
traffic from VM A is being DNAT'd to VM B. If I disable 
CONFIG_BRIDGE_NETFILTER, or I leave out the DNAT then I can't replicate 
the problem as I don't seem to be able to get the packets to go where I 
want them to go.

Let me try and explain it a little more clearly with made up IP 
addresses to illustrate the problem.

I have VM A (1.1.1.2) and VM B (1.1.1.3) on br1 (1.1.1.1)
I have public IP on ppp0 (2.2.2.2).

VM B can talk to VM A using its host address (1.1.1.2) and there is no 
problem.

The DNAT says anything destined for PPP0 that is on port 443 and coming 
from anywhere other than PPP0 (ie inside the network) is to be DNAT'd to 
1.1.1.3.

So VM B (1.1.1.3) tries to connect to ppp0 (2.2.2.2) on port 443, and 
this is redirected to VM B on 1.1.1.2.

Only under this specific circumstance does the problem occur. I can get 
VM B (1.1.1.3) to talk directly to VM A (1.1.1.2) all day long and there 
is no problem, it's only when VM B tries to talk to ppp0 that there is 
an issue (and it happens within seconds of the initial connection).

All these tests have been performed with VM B being a Windows XP guest. 
Tonight I'll try it with a Linux guest and see if I can make it happen. 
If that works I might be able to come up with some reproducible test 
case for you. I have a desktop machine that has Intel VT extensions, so 
I'll work toward making a portable test case.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/03/2011 04:38 PM, Brad Campbell wrote:
>
> Is there anyone who can point me at the appropriate cage to rattle? I 
> know it appears to be a netfilter issue, but I don't seem to be able 
> to get a message to the list (and I am subscribed to it and have been 
> getting mail for months) and I'm not sure who to pester. The other 
> alternative is I just stop doing "that" and wait for it to bite 
> someone else.

The mailing list might be set not to send your own mails back to you.  
Check the list archive.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 05/06/11 16:14, Avi Kivity wrote:
> On 06/03/2011 04:38 PM, Brad Campbell wrote:
>>
>> Is there anyone who can point me at the appropriate cage to rattle? I
>> know it appears to be a netfilter issue, but I don't seem to be able
>> to get a message to the list (and I am subscribed to it and have been
>> getting mail for months) and I'm not sure who to pester. The other
>> alternative is I just stop doing "that" and wait for it to bite
>> someone else.
>
> The mailing list might be set not to send your own mails back to you.
> Check the list archive.

Yep, I did that first..

Given the response to previous issues along the same line, it looks a 
bit like I just remember not to actually use the system in the way that 
triggers the bug and be happy that 99% of the time the kernel does not 
panic, but have that lovely feeling in the back of the skull that says 
"any time now, and without obvious reason the whole machine might just 
come crashing down"..

I guess it's still better than running Xen or Windows..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/05/2011 04:45 PM, Brad Campbell wrote:
>> The mailing list might be set not to send your own mails back to you.
>> Check the list archive.
>
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way 
> that triggers the bug and be happy that 99% of the time the kernel 
> does not panic, but have that lovely feeling in the back of the skull 
> that says "any time now, and without obvious reason the whole machine 
> might just come crashing down"..
>
> I guess it's still better than running Xen or Windows..

Not at all.  Can some networking/netfilter expert look at this?

Please file a bug with all the relevant information in this thread.  If 
you can look for a previous version that worked, that might increase the 
chances of the bug being resolved faster.

-- 
error compiling committee.c: too many arguments to function

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le dimanche 05 juin 2011 à 21:45 +0800, Brad Campbell a écrit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
> 
> Yep, I did that first..
> 
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way that 
> triggers the bug and be happy that 99% of the time the kernel does not 
> panic, but have that lovely feeling in the back of the skull that says 
> "any time now, and without obvious reason the whole machine might just 
> come crashing down"..
> 
> I guess it's still better than running Xen or Windows..

Could you please try latest linux-2.6 tree ?

We fixed many networking bugs that could explain your crash.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 04:22, Eric Dumazet wrote:

> Could you please try latest linux-2.6 tree ?
>
> We fixed many networking bugs that could explain your crash.
>
>
>
>

No good I'm afraid.

[  543.040056] 
=============================================================================
[  543.040136] BUG ip_dst_cache: Padding overwritten. 
0xffff8803e4217ffe-0xffff8803e4217fff
[  543.040194] 
-----------------------------------------------------------------------------
[  543.040198]
[  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x 
        (null) flags=0x8000000000004081
[  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
[  543.040415] Call Trace:
[  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
[  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
[  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
[  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
[  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
[  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
[  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
[  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
[  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
[  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
[  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
[  543.050021] FIX ip_dst_cache: Restoring 
0xffff8803e4217f40-0xffff8803e4217fff=0x5a
[  543.050021]

Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or 
ksm related.

I'll pare down the firewall and see if I can make it break easier with a 
smaller test set.

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
> On 07/06/11 04:22, Eric Dumazet wrote:
> 
> > Could you please try latest linux-2.6 tree ?
> >
> > We fixed many networking bugs that could explain your crash.
> >
> >
> >
> >
> 
> No good I'm afraid.
> 
> [  543.040056] 
> =============================================================================
> [  543.040136] BUG ip_dst_cache: Padding overwritten. 
> 0xffff8803e4217ffe-0xffff8803e4217fff
> [  543.040194] 

Thats pretty strange : These are the last two bytes of a page, set to
0x0000 (a 16 bit value)

There is no way a dst field could actually sit on this location (its a
padding), since a dst is a bit less than 256 bytes (0xe8), and each
entry is aligned on a 64byte address.

grep dst /proc/slabinfo 

ip_dst_cache       32823  62944    256   32    2 : tunables    0    0
0 : slabdata   1967   1967      0

sizeof(struct rtable)=0xe8


> -----------------------------------------------------------------------------
> [  543.040198]
> [  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x 
>         (null) flags=0x8000000000004081
> [  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
> [  543.040415] Call Trace:
> [  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
> [  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
> [  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
> [  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
> [  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
> [  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
> [  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
> [  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
> [  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
> [  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
> [  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
> [  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
> [  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
> [  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
> [  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
> [  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
> [  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
> [  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
> [  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
> [  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
> [  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
> [  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
> [  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
> [  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
> [  543.050021] FIX ip_dst_cache: Restoring 
> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
> [  543.050021]
> 
> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or 
> ksm related.
> 
> I'll pare down the firewall and see if I can make it break easier with a 
> smaller test set.

Hmm, not sure now :(

Could you reproduce another bug please ?

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [  543.040056]
>> =============================================================================
>> [  543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [  543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache       32823  62944    256   32    2 : tunables    0    0
> 0 : slabdata   1967   1967      0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [  543.040198]
>> [  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>>          (null) flags=0x8000000000004081
>> [  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [  543.040415] Call Trace:
>> [  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [  543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [  543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?

Here is the remainder of the log from that bug that I managed to scrape 
off the disk. Odd as usually the netconsole dies and so does the disk 
subsystem. In this case it managed to get the log to disk.

[  543.040056] 
=============================================================================
[  543.040136] BUG ip_dst_cache: Padding overwritten. 
0xffff8803e4217ffe-0xffff8803e4217fff
[  543.040194] 
-----------------------------------------------------------------------------
[  543.040198]
[  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x 
        (null) flags=0x8000000000004081
[  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
[  543.040415] Call Trace:
[  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
[  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
[  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
[  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
[  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
[  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
[  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
[  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
[  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
[  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
[  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
[  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
[  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a 
5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
[  543.050021] FIX ip_dst_cache: Restoring 
0xffff8803e4217f40-0xffff8803e4217fff=0x5a
[  543.050021]
[  885.504318] br1: port 4(vnet2) entering forwarding state
[  885.504797] device vnet2 left promiscuous mode
[  885.504868] br1: port 4(vnet2) entering disabled state
[  885.847186] br1: port 3(vnet1) entering forwarding state
[  885.847667] device vnet1 left promiscuous mode
[  885.847740] br1: port 3(vnet1) entering disabled state
[  885.940042] 
=============================================================================
[  885.940122] BUG kmalloc-512: Object padding overwritten
[  885.940171] 
-----------------------------------------------------------------------------
[  885.940175]
[  885.940273] INFO: 0xffff8803dc09322e-0xffff8803dc09322f. First byte 
0x0 instead of 0x5a
[  885.940344] INFO: Allocated in qdisc_alloc+0x59/0x110 age=66496 cpu=2 
pid=4948
[  885.940404] INFO: Freed in qdisc_alloc+0x49/0x110 age=66496 cpu=2 
pid=4948
[  885.940457] INFO: Slab 0xffffea000d821f80 objects=28 used=21 
fp=0xffff8803dc090b68 flags=0x8000000000004081
[  885.940519] INFO: Object 0xffff8803dc092fe8 @offset=12264 
fp=0xffff8803dc092238
[  885.940523]
[  885.940618] Bytes b4 0xffff8803dc092fd8:  d3 e0 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a Óàÿÿ....ZZZZZZZZ
[  885.941373]   Object 0xffff8803dc092fe8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.942120]   Object 0xffff8803dc092ff8:  00 00 00 00 00 00 00 00 70 
0a 34 81 ff ff ff ff ........p.4.ÿÿÿÿ
[  885.942847]   Object 0xffff8803dc093008:  b0 08 34 81 ff ff ff ff 04 
00 00 00 18 00 00 00 °.4.ÿÿÿÿ........
[  885.943585]   Object 0xffff8803dc093018:  60 ac 5d 81 ff ff ff ff 00 
00 00 00 00 00 00 00 `¬].ÿÿÿÿ........
[  885.944324]   Object 0xffff8803dc093028:  28 30 09 dc 03 88 ff ff 28 
30 09 dc 03 88 ff ff (0.Ü..ÿÿ(0.Ü..ÿÿ
[  885.945071]   Object 0xffff8803dc093038:  00 00 00 00 ff ff ff ff 00 
00 00 00 00 00 00 00 ....ÿÿÿÿ........
[  885.945794]   Object 0xffff8803dc093048:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.946530]   Object 0xffff8803dc093058:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.947281]   Object 0xffff8803dc093068:  98 f3 f9 0e 04 88 ff ff 00 
00 00 00 00 00 00 00 .óù...ÿÿ........
[  885.948011]   Object 0xffff8803dc093078:  00 00 00 00 00 00 00 00 02 
00 00 00 00 00 00 00 ................
[  885.948735]   Object 0xffff8803dc093088:  88 30 09 dc 03 88 ff ff 88 
30 09 dc 03 88 ff ff .0.Ü..ÿÿ.0.Ü..ÿÿ
[  885.949483]   Object 0xffff8803dc093098:  00 00 00 00 bf bf 00 00 ca 
29 01 00 00 00 00 00 ....¿¿..Ê)......
[  885.950007]   Object 0xffff8803dc0930a8:  de 01 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 Þ...............
[  885.950007]   Object 0xffff8803dc0930b8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0930c8:  10 cd 13 e7 03 88 ff ff a0 
13 34 81 ff ff ff ff .Í.ç..ÿÿ..4.ÿÿÿÿ
[  885.950007]   Object 0xffff8803dc0930d8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0930e8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0930f8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093108:  08 31 09 dc 03 88 ff ff 08 
31 09 dc 03 88 ff ff .1.Ü..ÿÿ.1.Ü..ÿÿ
[  885.950007]   Object 0xffff8803dc093118:  00 00 00 00 00 00 00 00 20 
31 09 dc 03 88 ff ff .........1.Ü..ÿÿ
[  885.950007]   Object 0xffff8803dc093128:  20 31 09 dc 03 88 ff ff 00 
00 00 00 00 00 00 00 .1.Ü..ÿÿ........
[  885.950007]   Object 0xffff8803dc093138:  38 31 09 dc 03 88 ff ff 38 
31 09 dc 03 88 ff ff 81.Ü..ÿÿ81.Ü..ÿÿ
[  885.950007]   Object 0xffff8803dc093148:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093158:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093168:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093178:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093188:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc093198:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0931a8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0931b8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0931c8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]   Object 0xffff8803dc0931d8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  885.950007]  Redzone 0xffff8803dc0931e8:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[  885.950007]  Padding 0xffff8803dc093228:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  885.950007] Pid: 11, comm: rcuc1 Not tainted 3.0.0-rc2 #1
[  885.950007] Call Trace:
[  885.950007]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[  885.950007]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[  885.950007]  [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[  885.950007]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[  885.950007]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[  885.950007]  [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[  885.950007]  [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[  885.950007]  [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[  885.950007]  [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[  885.950007]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[  885.950007]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[  885.950007]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[  885.950007]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[  885.950007]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[  885.950007]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[  885.950007] FIX kmalloc-512: Restoring 
0xffff8803dc09322e-0xffff8803dc09322f=0x5a
[  885.950007]
[  886.050119] 
=============================================================================
[  886.050198] BUG kmalloc-512: Object padding overwritten
[  886.050247] 
-----------------------------------------------------------------------------
[  886.050251]
[  886.050353] INFO: 0xffff8803dc091916-0xffff8803dc091917. First byte 
0x0 instead of 0x5a
[  886.050451] INFO: Allocated in kvm_ioapic_init+0x3c/0xd0 [kvm] 
age=66256 cpu=0 pid=5122
[  886.050517] INFO: Freed in __kfree_skb+0x11/0x90 age=66311 cpu=4 pid=5054
[  886.050568] INFO: Slab 0xffffea000d821f80 objects=28 used=17 
fp=0xffff8803dc091488 flags=0x8000000000004081
[  886.050630] INFO: Object 0xffff8803dc0916d0 @offset=5840 fp=0x 
    (null)
[  886.050634]
[  886.050726] Bytes b4 0xffff8803dc0916c0:  ea e4 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a êä......ZZZZZZZZ
[  886.051451]   Object 0xffff8803dc0916d0:  00 00 c0 fe 00 00 00 00 3f 
00 00 00 00 00 00 00 ..Àþ....?.......
[  886.052183]   Object 0xffff8803dc0916e0:  00 08 00 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.052898]   Object 0xffff8803dc0916f0:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.053613]   Object 0xffff8803dc091700:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.054325]   Object 0xffff8803dc091710:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.055032]   Object 0xffff8803dc091720:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.055743]   Object 0xffff8803dc091730:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.056462]   Object 0xffff8803dc091740:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.057177]   Object 0xffff8803dc091750:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.058210]   Object 0xffff8803dc091760:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.058957]   Object 0xffff8803dc091770:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.059667]   Object 0xffff8803dc091780:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091790:  00 00 01 00 00 00 00 00 00 
00 01 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917a0:  00 00 01 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917b0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917c0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917e0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0917f0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091800:  01 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091810:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091820:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091830:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091840:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091850:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc091860:  00 00 00 00 00 00 00 00 b0 
80 10 a0 ff ff ff ff ........°...ÿÿÿÿ
[  886.060104]   Object 0xffff8803dc091870:  00 00 df f5 03 88 ff ff 00 
00 00 00 00 00 00 00 ..ßõ..ÿÿ........
[  886.060104]   Object 0xffff8803dc091880:  43 43 00 00 00 00 00 00 01 
00 00 00 00 00 00 00 CC..............
[  886.060104]   Object 0xffff8803dc091890:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0918a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0918b0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]   Object 0xffff8803dc0918c0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  886.060104]  Redzone 0xffff8803dc0918d0:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[  886.060104]  Padding 0xffff8803dc091910:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  886.060104] Pid: 7359, comm: qemu Not tainted 3.0.0-rc2 #1
[  886.060104] Call Trace:
[  886.060104]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[  886.060104]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[  886.060104]  [<ffffffffa00eb8bf>] ? kvm_arch_destroy_vm+0x2f/0x150 [kvm]
[  886.060104]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[  886.060104]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[  886.060104]  [<ffffffffa00eb8bf>] ? kvm_arch_destroy_vm+0x2f/0x150 [kvm]
[  886.060104]  [<ffffffffa00e2dd9>] ? kvm_put_kvm+0xe9/0x130 [kvm]
[  886.060104]  [<ffffffffa00e3763>] ? kvm_vcpu_release+0x13/0x20 [kvm]
[  886.060104]  [<ffffffff810c65fd>] ? fput+0xdd/0x270
[  886.060104]  [<ffffffff810c2e1c>] ? filp_close+0x5c/0x90
[  886.060104]  [<ffffffff8103ae1d>] ? put_files_struct+0xcd/0x100
[  886.060104]  [<ffffffff8103ccc7>] ? do_exit+0x6d7/0x7e0
[  886.060104]  [<ffffffff8103ce20>] ? do_group_exit+0x50/0xc0
[  886.060104]  [<ffffffff81048914>] ? get_signal_to_deliver+0x1a4/0x3a0
[  886.060104]  [<ffffffff8100188a>] ? do_signal+0x8a/0x7d0
[  886.060104]  [<ffffffff81049148>] ? kill_pid_info+0x38/0x60
[  886.060104]  [<ffffffff810493e4>] ? sys_kill+0x94/0x1b0
[  886.060104]  [<ffffffff8100201d>] ? do_notify_resume+0x4d/0x70
[  886.060104]  [<ffffffff810c59c4>] ? sys_pread64+0x74/0xa0
[  886.060104]  [<ffffffff813e11db>] ? int_signal+0x12/0x17
[  886.060104] FIX kmalloc-512: Restoring 
0xffff8803dc091916-0xffff8803dc091917=0x5a
[  886.060104]
[  926.148014] br1: port 6(vnet4) entering forwarding state
[  926.148385] device vnet4 left promiscuous mode
[  926.148457] br1: port 6(vnet4) entering disabled state
[  945.659523] br1: port 2(vnet0) entering forwarding state
[  945.660041] device vnet0 left promiscuous mode
[  945.660116] br1: port 2(vnet0) entering disabled state
[ 1121.085267] br1: port 5(vnet3) entering forwarding state
[ 1121.085783] device vnet3 left promiscuous mode
[ 1121.085860] br1: port 5(vnet3) entering disabled state
[ 1176.307099] nfsd: last server has exited, flushing export cache
[ 1176.430113] 
=============================================================================
[ 1176.430194] BUG kmalloc-512: Object padding overwritten
[ 1176.430199] 
-----------------------------------------------------------------------------
[ 1176.430204]
[ 1176.430210] INFO: 0xffff8803f5df5b5e-0xffff8803f5df5b5f. First byte 
0x0 instead of 0x5a
[ 1176.430225] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=93370 cpu=0 pid=5126
[ 1176.430234] INFO: Freed in __kfree_skb+0x11/0x90 age=93370 cpu=0 pid=4495
[ 1176.430240] INFO: Slab 0xffffea000ddc8d60 objects=28 used=0 
fp=0xffff8803f5df76c0 flags=0x8000000000004080
[ 1176.430246] INFO: Object 0xffff8803f5df5918 @offset=6424 
fp=0xffff8803f5df7908
[ 1176.430249]
[ 1176.430253] Bytes b4 0xffff8803f5df5908:  ab f0 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a «ðÿÿ....ZZZZZZZZ
[ 1176.430270]   Object 0xffff8803f5df5918:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430285]   Object 0xffff8803f5df5928:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430300]   Object 0xffff8803f5df5938:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430314]   Object 0xffff8803f5df5948:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430328]   Object 0xffff8803f5df5958:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430343]   Object 0xffff8803f5df5968:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430357]   Object 0xffff8803f5df5978:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430371]   Object 0xffff8803f5df5988:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430386]   Object 0xffff8803f5df5998:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430400]   Object 0xffff8803f5df59a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430414]   Object 0xffff8803f5df59b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430429]   Object 0xffff8803f5df59c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430443]   Object 0xffff8803f5df59d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430457]   Object 0xffff8803f5df59e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430471]   Object 0xffff8803f5df59f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430485]   Object 0xffff8803f5df5a08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430500]   Object 0xffff8803f5df5a18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430514]   Object 0xffff8803f5df5a28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430528]   Object 0xffff8803f5df5a38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430542]   Object 0xffff8803f5df5a48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430557]   Object 0xffff8803f5df5a58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430571]   Object 0xffff8803f5df5a68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430585]   Object 0xffff8803f5df5a78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430599]   Object 0xffff8803f5df5a88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430614]   Object 0xffff8803f5df5a98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430628]   Object 0xffff8803f5df5aa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430642]   Object 0xffff8803f5df5ab8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430656]   Object 0xffff8803f5df5ac8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430671]   Object 0xffff8803f5df5ad8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430685]   Object 0xffff8803f5df5ae8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430699]   Object 0xffff8803f5df5af8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430714]   Object 0xffff8803f5df5b08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1176.430729]  Redzone 0xffff8803f5df5b18:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[ 1176.430742]  Padding 0xffff8803f5df5b58:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1176.430758] Pid: 24, comm: rcuc4 Not tainted 3.0.0-rc2 #1
[ 1176.430762] Call Trace:
[ 1176.430774]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1176.430782]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1176.430789]  [<ffffffff810bac15>] ? init_object+0x85/0xa0
[ 1176.430795]  [<ffffffff810ba816>] ? __free_slab+0x156/0x180
[ 1176.430803]  [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 1176.430810]  [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 1176.430818]  [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 1176.430825]  [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 1176.430832]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.430838]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.430846]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1176.430855]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1176.430863]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1176.430870]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1176.430876] FIX kmalloc-512: Restoring 
0xffff8803f5df5b5e-0xffff8803f5df5b5f=0x5a
[ 1176.430879]
[ 1176.430897] 
=============================================================================
[ 1176.430902] BUG kmalloc-512: Object padding overwritten
[ 1176.430905] 
-----------------------------------------------------------------------------
[ 1176.430909]
[ 1176.430912] INFO: 0xffff8803f5df6fe6-0xffff8803f5df6fe7. First byte 
0x0 instead of 0x5a
[ 1176.430921] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=93370 cpu=0 pid=5126
[ 1176.430928] INFO: Freed in __kfree_skb+0x11/0x90 age=93370 cpu=0 pid=4495
[ 1176.430935] INFO: Slab 0xffffea000ddc8d60 objects=28 used=0 
fp=0xffff8803f5df76c0 flags=0x8000000000004080
[ 1176.430940] INFO: Object 0xffff8803f5df6da0 @offset=11680 
fp=0xffff8803f5df5488
[ 1176.430943]
[ 1176.430946] Bytes b4 0xffff8803f5df6d90:  cd 7a 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a Íz......ZZZZZZZZ
[ 1176.430962]   Object 0xffff8803f5df6da0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430977]   Object 0xffff8803f5df6db0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.430992]   Object 0xffff8803f5df6dc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431006]   Object 0xffff8803f5df6dd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431021]   Object 0xffff8803f5df6de0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431036]   Object 0xffff8803f5df6df0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431050]   Object 0xffff8803f5df6e00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431065]   Object 0xffff8803f5df6e10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431080]   Object 0xffff8803f5df6e20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431094]   Object 0xffff8803f5df6e30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431109]   Object 0xffff8803f5df6e40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431124]   Object 0xffff8803f5df6e50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431138]   Object 0xffff8803f5df6e60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431153]   Object 0xffff8803f5df6e70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431168]   Object 0xffff8803f5df6e80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431182]   Object 0xffff8803f5df6e90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431197]   Object 0xffff8803f5df6ea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431213]   Object 0xffff8803f5df6eb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431228]   Object 0xffff8803f5df6ec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431243]   Object 0xffff8803f5df6ed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431258]   Object 0xffff8803f5df6ee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431273]   Object 0xffff8803f5df6ef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431287]   Object 0xffff8803f5df6f00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431303]   Object 0xffff8803f5df6f10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431317]   Object 0xffff8803f5df6f20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431332]   Object 0xffff8803f5df6f30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431346]   Object 0xffff8803f5df6f40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431361]   Object 0xffff8803f5df6f50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431376]   Object 0xffff8803f5df6f60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431391]   Object 0xffff8803f5df6f70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431405]   Object 0xffff8803f5df6f80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1176.431420]   Object 0xffff8803f5df6f90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1176.431435]  Redzone 0xffff8803f5df6fa0:  bb bb bb bb bb bb bb bb 
                      »»»»»»»»
[ 1176.431448]  Padding 0xffff8803f5df6fe0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1176.431463] Pid: 24, comm: rcuc4 Not tainted 3.0.0-rc2 #1
[ 1176.431467] Call Trace:
[ 1176.431474]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1176.431481]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1176.431487]  [<ffffffff810bac15>] ? init_object+0x85/0xa0
[ 1176.431494]  [<ffffffff810ba816>] ? __free_slab+0x156/0x180
[ 1176.431501]  [<ffffffff81075a49>] ? __rcu_process_callbacks+0x119/0x340
[ 1176.431508]  [<ffffffff8107666b>] ? rcu_cpu_kthread+0x27b/0x3a0
[ 1176.431515]  [<ffffffff8102d5b9>] ? ttwu_do_wakeup+0x19/0xa0
[ 1176.431522]  [<ffffffff81074a80>] ? invoke_rcu_cpu_kthread+0x30/0x30
[ 1176.431529]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.431535]  [<ffffffff810763f0>] ? rcu_check_callbacks+0x100/0x100
[ 1176.431543]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1176.431550]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1176.431558]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1176.431565]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1176.431571] FIX kmalloc-512: Restoring 
0xffff8803f5df6fe6-0xffff8803f5df6fe7=0x5a
[ 1176.431574]
[ 1179.773660] 
=============================================================================
[ 1179.773742] BUG kmalloc-1024: Object padding overwritten
[ 1179.773792] 
-----------------------------------------------------------------------------
[ 1179.773796]
[ 1179.773901] INFO: 0xffff8803e430e6be-0xffff8803e430e6bf. First byte 
0x0 instead of 0x5a
[ 1179.773972] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0 
age=93277 cpu=1 pid=5109
[ 1179.774042] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=93277 
cpu=5 pid=5146
[ 1179.774102] INFO: Slab 0xffffea000d9ea9c0 objects=29 used=14 
fp=0xffff8803e430d9e8 flags=0x8000000000004081
[ 1179.774163] INFO: Object 0xffff8803e430e278 @offset=25208 
fp=0xffff8803e4309df8
[ 1179.774166]
[ 1179.774260] Bytes b4 0xffff8803e430e268:  a5 57 01 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ¥W......ZZZZZZZZ
[ 1179.775297]   Object 0xffff8803e430e278:  08 eb 30 e4 03 88 ff ff a0 
e2 30 e4 03 88 ff ff .ë0ä..ÿÿ.â0ä..ÿÿ
[ 1179.776003]   Object 0xffff8803e430e288:  a0 e3 30 e4 03 88 ff ff be 
00 00 00 00 01 00 00 .ã0ä..ÿÿŸ.......
[ 1179.776974]   Object 0xffff8803e430e298:  be 00 00 00 be 00 00 00 5b 
20 20 20 20 32 2e 34 Ÿ...Ÿ...[....2.4
[ 1179.777656]   Object 0xffff8803e430e2a8:  31 30 32 34 36 5d 20 70 63 
69 20 30 30 30 30 3a 10246].pci.0000:
[ 1179.778609]   Object 0xffff8803e430e2b8:  30 30 3a 31 34 2e 34 3a 20 
20 20 62 72 69 64 67 00:14.4:...bridg
[ 1179.779289]   Object 0xffff8803e430e2c8:  65 20 77 69 6e 64 6f 77 20 
5b 69 6f 20 20 30 78 e.window.[io..0x
[ 1179.779976]   Object 0xffff8803e430e2d8:  30 64 30 30 2d 30 78 66 66 
66 66 5d 20 28 73 75 0d00-0xffff].(su
[ 1179.780327]   Object 0xffff8803e430e2e8:  62 74 72 61 63 74 69 76 65 
20 64 65 63 6f 64 65 btractive.decode
[ 1179.780327]   Object 0xffff8803e430e2f8:  29 0d 0a 5b 20 20 20 20 32 
2e 34 31 30 32 34 38 )..[....2.410248
[ 1179.780327]   Object 0xffff8803e430e308:  5d 20 70 63 69 20 30 30 30 
30 3a 30 30 3a 31 34 ].pci.0000:00:14
[ 1179.780327]   Object 0xffff8803e430e318:  2e 34 3a 20 20 20 62 72 69 
64 67 65 20 77 69 6e .4:...bridge.win
[ 1179.780327]   Object 0xffff8803e430e328:  64 6f 77 20 5b 6d 65 6d 20 
30 78 30 30 30 61 30 dow.[mem.0x000a0
[ 1179.780327]   Object 0xffff8803e430e338:  30 30 30 2d 30 78 30 30 30 
62 66 66 66 66 5d 20 000-0x000bffff].
[ 1179.780327]   Object 0xffff8803e430e348:  28 73 75 62 74 72 61 63 74 
69 76 65 20 64 65 63 (subtractive.dec
[ 1179.780327]   Object 0xffff8803e430e358:  6f 64 65 29 0d 0a 32 30 5d 
20 78 48 43 49 20 78 ode)..20].xHCI.x
[ 1179.780327]   Object 0xffff8803e430e368:  68 63 69 5f 63 68 65 63 6b 
5f 62 61 6e 64 77 69 hci_check_bandwi
[ 1179.780327]   Object 0xffff8803e430e378:  64 74 68 20 63 61 6c 6c 65 
64 20 66 6f 72 20 72 dth.called.for.r
[ 1179.780327]   Object 0xffff8803e430e388:  6f 6f 74 20 68 75 62 0d 0a 
0d 0a 0d 0a 0d 0a 0d oot.hub.........
[ 1179.780327]   Object 0xffff8803e430e398:  0a 30 30 0d 0a 0d 0a 0a 00 
00 00 00 00 00 00 00 .00.............
[ 1179.780327]   Object 0xffff8803e430e3a8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3b8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3c8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3d8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3e8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e3f8:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e408:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e418:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e428:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e438:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e448:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e458:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e468:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e478:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e488:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1179.780327]   Object 0xffff8803e430e498:  00 00 00 00 00 00 00 00 6b 
6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e4f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e508:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e518:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e528:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e538:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e548:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e558:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e568:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e578:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e588:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e598:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e5f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e608:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e618:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e628:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e638:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e648:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e658:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1179.780327]   Object 0xffff8803e430e668:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1179.780327]  Redzone 0xffff8803e430e678:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[ 1179.780327]  Padding 0xffff8803e430e6b8:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1179.780327] Pid: 2212, comm: kworker/5:2 Not tainted 3.0.0-rc2 #1
[ 1179.780327] Call Trace:
[ 1179.780327]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1179.780327]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1179.780327]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1179.780327]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1179.780327]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1179.780327]  [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1179.780327]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1179.780327]  [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1179.780327]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1179.780327]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1179.780327]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1179.780327]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1179.780327]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1179.780327]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1179.780327]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1179.780327]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1179.780327]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1179.780327] FIX kmalloc-1024: Restoring 
0xffff8803e430e6be-0xffff8803e430e6bf=0x5a
[ 1179.780327]
[ 1180.826212] 
=============================================================================
[ 1180.826289] BUG kmalloc-1024: Object padding overwritten
[ 1180.826338] 
-----------------------------------------------------------------------------
[ 1180.826341]
[ 1180.826439] INFO: 0xffff8804097a9566-0xffff8804097a9567. First byte 
0x0 instead of 0x5a
[ 1180.826505] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0 
age=69291 cpu=0 pid=5978
[ 1180.826567] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=80371 
cpu=5 pid=5713
[ 1180.826626] INFO: Slab 0xffffea000e212cc0 objects=29 used=3 
fp=0xffff8804097a8890 flags=0x8000000000004081
[ 1180.826686] INFO: Object 0xffff8804097a9120 @offset=4384 
fp=0xffff8804097a8890
[ 1180.826689]
[ 1180.826781] Bytes b4 0xffff8804097a9110:  91 48 01 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a .H......ZZZZZZZZ
[ 1180.827478]   Object 0xffff8804097a9120:  30 de 70 fc 03 88 ff ff 48 
91 7a 09 04 88 ff ff 0Þpü..ÿÿH.z...ÿÿ
[ 1180.828179]   Object 0xffff8804097a9130:  48 92 7a 09 04 88 ff ff ce 
00 00 00 00 01 00 00 H.z...ÿÿÎ.......
[ 1180.829149]   Object 0xffff8804097a9140:  ce 00 00 00 ce 00 00 00 5b 
20 20 20 20 32 2e 33 Î...Î...[....2.3
[ 1180.830162]   Object 0xffff8804097a9150:  32 31 31 35 36 5d 20 70 63 
69 20 30 30 30 30 3a 21156].pci.0000:
[ 1180.831118]   Object 0xffff8804097a9160:  30 30 3a 31 33 2e 30 3a 20 
5b 31 30 30 32 3a 34 00:13.0:.[1002:4
[ 1180.831796]   Object 0xffff8804097a9170:  33 39 37 5d 20 74 79 70 65 
20 30 20 63 6c 61 73 397].type.0.clas
[ 1180.832492]   Object 0xffff8804097a9180:  73 20 30 78 30 30 30 63 30 
33 0d 0a 5b 20 20 20 s.0x000c03..[...
[ 1180.833482]   Object 0xffff8804097a9190:  20 32 2e 33 32 31 31 36 36 
5d 20 70 63 69 20 30 .2.321166].pci.0
[ 1180.834173]   Object 0xffff8804097a91a0:  30 30 30 3a 30 30 3a 31 33 
2e 30 3a 20 72 65 67 000:00:13.0:.reg
[ 1180.835134]   Object 0xffff8804097a91b0:  20 31 30 3a 20 5b 6d 65 6d 
20 30 78 66 65 31 66 .10:.[mem.0xfe1f
[ 1180.835240]   Object 0xffff8804097a91c0:  64 30 30 30 2d 30 78 66 65 
31 66 64 66 66 66 5d d000-0xfe1fdfff]
[ 1180.835240]   Object 0xffff8804097a91d0:  0d 0a 5b 20 20 20 20 32 2e 
33 32 31 32 31 38 5d ..[....2.321218]
[ 1180.835240]   Object 0xffff8804097a91e0:  20 70 63 69 20 30 30 30 30 
3a 30 30 3a 31 33 2e .pci.0000:00:13.
[ 1180.835240]   Object 0xffff8804097a91f0:  32 3a 20 5b 31 30 30 32 3a 
34 33 39 36 5d 20 74 2:.[1002:4396].t
[ 1180.835240]   Object 0xffff8804097a9200:  79 70 65 20 30 20 63 6c 61 
73 73 20 30 78 30 30 ype.0.class.0x00
[ 1180.835240]   Object 0xffff8804097a9210:  30 63 30 33 0d 0a 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b 0c03..kkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9220:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9240:  6b 6b 6b 6b 6b 6b 6b 6b 00 
00 00 00 00 00 00 00 kkkkkkkk........
[ 1180.835240]   Object 0xffff8804097a9250:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9260:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9270:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9280:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9290:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92b0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92c0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92e0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a92f0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9300:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.835240]   Object 0xffff8804097a9310:  00 00 00 00 00 00 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b ......kkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a93f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9440:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9450:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9460:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9470:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9480:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9490:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a94f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9500:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.835240]   Object 0xffff8804097a9510:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1180.835240]  Redzone 0xffff8804097a9520:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[ 1180.835240]  Padding 0xffff8804097a9560:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1180.835240] Pid: 8295, comm: kworker/5:0 Not tainted 3.0.0-rc2 #1
[ 1180.835240] Call Trace:
[ 1180.835240]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1180.835240]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1180.835240]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.835240]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1180.835240]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1180.835240]  [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1180.835240]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.835240]  [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1180.835240]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1180.835240]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1180.835240]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1180.835240]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.835240]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.835240]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1180.835240]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1180.835240]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1180.835240]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1180.835240] FIX kmalloc-1024: Restoring 
0xffff8804097a9566-0xffff8804097a9567=0x5a
[ 1180.835240]
[ 1180.881652] 
=============================================================================
[ 1180.881713] BUG kmalloc-1024: Object padding overwritten
[ 1180.881760] 
-----------------------------------------------------------------------------
[ 1180.881763]
[ 1180.881859] INFO: 0xffff8803fc70eb06-0xffff8803fc70eb07. First byte 
0x0 instead of 0x5a
[ 1180.881920] INFO: Allocated in tty_buffer_request_room+0xcc/0x1a0 
age=69297 cpu=0 pid=5978
[ 1180.881981] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=69384 
cpu=5 pid=5146
[ 1180.882040] INFO: Slab 0xffffea000df389c0 objects=29 used=3 
fp=0xffff8803fc70de30 flags=0x8000000000004081
[ 1180.882099] INFO: Object 0xffff8803fc70e6c0 @offset=26304 
fp=0xffff8803fc70de30
[ 1180.882102]
[ 1180.882192] Bytes b4 0xffff8803fc70e6b0:  aa 40 01 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ª@......ZZZZZZZZ
[ 1180.882884]   Object 0xffff8803fc70e6c0:  68 95 6c fc 03 88 ff ff e8 
e6 70 fc 03 88 ff ff h.lü..ÿÿèæpü..ÿÿ
[ 1180.883583]   Object 0xffff8803fc70e6d0:  e8 e7 70 fc 03 88 ff ff 00 
01 00 00 00 01 00 00 èçpü..ÿÿ........
[ 1180.884542]   Object 0xffff8803fc70e6e0:  00 01 00 00 00 01 00 00 5b 
20 20 20 20 32 2e 33 ........[....2.3
[ 1180.885223]   Object 0xffff8803fc70e6f0:  32 30 38 31 38 5d 20 70 63 
69 20 30 30 30 30 3a 20818].pci.0000:
[ 1180.886169]   Object 0xffff8803fc70e700:  30 30 3a 30 34 2e 30 3a 20 
50 4d 45 23 20 73 75 00:04.0:.PME#.su
[ 1180.886850]   Object 0xffff8803fc70e710:  70 70 6f 72 74 65 64 20 66 
72 6f 6d 20 44 30 20 pported.from.D0.
[ 1180.887535]   Object 0xffff8803fc70e720:  44 33 68 6f 74 20 44 33 63 
6f 6c 64 0d 0a 5b 20 D3hot.D3cold..[.
[ 1180.888848]   Object 0xffff8803fc70e730:  20 20 20 32 2e 33 32 30 38 
32 30 5d 20 70 63 69 ...2.320820].pci
[ 1180.889537]   Object 0xffff8803fc70e740:  20 30 30 30 30 3a 30 30 3a 
30 34 2e 30 3a 20 50 .0000:00:04.0:.P
[ 1180.890481]   Object 0xffff8803fc70e750:  4d 45 23 20 64 69 73 61 62 
6c 65 64 0d 0a 5b 20 ME#.disabled..[.
[ 1180.891161]   Object 0xffff8803fc70e760:  20 20 20 32 2e 33 32 30 38 
33 36 5d 20 70 63 69 ...2.320836].pci
[ 1180.891365]   Object 0xffff8803fc70e770:  20 30 30 30 30 3a 30 30 3a 
30 39 2e 30 3a 20 5b .0000:00:09.0:.[
[ 1180.891365]   Object 0xffff8803fc70e780:  31 30 32 32 3a 39 36 30 38 
5d 20 74 79 70 65 20 1022:9608].type.
[ 1180.891365]   Object 0xffff8803fc70e790:  31 20 63 6c 61 73 73 20 30 
78 30 30 30 36 30 34 1.class.0x000604
[ 1180.891365]   Object 0xffff8803fc70e7a0:  0d 0a 5b 20 20 20 20 32 2e 
33 32 30 38 35 35 5d ..[....2.320855]
[ 1180.891365]   Object 0xffff8803fc70e7b0:  20 70 63 69 20 30 30 30 30 
3a 30 30 3a 30 39 2e .pci.0000:00:09.
[ 1180.891365]   Object 0xffff8803fc70e7c0:  30 3a 20 50 4d 45 23 20 73 
75 70 70 6f 72 74 65 0:.PME#.supporte
[ 1180.891365]   Object 0xffff8803fc70e7d0:  64 20 66 72 6f 6d 20 44 30 
20 44 33 68 6f 74 20 d.from.D0.D3hot.
[ 1180.891365]   Object 0xffff8803fc70e7e0:  44 33 63 6f 6c 64 0d 0a 00 
00 00 00 00 00 00 00 D3cold..........
[ 1180.891365]   Object 0xffff8803fc70e7f0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e800:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e810:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e820:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e830:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e840:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e850:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e860:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e870:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e880:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e890:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8b0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8c0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[ 1180.891365]   Object 0xffff8803fc70e8e0:  00 00 00 00 00 00 00 00 6b 
6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e8f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e900:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e910:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e920:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e930:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e940:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e950:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e960:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e970:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e980:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e990:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70e9f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70ea90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70eaa0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 1180.891365]   Object 0xffff8803fc70eab0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
[ 1180.891365]  Redzone 0xffff8803fc70eac0:  cc cc cc cc cc cc cc cc 
                      ÌÌÌÌÌÌÌÌ
[ 1180.891365]  Padding 0xffff8803fc70eb00:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[ 1180.891365] Pid: 8295, comm: kworker/5:0 Not tainted 3.0.0-rc2 #1
[ 1180.891365] Call Trace:
[ 1180.891365]  [<ffffffff810ba41d>] ? check_bytes_and_report+0x10d/0x150
[ 1180.891365]  [<ffffffff810ba509>] ? check_object+0xa9/0x260
[ 1180.891365]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.891365]  [<ffffffff810bb033>] ? free_debug_processing+0x133/0x250
[ 1180.891365]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
[ 1180.891365]  [<ffffffff81254c70>] ? free_tty_struct+0x30/0x30
[ 1180.891365]  [<ffffffff8125acf7>] ? tty_buffer_free_all+0x47/0x70
[ 1180.891365]  [<ffffffff81254c66>] ? free_tty_struct+0x26/0x30
[ 1180.891365]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
[ 1180.891365]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
[ 1180.891365]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
[ 1180.891365]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.891365]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
[ 1180.891365]  [<ffffffff81053436>] ? kthread+0x96/0xa0
[ 1180.891365]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
[ 1180.891365]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
[ 1180.891365]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
[ 1180.891365] FIX kmalloc-1024: Restoring 
0xffff8803fc70eb06-0xffff8803fc70eb07=0x5a
[ 1180.891365]

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 21:37, Eric Dumazet wrote:
> Le mardi 07 juin 2011 à 21:27 +0800, Brad Campbell a écrit :
>> On 07/06/11 04:22, Eric Dumazet wrote:
>>
>>> Could you please try latest linux-2.6 tree ?
>>>
>>> We fixed many networking bugs that could explain your crash.
>>>
>>>
>>>
>>>
>>
>> No good I'm afraid.
>>
>> [  543.040056]
>> =============================================================================
>> [  543.040136] BUG ip_dst_cache: Padding overwritten.
>> 0xffff8803e4217ffe-0xffff8803e4217fff
>> [  543.040194]
>
> Thats pretty strange : These are the last two bytes of a page, set to
> 0x0000 (a 16 bit value)
>
> There is no way a dst field could actually sit on this location (its a
> padding), since a dst is a bit less than 256 bytes (0xe8), and each
> entry is aligned on a 64byte address.
>
> grep dst /proc/slabinfo
>
> ip_dst_cache       32823  62944    256   32    2 : tunables    0    0
> 0 : slabdata   1967   1967      0
>
> sizeof(struct rtable)=0xe8
>
>
>> -----------------------------------------------------------------------------
>> [  543.040198]
>> [  543.040298] INFO: Slab 0xffffea000d9e74d0 objects=25 used=25 fp=0x
>>          (null) flags=0x8000000000004081
>> [  543.040364] Pid: 4576, comm: kworker/1:2 Not tainted 3.0.0-rc2 #1
>> [  543.040415] Call Trace:
>> [  543.040472]  [<ffffffff810b9c1d>] ? slab_err+0xad/0xd0
>> [  543.040528]  [<ffffffff8102e034>] ? check_preempt_wakeup+0xa4/0x160
>> [  543.040595]  [<ffffffff810ba206>] ? slab_pad_check+0x126/0x170
>> [  543.040650]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [  543.040701]  [<ffffffff810ba29a>] ? check_slab+0x4a/0xc0
>> [  543.040753]  [<ffffffff810baf2d>] ? free_debug_processing+0x2d/0x250
>> [  543.040808]  [<ffffffff810bb27b>] ? __slab_free+0x12b/0x140
>> [  543.040862]  [<ffffffff810bbe99>] ? kmem_cache_free+0x99/0xa0
>> [  543.040915]  [<ffffffff8133045b>] ? dst_destroy+0x8b/0x110
>> [  543.040967]  [<ffffffff813307f6>] ? dst_gc_task+0x196/0x1f0
>> [  543.041021]  [<ffffffff8104e954>] ? queue_delayed_work_on+0x154/0x160
>> [  543.041081]  [<ffffffff813066fe>] ? do_dbs_timer+0x20e/0x3d0
>> [  543.041133]  [<ffffffff81330660>] ? dst_alloc+0x180/0x180
>> [  543.041187]  [<ffffffff8104f28b>] ? process_one_work+0xfb/0x3b0
>> [  543.041242]  [<ffffffff8104f964>] ? worker_thread+0x144/0x3d0
>> [  543.041296]  [<ffffffff8102cc10>] ? __wake_up_common+0x50/0x80
>> [  543.041678]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [  543.041729]  [<ffffffff8104f820>] ? rescuer_thread+0x2e0/0x2e0
>> [  543.041782]  [<ffffffff81053436>] ? kthread+0x96/0xa0
>> [  543.041835]  [<ffffffff813e1d14>] ? kernel_thread_helper+0x4/0x10
>> [  543.041890]  [<ffffffff810533a0>] ? kthread_worker_fn+0x120/0x120
>> [  543.041944]  [<ffffffff813e1d10>] ? gs_change+0xb/0xb
>> [  543.041993]  Padding 0xffff8803e4217f40:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.042718]  Padding 0xffff8803e4217f50:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.043433]  Padding 0xffff8803e4217f60:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.044155]  Padding 0xffff8803e4217f70:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.044866]  Padding 0xffff8803e4217f80:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.045590]  Padding 0xffff8803e4217f90:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.046311]  Padding 0xffff8803e4217fa0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.047034]  Padding 0xffff8803e4217fb0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.047755]  Padding 0xffff8803e4217fc0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.048474]  Padding 0xffff8803e4217fd0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.049203]  Padding 0xffff8803e4217fe0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
>> [  543.049909]  Padding 0xffff8803e4217ff0:  5a 5a 5a 5a 5a 5a 5a 5a 5a
>> 5a 5a 5a 5a 5a 00 00 ZZZZZZZZZZZZZZ..
>> [  543.050021] FIX ip_dst_cache: Restoring
>> 0xffff8803e4217f40-0xffff8803e4217fff=0x5a
>> [  543.050021]
>>
>> Dropped -mm, Hugh and Andrea from CC as this does not appear to be mm or
>> ksm related.
>>
>> I'll pare down the firewall and see if I can make it break easier with a
>> smaller test set.
>
> Hmm, not sure now :(
>
> Could you reproduce another bug please ?

I know this is an old one, but I recently purchased a second system to 
allow me to test and bisect this off-line (the live system is too much 
of a headache to bisect on).

brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch 
'x86-cpu-for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add 
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove 
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix 
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove 
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the 
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config 
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to 
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop 
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect log
git bisect start
# good: [9fe6206f400646a2322096b56c59891d530e8d51] Linux 2.6.35
git bisect good 9fe6206f400646a2322096b56c59891d530e8d51
# bad: [da5cabf80e2433131bf0ed8993abc0f7ea618c73] Linux 2.6.36-rc1
git bisect bad da5cabf80e2433131bf0ed8993abc0f7ea618c73
# bad: [0f477dd0851bdcee82923da66a7fc4a44cb1bc3d] Merge branch 
'x86-cpu-for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 0f477dd0851bdcee82923da66a7fc4a44cb1bc3d
# bad: [3ff1c25927e3af61c6bf0e4ed959504058ae4565] phy/marvell: add 
88ec048 support
git bisect bad 3ff1c25927e3af61c6bf0e4ed959504058ae4565
# good: [05318bc905467237d4aa68a701f6e92a2b332218] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next-2.6
git bisect good 05318bc905467237d4aa68a701f6e92a2b332218
# bad: [2ba13ed678775195e8255b4e503c59d48b615bd8] Bluetooth: Remove 
check for supported mode
git bisect bad 2ba13ed678775195e8255b4e503c59d48b615bd8
# bad: [1e2cfeef060fa0270f9a2d66b1218c12c05062e0] Revert "tc35815: fix 
iomap leak"
git bisect bad 1e2cfeef060fa0270f9a2d66b1218c12c05062e0
# bad: [d9bed6bbd4f2a0120c93fed68605950651e1f225] isdn/gigaset: remove 
EXPERIMENTAL tag from GIGASET_CAPI
git bisect bad d9bed6bbd4f2a0120c93fed68605950651e1f225
# bad: [d117b6665847084cfe8a44b870f771153e18991d] fealnx: Use the 
instance of net_device_stats from net_device.
git bisect bad d117b6665847084cfe8a44b870f771153e18991d
# bad: [e490c1defec4236a6a131fe2d13bf7ba787c02f8] Merge branch 'master' 
of git://git.kernel.org/pub/scm/linux/kernel/git/kaber/nf-next-2.6
git bisect bad e490c1defec4236a6a131fe2d13bf7ba787c02f8
# bad: [0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba] ixgbe: use NETIF_F_LRO
git bisect bad 0a17d8c744e44617a3c22e7af68b4c5c9c1c5dba
# bad: [ede3ef0d940ef052466f42c849390b23c6859abc] igb: fix PHY config 
access on 82580
git bisect bad ede3ef0d940ef052466f42c849390b23c6859abc
# good: [ee3cb6295144b0adfa75ccaca307643a6998b1e2] be2net: changes to 
properly provide phy details
git bisect good ee3cb6295144b0adfa75ccaca307643a6998b1e2
# bad: [7475271004b66e9c22e1bb28f240a38c5d6fe76e] x86: Drop 
CONFIG_MCORE2 check around setting of NET_IP_ALIGN
git bisect bad 7475271004b66e9c22e1bb28f240a38c5d6fe76e
brad@test:/raid10/src/linux-2.6$ git bisect good
7475271004b66e9c22e1bb28f240a38c5d6fe76e is the first bad commit
commit 7475271004b66e9c22e1bb28f240a38c5d6fe76e
Author: Alexander Duyck <alexander.h.duyck@intel.com>
Date:   Thu Jul 1 13:28:27 2010 +0000

     x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN

     This patch removes the CONFIG_MCORE2 check from around 
NET_IP_ALIGN.  It is
     based on a suggestion from Andi Kleen.  The assumption is that 
there are
     not any x86 cores where unaligned access is really slow, and this 
change
     would allow for a performance improvement to still exist on 
configurations
     that are not necessarily optimized for Core 2.

     Cc: Andi Kleen <ak@linux.intel.com>
     Cc: Thomas Gleixner <tglx@linutronix.de>
     Cc: Ingo Molnar <mingo@redhat.com>
     Cc: "H. Peter Anvin" <hpa@zytor.com>
     Cc: x86@kernel.org
     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
     Acked-by: H. Peter Anvin <hpa@zytor.com>
     Signed-off-by: David S. Miller <davem@davemloft.net>

:040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a 
b98769348bd765731ca3ff03b33764257e23226c M	arch

I can confirm this bug exists in the 3.0 kernel, however I'm unable to 
reproduce it on todays git.

So anyone using netfilter, kvm and bridge on kernels between 2.6.36-rc1 
and 3.0 may hit this bug, but it looks like it is fixed in the current 
3.1-rc kernels.

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 08/20/2011 04:16 PM, Brad Campbell wrote:
> Author: Alexander Duyck <alexander.h.duyck@intel.com>
> Date:   Thu Jul 1 13:28:27 2010 +0000
>
>     x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
>
>     This patch removes the CONFIG_MCORE2 check from around 
> NET_IP_ALIGN.  It is
>     based on a suggestion from Andi Kleen.  The assumption is that 
> there are
>     not any x86 cores where unaligned access is really slow, and this 
> change
>     would allow for a performance improvement to still exist on 
> configurations
>     that are not necessarily optimized for Core 2.
>
>     Cc: Andi Kleen <ak@linux.intel.com>
>     Cc: Thomas Gleixner <tglx@linutronix.de>
>     Cc: Ingo Molnar <mingo@redhat.com>
>     Cc: "H. Peter Anvin" <hpa@zytor.com>
>     Cc: x86@kernel.org
>     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
>     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
>     Acked-by: H. Peter Anvin <hpa@zytor.com>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
>
> :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a 
> b98769348bd765731ca3ff03b33764257e23226c M    arch
>
> I can confirm this bug exists in the 3.0 kernel, however I'm unable to 
> reproduce it on todays git.
>
> So anyone using netfilter, kvm and bridge on kernels between 
> 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in 
> the current 3.1-rc kernels.
>

Thanks for this effort.  I don't think this patch is buggy in itself, it 
merely exposed another bug which was fixed later on.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le lundi 22 août 2011 à 09:36 +0300, Avi Kivity a écrit :
> On 08/20/2011 04:16 PM, Brad Campbell wrote:
> > Author: Alexander Duyck <alexander.h.duyck@intel.com>
> > Date:   Thu Jul 1 13:28:27 2010 +0000
> >
> >     x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
> >
> >     This patch removes the CONFIG_MCORE2 check from around 
> > NET_IP_ALIGN.  It is
> >     based on a suggestion from Andi Kleen.  The assumption is that 
> > there are
> >     not any x86 cores where unaligned access is really slow, and this 
> > change
> >     would allow for a performance improvement to still exist on 
> > configurations
> >     that are not necessarily optimized for Core 2.
> >
> >     Cc: Andi Kleen <ak@linux.intel.com>
> >     Cc: Thomas Gleixner <tglx@linutronix.de>
> >     Cc: Ingo Molnar <mingo@redhat.com>
> >     Cc: "H. Peter Anvin" <hpa@zytor.com>
> >     Cc: x86@kernel.org
> >     Signed-off-by: Alexander Duyck <alexander.h.duyck@intel.com>
> >     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> >     Acked-by: H. Peter Anvin <hpa@zytor.com>
> >     Signed-off-by: David S. Miller <davem@davemloft.net>
> >
> > :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a 
> > b98769348bd765731ca3ff03b33764257e23226c M    arch
> >
> > I can confirm this bug exists in the 3.0 kernel, however I'm unable to 
> > reproduce it on todays git.
> >
> > So anyone using netfilter, kvm and bridge on kernels between 
> > 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in 
> > the current 3.1-rc kernels.
> >
> 
> Thanks for this effort.  I don't think this patch is buggy in itself, it 
> merely exposed another bug which was fixed later on.
> 

Some piece of hardware has a 2-byte offset requirement, and driver
incorrectly assumed NET_IP_ALIGN was 2 on x86.

Brad, could you post your config (lsmod, dmesg) again ?

tg3.c code for example uses a private value, not related to NET_IP_ALIGN

#define TG3_RAW_IP_ALIGN 2

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 22/08/11 14:45, Eric Dumazet wrote:
> Le lundi 22 août 2011 à 09:36 +0300, Avi Kivity a écrit :
>> On 08/20/2011 04:16 PM, Brad Campbell wrote:
>>> Author: Alexander Duyck<alexander.h.duyck@intel.com>
>>> Date:   Thu Jul 1 13:28:27 2010 +0000
>>>
>>>      x86: Drop CONFIG_MCORE2 check around setting of NET_IP_ALIGN
>>>
>>>      This patch removes the CONFIG_MCORE2 check from around
>>> NET_IP_ALIGN.  It is
>>>      based on a suggestion from Andi Kleen.  The assumption is that
>>> there are
>>>      not any x86 cores where unaligned access is really slow, and this
>>> change
>>>      would allow for a performance improvement to still exist on
>>> configurations
>>>      that are not necessarily optimized for Core 2.
>>>
>>>      Cc: Andi Kleen<ak@linux.intel.com>
>>>      Cc: Thomas Gleixner<tglx@linutronix.de>
>>>      Cc: Ingo Molnar<mingo@redhat.com>
>>>      Cc: "H. Peter Anvin"<hpa@zytor.com>
>>>      Cc: x86@kernel.org
>>>      Signed-off-by: Alexander Duyck<alexander.h.duyck@intel.com>
>>>      Signed-off-by: Jeff Kirsher<jeffrey.t.kirsher@intel.com>
>>>      Acked-by: H. Peter Anvin<hpa@zytor.com>
>>>      Signed-off-by: David S. Miller<davem@davemloft.net>
>>>
>>> :040000 040000 5a15867789080a2f67a74b17c4422f85b7a9fb4a
>>> b98769348bd765731ca3ff03b33764257e23226c M    arch
>>>
>>> I can confirm this bug exists in the 3.0 kernel, however I'm unable to
>>> reproduce it on todays git.
>>>
>>> So anyone using netfilter, kvm and bridge on kernels between
>>> 2.6.36-rc1 and 3.0 may hit this bug, but it looks like it is fixed in
>>> the current 3.1-rc kernels.
>>>
>>
>> Thanks for this effort.  I don't think this patch is buggy in itself, it
>> merely exposed another bug which was fixed later on.
>>
>
> Some piece of hardware has a 2-byte offset requirement, and driver
> incorrectly assumed NET_IP_ALIGN was 2 on x86.
>
> Brad, could you post your config (lsmod, dmesg) again ?
>
> tg3.c code for example uses a private value, not related to NET_IP_ALIGN
>
> #define TG3_RAW_IP_ALIGN 2

Both sets of hardware were using Marvell 88e1116 chips (the r8169 driver).

Interestingly on the hardware I just used to reproduce the fault the 
network path is VM1 -> br1 -> eth0 -> DNAT -> br1 -> VM2

On the machine that originally exhibited the fault the network path was
VM1 -> br1 -> ppp0 -> DNAT -> br1 -> VM2

So on the original machine the packet never even goes near a physical 
interface.

The problem only exhibited itself if I tried to use the address of eth0 
or ppp0 to get to VM2 from VM1. If I went dirctly from VM1 to VM2 using 
local addresses and never traversed netfilter it never paniced.

I also agree the patch I identified simply unmasked an underlying issue, 
and of course on anything later than the linus 3.0 kernel I can't 
reproduce it now anyway.

It's nice to know its fixed, but it'd be nice to know *what* fixed it 
rather than it just went away after some code re-factoring.

I did have a perverse thought about doing a bisect between 3.0.0 and 
3.1-rc2 to see precisely *what* fixed the problem, but the thought of 
another 7 hours of crash and burn is not overly appealing.

[    0.000000] Linux version 3.0.0 (brad@srv) (gcc version 4.4.5 (Debian 
4.4.5-8) ) #5 SMP Mon Jul 25 22:43:34 WST 2011
[    0.000000] Command line: BOOT_IMAGE=/bzImage 
root=UUID=9162858d-df5e-43d6-8331-6d2b65082b4c ro panic=5 
netconsole=6666@192.168.2.1/eth0,5050@192.168.2.87/00:16:cb:a7:dd:d1
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009c400 (usable)
[    0.000000]  BIOS-e820: 000000000009c400 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000dbf80000 (usable)
[    0.000000]  BIOS-e820: 00000000dbf80000 - 00000000dffa0000 (reserved)
[    0.000000]  BIOS-e820: 00000000dffa0000 - 00000000dffb0000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000dffb0000 - 00000000dffe0000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000dffe0000 - 00000000e0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000420000000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] DMI: To Be Filled By O.E.M. To Be Filled By O.E.M./880G 
Extreme3, BIOS P1.70 06/21/2010
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 
(usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 
(usable)
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x420000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-EFFFF uncachable
[    0.000000]   F0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000000 mask FFFF80000000 write-back
[    0.000000]   1 base 000080000000 mask FFFFC0000000 write-back
[    0.000000]   2 base 0000C0000000 mask FFFFE0000000 write-back
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] TOM2: 0000000420000000 aka 16896M
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 
0x7010600070106
[    0.000000] e820 update range: 00000000e0000000 - 0000000100000000 
(usable) ==> (reserved)
[    0.000000] last_pfn = 0xdbf80 max_arch_pfn = 0x400000000
[    0.000000] initial memory mapped : 0 - 20000000
[    0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 8192
[    0.000000] Using GB pages for direct mapping
[    0.000000] init_memory_mapping: 0000000000000000-00000000dbf80000
[    0.000000]  0000000000 - 00c0000000 page 1G
[    0.000000]  00c0000000 - 00dbe00000 page 2M
[    0.000000]  00dbe00000 - 00dbf80000 page 4k
[    0.000000] kernel direct mapping tables up to dbf80000 @ 
dbf7d000-dbf80000
[    0.000000] init_memory_mapping: 0000000100000000-0000000420000000
[    0.000000]  0100000000 - 0400000000 page 1G
[    0.000000]  0400000000 - 0420000000 page 2M
[    0.000000] kernel direct mapping tables up to 420000000 @ 
41fffe000-420000000
[    0.000000] RAMDISK: 37cd8000 - 37ff0000
[    0.000000] ACPI: RSDP 00000000000fa940 00014 (v00 ACPIAM)
[    0.000000] ACPI: RSDT 00000000dffa0000 00044 (v01 062110 RSDT2249 
20100621 MSFT 00000097)
[    0.000000] ACPI: FACP 00000000dffa0200 00084 (v01 A_M_I  OEMFACP 
12000601 MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000dffa0450 07D9F (v01  AS368 AS368164 
00000164 INTL 20051117)
[    0.000000] ACPI: FACS 00000000dffb0000 00040
[    0.000000] ACPI: APIC 00000000dffa0390 0007C (v01 062110 APIC2249 
20100621 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffa0410 0003C (v01 062110 OEMMCFG 
20100621 MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000dffb0040 00072 (v01 062110 OEMB2249 
20100621 MSFT 00000097)
[    0.000000] ACPI: SRAT 00000000dffaa450 00108 (v03 AMD    FAM_F_10 
00000002 AMD  00000001)
[    0.000000] ACPI: AAFT 00000000dffaa560 00027 (v01 062110 OEMAAFT 
20100621 MSFT 00000097)
[    0.000000] ACPI: HPET 00000000dffaa590 00038 (v01 062110 OEMHPET 
20100621 MSFT 00000097)
[    0.000000] ACPI: SSDT 00000000dffaa5d0 00DA4 (v01 A M I  POWERNOW 
00000001 AMD  00000001)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000]  [ffffea0000000000-ffffea000e7fffff] PMD -> 
[ffff88040f600000-ffff88041d7fffff] on node 0
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00420000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[3] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x0000009c
[    0.000000]     0: 0x00000100 -> 0x000dbf80
[    0.000000]     0: 0x00100000 -> 0x00420000
[    0.000000] On node 0 totalpages: 4177676
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 2 pages reserved
[    0.000000]   DMA zone: 3922 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 14280 pages used for memmap
[    0.000000]   DMA32 zone: 882616 pages, LIFO batch:31
[    0.000000]   Normal zone: 44800 pages used for memmap
[    0.000000]   Normal zone: 3232000 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled)
[    0.000000] ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 
0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8300 base: 0xfed00000
[    0.000000] SMP: Allowing 6 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 40
[    0.000000] Allocating PCI resources starting at e0000000 (gap: 
e0000000:1ff00000)
[    0.000000] setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:6 
nr_node_ids:1
[    0.000000] PERCPU: Embedded 24 pages/cpu @ffff88041fc00000 s69248 
r8192 d20864 u262144
[    0.000000] pcpu-alloc: s69248 r8192 d20864 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 - -
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on. 
Total pages: 4118538
[    0.000000] Kernel command line: BOOT_IMAGE=/bzImage 
root=UUID=9162858d-df5e-43d6-8331-6d2b65082b4c ro panic=5 
netconsole=6666@192.168.2.1/eth0,5050@192.168.2.87/00:16:cb:a7:dd:d1
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 2097152 (order: 12, 
16777216 bytes)
[    0.000000] Inode-cache hash table entries: 1048576 (order: 11, 
8388608 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ d21e000000 size 32 MB
[    0.000000] Aperture beyond 4GB. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000] Mapping aperture over 65536 KB of RAM @ d0000000
[    0.000000] Memory: 16310976k/17301504k available (4036k kernel code, 
590800k absent, 399728k reserved, 2027k data, 468k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, 
CPUs=6, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:512
[    0.000000] spurious 8259A interrupt: IRQ7.
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty0] enabled
[    0.000000] hpet clockevent registered
[    0.000000] Fast TSC calibration using PIT
[    0.000000] Detected 3213.157 MHz processor.
[    0.000003] Calibrating delay loop (skipped), value calculated using 
timer frequency.. 6426.31 BogoMIPS (lpj=32131570)
[    0.000064] pid_max: default: 32768 minimum: 301
[    0.000126] Mount-cache hash table entries: 256
[    0.000267] tseg: 0000000000
[    0.000272] CPU: Physical Processor ID: 0
[    0.000304] CPU: Processor Core ID: 0
[    0.000334] mce: CPU supports 6 MCE banks
[    0.000420] Freeing SMP alternatives: 20k freed
[    0.000456] ACPI: Core revision 20110413
[    0.012256] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.112297] CPU0: AMD Phenom(tm) II X6 1090T Processor stepping 00
[    0.120000] Performance Events: AMD PMU driver.
[    0.120000] ... version:                0
[    0.120000] ... bit width:              48
[    0.120000] ... generic registers:      4
[    0.120000] ... value mask:             0000ffffffffffff
[    0.120000] ... max period:             00007fffffffffff
[    0.120000] ... fixed-purpose events:   0
[    0.120000] ... event mask:             000000000000000f
[    0.120000] Booting Node   0, Processors  #1
[    0.120000] smpboot cpu 1: start_ip = 9a000
[    0.280028]  #2
[    0.280056] smpboot cpu 2: start_ip = 9a000
[    0.450026]  #3
[    0.450054] smpboot cpu 3: start_ip = 9a000
[    0.620025]  #4
[    0.620066] smpboot cpu 4: start_ip = 9a000
[    0.790027]  #5 Ok.
[    0.790057] smpboot cpu 5: start_ip = 9a000
[    0.960003] Brought up 6 CPUs
[    0.960033] Total of 6 processors activated (38560.60 BogoMIPS).
[    0.963391] devtmpfs: initialized
[    0.963391] xor: automatically using best checksumming function: 
generic_sse
[    1.009895]    generic_sse: 14408.400 MB/sec
[    1.009928] xor: using function: generic_sse (14408.400 MB/sec)
[    1.010040] NET: Registered protocol family 16
[    1.010079] node 0 link 0: io port [1000, ffffff]
[    1.010079] TOM: 00000000e0000000 aka 3584M
[    1.010079] Fam 10h mmconf [e0000000, efffffff]
[    1.010079] node 0 link 0: mmio [a0000, bffff]
[    1.010079] node 0 link 0: mmio [e0000000, efffffff] ==> none
[    1.010079] node 0 link 0: mmio [f0000000, fe1fffff]
[    1.010079] node 0 link 0: mmio [fe200000, fe3fffff]
[    1.010079] node 0 link 0: mmio [fe400000, ffefffff]
[    1.010079] TOM2: 0000000420000000 aka 16896M
[    1.010079] bus: [00, 07] on node 0 link 0
[    1.010079] bus: 00 index 0 [io  0x0000-0xffff]
[    1.010080] bus: 00 index 1 [mem 0x000a0000-0x000bffff]
[    1.010081] bus: 00 index 2 [mem 0xf0000000-0xffffffff]
[    1.010082] bus: 00 index 3 [mem 0x420000000-0xfcffffffff]
[    1.010089] Extended Config Space enabled on 1 nodes
[    1.010151] ACPI: bus type pci registered
[    1.010185] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 
0xe0000000-0xefffffff] (base 0xe0000000)
[    1.010185] PCI: not using MMCONFIG
[    1.010185] PCI: Using configuration type 1 for base access
[    1.010185] PCI: Using configuration type 1 for extended access
[    1.010185] bio: create slab <bio-0> at 0
[    1.179907] raid6: int64x1   3168 MB/s
[    1.349907] raid6: int64x2   4039 MB/s
[    1.519903] raid6: int64x4   3205 MB/s
[    1.689900] raid6: int64x8   3017 MB/s
[    1.859895] raid6: sse2x1    4891 MB/s
[    2.029898] raid6: sse2x2    7971 MB/s
[    2.199900] raid6: sse2x4    9353 MB/s
[    2.199934] raid6: using algorithm sse2x4 (9353 MB/s)
[    2.200432] ACPI: EC: Look up EC in DSDT
[    2.200983] ACPI: Executed 4 blocks of module-level executable AML code
[    2.230239] ACPI: Interpreter enabled
[    2.230271] ACPI: (supports S0 S5)
[    2.230358] ACPI: Using IOAPIC for interrupt routing
[    2.230475] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 
0xe0000000-0xefffffff] (base 0xe0000000)
[    2.231052] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in 
ACPI motherboard resources
[    2.245403] ACPI: No dock devices found.
[    2.245403] PCI: Using host bridge windows from ACPI; if necessary, 
use "pci=nocrs" and report a bug
[    2.245403] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    2.245403] pci_root PNP0A03:00: host bridge window [io  0x0000-0x0cf7]
[    2.245403] pci_root PNP0A03:00: host bridge window [io  0x0d00-0xffff]
[    2.245403] pci_root PNP0A03:00: host bridge window [mem 
0x000a0000-0x000bffff]
[    2.245403] pci_root PNP0A03:00: host bridge window [mem 
0x000d0000-0x000dffff]
[    2.245403] pci_root PNP0A03:00: host bridge window [mem 
0xf0000000-0xfebfffff]
[    2.245403] pci_root PNP0A03:00: address space collision: host bridge 
window [mem 0x000d0000-0x000dffff] conflicts with Adapter ROM [mem 
0x000cf000-0x000d53ff]
[    2.245403] pci 0000:00:00.0: [1022:9601] type 0 class 0x000600
[    2.245403] pci 0000:00:01.0: [1849:9602] type 1 class 0x000604
[    2.245403] pci 0000:00:02.0: [1022:9603] type 1 class 0x000604
[    2.245403] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:02.0: PME# disabled
[    2.245403] pci 0000:00:03.0: [1022:960b] type 1 class 0x000604
[    2.245403] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:03.0: PME# disabled
[    2.245403] pci 0000:00:04.0: [1022:9604] type 1 class 0x000604
[    2.245403] pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:04.0: PME# disabled
[    2.245403] pci 0000:00:09.0: [1022:9608] type 1 class 0x000604
[    2.245403] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:09.0: PME# disabled
[    2.245403] pci 0000:00:0a.0: [1022:9609] type 1 class 0x000604
[    2.245403] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:0a.0: PME# disabled
[    2.245403] pci 0000:00:11.0: [1002:4391] type 0 class 0x000106
[    2.245403] pci 0000:00:11.0: reg 10: [io  0x9000-0x9007]
[    2.245403] pci 0000:00:11.0: reg 14: [io  0x8000-0x8003]
[    2.245403] pci 0000:00:11.0: reg 18: [io  0x7000-0x7007]
[    2.245403] pci 0000:00:11.0: reg 1c: [io  0x6000-0x6003]
[    2.245403] pci 0000:00:11.0: reg 20: [io  0x5000-0x500f]
[    2.245403] pci 0000:00:11.0: reg 24: [mem 0xfe1ffc00-0xfe1fffff]
[    2.245403] pci 0000:00:12.0: [1002:4397] type 0 class 0x000c03
[    2.245403] pci 0000:00:12.0: reg 10: [mem 0xfe1fe000-0xfe1fefff]
[    2.245403] pci 0000:00:12.2: [1002:4396] type 0 class 0x000c03
[    2.245403] pci 0000:00:12.2: reg 10: [mem 0xfe1ff800-0xfe1ff8ff]
[    2.245403] pci 0000:00:12.2: supports D1 D2
[    2.245403] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[    2.245403] pci 0000:00:12.2: PME# disabled
[    2.245403] pci 0000:00:13.0: [1002:4397] type 0 class 0x000c03
[    2.245403] pci 0000:00:13.0: reg 10: [mem 0xfe1fd000-0xfe1fdfff]
[    2.245403] pci 0000:00:13.2: [1002:4396] type 0 class 0x000c03
[    2.245403] pci 0000:00:13.2: reg 10: [mem 0xfe1ff400-0xfe1ff4ff]
[    2.245403] pci 0000:00:13.2: supports D1 D2
[    2.245403] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[    2.245403] pci 0000:00:13.2: PME# disabled
[    2.245403] pci 0000:00:14.0: [1002:4385] type 0 class 0x000c05
[    2.245403] pci 0000:00:14.2: [1002:4383] type 0 class 0x000403
[    2.245403] pci 0000:00:14.2: reg 10: [mem 0xfe1f8000-0xfe1fbfff 64bit]
[    2.245403] pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
[    2.245403] pci 0000:00:14.2: PME# disabled
[    2.245403] pci 0000:00:14.3: [1002:439d] type 0 class 0x000601
[    2.245403] pci 0000:00:14.4: [1002:4384] type 1 class 0x000604
[    2.245403] pci 0000:00:14.5: [1002:4399] type 0 class 0x000c03
[    2.245403] pci 0000:00:14.5: reg 10: [mem 0xfe1fc000-0xfe1fcfff]
[    2.245403] pci 0000:00:16.0: [1002:4397] type 0 class 0x000c03
[    2.245403] pci 0000:00:16.0: reg 10: [mem 0xfe1f7000-0xfe1f7fff]
[    2.245403] pci 0000:00:16.2: [1002:4396] type 0 class 0x000c03
[    2.245403] pci 0000:00:16.2: reg 10: [mem 0xfe1ff000-0xfe1ff0ff]
[    2.245403] pci 0000:00:16.2: supports D1 D2
[    2.245403] pci 0000:00:16.2: PME# supported from D0 D1 D2 D3hot
[    2.245403] pci 0000:00:16.2: PME# disabled
[    2.245403] pci 0000:00:18.0: [1022:1200] type 0 class 0x000600
[    2.245403] pci 0000:00:18.1: [1022:1201] type 0 class 0x000600
[    2.245403] pci 0000:00:18.2: [1022:1202] type 0 class 0x000600
[    2.245403] pci 0000:00:18.3: [1022:1203] type 0 class 0x000600
[    2.250003] pci 0000:00:18.4: [1022:1204] type 0 class 0x000600
[    2.250040] pci 0000:01:05.0: [1002:9715] type 0 class 0x000300
[    2.250046] pci 0000:01:05.0: reg 10: [mem 0xf0000000-0xf7ffffff pref]
[    2.250050] pci 0000:01:05.0: reg 14: [io  0xa000-0xa0ff]
[    2.250054] pci 0000:01:05.0: reg 18: [mem 0xfe3f0000-0xfe3fffff]
[    2.250061] pci 0000:01:05.0: reg 24: [mem 0xfe200000-0xfe2fffff]
[    2.250070] pci 0000:01:05.0: supports D1 D2
[    2.250097] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    2.250129] pci 0000:00:01.0:   bridge window [io  0xa000-0xafff]
[    2.250131] pci 0000:00:01.0:   bridge window [mem 0xfe200000-0xfe3fffff]
[    2.250134] pci 0000:00:01.0:   bridge window [mem 
0xf0000000-0xf7ffffff 64bit pref]
[    2.250158] pci 0000:04:00.0: [1000:0073] type 0 class 0x000104
[    2.250165] pci 0000:04:00.0: reg 10: [io  0xc000-0xc0ff]
[    2.250172] pci 0000:04:00.0: reg 14: [mem 0xfe6fc000-0xfe6fffff 64bit]
[    2.250180] pci 0000:04:00.0: reg 1c: [mem 0xfe680000-0xfe6bffff 64bit]
[    2.250190] pci 0000:04:00.0: reg 30: [mem 0xfe6c0000-0xfe6dffff pref]
[    2.250204] pci 0000:04:00.0: supports D1 D2
[    2.270016] pci 0000:00:02.0: PCI bridge to [bus 04-04]
[    2.270057] pci 0000:00:02.0:   bridge window [io  0xc000-0xcfff]
[    2.270059] pci 0000:00:02.0:   bridge window [mem 0xfe600000-0xfe6fffff]
[    2.270062] pci 0000:00:02.0:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.270088] pci 0000:05:00.0: [1000:0073] type 0 class 0x000104
[    2.270096] pci 0000:05:00.0: reg 10: [io  0xd000-0xd0ff]
[    2.270104] pci 0000:05:00.0: reg 14: [mem 0xfe7fc000-0xfe7fffff 64bit]
[    2.270112] pci 0000:05:00.0: reg 1c: [mem 0xfe780000-0xfe7bffff 64bit]
[    2.270121] pci 0000:05:00.0: reg 30: [mem 0xfe7c0000-0xfe7dffff pref]
[    2.270136] pci 0000:05:00.0: supports D1 D2
[    2.290016] pci 0000:00:03.0: PCI bridge to [bus 05-05]
[    2.290054] pci 0000:00:03.0:   bridge window [io  0xd000-0xdfff]
[    2.290057] pci 0000:00:03.0:   bridge window [mem 0xfe700000-0xfe7fffff]
[    2.290060] pci 0000:00:03.0:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.290091] pci 0000:07:00.0: [11ab:7042] type 0 class 0x000100
[    2.290104] pci 0000:07:00.0: reg 10: [mem 0xfe900000-0xfe9fffff 64bit]
[    2.290110] pci 0000:07:00.0: reg 18: [io  0xe800-0xe8ff]
[    2.290133] pci 0000:07:00.0: reg 30: [mem 0xfe800000-0xfe8fffff pref]
[    2.290161] pci 0000:07:00.0: disabling ASPM on pre-1.1 PCIe device. 
  You can enable it with 'pcie_aspm=force'
[    2.290198] pci 0000:00:04.0: PCI bridge to [bus 07-07]
[    2.290230] pci 0000:00:04.0:   bridge window [io  0xe000-0xefff]
[    2.290232] pci 0000:00:04.0:   bridge window [mem 0xfe800000-0xfe9fffff]
[    2.290235] pci 0000:00:04.0:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.290264] pci 0000:02:00.0: [1033:0194] type 0 class 0x000c03
[    2.290277] pci 0000:02:00.0: reg 10: [mem 0xfe4fe000-0xfe4fffff 64bit]
[    2.290327] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
[    2.290330] pci 0000:02:00.0: PME# disabled
[    2.310021] pci 0000:00:09.0: PCI bridge to [bus 02-02]
[    2.310056] pci 0000:00:09.0:   bridge window [io  0xf000-0x0000] 
(disabled)
[    2.310058] pci 0000:00:09.0:   bridge window [mem 0xfe400000-0xfe4fffff]
[    2.310061] pci 0000:00:09.0:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.310092] pci 0000:03:00.0: [10ec:8168] type 0 class 0x000200
[    2.310104] pci 0000:03:00.0: reg 10: [io  0xb800-0xb8ff]
[    2.310122] pci 0000:03:00.0: reg 18: [mem 0xfdfff000-0xfdffffff 
64bit pref]
[    2.310134] pci 0000:03:00.0: reg 20: [mem 0xfdff8000-0xfdffbfff 
64bit pref]
[    2.310142] pci 0000:03:00.0: reg 30: [mem 0xfe5e0000-0xfe5fffff pref]
[    2.310167] pci 0000:03:00.0: supports D1 D2
[    2.310169] pci 0000:03:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.310172] pci 0000:03:00.0: PME# disabled
[    2.330022] pci 0000:00:0a.0: PCI bridge to [bus 03-03]
[    2.330056] pci 0000:00:0a.0:   bridge window [io  0xb000-0xbfff]
[    2.330058] pci 0000:00:0a.0:   bridge window [mem 0xfe500000-0xfe5fffff]
[    2.330061] pci 0000:00:0a.0:   bridge window [mem 
0xfdf00000-0xfdffffff 64bit pref]
[    2.330109] pci 0000:00:14.4: PCI bridge to [bus 06-06] (subtractive 
decode)
[    2.330142] pci 0000:00:14.4:   bridge window [io  0xf000-0x0000] 
(disabled)
[    2.330145] pci 0000:00:14.4:   bridge window [mem 
0xfff00000-0x000fffff] (disabled)
[    2.330148] pci 0000:00:14.4:   bridge window [mem 
0xfff00000-0x000fffff pref] (disabled)
[    2.330150] pci 0000:00:14.4:   bridge window [io  0x0000-0x0cf7] 
(subtractive decode)
[    2.330152] pci 0000:00:14.4:   bridge window [io  0x0d00-0xffff] 
(subtractive decode)
[    2.330153] pci 0000:00:14.4:   bridge window [mem 
0x000a0000-0x000bffff] (subtractive decode)
[    2.330155] pci 0000:00:14.4:   bridge window [mem 
0xf0000000-0xfebfffff] (subtractive decode)
[    2.330168] pci_bus 0000:00: on NUMA node 0
[    2.330171] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    2.330251] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
[    2.330268] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE2._PRT]
[    2.330283] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE3._PRT]
[    2.330296] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE4._PRT]
[    2.330310] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE9._PRT]
[    2.330324] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCEA._PRT]
[    2.330343] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
[    2.330382]  pci0000:00: Requesting ACPI _OSC control (0x1d)
[    2.330420]  pci0000:00: ACPI _OSC request failed (AE_NOT_FOUND), 
returned control mask: 0x1d
[    2.330453] ACPI _OSC control for PCIe not granted, disabling ASPM
[    2.330576] ACPI: PCI Interrupt Link [LNKA] (IRQs 10 11 14 15) *0
[    2.330576] ACPI: PCI Interrupt Link [LNKB] (IRQs 10 11 14 15) *0
[    2.330576] ACPI: PCI Interrupt Link [LNKC] (IRQs 10 11 14 15) *0
[    2.330763] ACPI: PCI Interrupt Link [LNKD] (IRQs 10 11 14 15) *0
[    2.331002] ACPI: PCI Interrupt Link [LNKE] (IRQs 10 11 14 15) *0
[    2.331234] ACPI: PCI Interrupt Link [LNKF] (IRQs 10 11 14 15) *0
[    2.331466] ACPI: PCI Interrupt Link [LNKG] (IRQs 10 11 14 15) *0
[    2.331698] ACPI: PCI Interrupt Link [LNKH] (IRQs 10 11 14 15) *0
[    2.331926] vgaarb: device added: 
PCI:0000:01:05.0,decodes=io+mem,owns=io+mem,locks=none
[    2.331926] vgaarb: loaded
[    2.331926] vgaarb: bridge control possible 0000:01:05.0
[    2.331926] SCSI subsystem initialized
[    2.331926] libata version 3.00 loaded.
[    2.331926] PCI: Using ACPI for IRQ routing
[    2.341813] PCI: pci_cache_line_size set to 64 bytes
[    2.341867] reserve RAM buffer: 000000000009c400 - 000000000009ffff
[    2.341869] reserve RAM buffer: 00000000dbf80000 - 00000000dbffffff
[    2.341881] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    2.341881] hpet0: 3 comparators, 32-bit 14.318180 MHz counter
[    2.342205] Switching to clocksource hpet
[    2.342293] pnp: PnP ACPI init
[    2.342293] ACPI: bus type pnp registered
[    2.342293] pnp 00:00: [bus 00-ff]
[    2.342293] pnp 00:00: [io  0x0cf8-0x0cff]
[    2.342293] pnp 00:00: [io  0x0000-0x0cf7 window]
[    2.342293] pnp 00:00: [io  0x0d00-0xffff window]
[    2.342293] pnp 00:00: [mem 0x000a0000-0x000bffff window]
[    2.342293] pnp 00:00: [mem 0x000d0000-0x000dffff window]
[    2.342293] pnp 00:00: [mem 0xe0000000-0xdfffffff window disabled]
[    2.342293] pnp 00:00: [mem 0xf0000000-0xfebfffff window]
[    2.342293] pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
[    2.342293] pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
[    2.342293] pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
[    2.342293] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342293] pnp 00:02: [dma 4]
[    2.342293] pnp 00:02: [io  0x0000-0x000f]
[    2.342293] pnp 00:02: [io  0x0081-0x0083]
[    2.342293] pnp 00:02: [io  0x0087]
[    2.342293] pnp 00:02: [io  0x0089-0x008b]
[    2.342293] pnp 00:02: [io  0x008f]
[    2.342293] pnp 00:02: [io  0x00c0-0x00df]
[    2.342293] pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
[    2.342293] pnp 00:03: [io  0x0070-0x0071]
[    2.342293] pnp 00:03: [irq 8]
[    2.342293] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active)
[    2.342293] pnp 00:04: [io  0x0061]
[    2.342293] pnp 00:04: Plug and Play ACPI device, IDs PNP0800 (active)
[    2.342293] pnp 00:05: [io  0x00f0-0x00ff]
[    2.342293] pnp 00:05: [irq 13]
[    2.342293] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active)
[    2.342293] pnp 00:06: [io  0x02f8-0x02ff]
[    2.342293] pnp 00:06: [irq 3]
[    2.342293] pnp 00:06: [dma 0 disabled]
[    2.342293] pnp 00:06: Plug and Play ACPI device, IDs PNP0510 (active)
[    2.342293] pnp 00:07: [mem 0xfed00000-0xfed003ff]
[    2.342293] pnp 00:07: Plug and Play ACPI device, IDs PNP0103 (active)
[    2.342293] pnp 00:08: [mem 0xfec00000-0xfec00fff]
[    2.342293] pnp 00:08: [mem 0xfee00000-0xfee00fff]
[    2.342293] system 00:08: [mem 0xfec00000-0xfec00fff] could not be 
reserved
[    2.342293] system 00:08: [mem 0xfee00000-0xfee00fff] has been reserved
[    2.342293] system 00:08: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342293] pnp 00:09: [io  0x0010-0x001f]
[    2.342293] pnp 00:09: [io  0x0022-0x003f]
[    2.342293] pnp 00:09: [io  0x0062-0x0063]
[    2.342293] pnp 00:09: [io  0x0065-0x006f]
[    2.342293] pnp 00:09: [io  0x0072-0x007f]
[    2.342293] pnp 00:09: [io  0x0080]
[    2.342293] pnp 00:09: [io  0x0084-0x0086]
[    2.342293] pnp 00:09: [io  0x0088]
[    2.342293] pnp 00:09: [io  0x008c-0x008e]
[    2.342293] pnp 00:09: [io  0x0090-0x009f]
[    2.342293] pnp 00:09: [io  0x00a2-0x00bf]
[    2.342293] pnp 00:09: [io  0x00b1]
[    2.342293] pnp 00:09: [io  0x00e0-0x00ef]
[    2.342293] pnp 00:09: [io  0x04d0-0x04d1]
[    2.342293] pnp 00:09: [io  0x040b]
[    2.342293] pnp 00:09: [io  0x04d6]
[    2.342293] pnp 00:09: [io  0x0c00-0x0c01]
[    2.342293] pnp 00:09: [io  0x0c14]
[    2.342293] pnp 00:09: [io  0x0c50-0x0c51]
[    2.342293] pnp 00:09: [io  0x0c52]
[    2.342293] pnp 00:09: [io  0x0c6c]
[    2.342293] pnp 00:09: [io  0x0c6f]
[    2.342293] pnp 00:09: [io  0x0cd0-0x0cd1]
[    2.342293] pnp 00:09: [io  0x0cd2-0x0cd3]
[    2.342293] pnp 00:09: [io  0x0cd4-0x0cd5]
[    2.342293] pnp 00:09: [io  0x0cd6-0x0cd7]
[    2.342293] pnp 00:09: [io  0x0cd8-0x0cdf]
[    2.342293] pnp 00:09: [io  0x0800-0x089f]
[    2.342293] pnp 00:09: [io  0x0000-0xffffffffffffffff disabled]
[    2.342293] pnp 00:09: [io  0x0b00-0x0b1f]
[    2.342293] pnp 00:09: [io  0x0b20-0x0b3f]
[    2.342293] pnp 00:09: [io  0x0900-0x090f]
[    2.342293] pnp 00:09: [io  0x0910-0x091f]
[    2.342293] pnp 00:09: [io  0xfe00-0xfefe]
[    2.342293] pnp 00:09: [io  0x0060-0x005f disabled]
[    2.342293] pnp 00:09: [io  0x0064-0x0063 disabled]
[    2.342293] pnp 00:09: [mem 0xffb80000-0xffbfffff]
[    2.342293] pnp 00:09: [mem 0xfec10000-0xfec1001f]
[    2.342293] pnp 00:09: [mem 0xfed80000-0xfed80fff]
[    2.342293] pnp 00:09: [mem 0x00000000-0xffffffffffffffff disabled]
[    2.342293] system 00:09: [io  0x04d0-0x04d1] has been reserved
[    2.342293] system 00:09: [io  0x040b] has been reserved
[    2.342293] system 00:09: [io  0x04d6] has been reserved
[    2.342293] system 00:09: [io  0x0c00-0x0c01] has been reserved
[    2.342293] system 00:09: [io  0x0c14] has been reserved
[    2.342293] system 00:09: [io  0x0c50-0x0c51] has been reserved
[    2.342293] system 00:09: [io  0x0c52] has been reserved
[    2.342293] system 00:09: [io  0x0c6c] has been reserved
[    2.342293] system 00:09: [io  0x0c6f] has been reserved
[    2.342293] system 00:09: [io  0x0cd0-0x0cd1] has been reserved
[    2.342293] system 00:09: [io  0x0cd2-0x0cd3] has been reserved
[    2.342293] system 00:09: [io  0x0cd4-0x0cd5] has been reserved
[    2.342293] system 00:09: [io  0x0cd6-0x0cd7] has been reserved
[    2.342293] system 00:09: [io  0x0cd8-0x0cdf] has been reserved
[    2.342293] system 00:09: [io  0x0800-0x089f] has been reserved
[    2.342293] system 00:09: [io  0x0b00-0x0b1f] has been reserved
[    2.342293] system 00:09: [io  0x0b20-0x0b3f] has been reserved
[    2.342293] system 00:09: [io  0x0900-0x090f] has been reserved
[    2.342293] system 00:09: [io  0x0910-0x091f] has been reserved
[    2.342293] system 00:09: [io  0xfe00-0xfefe] has been reserved
[    2.342293] system 00:09: [mem 0xffb80000-0xffbfffff] has been reserved
[    2.342293] system 00:09: [mem 0xfec10000-0xfec1001f] has been reserved
[    2.342293] system 00:09: [mem 0xfed80000-0xfed80fff] has been reserved
[    2.342293] system 00:09: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342293] pnp 00:0a: [io  0x0060]
[    2.342293] pnp 00:0a: [io  0x0064]
[    2.342293] pnp 00:0a: [irq 1]
[    2.342293] pnp 00:0a: Plug and Play ACPI device, IDs PNP0303 PNP030b 
(active)
[    2.342293] pnp 00:0b: [io  0x03f8-0x03ff]
[    2.342293] pnp 00:0b: [irq 4]
[    2.342293] pnp 00:0b: [dma 0 disabled]
[    2.342293] pnp 00:0b: Plug and Play ACPI device, IDs PNP0501 (active)
[    2.342316] pnp 00:0c: [io  0x0000-0xffffffffffffffff disabled]
[    2.342317] pnp 00:0c: [io  0x0000-0xffffffffffffffff disabled]
[    2.342319] pnp 00:0c: [io  0x0290-0x029f]
[    2.342365] system 00:0c: [io  0x0290-0x029f] has been reserved
[    2.342396] system 00:0c: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342412] pnp 00:0d: [mem 0xe0000000-0xefffffff]
[    2.342457] system 00:0d: [mem 0xe0000000-0xefffffff] has been reserved
[    2.342489] system 00:0d: Plug and Play ACPI device, IDs PNP0c02 (active)
[    2.342549] pnp 00:0e: [mem 0x00000000-0x0009ffff]
[    2.342550] pnp 00:0e: [mem 0x000c0000-0x000cffff]
[    2.342551] pnp 00:0e: [mem 0x000e0000-0x000fffff]
[    2.342553] pnp 00:0e: [mem 0x00100000-0xdfffffff]
[    2.342554] pnp 00:0e: [mem 0xfec00000-0xffffffff]
[    2.342629] system 00:0e: [mem 0x00000000-0x0009ffff] could not be 
reserved
[    2.342660] system 00:0e: [mem 0x000c0000-0x000cffff] could not be 
reserved
[    2.342691] system 00:0e: [mem 0x000e0000-0x000fffff] could not be 
reserved
[    2.342723] system 00:0e: [mem 0x00100000-0xdfffffff] could not be 
reserved
[    2.342754] system 00:0e: [mem 0xfec00000-0xffffffff] could not be 
reserved
[    2.342785] system 00:0e: Plug and Play ACPI device, IDs PNP0c01 (active)
[    2.342835] pnp: PnP ACPI: found 15 devices
[    2.342865] ACPI: ACPI bus type pnp unregistered
[    2.345616] Switched to NOHz mode on CPU #0
[    2.345673] Switched to NOHz mode on CPU #3
[    2.345731] Switched to NOHz mode on CPU #1
[    2.345734] Switched to NOHz mode on CPU #2
[    2.345746] Switched to NOHz mode on CPU #4
[    2.345749] Switched to NOHz mode on CPU #5
[    2.349869] PCI: max bus depth: 1 pci_try_num: 2
[    2.349888] pci 0000:00:01.0: PCI bridge to [bus 01-01]
[    2.349919] pci 0000:00:01.0:   bridge window [io  0xa000-0xafff]
[    2.349951] pci 0000:00:01.0:   bridge window [mem 0xfe200000-0xfe3fffff]
[    2.349982] pci 0000:00:01.0:   bridge window [mem 
0xf0000000-0xf7ffffff 64bit pref]
[    2.350059] pci 0000:00:02.0: PCI bridge to [bus 04-04]
[    2.350090] pci 0000:00:02.0:   bridge window [io  0xc000-0xcfff]
[    2.350122] pci 0000:00:02.0:   bridge window [mem 0xfe600000-0xfe6fffff]
[    2.350153] pci 0000:00:02.0:   bridge window [mem pref disabled]
[    2.350185] pci 0000:00:03.0: PCI bridge to [bus 05-05]
[    2.350216] pci 0000:00:03.0:   bridge window [io  0xd000-0xdfff]
[    2.350249] pci 0000:00:03.0:   bridge window [mem 0xfe700000-0xfe7fffff]
[    2.350280] pci 0000:00:03.0:   bridge window [mem pref disabled]
[    2.350312] pci 0000:00:04.0: PCI bridge to [bus 07-07]
[    2.350343] pci 0000:00:04.0:   bridge window [io  0xe000-0xefff]
[    2.350375] pci 0000:00:04.0:   bridge window [mem 0xfe800000-0xfe9fffff]
[    2.350411] pci 0000:00:04.0:   bridge window [mem pref disabled]
[    2.350443] pci 0000:00:09.0: PCI bridge to [bus 02-02]
[    2.350473] pci 0000:00:09.0:   bridge window [io  disabled]
[    2.350504] pci 0000:00:09.0:   bridge window [mem 0xfe400000-0xfe4fffff]
[    2.350535] pci 0000:00:09.0:   bridge window [mem pref disabled]
[    2.350567] pci 0000:00:0a.0: PCI bridge to [bus 03-03]
[    2.350598] pci 0000:00:0a.0:   bridge window [io  0xb000-0xbfff]
[    2.350629] pci 0000:00:0a.0:   bridge window [mem 0xfe500000-0xfe5fffff]
[    2.350661] pci 0000:00:0a.0:   bridge window [mem 
0xfdf00000-0xfdffffff 64bit pref]
[    2.350694] pci 0000:00:14.4: PCI bridge to [bus 06-06]
[    2.350724] pci 0000:00:14.4:   bridge window [io  disabled]
[    2.350757] pci 0000:00:14.4:   bridge window [mem disabled]
[    2.350789] pci 0000:00:14.4:   bridge window [mem pref disabled]
[    2.350829] pci 0000:00:02.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    2.350861] pci 0000:00:02.0: setting latency timer to 64
[    2.350865] pci 0000:00:03.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
[    2.350897] pci 0000:00:03.0: setting latency timer to 64
[    2.350901] pci 0000:00:04.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    2.350932] pci 0000:00:04.0: setting latency timer to 64
[    2.350936] pci 0000:00:09.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[    2.350967] pci 0000:00:09.0: setting latency timer to 64
[    2.350970] pci 0000:00:0a.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    2.351001] pci 0000:00:0a.0: setting latency timer to 64
[    2.351006] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
[    2.351007] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
[    2.351009] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
[    2.351010] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfebfffff]
[    2.351012] pci_bus 0000:01: resource 0 [io  0xa000-0xafff]
[    2.351013] pci_bus 0000:01: resource 1 [mem 0xfe200000-0xfe3fffff]
[    2.351014] pci_bus 0000:01: resource 2 [mem 0xf0000000-0xf7ffffff 
64bit pref]
[    2.351016] pci_bus 0000:04: resource 0 [io  0xc000-0xcfff]
[    2.351017] pci_bus 0000:04: resource 1 [mem 0xfe600000-0xfe6fffff]
[    2.351019] pci_bus 0000:05: resource 0 [io  0xd000-0xdfff]
[    2.351020] pci_bus 0000:05: resource 1 [mem 0xfe700000-0xfe7fffff]
[    2.351021] pci_bus 0000:07: resource 0 [io  0xe000-0xefff]
[    2.351023] pci_bus 0000:07: resource 1 [mem 0xfe800000-0xfe9fffff]
[    2.351024] pci_bus 0000:02: resource 1 [mem 0xfe400000-0xfe4fffff]
[    2.351025] pci_bus 0000:03: resource 0 [io  0xb000-0xbfff]
[    2.351027] pci_bus 0000:03: resource 1 [mem 0xfe500000-0xfe5fffff]
[    2.351028] pci_bus 0000:03: resource 2 [mem 0xfdf00000-0xfdffffff 
64bit pref]
[    2.351030] pci_bus 0000:06: resource 4 [io  0x0000-0x0cf7]
[    2.351031] pci_bus 0000:06: resource 5 [io  0x0d00-0xffff]
[    2.351032] pci_bus 0000:06: resource 6 [mem 0x000a0000-0x000bffff]
[    2.351034] pci_bus 0000:06: resource 7 [mem 0xf0000000-0xfebfffff]
[    2.351057] NET: Registered protocol family 2
[    2.351135] IP route cache hash table entries: 524288 (order: 10, 
4194304 bytes)
[    2.352087] TCP established hash table entries: 262144 (order: 10, 
4194304 bytes)
[    2.353248] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    2.353567] TCP: Hash tables configured (established 262144 bind 65536)
[    2.353601] TCP reno registered
[    2.353632] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    2.353743] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    2.353926] NET: Registered protocol family 1
[    2.354056] RPC: Registered named UNIX socket transport module.
[    2.354087] RPC: Registered udp transport module.
[    2.354117] RPC: Registered tcp transport module.
[    2.354147] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    2.354182] pci 0000:00:01.0: MSI quirk detected; subordinate MSI 
disabled
[    2.500089] pci 0000:01:05.0: Boot video device
[    2.500118] PCI: CLS 64 bytes, default 64
[    2.500152] Unpacking initramfs...
[    2.539130] Freeing initrd memory: 3168k freed
[    2.539863] PCI-DMA: Disabling AGP.
[    2.540036] PCI-DMA: aperture base @ d0000000 size 65536 KB
[    2.540067] PCI-DMA: using GART IOMMU.
[    2.540098] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    2.574189] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    2.636100] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    2.636310] msgmni has been set to 31992
[    2.636457] Block layer SCSI generic (bsg) driver version 0.4 loaded 
(major 254)
[    2.636490] io scheduler noop registered
[    2.636519] io scheduler deadline registered
[    2.636606] io scheduler cfq registered (default)
[    2.636711] pcieport 0000:00:02.0: setting latency timer to 64
[    2.636730] pcieport 0000:00:02.0: irq 40 for MSI/MSI-X
[    2.636781] pcieport 0000:00:03.0: setting latency timer to 64
[    2.636795] pcieport 0000:00:03.0: irq 41 for MSI/MSI-X
[    2.636839] pcieport 0000:00:04.0: setting latency timer to 64
[    2.636854] pcieport 0000:00:04.0: irq 42 for MSI/MSI-X
[    2.636898] pcieport 0000:00:09.0: setting latency timer to 64
[    2.636912] pcieport 0000:00:09.0: irq 43 for MSI/MSI-X
[    2.636956] pcieport 0000:00:0a.0: setting latency timer to 64
[    2.636971] pcieport 0000:00:0a.0: irq 44 for MSI/MSI-X
[    2.637246] input: Power Button as 
/devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input0
[    2.637280] ACPI: Power Button [PWRB]
[    2.637353] input: Power Button as 
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[    2.637386] ACPI: Power Button [PWRF]
[    2.637446] ACPI: acpi_idle registered with cpuidle
[    2.658560] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    2.679110] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.699710] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[    2.720581] 00:0b: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.720829] Real Time Clock Driver v1.12b
[    2.721594] loop: module loaded
[    2.721752] tun: Universal TUN/TAP device driver, 1.6
[    2.721782] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    2.721866] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.721909] r8169 0000:03:00.0: PCI INT A -> GSI 18 (level, low) -> 
IRQ 18
[    2.721993] r8169 0000:03:00.0: setting latency timer to 64
[    2.722028] r8169 0000:03:00.0: irq 45 for MSI/MSI-X
[    2.722217] r8169 0000:03:00.0: eth0: RTL8168e/8111e at 
0xffffc9000000e000, 00:25:22:55:3e:2d, XID 0c100000 IRQ 45
[    2.722289] netconsole: local port 6666
[    2.722332] netconsole: local IP 192.168.2.1
[    2.722361] netconsole: interface 'eth0'
[    2.722391] netconsole: remote port 5050
[    2.722420] netconsole: remote IP 192.168.2.87
[    2.722451] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[    2.722482] netconsole: device eth0 not up yet, forcing it
[    3.540028] Refined TSC clocksource calibration: 3213.418 MHz.
[    3.540067] Switching to clocksource tsc
[   63.200106] r8169 0000:03:00.0: eth0: unable to load firmware patch 
rtl_nic/rtl8168e-1.fw (-2)
[   63.223480] r8169 0000:03:00.0: eth0: link down
[   63.223523] r8169 0000:03:00.0: eth0: link down
[   65.179915] r8169 0000:03:00.0: eth0: link up
[   65.182530] console [netcon0] enabled
[   65.182562] netconsole: network logging started
[   65.182738] i8042: PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
[   65.182771] i8042: PNP: PS/2 appears to have AUX port disabled, if 
this is incorrect please boot with i8042.nopnp
[   65.183304] serio: i8042 KBD port at 0x60,0x64 irq 1
[   65.183474] mousedev: PS/2 mouse device common for all mice
[   65.183532] i2c /dev entries driver
[   65.183588] md: linear personality registered for level -1
[   65.183619] md: raid0 personality registered for level 0
[   65.183651] md: raid1 personality registered for level 1
[   65.183683] md: raid10 personality registered for level 10
[   65.183714] md: raid6 personality registered for level 6
[   65.183746] md: raid5 personality registered for level 5
[   65.183779] md: raid4 personality registered for level 4
[   65.183968] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) 
initialised: dm-devel@redhat.com
[   65.184375] cpuidle: using governor ladder
[   65.185144] cpuidle: using governor menu
[   65.185339] TCP cubic registered
[   65.185377] NET: Registered protocol family 17
[   65.185431] Bridge firewalling registered
[   65.185468] 802.1Q VLAN Support v1.8
[   65.185502] Registering the dns_resolver key type
[   65.185730] registered taskstats version 1
[   65.216255] Freeing unused kernel memory: 468k freed
[   65.226393] udev[1318]: starting version 164
[   65.237596] input: AT Translated Set 2 keyboard as 
/devices/platform/i8042/serio0/input/input2
[   65.243627] megasas: 00.00.05.38-rc1 Wed. May. 11 17:00:00 PDT 2011
[   65.243694] megasas: 0x1000:0x0073:0x1014:0x03b1: bus 4:slot 0:func 0
[   65.243765] megaraid_sas 0000:04:00.0: PCI INT A -> GSI 18 (level, 
low) -> IRQ 18
[   65.243803] megaraid_sas 0000:04:00.0: setting latency timer to 64
[   65.244850] megasas: FW now in Ready state
[   65.245232] megaraid_sas 0000:04:00.0: irq 46 for MSI/MSI-X
[   65.246538] sata_mv 0000:07:00.0: version 1.28
[   65.246610] sata_mv 0000:07:00.0: PCI INT A -> GSI 16 (level, low) -> 
IRQ 16
[   65.246785] sata_mv 0000:07:00.0: Gen-IIE 32 slots 4 ports SCSI mode 
IRQ via INTx
[   65.246825] sata_mv 0000:07:00.0: setting latency timer to 64
[   65.247046] ahci 0000:00:11.0: version 3.0
[   65.247063] ahci 0000:00:11.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
[   65.247146] ahci 0000:00:11.0: irq 47 for MSI/MSI-X
[   65.247256] ahci 0000:00:11.0: AHCI 0001.0200 32 slots 6 ports 6 Gbps 
0x3f impl SATA mode
[   65.247293] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo 
pmp pio slum part
[   65.247385] usbcore: registered new interface driver usbfs
[   65.247443] usbcore: registered new interface driver hub
[   65.248070] scsi1 : sata_mv
[   65.248189] usbcore: registered new device driver usb
[   65.248450] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   65.248497] scsi3 : sata_mv
[   65.248728] scsi2 : ahci
[   65.249183] scsi4 : sata_mv
[   65.249270] scsi6 : sata_mv
[   65.249318] ata1: SATA max UDMA/133 mmio m1048576@0xfe900000 port 
0xfe922000 irq 16
[   65.249354] ata2: SATA max UDMA/133 mmio m1048576@0xfe900000 port 
0xfe924000 irq 16
[   65.249390] ata3: SATA max UDMA/133 mmio m1048576@0xfe900000 port 
0xfe926000 irq 16
[   65.249425] ata4: SATA max UDMA/133 mmio m1048576@0xfe900000 port 
0xfe928000 irq 16
[   65.249981] scsi5 : ahci
[   65.250068] scsi7 : ahci
[   65.250436] scsi8 : ahci
[   65.250534] scsi9 : ahci
[   65.250685] scsi10 : ahci
[   65.250746] ata5: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1ffd00 irq 47
[   65.250783] ata6: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1ffd80 irq 47
[   65.250820] ata7: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1ffe00 irq 47
[   65.250857] ata8: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1ffe80 irq 47
[   65.250895] ata9: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1fff00 irq 47
[   65.250933] ata10: SATA max UDMA/133 abar m1024@0xfe1ffc00 port 
0xfe1fff80 irq 47
[   65.250996] ehci_hcd 0000:00:12.2: PCI INT B -> GSI 17 (level, low) 
-> IRQ 17
[   65.251056] ehci_hcd 0000:00:12.2: EHCI Host Controller
[   65.251101] ehci_hcd 0000:00:12.2: new USB bus registered, assigned 
bus number 1
[   65.251148] ehci_hcd 0000:00:12.2: applying AMD 
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   65.251199] QUIRK: Enable AMD PLL fix
[   65.251208] ehci_hcd 0000:00:12.2: debug port 1
[   65.251257] ehci_hcd 0000:00:12.2: irq 17, io mem 0xfe1ff800
[   65.270037] ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
[   65.270214] hub 1-0:1.0: USB hub found
[   65.270254] hub 1-0:1.0: 5 ports detected
[   65.270351] ehci_hcd 0000:00:13.2: PCI INT B -> GSI 17 (level, low) 
-> IRQ 17
[   65.270404] ehci_hcd 0000:00:13.2: EHCI Host Controller
[   65.270454] ehci_hcd 0000:00:13.2: new USB bus registered, assigned 
bus number 2
[   65.270496] ehci_hcd 0000:00:13.2: applying AMD 
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   65.270551] ehci_hcd 0000:00:13.2: debug port 1
[   65.270595] ehci_hcd 0000:00:13.2: irq 17, io mem 0xfe1ff400
[   65.270605] megasas_init_mfi: fw_support_ieee=67108864
[   65.270606] megasas: INIT adapter done
[   65.290144] ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
[   65.290258] hub 2-0:1.0: USB hub found
[   65.290292] hub 2-0:1.0: 5 ports detected
[   65.290362] ehci_hcd 0000:00:16.2: PCI INT B -> GSI 17 (level, low) 
-> IRQ 17
[   65.290407] ehci_hcd 0000:00:16.2: EHCI Host Controller
[   65.290440] ehci_hcd 0000:00:16.2: new USB bus registered, assigned 
bus number 3
[   65.290478] ehci_hcd 0000:00:16.2: applying AMD 
SB700/SB800/Hudson-2/3 EHCI dummy qh workaround
[   65.290533] ehci_hcd 0000:00:16.2: debug port 1
[   65.290573] ehci_hcd 0000:00:16.2: irq 17, io mem 0xfe1ff000
[   65.320044] ehci_hcd 0000:00:16.2: USB 2.0 started, EHCI 1.00
[   65.320151] hub 3-0:1.0: USB hub found
[   65.320184] hub 3-0:1.0: 4 ports detected
[   65.348414] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[   65.348479] ohci_hcd 0000:00:12.0: PCI INT A -> GSI 18 (level, low) 
-> IRQ 18
[   65.348539] ohci_hcd 0000:00:12.0: OHCI Host Controller
[   65.348574] ohci_hcd 0000:00:12.0: new USB bus registered, assigned 
bus number 4
[   65.348629] ohci_hcd 0000:00:12.0: irq 18, io mem 0xfe1fe000
[   65.390058] scsi0 : LSI SAS based MegaRAID driver
[   65.402010] scsi 0:0:10:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.402165] sd 0:0:10:0: Attached scsi generic sg0 type 0
[   65.404137] hub 4-0:1.0: USB hub found
[   65.404173] hub 4-0:1.0: 5 ports detected
[   65.404224] ohci_hcd 0000:00:13.0: PCI INT A -> GSI 18 (level, low) 
-> IRQ 18
[   65.404265] ohci_hcd 0000:00:13.0: OHCI Host Controller
[   65.404298] ohci_hcd 0000:00:13.0: new USB bus registered, assigned 
bus number 5
[   65.404342] ohci_hcd 0000:00:13.0: irq 18, io mem 0xfe1fd000
[   65.408855] sd 0:0:10:0: [sda] physical block alignment offset: 4096
[   65.408893] sd 0:0:10:0: [sda] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.408928] sd 0:0:10:0: [sda] 4096-byte physical blocks
[   65.410683] scsi 0:0:11:0: Direct-Access     ATA      WDC 
WD20EARS-00M AB51 PQ: 0 ANSI: 5
[   65.410807] sd 0:0:11:0: Attached scsi generic sg1 type 0
[   65.415759] sd 0:0:11:0: [sdb] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.423407] scsi 0:0:12:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.423536] sd 0:0:12:0: Attached scsi generic sg2 type 0
[   65.430537] sd 0:0:12:0: [sdc] physical block alignment offset: 4096
[   65.430574] sd 0:0:12:0: [sdc] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.430608] sd 0:0:12:0: [sdc] 4096-byte physical blocks
[   65.437120] sd 0:0:11:0: [sdb] Write Protect is off
[   65.437156] sd 0:0:11:0: [sdb] Mode Sense: 73 00 00 08
[   65.437730] sd 0:0:10:0: [sda] Write Protect is off
[   65.437766] sd 0:0:10:0: [sda] Mode Sense: 73 00 00 08
[   65.441716] scsi 0:0:13:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.441835] sd 0:0:13:0: Attached scsi generic sg3 type 0
[   65.444683] sd 0:0:11:0: [sdb] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.445330] scsi 0:0:18:0: Direct-Access     SEAGATE  ST3300655SS 
   S515 PQ: 0 ANSI: 5
[   65.445467] sd 0:0:18:0: Attached scsi generic sg4 type 0
[   65.451625] sd 0:0:10:0: [sda] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.451711] sd 0:0:13:0: [sdd] physical block alignment offset: 4096
[   65.451747] sd 0:0:13:0: [sdd] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.451790] sd 0:0:13:0: [sdd] 4096-byte physical blocks
[   65.452285] sd 0:0:18:0: [sde] 585937500 512-byte logical blocks: 
(300 GB/279 GiB)
[   65.453321] scsi 0:0:19:0: Direct-Access     SEAGATE  ST3300655SS 
   A005 PQ: 0 ANSI: 5
[   65.453456] sd 0:0:19:0: Attached scsi generic sg5 type 0
[   65.458128] sd 0:0:18:0: [sde] Write Protect is off
[   65.458163] sd 0:0:18:0: [sde] Mode Sense: b3 00 10 08
[   65.459325] sd 0:0:19:0: [sdf] 585937500 512-byte logical blocks: 
(300 GB/279 GiB)
[   65.459448] scsi 0:0:20:0: Direct-Access     SEAGATE  ST3300655SS 
   A005 PQ: 0 ANSI: 5
[   65.459583] sd 0:0:20:0: Attached scsi generic sg6 type 0
[   65.464128] hub 5-0:1.0: USB hub found
[   65.464165] hub 5-0:1.0: 5 ports detected
[   65.464214] ohci_hcd 0000:00:14.5: PCI INT C -> GSI 18 (level, low) 
-> IRQ 18
[   65.464255] ohci_hcd 0000:00:14.5: OHCI Host Controller
[   65.464288] ohci_hcd 0000:00:14.5: new USB bus registered, assigned 
bus number 6
[   65.464331] ohci_hcd 0000:00:14.5: irq 18, io mem 0xfe1fc000
[   65.464611] sd 0:0:18:0: [sde] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[   65.465337] sd 0:0:20:0: [sdg] 585937500 512-byte logical blocks: 
(300 GB/279 GiB)
[   65.465748] scsi 0:0:21:0: Direct-Access     SEAGATE  ST3300655SS 
   S515 PQ: 0 ANSI: 5
[   65.465906] sd 0:0:21:0: Attached scsi generic sg7 type 0
[   65.470327] sd 0:0:19:0: [sdf] Write Protect is off
[   65.470362] sd 0:0:19:0: [sdf] Mode Sense: b3 00 10 08
[   65.476731] sd 0:0:21:0: [sdh] 585937500 512-byte logical blocks: 
(300 GB/279 GiB)
[   65.476846] sd 0:0:20:0: [sdg] Write Protect is off
[   65.476892] sd 0:0:20:0: [sdg] Mode Sense: b3 00 10 08
[   65.476916] sd 0:0:19:0: [sdf] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[   65.482548] sd 0:0:21:0: [sdh] Write Protect is off
[   65.482586] sd 0:0:21:0: [sdh] Mode Sense: b3 00 10 08
[   65.483436] sd 0:0:20:0: [sdg] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[   65.488248] sd 0:0:21:0: [sdh] Write cache: enabled, read cache: 
enabled, supports DPO and FUA
[   65.494395]  sde: unknown partition table
[   65.501262]  sdb: unknown partition table
[   65.503732] sd 0:0:12:0: [sdc] Write Protect is off
[   65.503776] sd 0:0:12:0: [sdc] Mode Sense: 73 00 00 08
[   65.505389]  sdf: unknown partition table
[   65.506463] sd 0:0:18:0: [sde] Attached SCSI disk
[   65.511538]  sdh: unknown partition table
[   65.512582]  sda: unknown partition table
[   65.519104] sd 0:0:12:0: [sdc] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.521490]  sdg: unknown partition table
[   65.524125] hub 6-0:1.0: USB hub found
[   65.524162] hub 6-0:1.0: 2 ports detected
[   65.524211] ohci_hcd 0000:00:16.0: PCI INT A -> GSI 18 (level, low) 
-> IRQ 18
[   65.524252] ohci_hcd 0000:00:16.0: OHCI Host Controller
[   65.524285] ohci_hcd 0000:00:16.0: new USB bus registered, assigned 
bus number 7
[   65.524332] ohci_hcd 0000:00:16.0: irq 18, io mem 0xfe1f7000
[   65.526953] sd 0:0:19:0: [sdf] Attached SCSI disk
[   65.529657] sd 0:0:21:0: [sdh] Attached SCSI disk
[   65.531531] sd 0:0:13:0: [sdd] Write Protect is off
[   65.531569] sd 0:0:13:0: [sdd] Mode Sense: 73 00 00 08
[   65.532606] sd 0:0:20:0: [sdg] Attached SCSI disk
[   65.537818] megasas: 0x1000:0x0073:0x1014:0x03b1: bus 5:slot 0:func 0
[   65.537887] megaraid_sas 0000:05:00.0: PCI INT A -> GSI 19 (level, 
low) -> IRQ 19
[   65.537923] megaraid_sas 0000:05:00.0: setting latency timer to 64
[   65.537974] megasas: FW now in Ready state
[   65.538034] megaraid_sas 0000:05:00.0: irq 48 for MSI/MSI-X
[   65.543918] sd 0:0:13:0: [sdd] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.545106] sd 0:0:11:0: [sdb] Attached SCSI disk
[   65.560038] megasas_init_mfi: fw_support_ieee=67108864
[   65.560079] megasas: INIT adapter done
[   65.563406] sd 0:0:10:0: [sda] Attached SCSI disk
[   65.570045]  sdc: unknown partition table
[   65.584124] hub 7-0:1.0: USB hub found
[   65.584161] hub 7-0:1.0: 4 ports detected
[   65.593514]  sdd: unknown partition table
[   65.600047] ata9: SATA link down (SStatus 0 SControl 300)
[   65.600107] ata7: SATA link down (SStatus 0 SControl 300)
[   65.600170] ata8: SATA link down (SStatus 0 SControl 300)
[   65.600261] ata10: SATA link down (SStatus 0 SControl 300)
[   65.623341] sd 0:0:12:0: [sdc] Attached SCSI disk
[   65.642020] sd 0:0:13:0: [sdd] Attached SCSI disk
[   65.650037] scsi11 : LSI SAS based MegaRAID driver
[   65.661988] scsi 11:0:9:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.662113] sd 11:0:9:0: Attached scsi generic sg8 type 0
[   65.668949] sd 11:0:9:0: [sdi] physical block alignment offset: 4096
[   65.668986] sd 11:0:9:0: [sdi] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.669020] sd 11:0:9:0: [sdi] 4096-byte physical blocks
[   65.675059] scsi 11:0:10:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.675182] sd 11:0:10:0: Attached scsi generic sg9 type 0
[   65.681812] sd 11:0:10:0: [sdj] physical block alignment offset: 4096
[   65.681849] sd 11:0:10:0: [sdj] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.681884] sd 11:0:10:0: [sdj] 4096-byte physical blocks
[   65.698957] sd 11:0:9:0: [sdi] Write Protect is off
[   65.698993] sd 11:0:9:0: [sdi] Mode Sense: 73 00 00 08
[   65.709639] scsi 11:0:13:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.709759] sd 11:0:13:0: Attached scsi generic sg10 type 0
[   65.711168] sd 11:0:9:0: [sdi] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.716961] sd 11:0:13:0: [sdk] physical block alignment offset: 4096
[   65.716998] sd 11:0:13:0: [sdk] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.717035] sd 11:0:13:0: [sdk] 4096-byte physical blocks
[   65.742927] scsi 11:0:14:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.743040] sd 11:0:14:0: Attached scsi generic sg11 type 0
[   65.747118] sd 11:0:13:0: [sdk] Write Protect is off
[   65.747153] sd 11:0:13:0: [sdk] Mode Sense: 73 00 00 08
[   65.749113] sd 11:0:14:0: [sdl] physical block alignment offset: 4096
[   65.749150] sd 11:0:14:0: [sdl] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.749184] sd 11:0:14:0: [sdl] 4096-byte physical blocks
[   65.749947] sd 11:0:10:0: [sdj] Write Protect is off
[   65.749982] sd 11:0:10:0: [sdj] Mode Sense: 73 00 00 08
[   65.750068] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   65.759269] sd 11:0:13:0: [sdk] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.759275]  sdi: unknown partition table
[   65.761631] sd 11:0:10:0: [sdj] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.777351] sd 11:0:14:0: [sdl] Write Protect is off
[   65.777386] sd 11:0:14:0: [sdl] Mode Sense: 73 00 00 08
[   65.787708] sd 11:0:14:0: [sdl] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.790285] ata1.00: ATA-8: FM-25S2S-60GBP2, 2.1, max UDMA/133
[   65.790321] ata1.00: 117231408 sectors, multi 1: LBA48 NCQ (depth 31/32)
[   65.800049] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   65.800102] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   65.803492] scsi 11:0:15:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.803611] sd 11:0:15:0: Attached scsi generic sg12 type 0
[   65.807891] sd 11:0:9:0: [sdi] Attached SCSI disk
[   65.808134]  sdj: unknown partition table
[   65.808954] ata5.00: ATA-8: WDC WD1002FAEX-00Z3A0, 05.01D05, max UDMA/133
[   65.808989] ata5.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 
31/32), AA
[   65.809689]  sdk: unknown partition table
[   65.810820] sd 11:0:15:0: [sdm] physical block alignment offset: 4096
[   65.810865] sd 11:0:15:0: [sdm] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.810902] sd 11:0:15:0: [sdm] 4096-byte physical blocks
[   65.811042] ata5.00: configured for UDMA/133
[   65.812159] ata6.00: ATA-8: WDC WD1002FAEX-00Z3A0, 05.01D05, max UDMA/133
[   65.812194] ata6.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 
31/32), AA
[   65.814325] ata6.00: configured for UDMA/133
[   65.833497]  sdl: unknown partition table
[   65.841554] sd 11:0:15:0: [sdm] Write Protect is off
[   65.841590] sd 11:0:15:0: [sdm] Mode Sense: 73 00 00 08
[   65.850283] ata1.00: configured for UDMA/133
[   65.850387] scsi 1:0:0:0: Direct-Access     ATA      FM-25S2S-60GBP2 
  2.1  PQ: 0 ANSI: 5
[   65.850502] sd 1:0:0:0: Attached scsi generic sg13 type 0
[   65.850529] sd 1:0:0:0: [sdn] 117231408 512-byte logical blocks: 
(60.0 GB/55.8 GiB)
[   65.850653] sd 1:0:0:0: [sdn] Write Protect is off
[   65.850687] sd 1:0:0:0: [sdn] Mode Sense: 00 3a 00 00
[   65.850712] sd 1:0:0:0: [sdn] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.851123]  sdn: unknown partition table
[   65.851320] sd 1:0:0:0: [sdn] Attached SCSI disk
[   65.853501] sd 11:0:15:0: [sdm] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   65.853583] sd 11:0:10:0: [sdj] Attached SCSI disk
[   65.860607] sd 11:0:13:0: [sdk] Attached SCSI disk
[   65.876691] sd 11:0:14:0: [sdl] Attached SCSI disk
[   65.890059] usb 4-2: new full speed USB device number 2 using ohci_hcd
[   65.902817]  sdm: unknown partition table
[   65.921026] scsi 11:0:16:0: Direct-Access     ATA      WDC 
WD20EARS-60M AB51 PQ: 0 ANSI: 5
[   65.921149] sd 11:0:16:0: Attached scsi generic sg14 type 0
[   65.928106] sd 11:0:16:0: [sdo] physical block alignment offset: 4096
[   65.928143] sd 11:0:16:0: [sdo] 3907029168 512-byte logical blocks: 
(2.00 TB/1.81 TiB)
[   65.928177] sd 11:0:16:0: [sdo] 4096-byte physical blocks
[   65.953117] sd 11:0:15:0: [sdm] Attached SCSI disk
[   66.030173] sd 11:0:16:0: [sdo] Write Protect is off
[   66.030211] sd 11:0:16:0: [sdo] Mode Sense: 73 00 00 08
[   66.043679] sd 11:0:16:0: [sdo] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   66.092356]  sdo: unknown partition table
[   66.182327] sd 11:0:16:0: [sdo] Attached SCSI disk
[   66.200059] ata2: SATA link down (SStatus 0 SControl 300)
[   66.550061] ata3: SATA link down (SStatus 0 SControl 300)
[   66.900053] ata4: SATA link down (SStatus 0 SControl 300)
[   66.900198] scsi 2:0:0:0: Direct-Access     ATA      WDC WD1002FAEX-0 
05.0 PQ: 0 ANSI: 5
[   66.900325] sd 2:0:0:0: [sdp] 1953525168 512-byte logical blocks: 
(1.00 TB/931 GiB)
[   66.900327] sd 2:0:0:0: Attached scsi generic sg15 type 0
[   66.900427] scsi 5:0:0:0: Direct-Access     ATA      WDC WD1002FAEX-0 
05.0 PQ: 0 ANSI: 5
[   66.900464] sd 2:0:0:0: [sdp] Write Protect is off
[   66.900467] sd 2:0:0:0: [sdp] Mode Sense: 00 3a 00 00
[   66.900491] sd 2:0:0:0: [sdp] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   66.900643] sd 5:0:0:0: [sdq] 1953525168 512-byte logical blocks: 
(1.00 TB/931 GiB)
[   66.900645] sd 5:0:0:0: Attached scsi generic sg16 type 0
[   66.900773] sd 5:0:0:0: [sdq] Write Protect is off
[   66.900808] sd 5:0:0:0: [sdq] Mode Sense: 00 3a 00 00
[   66.900829] sd 5:0:0:0: [sdq] Write cache: enabled, read cache: 
enabled, doesn't support DPO or FUA
[   66.904434]  sdq: sdq1 sdq2
[   66.904681] sd 5:0:0:0: [sdq] Attached SCSI disk
[   66.909577]  sdp: sdp1 sdp2
[   66.909810] sd 2:0:0:0: [sdp] Attached SCSI disk
[   67.191179] md: md1 stopped.
[   67.191764] md: bind<sdq1>
[   67.191895] md: bind<sdp1>
[   67.192591] bio: create slab <bio-1> at 1
[   67.192670] md/raid1:md1: active with 2 out of 2 mirrors
[   67.192714] md1: detected capacity change from 0 to 524222464
[   67.200588]  md1: unknown partition table
[   67.207511] md: md2 stopped.
[   67.208237] md: bind<sdq2>
[   67.209779] md: bind<sdp2>
[   67.210543] md/raid10:md2: active with 2 out of 2 devices
[   67.210673] created bitmap (8 pages) for device md2
[   67.210934] md2: bitmap initialized from disk: read 1/1 pages, set 2 
of 14897 bits
[   67.219474] md2: detected capacity change from 0 to 999677755392
[   67.224116]  md2: p1 p2 p3
[   67.230750] md: md10 stopped.
[   67.244399] md: bind<sde>
[   67.244619] md: bind<sdf>
[   67.244823] md: bind<sdg>
[   67.248205] md: bind<sdh>
[   67.248955] md/raid10:md10: active with 4 out of 4 devices
[   67.252191] created bitmap (5 pages) for device md10
[   67.252505] md10: bitmap initialized from disk: read 1/1 pages, set 0 
of 8941 bits
[   67.264422] md10: detected capacity change from 0 to 599997284352
[   67.271735]  md10: unknown partition table
[   67.296795] md: md0 stopped.
[   67.398685] md: bind<sdd>
[   67.404514] md: bind<sdm>
[   67.404703] md: bind<sdb>
[   67.408534] md: bind<sdj>
[   67.411132] md: bind<sdi>
[   67.411724] md: bind<sdl>
[   67.430316] md: bind<sda>
[   67.430607] md: bind<sdc>
[   67.430770] md: bind<sdk>
[   67.430922] md: bind<sdo>
[   67.431657] md/raid:md0: device sdo operational as raid disk 0
[   67.431694] md/raid:md0: device sdk operational as raid disk 9
[   67.431726] md/raid:md0: device sdc operational as raid disk 8
[   67.431758] md/raid:md0: device sda operational as raid disk 7
[   67.431793] md/raid:md0: device sdl operational as raid disk 6
[   67.431824] md/raid:md0: device sdi operational as raid disk 5
[   67.431856] md/raid:md0: device sdj operational as raid disk 4
[   67.431891] md/raid:md0: device sdb operational as raid disk 3
[   67.431925] md/raid:md0: device sdm operational as raid disk 2
[   67.431956] md/raid:md0: device sdd operational as raid disk 1
[   67.432356] md/raid:md0: allocated 10572kB
[   67.432418] md/raid:md0: raid level 6 active with 10 out of 10 
devices, algorithm 2
[   67.432457] RAID conf printout:
[   67.432458]  --- level:6 rd:10 wd:10
[   67.432459]  disk 0, o:1, dev:sdo
[   67.432461]  disk 1, o:1, dev:sdd
[   67.432462]  disk 2, o:1, dev:sdm
[   67.432463]  disk 3, o:1, dev:sdb
[   67.432463]  disk 4, o:1, dev:sdj
[   67.432464]  disk 5, o:1, dev:sdi
[   67.432465]  disk 6, o:1, dev:sdl
[   67.432466]  disk 7, o:1, dev:sda
[   67.432467]  disk 8, o:1, dev:sdc
[   67.432468]  disk 9, o:1, dev:sdk
[   67.466604] created bitmap (15 pages) for device md0
[   67.466953] md0: bitmap initialized from disk: read 1/1 pages, set 0 
of 29809 bits
[   67.472708] md0: detected capacity change from 0 to 16003181314048
[   67.511792]  md0: unknown partition table
[   67.680098] EXT4-fs (md2p1): INFO: recovery required on readonly 
filesystem
[   67.680135] EXT4-fs (md2p1): write access will be enabled during recovery
[   69.810256] EXT4-fs (md2p1): recovery complete
[   69.810604] EXT4-fs (md2p1): mounted filesystem with ordered data 
mode. Opts: (null)
[   70.548329] udev[1803]: starting version 164
[   70.676012] usbcore: registered new interface driver usbserial
[   70.676059] USB Serial support registered for generic
[   70.676103] usbcore: registered new interface driver usbserial_generic
[   70.676147] usbserial: USB Serial Driver core
[   70.683832] USB Serial support registered for pl2303
[   70.683881] pl2303 4-2:1.0: pl2303 converter detected
[   70.698118] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, 
revision 0
[   70.701760] xhci_hcd 0000:02:00.0: PCI INT A -> GSI 17 (level, low) 
-> IRQ 17
[   70.701834] xhci_hcd 0000:02:00.0: setting latency timer to 64
[   70.701837] xhci_hcd 0000:02:00.0: xHCI Host Controller
[   70.701877] xhci_hcd 0000:02:00.0: new USB bus registered, assigned 
bus number 8
[   70.702046] xhci_hcd 0000:02:00.0: irq 17, io mem 0xfe4fe000
[   70.702134] xhci_hcd 0000:02:00.0: irq 49 for MSI/MSI-X
[   70.702138] xhci_hcd 0000:02:00.0: irq 50 for MSI/MSI-X
[   70.702140] xhci_hcd 0000:02:00.0: irq 51 for MSI/MSI-X
[   70.702142] xhci_hcd 0000:02:00.0: irq 52 for MSI/MSI-X
[   70.702144] xhci_hcd 0000:02:00.0: irq 53 for MSI/MSI-X
[   70.702146] xhci_hcd 0000:02:00.0: irq 54 for MSI/MSI-X
[   70.702148] xhci_hcd 0000:02:00.0: irq 55 for MSI/MSI-X
[   70.702325] xHCI xhci_add_endpoint called for root hub
[   70.702327] xHCI xhci_check_bandwidth called for root hub
[   70.702347] hub 8-0:1.0: USB hub found
[   70.702384] hub 8-0:1.0: 2 ports detected
[   70.702526] xhci_hcd 0000:02:00.0: xHCI Host Controller
[   70.702567] xhci_hcd 0000:02:00.0: new USB bus registered, assigned 
bus number 9
[   70.704246] usb 4-2: pl2303 converter now attached to ttyUSB0
[   70.704320] usbcore: registered new interface driver pl2303
[   70.704367] pl2303: Prolific PL2303 USB to serial adaptor driver
[   70.705493] xHCI xhci_add_endpoint called for root hub
[   70.705495] xHCI xhci_check_bandwidth called for root hub
[   70.705526] hub 9-0:1.0: USB hub found
[   70.705569] hub 9-0:1.0: 2 ports detected
[   71.350688] EXT4-fs (md2p1): re-mounted. Opts: (null)
[   71.586339] EXT4-fs (md2p1): re-mounted. Opts: 
errors=remount-ro,commit=30
[   71.688584] kvm: Nested Virtualization enabled
[   71.688624] kvm: Nested Paging enabled
[   71.715947] powernow-k8: Found 1 AMD Phenom(tm) II X6 1090T Processor 
(6 cpu cores) (version 2.20.00)
[   71.715994] powernow-k8: Core Performance Boosting: on.
[   71.716050] powernow-k8:    0 : pstate 0 (3200 MHz)
[   71.716081] powernow-k8:    1 : pstate 1 (2400 MHz)
[   71.716112] powernow-k8:    2 : pstate 2 (1600 MHz)
[   71.716144] powernow-k8:    3 : pstate 3 (800 MHz)
[   71.741872] w83627ehf: Found NCT6775F chip at 0x290
[  113.081227] fuse init (API version 7.16)
[  113.463322] EXT4-fs (md2p3): mounted filesystem with ordered data 
mode. Opts: commit=30
[  113.523249] EXT4-fs (md2p2): mounted filesystem with ordered data 
mode. Opts: commit=30
[  113.603188] EXT4-fs (md10): mounted filesystem with ordered data 
mode. Opts: commit=30
[  114.034563] EXT4-fs (md0): mounted filesystem with ordered data mode. 
Opts: commit=30,acl
[  116.104972] Adding 20971516k swap on /raid10/SWAP/srv.swap. 
Priority:-1 extents:13 across:22421500k
[  116.578585] NET: Registered protocol family 15
[  138.220721] device br1_dummy entered promiscuous mode
[  180.833074] u32 classifier
[  180.833129]     Actions configured
[  191.222339] PPP generic driver version 2.4.2
[  192.068503] NET: Registered protocol family 24
[  192.759974] ip_tables: (C) 2000-2006 Netfilter Core Team
[  192.772071] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[  193.055922] HTB: quantum of class 10040 is small. Consider r2q change.
[  204.360085] sshd (3856): /proc/3856/oom_adj is deprecated, please use 
/proc/3856/oom_score_adj instead.
[  212.869525] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state 
recovery directory
[  212.875925] NFSD: starting 90-second grace period

brad@srv:~$ lsmod
Module                  Size  Used by
nfs                   294438  1
xt_length               1200  2
xt_CLASSIFY             1075  5
sch_sfq                 6676  2
xt_CHECKSUM             1213  2
iptable_mangle          1471  1
xt_TCPMSS               2551  2
ipt_REJECT              2229  1
ipt_MASQUERADE          1743  10
ipt_REDIRECT            1117  1
xt_recent               8207  2
xt_state                1210  6
xt_tcpudp               2451  28
iptable_filter          1426  1
iptable_nat             3977  1
nf_nat                 16789  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11852  9 iptable_nat,nf_nat
nf_conntrack           60821  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1401  1 nf_conntrack_ipv4
ip_tables              16538  3 iptable_mangle,iptable_filter,iptable_nat
x_tables               20342  14 
xt_length,xt_CLASSIFY,xt_CHECKSUM,iptable_mangle,xt_TCPMSS,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,xt_tcpudp,iptable_filter,iptable_nat,ip_tables
pppoe                   9558  2
pppox                   2156  1 pppoe
ppp_generic            24227  6 pppoe,pppox
slhc                    5277  1 ppp_generic
cls_u32                 6356  6
sch_htb                14352  2
deflate                 1921  0
zlib_deflate           21196  1 deflate
des_generic            16119  0
cbc                     2705  0
ecb                     1959  0
crypto_blkcipher       13629  2 cbc,ecb
sha1_generic            2079  0
md5                     3985  0
hmac                    2961  0
crypto_hash            14471  3 sha1_generic,md5,hmac
cryptomgr               2652  0
aead                    6121  1 cryptomgr
crypto_algapi          15273  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27356  0
fuse                   66684  1
w83627ehf              32036  0
hwmon_vid               2851  1 w83627ehf
vhost_net              17239  5
powernow_k8            12884  0
mperf                   1247  1 powernow_k8
kvm_amd                54109  20
kvm                   238605  1 kvm_amd
xhci_hcd               66960  0
k10temp                 3199  0
i2c_piix4               8375  0
pl2303                 12716  1
usbserial              34420  3 pl2303
ohci_hcd               17897  0
ehci_hcd               33721  0
usbcore               131192  5 xhci_hcd,pl2303,usbserial,ohci_hcd,ehci_hcd
ahci                   20732  4
libahci                21186  1 ahci
sata_mv                26923  0
megaraid_sas           71785  14