Re: KVM induced panic on 2.6.38[2367] & 2.6.39

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Borislav Petkov]

Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
machine from oopsing.

Adding linux-mm.

On Tue, May 31, 2011 at 09:24:03AM +0800, Brad Campbell wrote:
> G'day all,
> 
> I'm running a pretty standard home server
> x86_64 Phenom-II 6 Core
> 16GB DDR 3
> 
> I run some virtual machines under that. 3 x Debian 64 Bit, 1 x XP 32
> Bit. These run at boot.
> 
> When I fire up another XP 32 bit instance and play with it for more
> than about 2 minutes, I get the panics included in this mail.
> 
> I've included three of them here. The first and third are as booted.
> The second was with ksmd disabled just as a data point.
> 
> The machine passes every load test and memory test I can throw at
> it, but I still can't rule out this being a hardware issue.
> 
> Provided I don't start this XP VM the machine is quite stable, but
> running this VM will kill it within minutes.
> 
> This was tested with qemu-kvm.
> The last commit in the git tree was
> commit c007db193eb6b2557acb5caf2dc4d7023639e6f3
> Author: Avi Kivity <avi@redhat.com>
> Date:   Sun May 29 09:00:42 2011 -0400
> (I pulled it yesterday)
> 
> These panics were captured with netconsole to a remote syslog
> daemon, and the formatting was ruined, so I've reformatted them by
> hand prior to posting.
> 
> I've tested and reproduced this on 2.6.38.[2,3,6 & 7] and 2.6.39 obviously.
> 
> Can anyone help shed some light on this?
> 
> Regards,
> Brad
> 
> [ 438.632061] general protection fault: 0000 [#1] SMP
> [ 438.632196] last sysfs file: /sys/module/x_tables/initstate
> [ 438.632242] CPU 4
> [ 438.632282] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generic cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs= vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd k10temp
> i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd usbcore ahci
> libahci r8169 mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 438.634960]
> [ 438.635006] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 438.635170] RIP: 0010:[<ffffffff810b4596>] [<ffffffff810b4596>]
> remove_rmap_item_from_tree+0x96/0x150
> [ 438.635268] RSP: 0018:ffff88041c065e20  EFLAGS: 00010282
> [ 438.635314] RAX: ffff8804153bd8b0 RBX: ffff8804176c3fc0 RCX:
> 0000000000057754
> [ 438.635362] RDX: 0000880415418030 RSI: ffff880414b65003 RDI:
> ffffea000dde6030
> [ 438.635410] RBP: ffff880414b65000 R08: 0000000000057755 R09:
> 000000001bbde1bd
> [ 438.635458] R10: 000000001c28e0c3 R11: 0000000000000002 R12:
> ffffea000dde6030
> [ 438.635506] R13: ffff8804176c3f80 R14: ffff8804151ed7b0 R15:
> ffff88041bf23be0
> [ 438.635555] FS:  00007f617772e700(0000) GS:ffff88041fd00000(0000)
> knlGS:0000000000000000
> [ 438.635607] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 438.635654] CR2: 0000000000e70000 CR3: 0000000001583000 CR4:
> 00000000000006e0
> [ 438.635703] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 438.635750] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 438.635799] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d8caa70)
> [ 438.635851] Stack:
> [ 438.635893]  ffffea000c43c408 ffff8804176c3fc0 000000000000036c
> ffffffff810b58f2
> [ 438.636088]  ffff88041c782a00 00007fc2d81b5000 ffff88041c064000
> 00000000003808ff
> [ 438.636281]  ffff88041c064000 ffff88041c065e98 ffff88041d8caa70
> ffff8804165d4480
> [ 438.636479] Call Trace:
> [ 438.636528]  [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.636580]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.636628]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636679]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.636730]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.636781]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.636832]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.636882]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.636926] Code: 28 48 89 ef e8 6c fe ff ff 48 85 c0 49 89 c4 74
> d2 f0 0f ba 28 00 19 c0 85 c0 0f 85 ae 00 00 00 00 48 8b 43 30 48 8b
> 53 38 48 85 c0
> [ 438.638504] 89 02 74 04 48 89 50 08 48 ba 00 01 10 00 00 00 00 ad de 48 b8
> [ 438.639329] RIP [<ffffffff810b4596>] remove_rmap_item_from_tree+0x96/0x150
> [ 438.639414]  RSP <ffff88041c065e20>
> [ 438.639460] ---[ end trace c29fb871f6b874e3 ]---
> [ 438.639506] Kernel panic - not syncing: Fatal exception
> [ 438.639553] Pid: 551, comm: ksmd Tainted: G      D     2.6.39 #3
> [ 438.639598] Call Trace:
> [ 438.639644]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 438.639693]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 438.639745]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 438.639794]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 438.639842]  [<ffffffff810b4596>] ? remove_rmap_item_from_tree+0x96/0x150
> [ 438.639891]  [<ffffffff810b58f2>] ? ksm_scan_thread+0x4e2/0xc20
> [ 438.639941]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 438.640045]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640096]  [<ffffffff810b5410>] ? try_to_merge_with_ksm_page+0x570/0x570
> [ 438.640147]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 438.640196]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 438.640247]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 438.640297]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 438.640349] Rebooting in 60 seconds..
> 
> 
> 
> [ 469.706013] general protection fault: 0000 [#1] SMP
> [ 469.706125] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 469.706162] CPU 1
> [ 469.706195] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpmss
> xt_tcpudp iptable_mangle ip_tables x_tables pppoe pppox ppp_generic
> slhc cls_u32 sch_htb deflate zlib_deflate des_generi cbc ecb
> crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr aead
> crypto_algapi af_key fuse netconsole configfs vhost_net powernow_k8
> mperf i2c_nforce2 kvm_amd kvm pl2303 usbserial xhci_hcd i2c_piix4
> k10temp ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169
> libahci usbcore mii sata_mv megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 469.708573]
> [ 469.708608] Pid: 4942, comm: runbot Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 469.708753] RIP: 0010:[<ffffffff810db878>] [<ffffffff810db878>]
> dup_fd+0x168/0x300
> [ 469.708829] RSP: 0018:ffff880417279dd0  EFLAGS: 00010202
> [ 469.708870] RAX: 00000000000007f8 RBX: ffff88041c1658c0 RCX:
> bfffffffffffffff
> [ 469.708907] RDX: 0000880413155540 RSI: 00000000000000ff RDI:
> 0000000000000800
> [ 469.708945] RBP: ffff88041bf22000 R08: ffff880412839dc0 R09:
> 0000000000000003
> [ 469.708981] R10: 0000000000000001 R11: 4000000000000000 R12:
> 0000000000000100
> [ 469.709018] R13: ffff88041d3cc000 R14: ffff880412839d40 R15:
> ffff88041c72f440
> [ 469.709054] FS:  00007f510c0b4700(0000) GS:ffff88041fc40000(0000)
> knlGS:0000000000000000
> [ 469.709094] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 469.709130] CR2: 00007f510ba56810 CR3: 0000000417c93000 CR4:
> 00000000000006e0
> [ 469.709167] DR0: 00000000000000a0 DR1: 0000000000000000 DR2:
> 0000000000000003
> [ 469.709204] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 469.709240] Process runbot (pid: 4942, threadinfo
> ffff880417278000, task ffff880401351230)
> [ 469.709279] Stack:
> [ 469.709313]  ffff880401351230 0000000000000020 ffff880412e9ca80
> ffff88041c1658d0 ffff88041c1658d0
> [ 469.709477]  ffff88041723f330 0000000001200011 ffff88041723f330
> 0000000000000000
> [ 469.709641]  00007f510c0b49d0 ffff880401351230 0000000000000000
> ffffffff81037645
> [ 469.709812] Call Trace:
> [ 469.709849]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.709885]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.709921]  [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.709958]  [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.709998]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.710042]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.710079]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.710114] Code:  4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44
> 24 ff 31 f6 41 ba 01 00 00 00 00 48 8d 3c c5 08 00 00 00 00 31 c0 eb
> 1a 0f 1f 44 00 00 f0> 48 ff 42 30 48 89 54 05 00 48 83 c0
> May 30 16:39:36 192.168.2.1 08 ff c6 48 39 f8 74 3b
> [ 469.712247] RIP [<ffffffff810db878>] dup_fd+0x168/0x300
> [ 469.712317]  RSP <ffff880417279dd0>
> [ 469.712391] ---[ end trace b80596608cd6bb6b ]---
> [ 469.712432] Kernel panic - not syncing: Fatal exception
> [ 469.712470] Pid: 4942, comm: runbot Tainted: G      D     2.6.39 #3
> [ 469.712508] Call Trace:
> [ 469.712544]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 469.712581]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 469.712618]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 469.712654]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 469.712691]  [<ffffffff810db878>] ? dup_fd+0x168/0x300
> [ 469.712727]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
> [ 469.712767]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
> [ 469.712803]  [<ffffffff810dbc23>] ? alloc_fd+0x43/0x130
> [ 469.712839]  [<ffffffff810c08c5>] ? fd_install+0x35/0x70
> [ 469.712875]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
> [ 469.712912]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
> [ 469.712948]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
> [ 469.712989] Rebooting in 60 seconds..
> 
> 
> [ 1100.473612] general protection fault: 0000 [#1] SMP
> [ 1100.473718] last sysfs file:
> /sys/devices/pci0000:00/0000:00:18.3/temp1_input
> [ 1100.473755] CPU 0
> [ 1100.473788] Modules linked in: xt_iprange xt_DSCP xt_length
> xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE
> ipt_REDIRECT xt_recent xt_state iptable_nat nf_nat nf_conntrack_ipv4
> nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp
> iptable_mangle pppoe pppox iptable_filter ip_tables x_tables
> ppp_generic slhc cls_u32 sch_htb deflate zlib_deflate des_generic
> cbc ecb crypto_blkcipher sha1_generic md5 hmac crypto_hash cryptomgr
> aead crypto_algapi af_key fuse netconsole configfs vhost_net
> powernow_k8 mperf i2c_nforce2 kvm_amd kvm pl2303 xhci_hcd usbserial
> k10temp i2c_piix4 usb_storage usb_libusual ohci_hcd ehci_hcd ahci
> usbcore sata_mv r8169 mii libahci megaraid_sas [last unloaded:
> scsi_wait_scan]
> [ 1100.476102]
> [ 1100.476137] Pid: 551, comm: ksmd Not tainted 2.6.39 #3 To Be
> Filled By O.E.M. To Be Filled By O.E.M. /880G Extreme3
> [ 1100.476277] RIP: 0010:[<ffffffff810b5c71>] [<ffffffff810b5c71>]
> ksm_scan_thread+0x861/0xc20
> [ 1100.476351] RSP: 0018:ffff88041c065e40  EFLAGS: 00010282
> [ 1100.476387] RAX: 0000000000000000 RBX: ffffea000a5c8468 RCX:
> 0000000000000f01
> [ 1100.476424] RDX: 0000880403456828 RSI: ffff8802f5dcb002 RDI:
> ffffea000ce7eac8
> [ 1100.476460] RBP: ffff880418377fc0 R08: 6db6db6db6db6db7 R09:
> 0000160000000000
> [ 1100.476498] R10: ffffea000ce7eac8 R11: 0000000080100073 R12:
> 00000000000000ab
> [ 1100.476535] R13: ffff880418377ff8 R14: ffffea000ce7eac8 R15:
> ffff880418377fe8
> [ 1100.476572] FS:  00007f08c6498700(0000) GS:ffff88041fc00000(0000)
> knlGS:0000000000000000
> [ 1100.476611] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 1100.476646] CR2: 00000000e1dd3000 CR3: 00000004025f7000 CR4:
> 00000000000006f0
> [ 1100.477007] DR0: 0000000000000045 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 1100.477042] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 1100.477078] Process ksmd (pid: 551, threadinfo ffff88041c064000,
> task ffff88041d0454e0)
> [ 1100.477115] Stack:
> [ 1100.477148]  ffff880403a8b100 00007fc1863cb000 ffff88041c064000
> 00000000002f5dcb
> [ 1100.477317]  ffff88041c064000 ffff88041c065e98 ffff88041d0454e0
> ffff88041267a580
> [ 1100.477480]  0000000000000000 ffff88041d0454e0 ffffffff81052a20
> ffff88041c065e98
> [ 1100.477644] Call Trace:
> [ 1100.477681]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.477717]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477754]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.477791]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.477829]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.477865]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.477902]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.477937] Code: 05 99 6e 6e 00 4c 89 ff e8 ed 12 fa ff e9 6c f9
> ff ff 48 89 6c 24 38 49 83 7d 00 00 0f 84 63 01 00 00 e8 b3 8b 31 00
> 49 8b 55 00
> [ 1100.479306] 8b 42 e8 4c 8b 72 f0 48 8d 6a d8 4c 8d 78 60 48 89 04 24 4c
> [ 1100.480012] RIP [<ffffffff810b5c71>] ksm_scan_thread+0x861/0xc20
> [ 1100.480015]  RSP <ffff88041c065e40>
> [ 1100.480141] ---[ end trace 7382777d9e74ea23 ]---
> [ 1100.480178] Kernel panic - not syncing: Fatal exception
> [ 1100.480216] Pid: 551, comm: ksmd Tainted: G      D     2.6.39 #3
> [ 1100.480257] Call Trace:
> [ 1100.480297]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
> [ 1100.480340]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
> [ 1100.480384]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
> [ 1100.480425]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
> [ 1100.480465]  [<ffffffff810b5c71>] ? ksm_scan_thread+0x861/0xc20
> [ 1100.480507]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
> [ 1100.480545]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480584]  [<ffffffff810b5410>] ?
> try_to_merge_with_ksm_page+0x570/0x570
> [ 1100.480624]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
> [ 1100.480669]  [<ffffffff813d1794>] ? kernel_thread_helper+0x4/0x10
> [ 1100.480708]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
> [ 1100.480748]  [<ffffffff813d1790>] ? gs_change+0xb/0xb
> [ 1100.480794] Rebooting in 60 seconds..
> 
> brad@srv:~$ zcat /proc/config.gz | grep -v "#"
> CONFIG_64BIT=y
> CONFIG_X86_64=y
> CONFIG_X86=y
> CONFIG_INSTRUCTION_DECODER=y
> CONFIG_OUTPUT_FORMAT="elf64-x86-64"
> CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_NEED_DMA_MAP_STATE=y
> CONFIG_NEED_SG_DMA_LENGTH=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
> CONFIG_GENERIC_HWEIGHT=y
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> CONFIG_GENERIC_TIME_VSYSCALL=y
> CONFIG_ARCH_HAS_CPU_RELAX=y
> CONFIG_ARCH_HAS_DEFAULT_IDLE=y
> CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
> CONFIG_HAVE_SETUP_PER_CPU_AREA=y
> CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
> CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
> CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> CONFIG_ZONE_DMA32=y
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> CONFIG_AUDIT_ARCH=y
> CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
> CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
> CONFIG_X86_64_SMP=y
> CONFIG_X86_HT=y
> CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi
> -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9
> -fcall-saved-r10 -fcall-saved-r11"
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
> CONFIG_CONSTRUCTORS=y
> CONFIG_HAVE_IRQ_WORK=y
> CONFIG_IRQ_WORK=y
> 
> CONFIG_EXPERIMENTAL=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_CROSS_COMPILE=""
> CONFIG_LOCALVERSION=""
> CONFIG_LOCALVERSION_AUTO=y
> CONFIG_HAVE_KERNEL_GZIP=y
> CONFIG_HAVE_KERNEL_BZIP2=y
> CONFIG_HAVE_KERNEL_LZMA=y
> CONFIG_HAVE_KERNEL_XZ=y
> CONFIG_HAVE_KERNEL_LZO=y
> CONFIG_KERNEL_GZIP=y
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_BSD_PROCESS_ACCT=y
> CONFIG_TASKSTATS=y
> CONFIG_TASK_DELAY_ACCT=y
> CONFIG_TASK_XACCT=y
> CONFIG_TASK_IO_ACCOUNTING=y
> CONFIG_HAVE_GENERIC_HARDIRQS=y
> 
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_HAVE_SPARSE_IRQ=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_GENERIC_IRQ_SHOW=y
> CONFIG_GENERIC_PENDING_IRQ=y
> CONFIG_IRQ_FORCED_THREADING=y
> 
> CONFIG_TREE_RCU=y
> CONFIG_RCU_FANOUT=64
> CONFIG_IKCONFIG=y
> CONFIG_IKCONFIG_PROC=y
> CONFIG_LOG_BUF_SHIFT=20
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_NAMESPACES=y
> CONFIG_BLK_DEV_INITRD=y
> CONFIG_INITRAMFS_SOURCE=""
> CONFIG_RD_GZIP=y
> CONFIG_RD_BZIP2=y
> CONFIG_RD_LZMA=y
> CONFIG_RD_XZ=y
> CONFIG_RD_LZO=y
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> CONFIG_KALLSYMS_ALL=y
> CONFIG_KALLSYMS_EXTRA_PASS=y
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_EVENTS=y
> 
> CONFIG_PERF_EVENTS=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_PCI_QUIRKS=y
> CONFIG_SLUB_DEBUG=y
> CONFIG_SLUB=y
> CONFIG_HAVE_OPROFILE=y
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
> CONFIG_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_IOREMAP_PROT=y
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_HAVE_OPTPROBES=y
> CONFIG_HAVE_ARCH_TRACEHOOK=y
> CONFIG_HAVE_DMA_ATTRS=y
> CONFIG_USE_GENERIC_SMP_HELPERS=y
> CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
> CONFIG_HAVE_DMA_API_DEBUG=y
> CONFIG_HAVE_HW_BREAKPOINT=y
> CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
> CONFIG_HAVE_USER_RETURN_NOTIFIER=y
> CONFIG_HAVE_PERF_EVENTS_NMI=y
> CONFIG_HAVE_ARCH_JUMP_LABEL=y
> 
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_UNLOAD=y
> CONFIG_STOP_MACHINE=y
> CONFIG_BLOCK=y
> CONFIG_BLK_DEV_BSG=y
> CONFIG_BLOCK_COMPAT=y
> 
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_DEFAULT_CFQ=y
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_PREEMPT_NOTIFIERS=y
> CONFIG_INLINE_SPIN_UNLOCK=y
> CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> CONFIG_INLINE_READ_UNLOCK=y
> CONFIG_INLINE_READ_UNLOCK_IRQ=y
> CONFIG_INLINE_WRITE_UNLOCK=y
> CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> CONFIG_MUTEX_SPIN_ON_OWNER=y
> 
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ=y
> CONFIG_HIGH_RES_TIMERS=y
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> CONFIG_SMP=y
> CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_NO_BOOTMEM=y
> CONFIG_MK8=y
> CONFIG_X86_INTERNODE_CACHE_SHIFT=6
> CONFIG_X86_CMPXCHG=y
> CONFIG_CMPXCHG_LOCAL=y
> CONFIG_X86_L1_CACHE_SHIFT=6
> CONFIG_X86_XADD=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_INTEL_USERCOPY=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMPXCHG64=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=64
> CONFIG_X86_DEBUGCTLMSR=y
> CONFIG_CPU_SUP_INTEL=y
> CONFIG_CPU_SUP_AMD=y
> CONFIG_CPU_SUP_CENTAUR=y
> CONFIG_HPET_TIMER=y
> CONFIG_HPET_EMULATE_RTC=y
> CONFIG_DMI=y
> CONFIG_GART_IOMMU=y
> CONFIG_AMD_IOMMU=y
> CONFIG_SWIOTLB=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_NR_CPUS=8
> CONFIG_SCHED_MC=y
> CONFIG_PREEMPT_NONE=y
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> CONFIG_X86_MCE=y
> CONFIG_X86_MCE_AMD=y
> CONFIG_X86_MCE_THRESHOLD=y
> CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
> CONFIG_ARCH_DMA_ADDR_T_64BIT=y
> CONFIG_DIRECT_GBPAGES=y
> CONFIG_ARCH_SPARSEMEM_DEFAULT=y
> CONFIG_ARCH_SPARSEMEM_ENABLE=y
> CONFIG_ARCH_SELECT_MEMORY_MODEL=y
> CONFIG_ARCH_MEMORY_PROBE=y
> CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
> CONFIG_SELECT_MEMORY_MODEL=y
> CONFIG_SPARSEMEM_MANUAL=y
> CONFIG_SPARSEMEM=y
> CONFIG_HAVE_MEMORY_PRESENT=y
> CONFIG_SPARSEMEM_EXTREME=y
> CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
> CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
> CONFIG_SPARSEMEM_VMEMMAP=y
> CONFIG_HAVE_MEMBLOCK=y
> CONFIG_MEMORY_HOTPLUG=y
> CONFIG_MEMORY_HOTPLUG_SPARSE=y
> CONFIG_PAGEFLAGS_EXTENDED=y
> CONFIG_SPLIT_PTLOCK_CPUS=4
> CONFIG_COMPACTION=y
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> CONFIG_MMU_NOTIFIER=y
> CONFIG_KSM=y
> CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
> CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
> CONFIG_TRANSPARENT_HUGEPAGE=y
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
> CONFIG_X86_RESERVE_LOW=64
> CONFIG_MTRR=y
> CONFIG_MTRR_SANITIZER=y
> CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
> CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
> CONFIG_X86_PAT=y
> CONFIG_ARCH_USES_PG_UNCACHED=y
> CONFIG_SECCOMP=y
> CONFIG_HZ_100=y
> CONFIG_HZ=100
> CONFIG_SCHED_HRTICK=y
> CONFIG_PHYSICAL_START=0x1000000
> CONFIG_PHYSICAL_ALIGN=0x1000000
> CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
> CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
> 
> CONFIG_ACPI=y
> CONFIG_ACPI_BUTTON=y
> CONFIG_ACPI_DOCK=y
> CONFIG_ACPI_PROCESSOR=y
> CONFIG_ACPI_THERMAL=y
> CONFIG_ACPI_BLACKLIST_YEAR=0
> CONFIG_X86_PM_TIMER=y
> 
> CONFIG_CPU_FREQ=y
> CONFIG_CPU_FREQ_TABLE=y
> CONFIG_CPU_FREQ_STAT=y
> CONFIG_CPU_FREQ_STAT_DETAILS=y
> CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
> CONFIG_CPU_FREQ_GOV_POWERSAVE=y
> CONFIG_CPU_FREQ_GOV_USERSPACE=y
> CONFIG_CPU_FREQ_GOV_ONDEMAND=y
> CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
> 
> CONFIG_X86_ACPI_CPUFREQ=m
> CONFIG_X86_POWERNOW_K8=m
> 
> CONFIG_CPU_IDLE=y
> CONFIG_CPU_IDLE_GOV_LADDER=y
> CONFIG_CPU_IDLE_GOV_MENU=y
> 
> 
> CONFIG_PCI=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> CONFIG_PCIEPORTBUS=y
> CONFIG_PCIEAER=y
> CONFIG_PCIEASPM=y
> CONFIG_ARCH_SUPPORTS_MSI=y
> CONFIG_PCI_MSI=y
> CONFIG_HT_IRQ=y
> CONFIG_PCI_IOAPIC=y
> CONFIG_PCI_LABEL=y
> CONFIG_ISA_DMA_API=y
> CONFIG_AMD_NB=y
> 
> CONFIG_BINFMT_ELF=y
> CONFIG_COMPAT_BINFMT_ELF=y
> CONFIG_BINFMT_MISC=m
> CONFIG_IA32_EMULATION=y
> CONFIG_IA32_AOUT=m
> CONFIG_COMPAT=y
> CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
> CONFIG_SYSVIPC_COMPAT=y
> CONFIG_HAVE_TEXT_POKE_SMP=y
> CONFIG_NET=y
> 
> CONFIG_PACKET=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_MIGRATE=y
> CONFIG_XFRM_IPCOMP=m
> CONFIG_NET_KEY=m
> CONFIG_NET_KEY_MIGRATE=y
> CONFIG_INET=y
> CONFIG_IP_MULTICAST=y
> CONFIG_IP_ROUTE_CLASSID=y
> CONFIG_NET_IPGRE_DEMUX=m
> CONFIG_INET_AH=m
> CONFIG_INET_ESP=m
> CONFIG_INET_IPCOMP=m
> CONFIG_INET_XFRM_TUNNEL=m
> CONFIG_INET_TUNNEL=m
> CONFIG_INET_XFRM_MODE_TRANSPORT=m
> CONFIG_INET_XFRM_MODE_TUNNEL=m
> CONFIG_INET_XFRM_MODE_BEET=m
> CONFIG_INET_DIAG=y
> CONFIG_INET_TCP_DIAG=y
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_TCP_CONG="cubic"
> CONFIG_NETFILTER=y
> CONFIG_NETFILTER_ADVANCED=y
> CONFIG_BRIDGE_NETFILTER=y
> 
> CONFIG_NETFILTER_NETLINK=m
> CONFIG_NETFILTER_NETLINK_QUEUE=m
> CONFIG_NETFILTER_NETLINK_LOG=m
> CONFIG_NF_CONNTRACK=m
> CONFIG_NF_CONNTRACK_MARK=y
> CONFIG_NF_CONNTRACK_ZONES=y
> CONFIG_NF_CT_PROTO_DCCP=m
> CONFIG_NF_CT_PROTO_GRE=m
> CONFIG_NF_CT_PROTO_SCTP=m
> CONFIG_NF_CT_PROTO_UDPLITE=m
> CONFIG_NF_CONNTRACK_AMANDA=m
> CONFIG_NF_CONNTRACK_FTP=m
> CONFIG_NF_CONNTRACK_H323=m
> CONFIG_NF_CONNTRACK_IRC=m
> CONFIG_NF_CONNTRACK_BROADCAST=m
> CONFIG_NF_CONNTRACK_NETBIOS_NS=m
> CONFIG_NF_CONNTRACK_SNMP=m
> CONFIG_NF_CONNTRACK_PPTP=m
> CONFIG_NF_CONNTRACK_SANE=m
> CONFIG_NF_CONNTRACK_SIP=m
> CONFIG_NF_CONNTRACK_TFTP=m
> CONFIG_NF_CT_NETLINK=m
> CONFIG_NETFILTER_TPROXY=m
> CONFIG_NETFILTER_XTABLES=m
> 
> CONFIG_NETFILTER_XT_MARK=m
> CONFIG_NETFILTER_XT_CONNMARK=m
> 
> CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
> CONFIG_NETFILTER_XT_TARGET_CT=m
> CONFIG_NETFILTER_XT_TARGET_DSCP=m
> CONFIG_NETFILTER_XT_TARGET_HL=m
> CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
> CONFIG_NETFILTER_XT_TARGET_MARK=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
> CONFIG_NETFILTER_XT_TARGET_RATEEST=m
> CONFIG_NETFILTER_XT_TARGET_TEE=m
> CONFIG_NETFILTER_XT_TARGET_TPROXY=m
> CONFIG_NETFILTER_XT_TARGET_TRACE=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
> 
> CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
> CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
> CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
> CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
> CONFIG_NETFILTER_XT_MATCH_CPU=m
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_HELPER=m
> CONFIG_NETFILTER_XT_MATCH_HL=m
> CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=m
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=m
> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
> CONFIG_NETFILTER_XT_MATCH_OSF=m
> CONFIG_NETFILTER_XT_MATCH_OWNER=m
> CONFIG_NETFILTER_XT_MATCH_POLICY=m
> CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
> CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
> CONFIG_NETFILTER_XT_MATCH_QUOTA=m
> CONFIG_NETFILTER_XT_MATCH_RATEEST=m
> CONFIG_NETFILTER_XT_MATCH_REALM=m
> CONFIG_NETFILTER_XT_MATCH_RECENT=m
> CONFIG_NETFILTER_XT_MATCH_SCTP=m
> CONFIG_NETFILTER_XT_MATCH_SOCKET=m
> CONFIG_NETFILTER_XT_MATCH_STATE=m
> CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
> CONFIG_NETFILTER_XT_MATCH_STRING=m
> CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
> CONFIG_NETFILTER_XT_MATCH_TIME=m
> CONFIG_NETFILTER_XT_MATCH_U32=m
> 
> CONFIG_NF_DEFRAG_IPV4=m
> CONFIG_NF_CONNTRACK_IPV4=m
> CONFIG_NF_CONNTRACK_PROC_COMPAT=y
> CONFIG_IP_NF_QUEUE=m
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP_NF_MATCH_AH=m
> CONFIG_IP_NF_MATCH_ECN=m
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_FILTER=m
> CONFIG_IP_NF_TARGET_REJECT=m
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=m
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_NF_NAT_SNMP_BASIC=m
> CONFIG_NF_NAT_PROTO_DCCP=m
> CONFIG_NF_NAT_PROTO_GRE=m
> CONFIG_NF_NAT_PROTO_UDPLITE=m
> CONFIG_NF_NAT_PROTO_SCTP=m
> CONFIG_NF_NAT_FTP=m
> CONFIG_NF_NAT_IRC=m
> CONFIG_NF_NAT_TFTP=m
> CONFIG_NF_NAT_AMANDA=m
> CONFIG_NF_NAT_PPTP=m
> CONFIG_NF_NAT_H323=m
> CONFIG_NF_NAT_SIP=m
> CONFIG_IP_NF_MANGLE=m
> CONFIG_IP_NF_TARGET_CLUSTERIP=m
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> CONFIG_IP_NF_RAW=m
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
> CONFIG_IP_SCTP=m
> CONFIG_SCTP_HMAC_MD5=y
> CONFIG_STP=y
> CONFIG_BRIDGE=y
> CONFIG_BRIDGE_IGMP_SNOOPING=y
> CONFIG_VLAN_8021Q=y
> CONFIG_LLC=y
> CONFIG_NET_SCHED=y
> 
> CONFIG_NET_SCH_CBQ=m
> CONFIG_NET_SCH_HTB=m
> CONFIG_NET_SCH_HFSC=m
> CONFIG_NET_SCH_PRIO=m
> CONFIG_NET_SCH_RED=m
> CONFIG_NET_SCH_SFB=m
> CONFIG_NET_SCH_SFQ=m
> CONFIG_NET_SCH_TEQL=m
> CONFIG_NET_SCH_TBF=m
> CONFIG_NET_SCH_GRED=m
> CONFIG_NET_SCH_DSMARK=m
> CONFIG_NET_SCH_NETEM=m
> CONFIG_NET_SCH_MQPRIO=m
> CONFIG_NET_SCH_CHOKE=m
> CONFIG_NET_SCH_INGRESS=m
> 
> CONFIG_NET_CLS=y
> CONFIG_NET_CLS_BASIC=m
> CONFIG_NET_CLS_TCINDEX=m
> CONFIG_NET_CLS_ROUTE4=m
> CONFIG_NET_CLS_FW=m
> CONFIG_NET_CLS_U32=m
> CONFIG_NET_CLS_FLOW=m
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> CONFIG_NET_EMATCH_CMP=m
> CONFIG_NET_EMATCH_NBYTE=m
> CONFIG_NET_EMATCH_U32=m
> CONFIG_NET_EMATCH_META=m
> CONFIG_NET_EMATCH_TEXT=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_NET_ACT_POLICE=m
> CONFIG_NET_ACT_GACT=m
> CONFIG_GACT_PROB=y
> CONFIG_NET_ACT_MIRRED=m
> CONFIG_NET_ACT_IPT=m
> CONFIG_NET_ACT_NAT=m
> CONFIG_NET_ACT_PEDIT=m
> CONFIG_NET_ACT_SIMP=m
> CONFIG_NET_ACT_SKBEDIT=m
> CONFIG_NET_ACT_CSUM=m
> CONFIG_NET_SCH_FIFO=y
> CONFIG_RPS=y
> CONFIG_RFS_ACCEL=y
> CONFIG_XPS=y
> 
> 
> 
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> CONFIG_DEVTMPFS=y
> CONFIG_STANDALONE=y
> CONFIG_PREVENT_FIRMWARE_BUILD=y
> CONFIG_FW_LOADER=y
> CONFIG_FIRMWARE_IN_KERNEL=y
> CONFIG_EXTRA_FIRMWARE=""
> CONFIG_ARCH_NO_SYSDEV_OPS=y
> CONFIG_PNP=y
> CONFIG_PNP_DEBUG_MESSAGES=y
> 
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_LOOP=y
> 
> CONFIG_HAVE_IDE=y
> 
> CONFIG_SCSI_MOD=y
> CONFIG_RAID_ATTRS=m
> CONFIG_SCSI=y
> CONFIG_SCSI_DMA=y
> CONFIG_SCSI_PROC_FS=y
> 
> CONFIG_BLK_DEV_SD=y
> CONFIG_BLK_DEV_SR=y
> CONFIG_CHR_DEV_SG=y
> CONFIG_SCSI_WAIT_SCAN=m
> 
> CONFIG_SCSI_SAS_ATTRS=m
> CONFIG_SCSI_LOWLEVEL=y
> CONFIG_MEGARAID_SAS=m
> CONFIG_ATA=y
> CONFIG_ATA_VERBOSE_ERROR=y
> CONFIG_ATA_ACPI=y
> CONFIG_SATA_PMP=y
> 
> CONFIG_SATA_AHCI=m
> CONFIG_SATA_SIL24=m
> CONFIG_ATA_SFF=y
> 
> CONFIG_ATA_BMDMA=y
> 
> CONFIG_SATA_MV=m
> CONFIG_SATA_NV=m
> CONFIG_SATA_PROMISE=m
> CONFIG_SATA_SIL=m
> 
> 
> 
> CONFIG_MD=y
> CONFIG_BLK_DEV_MD=y
> CONFIG_MD_LINEAR=y
> CONFIG_MD_RAID0=y
> CONFIG_MD_RAID1=y
> CONFIG_MD_RAID10=y
> CONFIG_MD_RAID456=y
> CONFIG_BLK_DEV_DM=y
> 
> CONFIG_NETDEVICES=y
> CONFIG_DUMMY=m
> CONFIG_BONDING=m
> CONFIG_TUN=y
> CONFIG_MII=m
> CONFIG_NET_ETHERNET=y
> CONFIG_NET_PCI=y
> CONFIG_FORCEDETH=m
> CONFIG_NETDEV_1000=y
> CONFIG_R8169=m
> 
> 
> 
> CONFIG_PPP=m
> CONFIG_PPP_MULTILINK=y
> CONFIG_PPP_FILTER=y
> CONFIG_PPP_ASYNC=m
> CONFIG_PPP_SYNC_TTY=m
> CONFIG_PPP_DEFLATE=m
> CONFIG_PPP_BSDCOMP=m
> CONFIG_PPP_MPPE=m
> CONFIG_PPPOE=m
> CONFIG_SLHC=m
> CONFIG_NETCONSOLE=m
> CONFIG_NETCONSOLE_DYNAMIC=y
> CONFIG_NETPOLL=y
> CONFIG_NET_POLL_CONTROLLER=y
> 
> CONFIG_INPUT=y
> 
> CONFIG_INPUT_MOUSEDEV=y
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
> 
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
> 
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> CONFIG_SERIO_LIBPS2=y
> 
> CONFIG_VT=y
> CONFIG_CONSOLE_TRANSLATIONS=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> CONFIG_UNIX98_PTYS=y
> CONFIG_LEGACY_PTYS=y
> CONFIG_LEGACY_PTY_COUNT=256
> 
> CONFIG_SERIAL_8250=y
> CONFIG_SERIAL_8250_CONSOLE=y
> CONFIG_FIX_EARLYCON_MEM=y
> CONFIG_SERIAL_8250_PCI=y
> CONFIG_SERIAL_8250_PNP=y
> CONFIG_SERIAL_8250_NR_UARTS=4
> CONFIG_SERIAL_8250_RUNTIME_UARTS=4
> 
> CONFIG_SERIAL_CORE=y
> CONFIG_SERIAL_CORE_CONSOLE=y
> CONFIG_RTC=y
> CONFIG_HPET=y
> CONFIG_HPET_MMAP=y
> CONFIG_DEVPORT=y
> CONFIG_I2C=y
> CONFIG_I2C_BOARDINFO=y
> CONFIG_I2C_COMPAT=y
> CONFIG_I2C_CHARDEV=y
> CONFIG_I2C_HELPER_AUTO=y
> CONFIG_I2C_ALGOBIT=y
> 
> 
> CONFIG_I2C_AMD756=m
> CONFIG_I2C_PIIX4=m
> CONFIG_I2C_NFORCE2=m
> CONFIG_I2C_VIA=m
> CONFIG_I2C_VIAPRO=m
> 
> CONFIG_I2C_SCMI=m
> 
> 
> 
> 
> 
> CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
> CONFIG_HWMON=y
> CONFIG_HWMON_VID=m
> 
> CONFIG_SENSORS_K8TEMP=m
> CONFIG_SENSORS_K10TEMP=m
> CONFIG_SENSORS_W83781D=m
> CONFIG_SENSORS_W83791D=m
> CONFIG_SENSORS_W83792D=m
> CONFIG_SENSORS_W83793=m
> CONFIG_SENSORS_W83795=m
> CONFIG_SENSORS_W83795_FANCTRL=y
> CONFIG_SENSORS_W83L785TS=m
> CONFIG_SENSORS_W83L786NG=m
> CONFIG_SENSORS_W83627HF=m
> CONFIG_SENSORS_W83627EHF=m
> 
> CONFIG_SENSORS_ATK0110=m
> CONFIG_THERMAL=y
> CONFIG_THERMAL_HWMON=y
> CONFIG_SSB_POSSIBLE=y
> 
> 
> CONFIG_VGA_ARB=y
> CONFIG_VGA_ARB_MAX_GPUS=16
> CONFIG_VGASTATE=y
> CONFIG_FB=y
> CONFIG_FB_DDC=y
> CONFIG_FB_CFB_FILLRECT=y
> CONFIG_FB_CFB_COPYAREA=y
> CONFIG_FB_CFB_IMAGEBLIT=y
> CONFIG_FB_MODE_HELPERS=y
> 
> CONFIG_FB_NVIDIA=y
> CONFIG_FB_NVIDIA_I2C=y
> CONFIG_FB_RADEON=y
> CONFIG_FB_RADEON_I2C=y
> 
> 
> CONFIG_VGA_CONSOLE=y
> CONFIG_DUMMY_CONSOLE=y
> CONFIG_FRAMEBUFFER_CONSOLE=y
> CONFIG_FONT_8x8=y
> CONFIG_FONT_8x16=y
> CONFIG_LOGO=y
> CONFIG_LOGO_LINUX_MONO=y
> CONFIG_LOGO_LINUX_VGA16=y
> CONFIG_LOGO_LINUX_CLUT224=y
> CONFIG_HID_SUPPORT=y
> CONFIG_HID=y
> CONFIG_HIDRAW=y
> 
> CONFIG_USB_HID=m
> CONFIG_USB_HIDDEV=y
> 
> CONFIG_HID_A4TECH=m
> CONFIG_HID_APPLE=m
> CONFIG_HID_BELKIN=m
> CONFIG_HID_CHERRY=m
> CONFIG_HID_CHICONY=m
> CONFIG_HID_CYPRESS=m
> CONFIG_HID_EZKEY=m
> CONFIG_HID_KYE=m
> CONFIG_HID_KENSINGTON=m
> CONFIG_HID_LOGITECH=m
> CONFIG_HID_MICROSOFT=m
> CONFIG_HID_MONTEREY=m
> CONFIG_USB_SUPPORT=y
> CONFIG_USB_ARCH_HAS_HCD=y
> CONFIG_USB_ARCH_HAS_OHCI=y
> CONFIG_USB_ARCH_HAS_EHCI=y
> CONFIG_USB=m
> 
> 
> CONFIG_USB_XHCI_HCD=m
> CONFIG_USB_EHCI_HCD=m
> CONFIG_USB_OHCI_HCD=m
> CONFIG_USB_OHCI_LITTLE_ENDIAN=y
> CONFIG_USB_UHCI_HCD=m
> 
> 
> CONFIG_USB_ACM=m
> 
> 
> CONFIG_USB_STORAGE=m
> CONFIG_USB_LIBUSUAL=y
> 
> 
> CONFIG_USB_SERIAL=m
> CONFIG_USB_SERIAL_GENERIC=y
> CONFIG_USB_SERIAL_FTDI_SIO=m
> CONFIG_USB_SERIAL_IPAQ=m
> CONFIG_USB_SERIAL_PL2303=m
> 
> 
> 
> CONFIG_FIRMWARE_MEMMAP=y
> CONFIG_DMIID=y
> 
> CONFIG_EXT2_FS=y
> CONFIG_EXT2_FS_XATTR=y
> CONFIG_EXT3_FS=y
> CONFIG_EXT3_FS_XATTR=y
> CONFIG_EXT4_FS=y
> CONFIG_EXT4_FS_XATTR=y
> CONFIG_JBD=y
> CONFIG_JBD2=y
> CONFIG_FS_MBCACHE=y
> CONFIG_EXPORTFS=y
> CONFIG_FILE_LOCKING=y
> CONFIG_FSNOTIFY=y
> CONFIG_DNOTIFY=y
> CONFIG_INOTIFY_USER=y
> CONFIG_FANOTIFY=y
> CONFIG_FUSE_FS=m
> CONFIG_CUSE=m
> 
> CONFIG_FSCACHE=m
> CONFIG_FSCACHE_DEBUG=y
> CONFIG_CACHEFILES=m
> 
> CONFIG_ISO9660_FS=m
> CONFIG_JOLIET=y
> CONFIG_ZISOFS=y
> CONFIG_UDF_FS=m
> CONFIG_UDF_NLS=y
> 
> CONFIG_FAT_FS=m
> CONFIG_MSDOS_FS=m
> CONFIG_VFAT_FS=m
> CONFIG_FAT_DEFAULT_CODEPAGE=437
> CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
> CONFIG_NTFS_FS=m
> 
> CONFIG_PROC_FS=y
> CONFIG_PROC_SYSCTL=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SYSFS=y
> CONFIG_TMPFS=y
> CONFIG_HUGETLBFS=y
> CONFIG_HUGETLB_PAGE=y
> CONFIG_CONFIGFS_FS=m
> CONFIG_NETWORK_FILESYSTEMS=y
> CONFIG_NFS_FS=y
> CONFIG_NFS_V3=y
> CONFIG_NFSD=y
> CONFIG_NFSD_DEPRECATED=y
> CONFIG_NFSD_V3=y
> CONFIG_LOCKD=y
> CONFIG_LOCKD_V4=y
> CONFIG_NFS_COMMON=y
> CONFIG_SUNRPC=y
> 
> CONFIG_PARTITION_ADVANCED=y
> CONFIG_MSDOS_PARTITION=y
> CONFIG_EFI_PARTITION=y
> CONFIG_NLS=y
> CONFIG_NLS_DEFAULT="iso8859-1"
> CONFIG_NLS_CODEPAGE_437=y
> CONFIG_NLS_ISO8859_1=y
> CONFIG_DLM=m
> 
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> CONFIG_PRINTK_TIME=y
> CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
> CONFIG_ENABLE_WARN_DEPRECATED=y
> CONFIG_ENABLE_MUST_CHECK=y
> CONFIG_FRAME_WARN=2048
> CONFIG_MAGIC_SYSRQ=y
> CONFIG_DEBUG_FS=y
> CONFIG_DEBUG_KERNEL=y
> CONFIG_DETECT_HUNG_TASK=y
> CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
> CONFIG_TIMER_STATS=y
> CONFIG_DEBUG_BUGVERBOSE=y
> CONFIG_DEBUG_MEMORY_INIT=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_SYSCTL_SYSCALL_CHECK=y
> CONFIG_USER_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_FUNCTION_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
> CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
> CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
> CONFIG_HAVE_DYNAMIC_FTRACE=y
> CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
> CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
> CONFIG_HAVE_C_RECORDMCOUNT=y
> CONFIG_TRACING_SUPPORT=y
> CONFIG_HAVE_ARCH_KGDB=y
> CONFIG_HAVE_ARCH_KMEMCHECK=y
> CONFIG_X86_VERBOSE_BOOTUP=y
> CONFIG_EARLY_PRINTK=y
> CONFIG_HAVE_MMIOTRACE_SUPPORT=y
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
> 
> CONFIG_DEFAULT_SECURITY_DAC=y
> CONFIG_DEFAULT_SECURITY=""
> CONFIG_XOR_BLOCKS=y
> CONFIG_ASYNC_CORE=y
> CONFIG_ASYNC_MEMCPY=y
> CONFIG_ASYNC_XOR=y
> CONFIG_ASYNC_PQ=y
> CONFIG_ASYNC_RAID6_RECOV=y
> CONFIG_CRYPTO=y
> 
> CONFIG_CRYPTO_ALGAPI=m
> CONFIG_CRYPTO_ALGAPI2=m
> CONFIG_CRYPTO_AEAD=m
> CONFIG_CRYPTO_AEAD2=m
> CONFIG_CRYPTO_BLKCIPHER=m
> CONFIG_CRYPTO_BLKCIPHER2=m
> CONFIG_CRYPTO_HASH=m
> CONFIG_CRYPTO_HASH2=m
> CONFIG_CRYPTO_RNG2=m
> CONFIG_CRYPTO_PCOMP2=m
> CONFIG_CRYPTO_MANAGER=m
> CONFIG_CRYPTO_MANAGER2=m
> CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
> CONFIG_CRYPTO_WORKQUEUE=m
> CONFIG_CRYPTO_AUTHENC=m
> 
> 
> CONFIG_CRYPTO_CBC=m
> CONFIG_CRYPTO_ECB=m
> 
> CONFIG_CRYPTO_HMAC=m
> 
> CONFIG_CRYPTO_CRC32C=m
> CONFIG_CRYPTO_MD5=m
> CONFIG_CRYPTO_SHA1=m
> 
> CONFIG_CRYPTO_ARC4=m
> CONFIG_CRYPTO_DES=m
> 
> CONFIG_CRYPTO_DEFLATE=m
> 
> CONFIG_HAVE_KVM=y
> CONFIG_HAVE_KVM_IRQCHIP=y
> CONFIG_HAVE_KVM_EVENTFD=y
> CONFIG_KVM_APIC_ARCHITECTURE=y
> CONFIG_KVM_MMIO=y
> CONFIG_KVM_ASYNC_PF=y
> CONFIG_VIRTUALIZATION=y
> CONFIG_KVM=m
> CONFIG_KVM_INTEL=m
> CONFIG_KVM_AMD=m
> CONFIG_VHOST_NET=m
> 
> CONFIG_RAID6_PQ=y
> CONFIG_BITREVERSE=y
> CONFIG_GENERIC_FIND_FIRST_BIT=y
> CONFIG_GENERIC_FIND_NEXT_BIT=y
> CONFIG_GENERIC_FIND_LAST_BIT=y
> CONFIG_CRC_CCITT=m
> CONFIG_CRC16=y
> CONFIG_CRC_T10DIF=y
> CONFIG_CRC_ITU_T=y
> CONFIG_CRC32=y
> CONFIG_LIBCRC32C=m
> CONFIG_ZLIB_INFLATE=y
> CONFIG_ZLIB_DEFLATE=m
> CONFIG_LZO_DECOMPRESS=y
> CONFIG_XZ_DEC=y
> CONFIG_XZ_DEC_X86=y
> CONFIG_XZ_DEC_POWERPC=y
> CONFIG_XZ_DEC_IA64=y
> CONFIG_XZ_DEC_ARM=y
> CONFIG_XZ_DEC_ARMTHUMB=y
> CONFIG_XZ_DEC_SPARC=y
> CONFIG_XZ_DEC_BCJ=y
> CONFIG_DECOMPRESS_GZIP=y
> CONFIG_DECOMPRESS_BZIP2=y
> CONFIG_DECOMPRESS_LZMA=y
> CONFIG_DECOMPRESS_XZ=y
> CONFIG_DECOMPRESS_LZO=y
> CONFIG_TEXTSEARCH=y
> CONFIG_TEXTSEARCH_KMP=m
> CONFIG_TEXTSEARCH_BM=m
> CONFIG_TEXTSEARCH_FSM=m
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> CONFIG_CPU_RMAP=y
> CONFIG_NLATTR=y
> 
> 00:00.0 Host bridge: Advanced Micro Devices [AMD] RS880 Host Bridge
> 00:01.0 PCI bridge: ASRock Incorporation Device 9602
> 00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 0)
> 00:03.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (ext gfx port 1)
> 00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 0)
> 00:09.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 4)
> 00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
> bridge (PCIE port 5)
> 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
> Controller [AHCI mode] (rev 40)
> 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 41)
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host
> controller (rev 40)
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
> Controller
> 00:16.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
> Controller
> 00:16.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor HyperTransport Configuration
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Address Map
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor DRAM Controller
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Miscellaneous Control
> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h
> Processor Link Control
> 01:05.0 VGA compatible controller: ATI Technologies Inc RS880
> [Radeon HD 4250]
> 02:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host
> Controller (rev 03)
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> 04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> 9240 (rev 02)
> 07:00.0 SCSI storage controller: Marvell Technology Group Ltd.
> 88SX7042 PCI-e 4-port SATA-II (rev 02)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Regards/Gruss,
    Boris.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 31/05/11 13:47, Borislav Petkov wrote:
> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> machine from oopsing.
>
> Adding linux-mm.
>

I initially thought that, so the second panic was produced with KSM 
disabled from boot.

echo 0 > /sys/kernel/mm/ksm/run

If you still think that compiling ksm out of the kernel will prevent it 
then I'm willing to give it a go.

It's a production server, so I can only really bounce it around after 
about 9PM - GMT+8.

Regards,
Brad

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Borislav Petkov]

On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> On 31/05/11 13:47, Borislav Petkov wrote:
> >Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> >machine from oopsing.
> >
> >Adding linux-mm.
> >
> 
> I initially thought that, so the second panic was produced with KSM
> disabled from boot.
> 
> echo 0 > /sys/kernel/mm/ksm/run
> 
> If you still think that compiling ksm out of the kernel will prevent
> it then I'm willing to give it a go.

Ok, from looking at the code, when KSM inits, it starts the ksm kernel
thread and it looks like your oops comes from the function that is run
in the kernel thread - ksm_scan_thread.

So even if you disable it from sysfs, it runs at least once.

Let's add some more people to Cc and see what happens :).

-- 
Regards/Gruss,
Boris.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 31/05/11 18:38, Borislav Petkov wrote:
> On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
>> On 31/05/11 13:47, Borislav Petkov wrote:
>>> Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
>>> machine from oopsing.
>>>
>>> Adding linux-mm.
>>>
>>
>> I initially thought that, so the second panic was produced with KSM
>> disabled from boot.
>>
>> echo 0>  /sys/kernel/mm/ksm/run
>>
>> If you still think that compiling ksm out of the kernel will prevent
>> it then I'm willing to give it a go.
>
> Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> thread and it looks like your oops comes from the function that is run
> in the kernel thread - ksm_scan_thread.
>
> So even if you disable it from sysfs, it runs at least once.
>

Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been 
unable to reproduce the bug, so it looks like you were on the money.

I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics 
about 75% of boots, and the reboot cycle required to get luck my way 
into a working kernel is just too much hassle.

It would appear that XP zero's its memory space on bootup, so there 
would be lots of pages to merge with a couple of relatively freshly 
booted XP machines running.

Regards,
Brad.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Hugh Dickins]

On Tue, 31 May 2011, Brad Campbell wrote:
> On 31/05/11 18:38, Borislav Petkov wrote:
> > On Tue, May 31, 2011 at 05:26:10PM +0800, Brad Campbell wrote:
> > > On 31/05/11 13:47, Borislav Petkov wrote:
> > > > Looks like a KSM issue. Disabling CONFIG_KSM should at least stop your
> > > > machine from oopsing.
> > > > 
> > > > Adding linux-mm.
> > > > 
> > > 
> > > I initially thought that, so the second panic was produced with KSM
> > > disabled from boot.
> > > 
> > > echo 0>  /sys/kernel/mm/ksm/run
> > > 
> > > If you still think that compiling ksm out of the kernel will prevent
> > > it then I'm willing to give it a go.
> > 
> > Ok, from looking at the code, when KSM inits, it starts the ksm kernel
> > thread and it looks like your oops comes from the function that is run
> > in the kernel thread - ksm_scan_thread.
> > 
> > So even if you disable it from sysfs, it runs at least once.
> > 
> 
> Just to confirm, I recompiled 2.6.38.7 without KSM enabled and I've been
> unable to reproduce the bug, so it looks like you were on the money.
> 
> I've moved back to 2.6.38.7 as 2.6.39 has a painful SCSI bug that panics
> about 75% of boots, and the reboot cycle required to get luck my way into a
> working kernel is just too much hassle.
> 
> It would appear that XP zero's its memory space on bootup, so there would be
> lots of pages to merge with a couple of relatively freshly booted XP
> machines running.

Thanks for the Cc, Borislav.

Brad, my suspicion is that in each case the top 16 bits of RDX have been
mysteriously corrupted from ffff to 0000, causing the general protection
faults.  I don't understand what that has to do with KSM.

But it's only a suspicion, because I can't make sense of the "Code:"
lines in your traces, they have more than the expected 64 bytes, and
only one of them has a ">" (with no "<") to mark faulting instruction.

I did try compiling the 2.6.39 kernel from your config, but of course
we have different compilers, so although I got close, it wasn't exact.

Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
output for your original vmlinux (with KSM on)?  (Those -trd options are
the ones I'm used to typing, I bet not they're not all relevant.)

Of course, it's only a tiny fraction of that output that I need,
might be better to cut it down to remove_rmap_item_from_tree and
dup_fd and ksm_scan_thread, if you have the time to do so.

Thanks,
Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 01/06/11 06:31, Hugh Dickins wrote:
>
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults.  I don't understand what that has to do with KSM.

No, nor do I. The panic I reproduced with KSM off was in a completely 
unrelated code path. To be honest I would not be surprised if it turns 
out I have dodgy RAM, although it has passed multiple memtests and I've 
tried clocking it down. Just a gut feeling.

> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.

Yeah, with hindsight I must have removed them when I re-formatted the 
code from the oops. Each byte was one line in the syslog so there was a 
lot of deleting to get it to a postable format.

> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)?  (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.

Ok, so since my initial posting I've figured out how to get a clean oops 
out of netconsole, so tonight (after 9PM GMT+8) I'll reproduce the oops 
a couple of times. What about I upload the oops, plus the vmlinux, plus 
.config and System.map to a server with a fat pipe and give you a link 
to it?

At least I can reproduce it quickly and easily.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 01/06/11 06:31, Hugh Dickins wrote:
> Brad, my suspicion is that in each case the top 16 bits of RDX have been
> mysteriously corrupted from ffff to 0000, causing the general protection
> faults.  I don't understand what that has to do with KSM.
>
> But it's only a suspicion, because I can't make sense of the "Code:"
> lines in your traces, they have more than the expected 64 bytes, and
> only one of them has a ">" (with no"<") to mark faulting instruction.
>
> I did try compiling the 2.6.39 kernel from your config, but of course
> we have different compilers, so although I got close, it wasn't exact.
>
> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> output for your original vmlinux (with KSM on)?  (Those -trd options are
> the ones I'm used to typing, I bet not they're not all relevant.)
>
> Of course, it's only a tiny fraction of that output that I need,
> might be better to cut it down to remove_rmap_item_from_tree and
> dup_fd and ksm_scan_thread, if you have the time to do so.

Would you believe about 20 seconds after I pressed send the kernel oopsed.

http://www.fnarfbargle.com/private/003_kernel_oops/

oops reproduced here, but an un-munged version is in that directory 
alongside the kernel.

[36542.880228] general protection fault: 0000 [#1] SMP
[36542.880271] last sysfs file: 
/sys/devices/pci0000:00/0000:00:18.3/temp1_input
[36542.880290] CPU 4
[36542.880301] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUER
ADE ipt_REDIRECT xt_recent xt_state iptable_filter iptable_nat nf_nat 
nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 x
t_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle ip_tables x_tables pppoe 
pppox ppp_generic slhc cls_u32 sch_htb deflate z
lib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac 
crypto_hash cryptomgr aead crypto_algapi af_ke
y fuse hwmon_vid netconsole configfs vhost_net powernow_k8 mperf kvm_amd 
kvm pl2303 usbserial xhci_hcd k10temp i2c_pi
ix4 ahci usb_storage usb_libusual ohci_hcd ehci_hcd r8169 libahci 
usbcore mii sata_mv megaraid_sas [last unloaded: sc
si_wait_scan]
[36542.880842]
[36542.880858] Pid: 13346, comm: bash Not tainted 2.6.38.7 #29 To Be 
Filled By O.E.M. To Be Filled By O.E.M./880G Ext
reme3
[36542.880911] RIP: 0010:[<ffffffff810cf0de>]  [<ffffffff810cf0de>] 
do_vfs_ioctl+0x5e/0x510
[36542.880948] RSP: 0018:ffff8802d25a1ec8  EFLAGS: 00010206
[36542.880965] RAX: fffffffffffffff7 RBX: 000088040eb12840 RCX: 
00007fff4fe4a4c0
[36542.880984] RDX: 0000000000005413 RSI: 0000000000005413 RDI: 
00000000000000ff
[36542.881002] RBP: 00000000000000ff R08: 00007fff4fe4a400 R09: 
0000000000000000
[36542.881020] R10: 00007fff4fe4a380 R11: 0000000000000246 R12: 
00007fff4fe4a4c0
[36542.881038] R13: 00007fff4fe4a4c0 R14: 0000000000000000 R15: 
0000000000000001
[36542.881058] FS:  00007f65f725b700(0000) GS:ffff8800dbd00000(0000) 
knlGS:0000000000000000
[36542.881081] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36542.881098] CR2: 0000000001f01008 CR3: 00000002d25c3000 CR4: 
00000000000006e0
[36542.881116] DR0: 00000000000000a0 DR1: 0000000000000000 DR2: 
0000000000000003
[36542.881133] DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[36542.881152] Process bash (pid: 13346, threadinfo ffff8802d25a0000, 
task ffff88041df88000)
[36542.881172] Stack:
[36542.881183]  0000000000000000 ffff88041df88218 0000000000100000 
0000000000000001
[36542.881225]  0000000000000002 00007fff4fe4a2c0 00007fff4fe4a220 
0000000000000002
[36542.881268]  0000000000000000 ffffffff81046d6a 000088040eb12840 
00000000000000ff
[36542.881312] Call Trace:
[36542.881333]  [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.881351]  [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.881373]  [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.881389] Code: 76 7b 81 fa 77 58 04 c0 0f 84 77 01 00 00 0f 1f 80 
00 00 00 00 0f 87 a2 00 00 00 81 fa 60 54 00 00 0f 1f 40 00 0f 84 ba 01 
00 00 <48> 8b 43 18 48 8b 50 30 0f b7 02 25 00 f0 00 00 3d 00 80 00 00
[36542.881793] RIP  [<ffffffff810cf0de>] do_vfs_ioctl+0x5e/0x510
[36542.881818]  RSP <ffff8802d25a1ec8>
[36542.882082] ---[ end trace 1b8d730cd479e388 ]---
[36542.882126] Kernel panic - not syncing: Fatal exception
[36542.882175] Pid: 13346, comm: bash Tainted: G      D     2.6.38.7 #29
[36542.882222] Call Trace:
[36542.882269]  [<ffffffff813c7f42>] ? panic+0x92/0x18a
[36542.882318]  [<ffffffff81039a41>] ? kmsg_dump+0x41/0xf0
[36542.882366]  [<ffffffff810062bd>] ? oops_end+0x8d/0xa0
[36542.882414]  [<ffffffff813caeef>] ? general_protection+0x1f/0x30
[36542.882463]  [<ffffffff810cf0de>] ? do_vfs_ioctl+0x5e/0x510
[36542.882511]  [<ffffffff81046d6a>] ? sys_rt_sigaction+0x8a/0xc0
[36542.882560]  [<ffffffff810cf5d9>] ? sys_ioctl+0x49/0x80
[36542.882608]  [<ffffffff810023fb>] ? system_call_fastpath+0x16/0x1b
[36542.882688] Rebooting in 60 seconds..[   33.104725] fuse init (API 
version 7.16)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Andrea Arcangeli]

Hello,

On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> On 01/06/11 06:31, Hugh Dickins wrote:
> > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > mysteriously corrupted from ffff to 0000, causing the general protection
> > faults.  I don't understand what that has to do with KSM.
> >
> > But it's only a suspicion, because I can't make sense of the "Code:"
> > lines in your traces, they have more than the expected 64 bytes, and
> > only one of them has a ">" (with no"<") to mark faulting instruction.
> >
> > I did try compiling the 2.6.39 kernel from your config, but of course
> > we have different compilers, so although I got close, it wasn't exact.
> >
> > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > output for your original vmlinux (with KSM on)?  (Those -trd options are
> > the ones I'm used to typing, I bet not they're not all relevant.)
> >
> > Of course, it's only a tiny fraction of that output that I need,
> > might be better to cut it down to remove_rmap_item_from_tree and
> > dup_fd and ksm_scan_thread, if you have the time to do so.
> 
> Would you believe about 20 seconds after I pressed send the kernel oopsed.
> 
> http://www.fnarfbargle.com/private/003_kernel_oops/
> 
> oops reproduced here, but an un-munged version is in that directory 
> alongside the kernel.
> 
> [36542.880228] general protection fault: 0000 [#1] SMP

Reminds me of another oops that was reported on the kvm list for
2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
rsi were flipped and it was a general protection too because of
hitting on the not mappable virtual range.

http://www.virtall.com/files/temp/kvm.txt
http://www.virtall.com/files/temp/config-2.6.38.1
http://virtall.com/files/temp/mmu-objdump.txt

That oops happened in kvm_unmap_rmapp though, but it looked memory
corruption (Avi suggested use after free) but it was a production
system so we couldn't debug it further.

I recommend next thing to reproduce again with 2.6.39 or
3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
test with 2.6.39.

We'd need chmod +r vmlinux on private/003_kernel_oops/

Thanks,
Andrea

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 01/06/11 09:15, Andrea Arcangeli wrote:
> Hello,
>
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
>> On 01/06/11 06:31, Hugh Dickins wrote:
>>> Brad, my suspicion is that in each case the top 16 bits of RDX have been
>>> mysteriously corrupted from ffff to 0000, causing the general protection
>>> faults.  I don't understand what that has to do with KSM.
>>>
>>> But it's only a suspicion, because I can't make sense of the "Code:"
>>> lines in your traces, they have more than the expected 64 bytes, and
>>> only one of them has a ">" (with no"<") to mark faulting instruction.
>>>
>>> I did try compiling the 2.6.39 kernel from your config, but of course
>>> we have different compilers, so although I got close, it wasn't exact.
>>>
>>> Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
>>> output for your original vmlinux (with KSM on)?  (Those -trd options are
>>> the ones I'm used to typing, I bet not they're not all relevant.)
>>>
>>> Of course, it's only a tiny fraction of that output that I need,
>>> might be better to cut it down to remove_rmap_item_from_tree and
>>> dup_fd and ksm_scan_thread, if you have the time to do so.
>>
>> Would you believe about 20 seconds after I pressed send the kernel oopsed.
>>
>> http://www.fnarfbargle.com/private/003_kernel_oops/
>>
>> oops reproduced here, but an un-munged version is in that directory
>> alongside the kernel.
>>
>> [36542.880228] general protection fault: 0000 [#1] SMP
>
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
>
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
>
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
>
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.
>
> We'd need chmod +r vmlinux on private/003_kernel_oops/

Ok, here we go then.

http://www.fnarfbargle.com/private/004_kernel_oops/

The permissions are right this time.
2.6.39 + KSM

[  694.227866] general protection fault: 0000 [#1] SMP
[  694.228001] last sysfs file: /sys/devices/platform/w83627ehf.656/cpu0_vid
[  694.228050] CPU 3
[  694.228091] Modules linked in: xt_iprange xt_DSCP xt_length 
xt_CLASSIFY sch_sfq xt_CHECKSUM ipt_REJECT ipt_MASQUERADE ipt_REDIRECT 
xt_recent xt_state iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 
nf_conntrack nf_defrag_ipv4 xt_TCPMSS xt_tcpmss xt_tcpudp iptable_mangle 
ip_tables x_tables pppoe pppox ppp_generic slhc cls_u32 sch_htb deflate 
zlib_deflate des_generic cbc ecb crypto_blkcipher sha1_generic md5 hmac 
crypto_hash cryptomgr aead crypto_algapi af_key fuse w83627ehf hwmon_vid 
netconsole configfs vhost_net powernow_k8 mperf kvm_amd kvm pl2303 
usbserial i2c_piix4 k10temp xhci_hcd usb_storage usb_libusual ohci_hcd 
r8169 ehci_hcd ahci usbcore sata_mv mii libahci megaraid_sas [last 
unloaded: scsi_wait_scan]
[  694.230897]
[  694.230944] Pid: 11841, comm: keepalive Not tainted 2.6.39 #3 To Be 
Filled By O.E.M. To Be Filled By O.E.M./880G Extreme3
[  694.231111] RIP: 0010:[<ffffffff810db878>]  [<ffffffff810db878>] 
dup_fd+0x168/0x300
[  694.231210] RSP: 0018:ffff8802f524fdd0  EFLAGS: 00010206
[  694.231258] RAX: 00000000000007f8 RBX: ffff8802f5721b80 RCX: 
bfffffffffffffff
[  694.231308] RDX: 00008802f51cacc0 RSI: 00000000000000ff RDI: 
0000000000000800
[  694.231358] RBP: ffff8803bf419800 R08: ffff88030167f6c0 R09: 
0000000000000003
[  694.231407] R10: 0000000000000001 R11: 4000000000000000 R12: 
0000000000000100
[  694.231457] R13: ffff880417aa9800 R14: ffff88030167f440 R15: 
ffff8803bd8c1600
[  694.231507] FS:  00007f02cfc32700(0000) GS:ffff88041fcc0000(0000) 
knlGS:0000000000000000
[  694.231560] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  694.231609] CR2: 00007f02cf5d4810 CR3: 00000002f52c3000 CR4: 
00000000000006e0
[  694.231657] DR0: 0000000000000045 DR1: 0000000000000000 DR2: 
0000000000000000
[  694.231707] DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[  694.231757] Process keepalive (pid: 11841, threadinfo 
ffff8802f524e000, task ffff8802f5143690)
[  694.231809] Stack:
[  694.231852]  ffff8802f5143690 0000000000000020 ffff8802f56badc0 
ffff8802f5721b90
[  694.232050]  ffff880417aa54e0 0000000001200011 ffff880417aa54e0 
0000000000000000
[  694.232248]  00007f02cfc329d0 ffff8802f5143690 0000000000000000 
ffffffff81037645
[  694.232448] Call Trace:
[  694.232499]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[  694.232549]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[  694.232599]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[  694.232651]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[  694.232699]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[  694.232745] Code: 4c 89 c2 e8 6b e5 0f 00 45 85 e4 74 78 41 8d 44 24 
ff 31 f6 41 ba 01 00 00 00 48 8d 3c c5 08 00 00 00 31 c0 eb 1a 0f 1f 44 
00 00 <f0> 48 ff 42 30 48 89 54 05 00 48 83 c0 08 ff c6 48 39 f8 74 3b
[  694.235190] RIP  [<ffffffff810db878>] dup_fd+0x168/0x300
[  694.235282]  RSP <ffff8802f524fdd0>
[  694.235379] ---[ end trace 949fad05591fcdb3 ]---
[  694.235428] Kernel panic - not syncing: Fatal exception
[  694.235478] Pid: 11841, comm: keepalive Tainted: G      D     2.6.39 #3
[  694.235525] Call Trace:
[  694.235573]  [<ffffffff813cd6f5>] ? panic+0x92/0x18a
[  694.235624]  [<ffffffff81038b61>] ? kmsg_dump+0x41/0xf0
[  694.235676]  [<ffffffff810050ad>] ? oops_end+0x8d/0xa0
[  694.235726]  [<ffffffff813d05ef>] ? general_protection+0x1f/0x30
[  694.235778]  [<ffffffff810db878>] ? dup_fd+0x168/0x300
[  694.235827]  [<ffffffff81037645>] ? copy_process+0xa75/0xfd0
[  694.235877]  [<ffffffff81037c0d>] ? do_fork+0x6d/0x2b0
[  694.235926]  [<ffffffff810457a9>] ? sigprocmask+0x69/0x100
[  694.235978]  [<ffffffff813d0ca3>] ? stub_clone+0x13/0x20
[  694.236028]  [<ffffffff813d0a3b>] ? system_call_fastpath+0x16/0x1b
[  694.236083] Rebooting in 60 seconds..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Hugh Dickins]

On Wed, 1 Jun 2011, Andrea Arcangeli wrote:
> On Wed, Jun 01, 2011 at 08:37:25AM +0800, Brad Campbell wrote:
> > On 01/06/11 06:31, Hugh Dickins wrote:
> > > Brad, my suspicion is that in each case the top 16 bits of RDX have been
> > > mysteriously corrupted from ffff to 0000, causing the general protection
> > > faults.  I don't understand what that has to do with KSM.
> > >
> > > But it's only a suspicion, because I can't make sense of the "Code:"
> > > lines in your traces, they have more than the expected 64 bytes, and
> > > only one of them has a ">" (with no"<") to mark faulting instruction.
> > >
> > > I did try compiling the 2.6.39 kernel from your config, but of course
> > > we have different compilers, so although I got close, it wasn't exact.
> > >
> > > Would you mind mailing me privately (it's about 73MB) the "objdump -trd"
> > > output for your original vmlinux (with KSM on)?  (Those -trd options are
> > > the ones I'm used to typing, I bet not they're not all relevant.)
> > >
> > > Of course, it's only a tiny fraction of that output that I need,
> > > might be better to cut it down to remove_rmap_item_from_tree and
> > > dup_fd and ksm_scan_thread, if you have the time to do so.
> > 
> > Would you believe about 20 seconds after I pressed send the kernel oopsed.
> > 
> > http://www.fnarfbargle.com/private/003_kernel_oops/
> > 
> > oops reproduced here, but an un-munged version is in that directory 
> > alongside the kernel.
> > 
> > [36542.880228] general protection fault: 0000 [#1] SMP
> 
> Reminds me of another oops that was reported on the kvm list for
> 2.6.38.1 with message id 4D8C6110.6090204. There the top 16 bits of
> rsi were flipped and it was a general protection too because of
> hitting on the not mappable virtual range.
> 
> http://www.virtall.com/files/temp/kvm.txt
> http://www.virtall.com/files/temp/config-2.6.38.1
> http://virtall.com/files/temp/mmu-objdump.txt
> 
> That oops happened in kvm_unmap_rmapp though, but it looked memory
> corruption (Avi suggested use after free) but it was a production
> system so we couldn't debug it further.
> 
> I recommend next thing to reproduce again with 2.6.39 or
> 3.0.0-rc1. Let's fix your scsi trouble if needed but it's better you
> test with 2.6.39.

Brad, thanks for this and the other further crash, with vmlinux etc:
very helpful info.

Andrea, I'm pretty sure you're right to connect Brad's report with
the one above.

In four out of five of Brad's reports (cannot tell in the fifth),
the bad pointer (with top 16 bits 0000 instead of ffff) had been
loaded from SLUB memory at an address offset 0x7f8 (1 case) or
0xff8 (3 cases) i.e. it's the short at 0x7fe or 0xffe that has
been zeroed.

No reason to suspect KSM's rmap_item code, or file table handling:
they just seem to be the victims of corruption from elsewhere.

I notice %rax and %rsi, the corrupted pointer in your kvm.txt
case, is itself a ...7f8 address; and %r13 an ...ff8 address.
I've not even glanced at the code, but I wonder if that implies
that KVM is close to the origin of the corruption.

I doubt I'll be able to spend more time on this, hope you can
take over.

I guess Brad could try SLUB debugging, boot with slub_debug=P
for poisoning perhaps; though it might upset alignments and
drive the problem underground.  Or see if the same happens
with SLAB instead of SLUB.

But I rather hope that you or someone will understand the 7fe clue.

Thanks,
Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 01/06/11 12:52, Hugh Dickins wrote:

>
> I guess Brad could try SLUB debugging, boot with slub_debug=P
> for poisoning perhaps; though it might upset alignments and
> drive the problem underground.  Or see if the same happens
> with SLAB instead of SLUB.

Not much use I'm afraid.
This is all I get in the log

[ 3161.300073] 
=============================================================================
[ 3161.300147] BUG kmalloc-512: Freechain corrupt

The qemu process is then frozen, unkillable but reported in state "R"

13881 ?        R      3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 
1024 -smp 2,sockets=2,cores=1,threads=1 -nam

The machine then progressively dies until it's frozen solid with no 
further error messages.

I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b, 
but at least it responded to that.

On the bright side I can reproduce it at will.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/01/2011 09:31 AM, Brad Campbell wrote:
> On 01/06/11 12:52, Hugh Dickins wrote:
>
>>
>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>> for poisoning perhaps; though it might upset alignments and
>> drive the problem underground.  Or see if the same happens
>> with SLAB instead of SLUB.
>
> Not much use I'm afraid.
> This is all I get in the log
>
> [ 3161.300073] 
> =============================================================================
> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>
> The qemu process is then frozen, unkillable but reported in state "R"
>
> 13881 ?        R      3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 
> 1024 -smp 2,sockets=2,cores=1,threads=1 -nam
>
> The machine then progressively dies until it's frozen solid with no 
> further error messages.
>
> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b, 
> but at least it responded to that.
>
> On the bright side I can reproduce it at will.

Please try slub_debug=FZPU; that should point the finger (hopefully at 
somebody else).

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 01/06/11 14:56, Avi Kivity wrote:
> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>> On 01/06/11 12:52, Hugh Dickins wrote:
>>
>>>
>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>> for poisoning perhaps; though it might upset alignments and
>>> drive the problem underground. Or see if the same happens
>>> with SLAB instead of SLUB.
>>
>> Not much use I'm afraid.
>> This is all I get in the log
>>
>> [ 3161.300073]
>> =============================================================================
>>
>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>
>> The qemu process is then frozen, unkillable but reported in state "R"
>>
>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>> 2,sockets=2,cores=1,threads=1 -nam
>>
>> The machine then progressively dies until it's frozen solid with no
>> further error messages.
>>
>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>> but at least it responded to that.
>>
>> On the bright side I can reproduce it at will.
>
> Please try slub_debug=FZPU; that should point the finger (hopefully at
> somebody else).
>

Well the first attempt locked the machine solid. No network, no console..

I saw 
"=========================================================================="

on the console.. nothing after that. Would not respond to sysrq-t or any 
other sysrq combination other than -b, which rebooted the box.


No output on netconsole at all, I had to walk to the other building to 
look at the monitor and reboot it.

The second attempt jammed netconsole again, but I managed to get this 
from an ssh session I already had established. The machine died a slow 
and horrible death, but remained interactive enough for me to reboot it with

echo b > /proc/sysrq-trigger

Nothing else worked.



[  376.269051] 
=============================================================================
[  413.755328] 
=============================================================================
[  413.755337] BUG kmalloc-1024: Object padding overwritten
[  413.755341] 
-----------------------------------------------------------------------------
[  413.755344]
[  413.755349] INFO: 0xffff880417179566-0xffff880417179567. First byte 
0x0 instead of 0x5a
[  413.755363] INFO: Allocated in tcp_send_ack+0x26/0x120 age=5320 cpu=5 
pid=0
[  413.755374] INFO: Freed in __kfree_skb+0x11/0x90 age=5320 cpu=5 pid=0
[  413.755380] INFO: Slab 0xffffea000e50d240 objects=29 used=5 
fp=0xffff880417179120 flags=0x80000000000040c1
[  413.755386] INFO: Object 0xffff880417179120 @offset=4384 
fp=0xffff8804171799b0
[  413.755389]
[  413.755392] Bytes b4 0xffff880417179110:  d9 2b 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a A?+......ZZZZZZZZ
[  413.755408]   Object 0xffff880417179120:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755423]   Object 0xffff880417179130:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755438]   Object 0xffff880417179140:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755452]   Object 0xffff880417179150:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755466]   Object 0xffff880417179160:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755480]   Object 0xffff880417179170:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755494]   Object 0xffff880417179180:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755508]   Object 0xffff880417179190:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755522]   Object 0xffff8804171791a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755536]   Object 0xffff8804171791b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755551]   Object 0xffff8804171791c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755565]   Object 0xffff8804171791d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755579]   Object 0xffff8804171791e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755593]   Object 0xffff8804171791f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755607]   Object 0xffff880417179200:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755621]   Object 0xffff880417179210:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755635]   Object 0xffff880417179220:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755650]   Object 0xffff880417179230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755664]   Object 0xffff880417179240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755678]   Object 0xffff880417179250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755692]   Object 0xffff880417179260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755706]   Object 0xffff880417179270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755720]   Object 0xffff880417179280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755734]   Object 0xffff880417179290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755749]   Object 0xffff8804171792a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755763]   Object 0xffff8804171792b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755777]   Object 0xffff8804171792c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755791]   Object 0xffff8804171792d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755805]   Object 0xffff8804171792e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755819]   Object 0xffff8804171792f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755834]   Object 0xffff880417179300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755848]   Object 0xffff880417179310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755862]   Object 0xffff880417179320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755876]   Object 0xffff880417179330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755890]   Object 0xffff880417179340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755904]   Object 0xffff880417179350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755919]   Object 0xffff880417179360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755933]   Object 0xffff880417179370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755947]   Object 0xffff880417179380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755961]   Object 0xffff880417179390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755975]   Object 0xffff8804171793a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.755989]   Object 0xffff8804171793b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756004]   Object 0xffff8804171793c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756018]   Object 0xffff8804171793d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756032]   Object 0xffff8804171793e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756046]   Object 0xffff8804171793f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756060]   Object 0xffff880417179400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756074]   Object 0xffff880417179410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756089]   Object 0xffff880417179420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756103]   Object 0xffff880417179430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756117]   Object 0xffff880417179440:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756131]   Object 0xffff880417179450:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756145]   Object 0xffff880417179460:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756160]   Object 0xffff880417179470:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756174]   Object 0xffff880417179480:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756188]   Object 0xffff880417179490:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756202]   Object 0xffff8804171794a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756216]   Object 0xffff8804171794b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756230]   Object 0xffff8804171794c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756245]   Object 0xffff8804171794d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756259]   Object 0xffff8804171794e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756273]   Object 0xffff8804171794f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756287]   Object 0xffff880417179500:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  413.756301]   Object 0xffff880417179510:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  413.756316]  Redzone 0xffff880417179520:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  413.756329]  Padding 0xffff880417179560:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  413.756345] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[  413.756349] Call Trace:
[  413.756353]  <IRQ>  [<ffffffff810b7ccd>] ? 
check_bytes_and_report+0x10d/0x150
[  413.756372]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756379]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  413.756387]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756393]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  413.756400]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  413.756409]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  413.756416]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
[  413.756424]  [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
[  413.756431]  [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
[  413.756439]  [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
[  413.756447]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756453]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[  413.756459]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756465]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
[  413.756472]  [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
[  413.756479]  [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
[  413.756487]  [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
[  413.756492]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756498]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[  413.756504]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756510]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
[  413.756516]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[  413.756522]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
[  413.756528]  [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
[  413.756534]  [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
[  413.756540]  [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
[  413.756546]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756552]  [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
[  413.756560]  [<ffffffff813a60f0>] ? br_nf_pre_routing_finish+0x1a0/0x330
[  413.756568]  [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
[  413.756577]  [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
[  413.756582]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
[  413.756589]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756594]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
[  413.756600]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
[  413.756607]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  413.756613]  [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
[  413.756620]  [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
[  413.756627]  [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
[  413.756634]  [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
[  413.756641]  [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
[  413.756650]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[  413.756657]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[  413.756660]  <EOI>  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[  413.756673]  [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
[  413.756681]  [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
[  413.756688]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[  413.756697]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  413.756704]  [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
[  413.756711]  [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
[  413.756719]  [<ffffffff81056ab7>] ? __hrtimer_start_range_ns+0x1e7/0x410
[  413.756726]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  413.756734]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  413.756742]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[  413.756750]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  413.756756] FIX kmalloc-1024: Restoring 
0xffff880417179566-0xffff880417179567=0x5a
[  413.756760]
[  556.640033] 
=============================================================================
[  556.640041] BUG kmalloc-512: Object padding overwritten
[  556.640045] 
-----------------------------------------------------------------------------
[  556.640048]
[  556.640053] INFO: 0xffff880403bf091e-0xffff880403bf091f. First byte 
0x0 instead of 0x5a
[  556.640069] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=21401 cpu=2 pid=5630
[  556.640080] INFO: Freed in __kfree_skb+0x11/0x90 age=21386 cpu=2 pid=3753
[  556.640086] INFO: Slab 0xffffea000e0d1c80 objects=28 used=10 
fp=0xffff880403bf06d8 flags=0x80000000000040c1
[  556.640092] INFO: Object 0xffff880403bf06d8 @offset=1752 
fp=0xffff880403bf1488
[  556.640096]
[  556.640099] Bytes b4 0xffff880403bf06c8:  90 c4 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a .A?A?A?....ZZZZZZZZ
[  556.640115]   Object 0xffff880403bf06d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640130]   Object 0xffff880403bf06e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640144]   Object 0xffff880403bf06f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640158]   Object 0xffff880403bf0708:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640172]   Object 0xffff880403bf0718:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640187]   Object 0xffff880403bf0728:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640201]   Object 0xffff880403bf0738:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640215]   Object 0xffff880403bf0748:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640229]   Object 0xffff880403bf0758:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640243]   Object 0xffff880403bf0768:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640257]   Object 0xffff880403bf0778:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640271]   Object 0xffff880403bf0788:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640286]   Object 0xffff880403bf0798:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640300]   Object 0xffff880403bf07a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640314]   Object 0xffff880403bf07b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640328]   Object 0xffff880403bf07c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640343]   Object 0xffff880403bf07d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640357]   Object 0xffff880403bf07e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640371]   Object 0xffff880403bf07f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640385]   Object 0xffff880403bf0808:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640399]   Object 0xffff880403bf0818:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640413]   Object 0xffff880403bf0828:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640428]   Object 0xffff880403bf0838:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640442]   Object 0xffff880403bf0848:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640456]   Object 0xffff880403bf0858:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640471]   Object 0xffff880403bf0868:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640485]   Object 0xffff880403bf0878:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640499]   Object 0xffff880403bf0888:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640513]   Object 0xffff880403bf0898:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640527]   Object 0xffff880403bf08a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640542]   Object 0xffff880403bf08b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  556.640556]   Object 0xffff880403bf08c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  556.640570]  Redzone 0xffff880403bf08d8:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  556.640583]  Padding 0xffff880403bf0918:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  556.640599] Pid: 4809, comm: qemu Not tainted 2.6.39 #2
[  556.640603] Call Trace:
[  556.640607]  <IRQ>  [<ffffffff810b7ccd>] ? 
check_bytes_and_report+0x10d/0x150
[  556.640626]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640632]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  556.640640]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640647]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  556.640654]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  556.640661]  [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[  556.640668]  [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[  556.640676]  [<ffffffff8131aae7>] ? __netdev_alloc_skb+0x17/0x40
[  556.640684]  [<ffffffff813a7b93>] ? 
br_ip4_multicast_alloc_query+0x23/0x1c0
[  556.640690]  [<ffffffff813a7e76>] ? br_multicast_send_query+0x76/0x130
[  556.640698]  [<ffffffff8104dba8>] ? wq_worker_waking_up+0x8/0x30
[  556.640706]  [<ffffffff8102ca81>] ? sched_slice+0x51/0x80
[  556.640711]  [<ffffffff813a7fb5>] ? 
br_multicast_port_query_expired+0x45/0x70
[  556.640719]  [<ffffffff81043f77>] ? run_timer_softirq+0x137/0x270
[  556.640725]  [<ffffffff81033fe9>] ? scheduler_tick+0x289/0x2d0
[  556.640731]  [<ffffffff813a7f70>] ? br_multicast_query_expired+0x40/0x40
[  556.640740]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
[  556.640747]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
[  556.640755]  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
[  556.640762]  [<ffffffff8103ddbe>] ? irq_exit+0x8e/0xb0
[  556.640770]  [<ffffffff81019d1a>] ? smp_apic_timer_interrupt+0x6a/0xa0
[  556.640776]  [<ffffffff813d7e53>] ? apic_timer_interrupt+0x13/0x20
[  556.640780]  <EOI>  [<ffffffff8102965a>] ? 
flush_tlb_others_ipi+0x11a/0x130
[  556.640791]  [<ffffffff81029647>] ? flush_tlb_others_ipi+0x107/0x130
[  556.640799]  [<ffffffff810aadab>] ? ptep_clear_flush+0xb/0x10
[  556.640806]  [<ffffffff8109a40b>] ? do_wp_page+0x34b/0x7a0
[  556.640836]  [<ffffffffa00a188a>] ? kvm_read_guest_page+0x5a/0x70 [kvm]
[  556.640863]  [<ffffffffa00b7db4>] ? 
paging64_walk_addr_generic+0x264/0x4c0 [kvm]
[  556.640871]  [<ffffffff8109acad>] ? handle_pte_fault+0x44d/0x990
[  556.640878]  [<ffffffff8109b4e8>] ? follow_page+0x268/0x440
[  556.640886]  [<ffffffff8109bd4d>] ? __get_user_pages+0x12d/0x530
[  556.640892]  [<ffffffff81028dbb>] ? gup_pud_range+0x12b/0x1b0
[  556.640914]  [<ffffffffa00a2d2e>] ? get_user_page_nowait+0x2e/0x40 [kvm]
[  556.640935]  [<ffffffffa00a2e7b>] ? hva_to_pfn+0x13b/0x440 [kvm]
[  556.640956]  [<ffffffffa00a31e9>] ? __gfn_to_pfn+0x39/0xf0 [kvm]
[  556.640979]  [<ffffffffa00b825d>] ? try_async_pf+0x4d/0x190 [kvm]
[  556.641002]  [<ffffffffa00b92be>] ? tdp_page_fault+0x10e/0x200 [kvm]
[  556.641026]  [<ffffffffa00b968c>] ? kvm_mmu_page_fault+0x1c/0x80 [kvm]
[  556.641050]  [<ffffffffa00b0a7a>] ? 
kvm_arch_vcpu_ioctl_run+0x3fa/0xcf0 [kvm]
[  556.641058]  [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[  556.641065]  [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[  556.641086]  [<ffffffffa00a517f>] ? kvm_vcpu_ioctl+0x4df/0x5e0 [kvm]
[  556.641097]  [<ffffffff81046671>] ? __dequeue_signal+0xe1/0x170
[  556.641104]  [<ffffffff8104821c>] ? do_send_sig_info+0x6c/0x90
[  556.641112]  [<ffffffff81046aac>] ? dequeue_signal+0x3c/0x170
[  556.641119]  [<ffffffff8104551f>] ? copy_siginfo_to_user+0xef/0x1d0
[  556.641125]  [<ffffffff810d26db>] ? do_vfs_ioctl+0x9b/0x4f0
[  556.641132]  [<ffffffff8106494a>] ? sys_futex+0x7a/0x180
[  556.641137]  [<ffffffff810d2b79>] ? sys_ioctl+0x49/0x80
[  556.641145]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  556.641152] FIX kmalloc-512: Restoring 
0xffff880403bf091e-0xffff880403bf091f=0x5a
[  556.641155]
[  602.610062] INFO: task ksmd:552 blocked for more than 120 seconds.
[  602.610068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  602.610073] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  602.610083]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  602.610090]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  602.610097]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  602.610104] Call Trace:
[  602.610121]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  602.610132]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  602.610141]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  602.610148]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  602.610154]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  602.610162]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  602.610167]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  602.610174]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  602.610181]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  602.610189]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  602.610194]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  602.610200]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  602.610207]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  602.610214]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  602.610221]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  602.610227]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  602.610233] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  602.610237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  602.610241] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  602.610248]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  602.610255]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  602.610262]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  602.610268] Call Trace:
[  602.610275]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  602.610282]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  602.610289]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  602.610295]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  602.610302]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  602.610307]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  602.610314]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  602.610320]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  602.610326]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  602.610333]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  602.610342]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  602.610348]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  602.610356]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  602.610364]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  602.610371]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  602.610377]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  602.610385]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  602.610390]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  644.689408] 
=============================================================================
[  644.689416] BUG kmalloc-512: Object padding overwritten
[  644.689420] 
-----------------------------------------------------------------------------
[  644.689423]
[  644.689428] INFO: 0xffff88041411f476-0xffff88041411f477. First byte 
0x0 instead of 0x5a
[  644.689444] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=18648 
cpu=3 pid=5915
[  644.689454] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=18648 
cpu=3 pid=5915
[  644.689461] INFO: Slab 0xffffea000e463e20 objects=28 used=10 
fp=0xffff88041411f230 flags=0x80000000000040c1
[  644.689467] INFO: Object 0xffff88041411f230 @offset=12848 
fp=0xffff88041411eda0
[  644.689470]
[  644.689474] Bytes b4 0xffff88041411f220:  22 6f 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a "o......ZZZZZZZZ
[  644.689490]   Object 0xffff88041411f230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689505]   Object 0xffff88041411f240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689519]   Object 0xffff88041411f250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689533]   Object 0xffff88041411f260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689547]   Object 0xffff88041411f270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689562]   Object 0xffff88041411f280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689576]   Object 0xffff88041411f290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689590]   Object 0xffff88041411f2a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689604]   Object 0xffff88041411f2b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689618]   Object 0xffff88041411f2c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689632]   Object 0xffff88041411f2d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689647]   Object 0xffff88041411f2e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689661]   Object 0xffff88041411f2f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689675]   Object 0xffff88041411f300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689689]   Object 0xffff88041411f310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689703]   Object 0xffff88041411f320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689717]   Object 0xffff88041411f330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689732]   Object 0xffff88041411f340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689746]   Object 0xffff88041411f350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689760]   Object 0xffff88041411f360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689774]   Object 0xffff88041411f370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689788]   Object 0xffff88041411f380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689802]   Object 0xffff88041411f390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689817]   Object 0xffff88041411f3a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689831]   Object 0xffff88041411f3b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689845]   Object 0xffff88041411f3c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689859]   Object 0xffff88041411f3d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689873]   Object 0xffff88041411f3e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689888]   Object 0xffff88041411f3f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689902]   Object 0xffff88041411f400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689916]   Object 0xffff88041411f410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  644.689930]   Object 0xffff88041411f420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  644.689945]  Redzone 0xffff88041411f430:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  644.689957]  Padding 0xffff88041411f470:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  644.689974] Pid: 6597, comm: awk Not tainted 2.6.39 #2
[  644.689978] Call Trace:
[  644.689989]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  644.689998]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690005]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  644.690012]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690019]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  644.690026]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  644.690032]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  644.690039]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  644.690047]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  644.690055]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  644.690062]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  644.690069]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  644.690076]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  644.690085]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  644.690092] FIX kmalloc-512: Restoring 
0xffff88041411f476-0xffff88041411f477=0x5a
[  644.690095]
[  704.979481] 
=============================================================================
[  704.979488] BUG kmalloc-512: Object padding overwritten
[  704.979492] 
-----------------------------------------------------------------------------
[  704.979496]
[  704.979501] INFO: 0xffff8804168c1fee-0xffff8804168c1fef. First byte 
0x0 instead of 0x5a
[  704.979517] INFO: Allocated in load_elf_binary+0xa6c/0x1c00 age=24184 
cpu=0 pid=5999
[  704.979527] INFO: Freed in load_elf_binary+0xab8/0x1c00 age=24184 
cpu=0 pid=5999
[  704.979534] INFO: Slab 0xffffea000e4eea00 objects=28 used=10 
fp=0xffff8804168c1da8 flags=0x80000000000040c1
[  704.979540] INFO: Object 0xffff8804168c1da8 @offset=7592 
fp=0xffff8804168c2910
[  704.979543]
[  704.979547] Bytes b4 0xffff8804168c1d98:  93 98 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ..A?A?....ZZZZZZZZ
[  704.979563]   Object 0xffff8804168c1da8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979578]   Object 0xffff8804168c1db8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979592]   Object 0xffff8804168c1dc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979606]   Object 0xffff8804168c1dd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979620]   Object 0xffff8804168c1de8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979634]   Object 0xffff8804168c1df8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979648]   Object 0xffff8804168c1e08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979662]   Object 0xffff8804168c1e18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979676]   Object 0xffff8804168c1e28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979690]   Object 0xffff8804168c1e38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979704]   Object 0xffff8804168c1e48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979719]   Object 0xffff8804168c1e58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979733]   Object 0xffff8804168c1e68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979747]   Object 0xffff8804168c1e78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979761]   Object 0xffff8804168c1e88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979775]   Object 0xffff8804168c1e98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979789]   Object 0xffff8804168c1ea8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979803]   Object 0xffff8804168c1eb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979817]   Object 0xffff8804168c1ec8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979831]   Object 0xffff8804168c1ed8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979845]   Object 0xffff8804168c1ee8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979859]   Object 0xffff8804168c1ef8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979873]   Object 0xffff8804168c1f08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979888]   Object 0xffff8804168c1f18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979902]   Object 0xffff8804168c1f28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979916]   Object 0xffff8804168c1f38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979930]   Object 0xffff8804168c1f48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979944]   Object 0xffff8804168c1f58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979958]   Object 0xffff8804168c1f68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979972]   Object 0xffff8804168c1f78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.979986]   Object 0xffff8804168c1f88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  704.980000]   Object 0xffff8804168c1f98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  704.980015]  Redzone 0xffff8804168c1fa8:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  704.980028]  Padding 0xffff8804168c1fe8:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  704.980044] Pid: 6812, comm: get-rrdtool-dat Not tainted 2.6.39 #2
[  704.980048] Call Trace:
[  704.980060]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  704.980069]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980075]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  704.980083]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980090]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  704.980096]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  704.980102]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  704.980109]  [<ffffffff81106e49>] ? load_elf_binary+0x179/0x1c00
[  704.980117]  [<ffffffff8109b782>] ? __pte_alloc+0x42/0x130
[  704.980123]  [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[  704.980130]  [<ffffffff8109b5a5>] ? follow_page+0x325/0x440
[  704.980137]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  704.980145]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  704.980152]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  704.980158]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  704.980166]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  704.980175]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  704.980182] FIX kmalloc-512: Restoring 
0xffff8804168c1fee-0xffff8804168c1fef=0x5a
[  704.980185]
[  722.610058] INFO: task ksmd:552 blocked for more than 120 seconds.
[  722.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610069] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  722.610078]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  722.610085]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  722.610092]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  722.610099] Call Trace:
[  722.610115]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  722.610127]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  722.610135]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  722.610142]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  722.610148]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610156]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  722.610162]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  722.610168]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  722.610175]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  722.610182]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610188]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  722.610194]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  722.610201]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610208]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610215]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610221]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610226] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  722.610230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610235] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  722.610241]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  722.610248]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  722.610255]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  722.610261] Call Trace:
[  722.610268]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  722.610275]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  722.610282]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  722.610289]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  722.610295]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  722.610301]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610308]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  722.610313]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  722.610319]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  722.610326]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  722.610335]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  722.610342]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610350]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  722.610358]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  722.610365]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610371]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610378]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610384]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610392] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[  722.610396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610400] jbd2/md0-8      D 0000000000000005     0  2528      2 
0x00000000
[  722.610407]  ffff88041d91c570 0000000000000046 ffff88041baeea28 
ffffea0000000000
[  722.610413]  ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000 
ffff880419f9a010
[  722.610420]  ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28 
ffffffff810b8495
[  722.610426] Call Trace:
[  722.610432]  [<ffffffff810b8495>] ? init_object+0x85/0xa0
[  722.610438]  [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[  722.610447]  [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[  722.610455]  [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[  722.610461]  [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[  722.610469]  [<ffffffff811d4403>] ? 
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[  722.610476]  [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[  722.610482]  [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[  722.610489]  [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[  722.610496]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  722.610504]  [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[  722.610509]  [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[  722.610518]  [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[  722.610526]  [<ffffffff81176276>] ? 
jbd2_journal_commit_transaction+0x796/0x1270
[  722.610536]  [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[  722.610542]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  722.610549]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  722.610556]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  722.610563]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610569]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610576]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610582]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  722.610591] INFO: task nfsd:4326 blocked for more than 120 seconds.
[  722.610594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  722.610598] nfsd            D 0000000000000000     0  4326      2 
0x00000000
[  722.610605]  ffff88041b604bc0 0000000000000046 0000000000000016 
ffffffff00000000
[  722.610611]  ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000 
ffff88041c526010
[  722.610617]  ffff88041c527fd8 ffff88041b604bc0 000000000000009c 
ffff880000000000
[  722.610623] Call Trace:
[  722.610630]  [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[  722.610637]  [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[  722.610643]  [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[  722.610650]  [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[  722.610657]  [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[  722.610665]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  722.610672]  [<ffffffff81174ef9>] ? 
jbd2_journal_get_write_access+0x29/0x50
[  722.610680]  [<ffffffff8115f122>] ? 
__ext4_journal_get_write_access+0x32/0x80
[  722.610689]  [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[  722.610696]  [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[  722.610703]  [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[  722.610711]  [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[  722.610719]  [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[  722.610726]  [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[  722.610733]  [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[  722.610741]  [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[  722.610747]  [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[  722.610755]  [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[  722.610763]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  722.610769]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  722.610776]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  722.610783]  [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[  722.610789]  [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[  722.610795]  [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[  722.610803]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  722.610811]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  722.610819]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  722.610825]  [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[  722.610831]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  722.610838]  [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[  722.610844]  [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[  722.610851]  [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[  722.610857]  [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[  722.610862]  [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[  722.610868]  [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[  722.610876]  [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[  722.610882]  [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[  722.610887]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  722.610894]  [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[  722.610901]  [<ffffffff8119784d>] ? nfsd+0xad/0x150
[  722.610907]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  722.610913]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  722.610921]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  722.610926]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  765.241265] 
=============================================================================
[  765.241267] BUG kmalloc-512: Object padding overwritten
[  765.241268] 
-----------------------------------------------------------------------------
[  765.241269]
[  765.241271] INFO: 0xffff8804168c1486-0xffff8804168c1487. First byte 
0x0 instead of 0x5a
[  765.241277] INFO: Allocated in bio_kmalloc+0x2d/0x70 age=73010 cpu=5 
pid=2495
[  765.241281] INFO: Freed in r10buf_pool_free+0x71/0x90 age=72999 cpu=5 
pid=1728
[  765.241283] INFO: Slab 0xffffea000e4eea00 objects=28 used=13 
fp=0xffff8804168c1240 flags=0x80000000000040c1
[  765.241284] INFO: Object 0xffff8804168c1240 @offset=4672 
fp=0xffff8804168c0ff8
[  765.241285]
[  765.241286] Bytes b4 0xffff8804168c1230:  95 98 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ..A?A?....ZZZZZZZZ
[  765.241291]   Object 0xffff8804168c1240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241294]   Object 0xffff8804168c1250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241298]   Object 0xffff8804168c1260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241301]   Object 0xffff8804168c1270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241305]   Object 0xffff8804168c1280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241308]   Object 0xffff8804168c1290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241312]   Object 0xffff8804168c12a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241315]   Object 0xffff8804168c12b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241319]   Object 0xffff8804168c12c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241322]   Object 0xffff8804168c12d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241326]   Object 0xffff8804168c12e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241329]   Object 0xffff8804168c12f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241332]   Object 0xffff8804168c1300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241336]   Object 0xffff8804168c1310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241339]   Object 0xffff8804168c1320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241343]   Object 0xffff8804168c1330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241346]   Object 0xffff8804168c1340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241350]   Object 0xffff8804168c1350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241353]   Object 0xffff8804168c1360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241357]   Object 0xffff8804168c1370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241360]   Object 0xffff8804168c1380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241364]   Object 0xffff8804168c1390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241367]   Object 0xffff8804168c13a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241371]   Object 0xffff8804168c13b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241374]   Object 0xffff8804168c13c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241378]   Object 0xffff8804168c13d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241381]   Object 0xffff8804168c13e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241385]   Object 0xffff8804168c13f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241388]   Object 0xffff8804168c1400:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241392]   Object 0xffff8804168c1410:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241395]   Object 0xffff8804168c1420:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  765.241399]   Object 0xffff8804168c1430:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  765.241402]  Redzone 0xffff8804168c1440:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  765.241406]  Padding 0xffff8804168c1480:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  765.241410] Pid: 7147, comm: apcaccess Not tainted 2.6.39 #2
[  765.241411] Call Trace:
[  765.241416]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  765.241419]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241421]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  765.241423]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241424]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  765.241426]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  765.241428]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  765.241429]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  765.241432]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  765.241434]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  765.241437]  [<ffffffff811da4ad>] ? strnlen_user+0x2d/0x80
[  765.241439]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  765.241440]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  765.241443]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  765.241446]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  765.241448] FIX kmalloc-512: Restoring 
0xffff8804168c1486-0xffff8804168c1487=0x5a
[  765.241449]
[  789.536436] 
=============================================================================
[  789.536444] BUG kmalloc-2048: Object padding overwritten
[  789.536448] 
-----------------------------------------------------------------------------
[  789.536451]
[  789.536456] INFO: 0xffff8802c8b1a11e-0xffff8802c8b1a11f. First byte 
0x0 instead of 0x5a
[  789.536471] INFO: Allocated in sk_stream_alloc_skb+0x3a/0x110 age=63 
cpu=3 pid=5169
[  789.536482] INFO: Freed in __kfree_skb+0x11/0x90 age=63 cpu=5 pid=0
[  789.536488] INFO: Slab 0xffffea0009be6d40 objects=15 used=4 
fp=0xffff8802c8b198d8 flags=0x80000000000040c1
[  789.536494] INFO: Object 0xffff8802c8b198d8 @offset=6360 
fp=0xffff8802c8b1a968
[  789.536497]
[  789.536501] Bytes b4 0xffff8802c8b198c8:  cc be 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a A?A,......ZZZZZZZZ
[  789.536517]   Object 0xffff8802c8b198d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536532]   Object 0xffff8802c8b198e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536546]   Object 0xffff8802c8b198f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536560]   Object 0xffff8802c8b19908:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536575]   Object 0xffff8802c8b19918:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536589]   Object 0xffff8802c8b19928:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536603]   Object 0xffff8802c8b19938:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536617]   Object 0xffff8802c8b19948:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536631]   Object 0xffff8802c8b19958:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536645]   Object 0xffff8802c8b19968:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536660]   Object 0xffff8802c8b19978:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536674]   Object 0xffff8802c8b19988:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536688]   Object 0xffff8802c8b19998:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536702]   Object 0xffff8802c8b199a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536716]   Object 0xffff8802c8b199b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536730]   Object 0xffff8802c8b199c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536745]   Object 0xffff8802c8b199d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536759]   Object 0xffff8802c8b199e8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536773]   Object 0xffff8802c8b199f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536787]   Object 0xffff8802c8b19a08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536801]   Object 0xffff8802c8b19a18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536816]   Object 0xffff8802c8b19a28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536830]   Object 0xffff8802c8b19a38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536844]   Object 0xffff8802c8b19a48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536858]   Object 0xffff8802c8b19a58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536872]   Object 0xffff8802c8b19a68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536887]   Object 0xffff8802c8b19a78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536901]   Object 0xffff8802c8b19a88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536915]   Object 0xffff8802c8b19a98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536929]   Object 0xffff8802c8b19aa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536943]   Object 0xffff8802c8b19ab8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536958]   Object 0xffff8802c8b19ac8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536972]   Object 0xffff8802c8b19ad8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.536986]   Object 0xffff8802c8b19ae8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537000]   Object 0xffff8802c8b19af8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537014]   Object 0xffff8802c8b19b08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537029]   Object 0xffff8802c8b19b18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537043]   Object 0xffff8802c8b19b28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537057]   Object 0xffff8802c8b19b38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537071]   Object 0xffff8802c8b19b48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537085]   Object 0xffff8802c8b19b58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537100]   Object 0xffff8802c8b19b68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537114]   Object 0xffff8802c8b19b78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537128]   Object 0xffff8802c8b19b88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537143]   Object 0xffff8802c8b19b98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537158]   Object 0xffff8802c8b19ba8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537172]   Object 0xffff8802c8b19bb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537187]   Object 0xffff8802c8b19bc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537201]   Object 0xffff8802c8b19bd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537215]   Object 0xffff8802c8b19be8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537229]   Object 0xffff8802c8b19bf8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537244]   Object 0xffff8802c8b19c08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537258]   Object 0xffff8802c8b19c18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537272]   Object 0xffff8802c8b19c28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537286]   Object 0xffff8802c8b19c38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537301]   Object 0xffff8802c8b19c48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537315]   Object 0xffff8802c8b19c58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537329]   Object 0xffff8802c8b19c68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537343]   Object 0xffff8802c8b19c78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537357]   Object 0xffff8802c8b19c88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537372]   Object 0xffff8802c8b19c98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537386]   Object 0xffff8802c8b19ca8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537400]   Object 0xffff8802c8b19cb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537414]   Object 0xffff8802c8b19cc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537428]   Object 0xffff8802c8b19cd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537443]   Object 0xffff8802c8b19ce8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537457]   Object 0xffff8802c8b19cf8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537471]   Object 0xffff8802c8b19d08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537485]   Object 0xffff8802c8b19d18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537500]   Object 0xffff8802c8b19d28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537514]   Object 0xffff8802c8b19d38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537528]   Object 0xffff8802c8b19d48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537542]   Object 0xffff8802c8b19d58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537557]   Object 0xffff8802c8b19d68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537571]   Object 0xffff8802c8b19d78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537585]   Object 0xffff8802c8b19d88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537599]   Object 0xffff8802c8b19d98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537613]   Object 0xffff8802c8b19da8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537628]   Object 0xffff8802c8b19db8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537642]   Object 0xffff8802c8b19dc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537656]   Object 0xffff8802c8b19dd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537670]   Object 0xffff8802c8b19de8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537685]   Object 0xffff8802c8b19df8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537699]   Object 0xffff8802c8b19e08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537713]   Object 0xffff8802c8b19e18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537727]   Object 0xffff8802c8b19e28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537742]   Object 0xffff8802c8b19e38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537756]   Object 0xffff8802c8b19e48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537770]   Object 0xffff8802c8b19e58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537784]   Object 0xffff8802c8b19e68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537799]   Object 0xffff8802c8b19e78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537813]   Object 0xffff8802c8b19e88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537827]   Object 0xffff8802c8b19e98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537841]   Object 0xffff8802c8b19ea8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537856]   Object 0xffff8802c8b19eb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537870]   Object 0xffff8802c8b19ec8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537884]   Object 0xffff8802c8b19ed8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537898]   Object 0xffff8802c8b19ee8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537912]   Object 0xffff8802c8b19ef8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537927]   Object 0xffff8802c8b19f08:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537941]   Object 0xffff8802c8b19f18:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537955]   Object 0xffff8802c8b19f28:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537969]   Object 0xffff8802c8b19f38:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537984]   Object 0xffff8802c8b19f48:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.537998]   Object 0xffff8802c8b19f58:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538012]   Object 0xffff8802c8b19f68:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538026]   Object 0xffff8802c8b19f78:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538041]   Object 0xffff8802c8b19f88:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538055]   Object 0xffff8802c8b19f98:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538069]   Object 0xffff8802c8b19fa8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538083]   Object 0xffff8802c8b19fb8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538097]   Object 0xffff8802c8b19fc8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538112]   Object 0xffff8802c8b19fd8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538126]   Object 0xffff8802c8b19fe8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538140]   Object 0xffff8802c8b19ff8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538154]   Object 0xffff8802c8b1a008:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538169]   Object 0xffff8802c8b1a018:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538183]   Object 0xffff8802c8b1a028:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538197]   Object 0xffff8802c8b1a038:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538211]   Object 0xffff8802c8b1a048:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538226]   Object 0xffff8802c8b1a058:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538240]   Object 0xffff8802c8b1a068:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538254]   Object 0xffff8802c8b1a078:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538268]   Object 0xffff8802c8b1a088:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538283]   Object 0xffff8802c8b1a098:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538297]   Object 0xffff8802c8b1a0a8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538311]   Object 0xffff8802c8b1a0b8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  789.538325]   Object 0xffff8802c8b1a0c8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  789.538340]  Redzone 0xffff8802c8b1a0d8:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  789.538353]  Padding 0xffff8802c8b1a118:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  789.538369] Pid: 5247, comm: qemu Not tainted 2.6.39 #2
[  789.538373] Call Trace:
[  789.538385]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  789.538394]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538400]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  789.538407]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538414]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  789.538420]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  789.538427]  [<ffffffff810bb70d>] ? __kmalloc_track_caller+0x10d/0x160
[  789.538434]  [<ffffffff813198c2>] ? __alloc_skb+0x72/0x160
[  789.538441]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  789.538450]  [<ffffffff8105af7b>] ? getnstimeofday+0x4b/0xd0
[  789.538457]  [<ffffffff81320d32>] ? netif_rx+0xb2/0x120
[  789.538465]  [<ffffffff812b393e>] ? tun_chr_aio_write+0x18e/0x4e0
[  789.538472]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
[  789.538481]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  789.538489]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  789.538496]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  789.538504]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
[  789.538513]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  789.538519] FIX kmalloc-2048: Restoring 
0xffff8802c8b1a11e-0xffff8802c8b1a11f=0x5a
[  789.538522]
[  825.601291] 
=============================================================================
[  825.601298] BUG kmalloc-512: Object padding overwritten
[  825.601302] 
-----------------------------------------------------------------------------
[  825.601306]
[  825.601311] INFO: 0xffff88041411efe6-0xffff88041411efe7. First byte 
0x0 instead of 0x5a
[  825.601327] INFO: Allocated in sock_alloc_send_pskb+0x1d0/0x320 
age=36742 cpu=3 pid=4846
[  825.601338] INFO: Freed in __kfree_skb+0x11/0x90 age=36742 cpu=3 pid=3781
[  825.601344] INFO: Slab 0xffffea000e463e20 objects=28 used=11 
fp=0xffff88041411eda0 flags=0x80000000000040c1
[  825.601350] INFO: Object 0xffff88041411eda0 @offset=11680 
fp=0xffff88041411cdb0
[  825.601353]
[  825.601357] Bytes b4 0xffff88041411ed90:  e2 96 ff ff 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a Ac.A?A?....ZZZZZZZZ
[  825.601373]   Object 0xffff88041411eda0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601388]   Object 0xffff88041411edb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601402]   Object 0xffff88041411edc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601416]   Object 0xffff88041411edd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601430]   Object 0xffff88041411ede0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601444]   Object 0xffff88041411edf0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601459]   Object 0xffff88041411ee00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601473]   Object 0xffff88041411ee10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601487]   Object 0xffff88041411ee20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601501]   Object 0xffff88041411ee30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601515]   Object 0xffff88041411ee40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601529]   Object 0xffff88041411ee50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601543]   Object 0xffff88041411ee60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601557]   Object 0xffff88041411ee70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601571]   Object 0xffff88041411ee80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601586]   Object 0xffff88041411ee90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601600]   Object 0xffff88041411eea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601614]   Object 0xffff88041411eeb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601628]   Object 0xffff88041411eec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601642]   Object 0xffff88041411eed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601656]   Object 0xffff88041411eee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601670]   Object 0xffff88041411eef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601684]   Object 0xffff88041411ef00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601699]   Object 0xffff88041411ef10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601713]   Object 0xffff88041411ef20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601727]   Object 0xffff88041411ef30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601741]   Object 0xffff88041411ef40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601755]   Object 0xffff88041411ef50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601769]   Object 0xffff88041411ef60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601783]   Object 0xffff88041411ef70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601797]   Object 0xffff88041411ef80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  825.601812]   Object 0xffff88041411ef90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  825.601826]  Redzone 0xffff88041411efa0:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  825.601839]  Padding 0xffff88041411efe0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  825.601855] Pid: 7353, comm: apcaccess Not tainted 2.6.39 #2
[  825.601859] Call Trace:
[  825.601871]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  825.601881]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601887]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  825.601895]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601902]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  825.601908]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  825.601914]  [<ffffffff810ba59d>] ? __kmalloc+0x10d/0x160
[  825.601922]  [<ffffffff8110773c>] ? load_elf_binary+0xa6c/0x1c00
[  825.601930]  [<ffffffff8109bd9d>] ? __get_user_pages+0x17d/0x530
[  825.601938]  [<ffffffff810c9556>] ? get_arg_page+0x56/0x100
[  825.601945]  [<ffffffff810c8070>] ? search_binary_handler+0x90/0x240
[  825.601951]  [<ffffffff810c9e9f>] ? do_execve+0x22f/0x2f0
[  825.601959]  [<ffffffff810094a6>] ? sys_execve+0x36/0x60
[  825.601968]  [<ffffffff813d78dc>] ? stub_execve+0x6c/0xc0
[  825.601975] FIX kmalloc-512: Restoring 
0xffff88041411efe6-0xffff88041411efe7=0x5a
[  825.601978]
[  842.610057] INFO: task ksmd:552 blocked for more than 120 seconds.
[  842.610063] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610069] ksmd            D 0000000000000000     0   552      2 
0x00000000
[  842.610078]  ffff88041db68ca0 0000000000000046 000212d01fc11d00 
ffff880400000000
[  842.610086]  ffffffff81593020 ffff88041c1a9fd8 0000000000004000 
ffff88041c1a8010
[  842.610093]  ffff88041c1a9fd8 ffff88041db68ca0 0000000000000000 
0000000200000002
[  842.610100] Call Trace:
[  842.610116]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  842.610128]  [<ffffffff8102f69c>] ? enqueue_task_fair+0x14c/0x190
[  842.610137]  [<ffffffff8102db27>] ? task_rq_lock+0x47/0x90
[  842.610143]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  842.610150]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610157]  [<ffffffff8104cb94>] ? flush_work+0x24/0x30
[  842.610163]  [<ffffffff8104be10>] ? do_work_for_cpu+0x20/0x20
[  842.610170]  [<ffffffff8104e0ab>] ? schedule_on_each_cpu+0xab/0xe0
[  842.610177]  [<ffffffff810b5c05>] ? ksm_scan_thread+0x7f5/0xc20
[  842.610184]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610190]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  842.610196]  [<ffffffff810b5410>] ? 
try_to_merge_with_ksm_page+0x570/0x570
[  842.610203]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610210]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610218]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610223]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610229] INFO: task fsnotify_mark:662 blocked for more than 120 
seconds.
[  842.610233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610237] fsnotify_mark   D 0000000000000000     0   662      2 
0x00000000
[  842.610244]  ffff88041c31d860 0000000000000046 0000000000000000 
0000000000000000
[  842.610251]  ffffffff81593020 ffff88041c365fd8 0000000000004000 
ffff88041c364010
[  842.610257]  ffff88041c365fd8 ffff88041c31d860 0000000000000000 
0000000000000000
[  842.610263] Call Trace:
[  842.610270]  [<ffffffff81035521>] ? load_balance+0x91/0x5e0
[  842.610278]  [<ffffffff813d5595>] ? schedule_timeout+0x1c5/0x230
[  842.610285]  [<ffffffff8102f323>] ? pick_next_task_fair+0x103/0x190
[  842.610291]  [<ffffffff813d4a6d>] ? schedule+0x28d/0x910
[  842.610297]  [<ffffffff813d4562>] ? wait_for_common+0xd2/0x180
[  842.610303]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610310]  [<ffffffff81074420>] ? synchronize_rcu_bh+0x50/0x50
[  842.610316]  [<ffffffff8107446a>] ? synchronize_sched+0x4a/0x50
[  842.610322]  [<ffffffff8104f940>] ? find_ge_pid+0x40/0x40
[  842.610329]  [<ffffffff8105742b>] ? __synchronize_srcu+0x5b/0xc0
[  842.610338]  [<ffffffff810f5f63>] ? fsnotify_mark_destroy+0x83/0x150
[  842.610344]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610352]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  842.610360]  [<ffffffff810f5ee0>] ? 
fsnotify_set_mark_ignored_mask_locked+0x20/0x20
[  842.610367]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610373]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610381]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610386]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610395] INFO: task jbd2/md0-8:2528 blocked for more than 120 seconds.
[  842.610399] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610403] jbd2/md0-8      D 0000000000000005     0  2528      2 
0x00000000
[  842.610410]  ffff88041d91c570 0000000000000046 ffff88041baeea28 
ffffea0000000000
[  842.610417]  ffff88041d91a5e0 ffff880419f9bfd8 0000000000004000 
ffff880419f9a010
[  842.610423]  ffff880419f9bfd8 ffff88041d91c570 ffff88041baeea28 
ffffffff810b8495
[  842.610429] Call Trace:
[  842.610435]  [<ffffffff810b8495>] ? init_object+0x85/0xa0
[  842.610442]  [<ffffffff810b8916>] ? free_debug_processing+0x196/0x250
[  842.610450]  [<ffffffff8105adae>] ? ktime_get_ts+0x6e/0xf0
[  842.610458]  [<ffffffff810810f0>] ? __lock_page+0x70/0x70
[  842.610464]  [<ffffffff813d5174>] ? io_schedule+0x84/0xd0
[  842.610473]  [<ffffffff811d4403>] ? 
radix_tree_gang_lookup_tag_slot+0x93/0xf0
[  842.610480]  [<ffffffff810810f9>] ? sleep_on_page+0x9/0x10
[  842.610486]  [<ffffffff813d57df>] ? __wait_on_bit+0x4f/0x80
[  842.610492]  [<ffffffff810812eb>] ? wait_on_page_bit+0x6b/0x80
[  842.610499]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  842.610507]  [<ffffffff8108a458>] ? pagevec_lookup_tag+0x18/0x20
[  842.610512]  [<ffffffff81081f2a>] ? filemap_fdatawait_range+0xfa/0x180
[  842.610521]  [<ffffffff811be09f>] ? submit_bio+0x6f/0xf0
[  842.610530]  [<ffffffff81176276>] ? 
jbd2_journal_commit_transaction+0x796/0x1270
[  842.610539]  [<ffffffff81179ed1>] ? kjournald2+0xb1/0x1e0
[  842.610546]  [<ffffffff81052a20>] ? wake_up_bit+0x40/0x40
[  842.610553]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  842.610560]  [<ffffffff81179e20>] ? commit_timeout+0x10/0x10
[  842.610566]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610572]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610580]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610585]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  842.610594] INFO: task nfsd:4326 blocked for more than 120 seconds.
[  842.610598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  842.610602] nfsd            D 0000000000000000     0  4326      2 
0x00000000
[  842.610608]  ffff88041b604bc0 0000000000000046 0000000000000016 
ffffffff00000000
[  842.610614]  ffff88041d8fa5e0 ffff88041c527fd8 0000000000004000 
ffff88041c526010
[  842.610621]  ffff88041c527fd8 ffff88041b604bc0 000000000000009c 
ffff880000000000
[  842.610626] Call Trace:
[  842.610634]  [<ffffffff8101e845>] ? amd_flush_garts+0x105/0x140
[  842.610640]  [<ffffffff8101fa80>] ? gart_map_sg+0x480/0x480
[  842.610646]  [<ffffffff8101f5d3>] ? flush_gart+0x23/0x50
[  842.610653]  [<ffffffff81080f98>] ? find_get_page+0x18/0x90
[  842.610660]  [<ffffffff81174b95>] ? do_get_write_access+0x265/0x4a0
[  842.610668]  [<ffffffff81052a50>] ? autoremove_wake_function+0x30/0x30
[  842.610675]  [<ffffffff81174ef9>] ? 
jbd2_journal_get_write_access+0x29/0x50
[  842.610684]  [<ffffffff8115f122>] ? 
__ext4_journal_get_write_access+0x32/0x80
[  842.610692]  [<ffffffff81143908>] ? ext4_reserve_inode_write+0x78/0xa0
[  842.610700]  [<ffffffff81143970>] ? ext4_mark_inode_dirty+0x40/0x1e0
[  842.610706]  [<ffffffff81156c0b>] ? ext4_journal_start_sb+0x6b/0x160
[  842.610715]  [<ffffffff81322645>] ? dev_hard_start_xmit+0x305/0x5f0
[  842.610723]  [<ffffffff81352fd0>] ? ip_finish_output2+0x290/0x290
[  842.610730]  [<ffffffff81143c65>] ? ext4_dirty_inode+0x35/0x70
[  842.610738]  [<ffffffff810e4a08>] ? __mark_inode_dirty+0x38/0x210
[  842.610745]  [<ffffffff810d9317>] ? file_update_time+0xf7/0x180
[  842.610751]  [<ffffffff81082458>] ? __generic_file_aio_write+0x1f8/0x430
[  842.610760]  [<ffffffff81373ef9>] ? udp_sendmsg+0x3c9/0x7e0
[  842.610767]  [<ffffffff81314260>] ? sock_alloc_send_pskb+0x1d0/0x320
[  842.610774]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  842.610781]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  842.610788]  [<ffffffff810da8ac>] ? iget_locked+0x4c/0x140
[  842.610794]  [<ffffffff8119a590>] ? fh_compose+0x4c0/0x4c0
[  842.610800]  [<ffffffff8113f6e0>] ? ext4_llseek+0x110/0x110
[  842.610808]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  842.610816]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  842.610824]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  842.610830]  [<ffffffff810b8735>] ? alloc_debug_processing+0x145/0x190
[  842.610836]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  842.610843]  [<ffffffff8113f463>] ? ext4_file_open+0x63/0x180
[  842.610849]  [<ffffffff8119c21d>] ? nfsd_vfs_write+0xed/0x3a0
[  842.610856]  [<ffffffff810c1127>] ? __dentry_open+0x1f7/0x2b0
[  842.610862]  [<ffffffff8119c892>] ? nfsd_open+0xf2/0x1b0
[  842.610867]  [<ffffffff8119cd34>] ? nfsd_write+0xf4/0x110
[  842.610873]  [<ffffffff81199930>] ? nfsd_proc_write+0xb0/0x120
[  842.610880]  [<ffffffff811971c5>] ? nfsd_dispatch+0xf5/0x230
[  842.610886]  [<ffffffff813b456f>] ? svc_process+0x4af/0x820
[  842.610892]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
[  842.610899]  [<ffffffff811977a0>] ? nfsd_svc+0x1b0/0x1b0
[  842.610906]  [<ffffffff8119784d>] ? nfsd+0xad/0x150
[  842.610912]  [<ffffffff810525b6>] ? kthread+0x96/0xa0
[  842.610918]  [<ffffffff813d8294>] ? kernel_thread_helper+0x4/0x10
[  842.610926]  [<ffffffff81052520>] ? kthread_worker_fn+0x120/0x120
[  842.610931]  [<ffffffff813d8290>] ? gs_change+0xb/0xb
[  854.241450] 
=============================================================================
[  854.241459] BUG kmalloc-1024: Object padding overwritten
[  854.241463] 
-----------------------------------------------------------------------------
[  854.241466]
[  854.241471] INFO: 0xffff8802c86059e6-0xffff8802c86059e7. First byte 
0x0 instead of 0x5a
[  854.241488] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90 
age=518 cpu=5 pid=7094
[  854.241500] INFO: Freed in __kfree_skb+0x11/0x90 age=519 cpu=5 pid=5247
[  854.241507] INFO: Slab 0xffffea0009bd5000 objects=29 used=22 
fp=0xffff8802c8606f50 flags=0x80000000000040c1
[  854.241513] INFO: Object 0xffff8802c86055a0 @offset=21920 
fp=0xffff8802c86066c0
[  854.241516]
[  854.241520] Bytes b4 0xffff8802c8605590:  00 00 00 00 00 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
[  854.241536]   Object 0xffff8802c86055a0:  00 00 00 00 00 00 00 00 e0 
61 29 fa 03 88 ff ff ........A a)Ao..A?A?
[  854.241551]   Object 0xffff8802c86055b0:  00 00 00 00 00 00 00 00 00 
00 41 ce 04 00 00 00 ..........AA?....
[  854.241565]   Object 0xffff8802c86055c0:  03 00 00 00 0c 00 00 00 00 
00 00 00 01 00 00 00 ................
[  854.241579]   Object 0xffff8802c86055d0:  00 00 00 00 00 00 00 00 00 
60 00 00 00 00 00 00 .........`......
[  854.241593]   Object 0xffff8802c86055e0:  b7 2c 0a 00 00 00 00 00 00 
00 00 00 00 00 00 00 A.,..............
[  854.241607]   Object 0xffff8802c86055f0:  b7 2c 0a 00 00 00 00 00 00 
00 00 00 00 00 00 00 A.,..............
[  854.241621]   Object 0xffff8802c8605600:  01 00 00 00 00 00 00 00 f0 
77 14 81 ff ff ff ff ........A?w..A?A?A?A?
[  854.241635]   Object 0xffff8802c8605610:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241649]   Object 0xffff8802c8605620:  00 00 00 00 00 00 00 00 b7 
54 3b 02 00 00 00 00 ........A.T;.....
[  854.241663]   Object 0xffff8802c8605630:  b7 54 3b 02 00 00 00 00 20 
00 00 00 00 00 00 00 A.T;.............
[  854.241677]   Object 0xffff8802c8605640:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241691]   Object 0xffff8802c8605650:  b6 54 3b 02 00 00 00 00 00 
10 00 00 00 00 00 00 A?T;.............
[  854.241705]   Object 0xffff8802c8605660:  00 00 00 00 00 00 00 00 00 
80 8e 1a 04 88 ff ff ..............A?A?
[  854.241719]   Object 0xffff8802c8605670:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241733]   Object 0xffff8802c8605680:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241747]   Object 0xffff8802c8605690:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241761]   Object 0xffff8802c86056a0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 10 00 00 ................
[  854.241775]   Object 0xffff8802c86056b0:  b6 54 3b 02 00 00 00 00 00 
60 cb a2 00 00 00 00 A?T;......`A?Ac....
[  854.241789]   Object 0xffff8802c86056c0:  06 06 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241803]   Object 0xffff8802c86056d0:  00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 ................
[  854.241817]   Object 0xffff8802c86056e0:  68 fd 1c c9 02 88 ff ff 00 
00 00 00 00 00 00 00 hA 1/2 .A?..A?A?........
[  854.241831]   Object 0xffff8802c86056f0:  00 60 00 00 00 00 00 00 01 
00 00 00 01 00 00 00 .`..............
[  854.241845]   Object 0xffff8802c8605700:  00 80 c9 dd 39 7f 00 00 01 
00 00 00 01 00 00 00 ..A?A?9...........
[  854.241860]   Object 0xffff8802c8605710:  00 00 00 00 00 00 00 00 08 
81 bb 0c 00 ea ff ff ..........A>>..AaA?A?
[  854.241874]   Object 0xffff8802c8605720:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241888]   Object 0xffff8802c8605730:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241902]   Object 0xffff8802c8605740:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241916]   Object 0xffff8802c8605750:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241930]   Object 0xffff8802c8605760:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241944]   Object 0xffff8802c8605770:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241959]   Object 0xffff8802c8605780:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241973]   Object 0xffff8802c8605790:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.241987]   Object 0xffff8802c86057a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242001]   Object 0xffff8802c86057b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242015]   Object 0xffff8802c86057c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242029]   Object 0xffff8802c86057d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242043]   Object 0xffff8802c86057e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242058]   Object 0xffff8802c86057f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242072]   Object 0xffff8802c8605800:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242086]   Object 0xffff8802c8605810:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242100]   Object 0xffff8802c8605820:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242114]   Object 0xffff8802c8605830:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242128]   Object 0xffff8802c8605840:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242142]   Object 0xffff8802c8605850:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242157]   Object 0xffff8802c8605860:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242171]   Object 0xffff8802c8605870:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242185]   Object 0xffff8802c8605880:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242199]   Object 0xffff8802c8605890:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242213]   Object 0xffff8802c86058a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242227]   Object 0xffff8802c86058b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242241]   Object 0xffff8802c86058c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242256]   Object 0xffff8802c86058d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242270]   Object 0xffff8802c86058e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242284]   Object 0xffff8802c86058f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242298]   Object 0xffff8802c8605900:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242312]   Object 0xffff8802c8605910:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242326]   Object 0xffff8802c8605920:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242340]   Object 0xffff8802c8605930:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242355]   Object 0xffff8802c8605940:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242369]   Object 0xffff8802c8605950:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242383]   Object 0xffff8802c8605960:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242397]   Object 0xffff8802c8605970:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242411]   Object 0xffff8802c8605980:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  854.242425]   Object 0xffff8802c8605990:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  854.242440]  Redzone 0xffff8802c86059a0:  cc cc cc cc cc cc cc cc 
                      A?A?A?A?A?A?A?A?
[  854.242453]  Padding 0xffff8802c86059e0:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  854.242470] Pid: 7094, comm: qemu Not tainted 2.6.39 #2
[  854.242474] Call Trace:
[  854.242485]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  854.242492]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  854.242500]  [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[  854.242507]  [<ffffffff810b88b3>] ? free_debug_processing+0x133/0x250
[  854.242513]  [<ffffffff810b8afb>] ? __slab_free+0x12b/0x140
[  854.242520]  [<ffffffff810f3126>] ? __blockdev_direct_IO+0xa16/0xa90
[  854.242530]  [<ffffffff81144a17>] ? ext4_ind_direct_IO+0xf7/0x410
[  854.242536]  [<ffffffff811477f0>] ? noalloc_get_block_write+0x30/0x30
[  854.242543]  [<ffffffff81082029>] ? __filemap_fdatawrite_range+0x49/0x50
[  854.242550]  [<ffffffff810830c3>] ? generic_file_aio_read+0x653/0x6d0
[  854.242558]  [<ffffffff810567a2>] ? hrtimer_cancel+0x12/0x20
[  854.242565]  [<ffffffff81062137>] ? futex_wait+0x197/0x240
[  854.242572]  [<ffffffff81082a70>] ? delete_from_page_cache+0x70/0x70
[  854.242580]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
[  854.242587]  [<ffffffff8106239f>] ? futex_wake+0x10f/0x120
[  854.242594]  [<ffffffff81063f7b>] ? do_futex+0x11b/0xa70
[  854.242601]  [<ffffffff810471cc>] ? T.680+0x13c/0x280
[  854.242608]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
[  854.242616]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
[  854.242624]  [<ffffffff810c3003>] ? sys_preadv+0xc3/0xd0
[  854.242633]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  854.242640] FIX kmalloc-1024: Restoring 
0xffff8802c86059e6-0xffff8802c86059e7=0x5a
[  854.242643]
[  860.203918] 
=============================================================================
[  860.203927] BUG kmalloc-1024: Object padding overwritten
[  860.203931] 
-----------------------------------------------------------------------------
[  860.203934]
[  860.203939] INFO: 0xffff88041ad15156-0xffff88041ad15157. First byte 
0x0 instead of 0x5a
[  860.203955] INFO: Allocated in __blockdev_direct_IO+0x16c/0xa90 
age=1609 cpu=3 pid=7094
[  860.203965] INFO: Freed in __blockdev_direct_IO+0xa16/0xa90 age=1609 
cpu=3 pid=7094
[  860.203972] INFO: Slab 0xffffea000e5ddb80 objects=29 used=25 
fp=0xffff88041ad14d10 flags=0x80000000000040c1
[  860.203977] INFO: Object 0xffff88041ad14d10 @offset=19728 
fp=0xffff88041ad10000
[  860.203981]
[  860.203984] Bytes b4 0xffff88041ad14d00:  62 28 00 00 01 00 00 00 5a 
5a 5a 5a 5a 5a 5a 5a b(......ZZZZZZZZ
[  860.204000]   Object 0xffff88041ad14d10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204015]   Object 0xffff88041ad14d20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204030]   Object 0xffff88041ad14d30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204044]   Object 0xffff88041ad14d40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204058]   Object 0xffff88041ad14d50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204073]   Object 0xffff88041ad14d60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204087]   Object 0xffff88041ad14d70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204101]   Object 0xffff88041ad14d80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204116]   Object 0xffff88041ad14d90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204130]   Object 0xffff88041ad14da0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204144]   Object 0xffff88041ad14db0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204159]   Object 0xffff88041ad14dc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204173]   Object 0xffff88041ad14dd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204187]   Object 0xffff88041ad14de0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204202]   Object 0xffff88041ad14df0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204216]   Object 0xffff88041ad14e00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204230]   Object 0xffff88041ad14e10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204245]   Object 0xffff88041ad14e20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204259]   Object 0xffff88041ad14e30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204273]   Object 0xffff88041ad14e40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204288]   Object 0xffff88041ad14e50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204302]   Object 0xffff88041ad14e60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204316]   Object 0xffff88041ad14e70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204331]   Object 0xffff88041ad14e80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204345]   Object 0xffff88041ad14e90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204359]   Object 0xffff88041ad14ea0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204374]   Object 0xffff88041ad14eb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204388]   Object 0xffff88041ad14ec0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204402]   Object 0xffff88041ad14ed0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204417]   Object 0xffff88041ad14ee0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204431]   Object 0xffff88041ad14ef0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204445]   Object 0xffff88041ad14f00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204460]   Object 0xffff88041ad14f10:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204474]   Object 0xffff88041ad14f20:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204488]   Object 0xffff88041ad14f30:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204503]   Object 0xffff88041ad14f40:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204517]   Object 0xffff88041ad14f50:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204531]   Object 0xffff88041ad14f60:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204546]   Object 0xffff88041ad14f70:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204560]   Object 0xffff88041ad14f80:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204574]   Object 0xffff88041ad14f90:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204589]   Object 0xffff88041ad14fa0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204603]   Object 0xffff88041ad14fb0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204618]   Object 0xffff88041ad14fc0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204632]   Object 0xffff88041ad14fd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204646]   Object 0xffff88041ad14fe0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204661]   Object 0xffff88041ad14ff0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204675]   Object 0xffff88041ad15000:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204689]   Object 0xffff88041ad15010:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204704]   Object 0xffff88041ad15020:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204718]   Object 0xffff88041ad15030:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204732]   Object 0xffff88041ad15040:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204747]   Object 0xffff88041ad15050:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204761]   Object 0xffff88041ad15060:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204775]   Object 0xffff88041ad15070:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204790]   Object 0xffff88041ad15080:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204804]   Object 0xffff88041ad15090:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204818]   Object 0xffff88041ad150a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204833]   Object 0xffff88041ad150b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204847]   Object 0xffff88041ad150c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204861]   Object 0xffff88041ad150d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204876]   Object 0xffff88041ad150e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204890]   Object 0xffff88041ad150f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[  860.204904]   Object 0xffff88041ad15100:  6b 6b 6b 6b 6b 6b 6b 6b 6b 
6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkkAJPY
[  860.204919]  Redzone 0xffff88041ad15110:  bb bb bb bb bb bb bb bb 
                      A>>A>>A>>A>>A>>A>>A>>A>>
[  860.204932]  Padding 0xffff88041ad15150:  5a 5a 5a 5a 5a 5a 00 00 
                      ZZZZZZ..
[  860.204949] Pid: 7100, comm: qemu Not tainted 2.6.39 #2
[  860.204953] Call Trace:
[  860.204964]  [<ffffffff810b7ccd>] ? check_bytes_and_report+0x10d/0x150
[  860.204973]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.204979]  [<ffffffff810b7db9>] ? check_object+0xa9/0x260
[  860.204986]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.204993]  [<ffffffff810b86f4>] ? alloc_debug_processing+0x104/0x190
[  860.204999]  [<ffffffff810b9ac2>] ? T.912+0x272/0x2d0
[  860.205006]  [<ffffffff810b9bc5>] ? kmem_cache_alloc+0xa5/0xb0
[  860.205013]  [<ffffffff810f287c>] ? __blockdev_direct_IO+0x16c/0xa90
[  860.205022]  [<ffffffff8108a475>] ? pagevec_lookup+0x15/0x20
[  860.205029]  [<ffffffff8108b5b7>] ? 
invalidate_inode_pages2_range+0x87/0x300
[  860.205039]  [<ffffffff81144e41>] ? ext4_direct_IO+0x111/0x1f0
[  860.205045]  [<ffffffff811477a0>] ? _ext4_get_block+0x160/0x160
[  860.205052]  [<ffffffff81144f20>] ? ext4_direct_IO+0x1f0/0x1f0
[  860.205058]  [<ffffffff81082182>] ? generic_file_direct_write+0xd2/0x1b0
[  860.205064]  [<ffffffff8108252b>] ? __generic_file_aio_write+0x2cb/0x430
[  860.205071]  [<ffffffff81082703>] ? generic_file_aio_write+0x73/0xf0
[  860.205079]  [<ffffffff8113f74e>] ? ext4_file_write+0x6e/0x2b0
[  860.205087]  [<ffffffff810c25ef>] ? do_sync_write+0xbf/0x100
[  860.205095]  [<ffffffff810471cc>] ? T.680+0x13c/0x280
[  860.205103]  [<ffffffff810485a8>] ? kill_pid_info+0x38/0x60
[  860.205110]  [<ffffffff81048854>] ? sys_kill+0x94/0x1b0
[  860.205117]  [<ffffffff810c3198>] ? vfs_write+0xc8/0x190
[  860.205124]  [<ffffffff810c32fb>] ? sys_pwrite64+0x9b/0xb0
[  860.205134]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
[  860.205140] FIX kmalloc-1024: Restoring 
0xffff88041ad15156-0xffff88041ad15157=0x5a
[  860.205143]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/01/2011 12:29 PM, Brad Campbell wrote:
> On 01/06/11 14:56, Avi Kivity wrote:
>> On 06/01/2011 09:31 AM, Brad Campbell wrote:
>>> On 01/06/11 12:52, Hugh Dickins wrote:
>>>
>>>>
>>>> I guess Brad could try SLUB debugging, boot with slub_debug=P
>>>> for poisoning perhaps; though it might upset alignments and
>>>> drive the problem underground. Or see if the same happens
>>>> with SLAB instead of SLUB.
>>>
>>> Not much use I'm afraid.
>>> This is all I get in the log
>>>
>>> [ 3161.300073]
>>> ============================================================================= 
>>>
>>>
>>> [ 3161.300147] BUG kmalloc-512: Freechain corrupt
>>>
>>> The qemu process is then frozen, unkillable but reported in state "R"
>>>
>>> 13881 ? R 3:27 /usr/bin/qemu -S -M pc-0.13 -enable-kvm -m 1024 -smp
>>> 2,sockets=2,cores=1,threads=1 -nam
>>>
>>> The machine then progressively dies until it's frozen solid with no
>>> further error messages.
>>>
>>> I stupidly forgot to do an alt-sysrq-t prior to doing an alt-sysrq-b,
>>> but at least it responded to that.
>>>
>>> On the bright side I can reproduce it at will.
>>
>> Please try slub_debug=FZPU; that should point the finger (hopefully at
>> somebody else).
>>
>
> Well the first attempt locked the machine solid. No network, no console..
>
> I saw 
> "=========================================================================="
>
> on the console.. nothing after that. Would not respond to sysrq-t or 
> any other sysrq combination other than -b, which rebooted the box.
>
>
> No output on netconsole at all, I had to walk to the other building to 
> look at the monitor and reboot it.
>
> The second attempt jammed netconsole again, but I managed to get this 
> from an ssh session I already had established. The machine died a slow 
> and horrible death, but remained interactive enough for me to reboot 
> it with
>
> echo b > /proc/sysrq-trigger
>
> Nothing else worked.
>
>
> [  413.756416]  [<ffffffff81318f1c>] ? pskb_expand_head+0x15c/0x250
> [  413.756424]  [<ffffffff813a6c45>] ? nf_bridge_copy_header+0x145/0x160
> [  413.756431]  [<ffffffff8139f78d>] ? br_dev_queue_push_xmit+0x6d/0x80
> [  413.756439]  [<ffffffff813a55a0>] ? br_nf_post_routing+0x2a0/0x2f0
> [  413.756447]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756453]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [  413.756459]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756465]  [<ffffffff8139f720>] ? br_flood_deliver+0x20/0x20
> [  413.756472]  [<ffffffff8139f7da>] ? br_forward_finish+0x3a/0x60
> [  413.756479]  [<ffffffff813a5758>] ? br_nf_forward_finish+0x168/0x170
> [  413.756487]  [<ffffffff813a5c90>] ? br_nf_forward_ip+0x360/0x3a0
> [  413.756492]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756498]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [  413.756504]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756510]  [<ffffffff8139f7a0>] ? br_dev_queue_push_xmit+0x80/0x80
> [  413.756516]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [  413.756522]  [<ffffffff8139f800>] ? br_forward_finish+0x60/0x60
> [  413.756528]  [<ffffffff8139f875>] ? __br_forward+0x75/0xc0
> [  413.756534]  [<ffffffff8139f426>] ? deliver_clone+0x36/0x60
> [  413.756540]  [<ffffffff8139f69d>] ? br_flood+0xbd/0x100
> [  413.756546]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756552]  [<ffffffff813a080e>] ? br_handle_frame_finish+0x25e/0x280
> [  413.756560]  [<ffffffff813a60f0>] ? 
> br_nf_pre_routing_finish+0x1a0/0x330
> [  413.756568]  [<ffffffff813a6958>] ? br_nf_pre_routing+0x6d8/0x800
> [  413.756577]  [<ffffffff8102d46a>] ? enqueue_task+0x3a/0x90
> [  413.756582]  [<ffffffff81346bc4>] ? nf_iterate+0x84/0xb0
> [  413.756589]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756594]  [<ffffffff81346c64>] ? nf_hook_slow+0x74/0x120
> [  413.756600]  [<ffffffff813a05b0>] ? br_handle_local_finish+0x40/0x40
> [  413.756607]  [<ffffffff810339b0>] ? try_to_wake_up+0x2c0/0x2c0
> [  413.756613]  [<ffffffff813a09d9>] ? br_handle_frame+0x1a9/0x280
> [  413.756620]  [<ffffffff813a0830>] ? br_handle_frame_finish+0x280/0x280
> [  413.756627]  [<ffffffff81320ef7>] ? __netif_receive_skb+0x157/0x5c0
> [  413.756634]  [<ffffffff81321443>] ? process_backlog+0xe3/0x1d0
> [  413.756641]  [<ffffffff81321da5>] ? net_rx_action+0xc5/0x1d0
> [  413.756650]  [<ffffffff8103df11>] ? __do_softirq+0x91/0x120
> [  413.756657]  [<ffffffff813d838c>] ? call_softirq+0x1c/0x30
> [  413.756660] <EOI>  [<ffffffff81003cbd>] ? do_softirq+0x4d/0x80
> [  413.756673]  [<ffffffff81321ece>] ? netif_rx_ni+0x1e/0x30
> [  413.756681]  [<ffffffff812b3ae2>] ? tun_chr_aio_write+0x332/0x4e0
> [  413.756688]  [<ffffffff812b37b0>] ? tun_sendmsg+0x4d0/0x4d0
> [  413.756697]  [<ffffffff810c24e9>] ? do_sync_readv_writev+0xa9/0xf0
> [  413.756704]  [<ffffffff81063f9c>] ? do_futex+0x13c/0xa70
> [  413.756711]  [<ffffffff811d6730>] ? timerqueue_add+0x60/0xb0
> [  413.756719]  [<ffffffff81056ab7>] ? 
> __hrtimer_start_range_ns+0x1e7/0x410
> [  413.756726]  [<ffffffff810c231b>] ? rw_copy_check_uvector+0x7b/0x140
> [  413.756734]  [<ffffffff810c2bcf>] ? do_readv_writev+0xdf/0x210
> [  413.756742]  [<ffffffff810c2e7e>] ? sys_writev+0x4e/0xc0
> [  413.756750]  [<ffffffff813d753b>] ? system_call_fastpath+0x16/0x1b
> [  413.756756] FIX kmalloc-1024: Restoring 
> 0xffff880417179566-0xffff880417179567=0x5a

bridge and netfilter, IIRC this was also the problem last time.

Do you have any ebtables loaded?

Can you try building a kernel without ebtables?  Without netfilter at all?

Please run all tests with slub_debug=FZPU.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/01/2011 12:40 PM, Avi Kivity wrote:
>
> bridge and netfilter, IIRC this was also the problem last time.
>
> Do you have any ebtables loaded?
>
> Can you try building a kernel without ebtables?  Without netfilter at 
> all?
>
> Please run all tests with slub_debug=FZPU.
>

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 01/06/11 17:41, Avi Kivity wrote:
> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>
>> bridge and netfilter, IIRC this was also the problem last time.
>>
>> Do you have any ebtables loaded?

Never heard of them, but making a cursory check just in case..

brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
# CONFIG_BRIDGE_NF_EBTABLES is not set

>> Can you try building a kernel without ebtables? Without netfilter at all?

Well, without netfilter I can't get it to crash. The problem is without 
netfilter I can't actually use it the way I use it to get it to crash.

I rebooted into a netfilter kernel, and did all the steps I'd used on 
the no-netfilter kernel and it ticked along happily.

So the result of the experiment is inconclusive. Having said that, the 
backtraces certainly smell networky.

To get it to crash, I have to start IE in the VM and https to the public 
address of the machine, which is then redirected by netfilter back into 
another of the VM's.

I can https directly to the other VM's address, but that does not cause 
it to crash, however without netfilter loaded I can't bounce off the 
public IP. It's all rather confusing really.

What next Sherlock?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/01/2011 01:53 PM, Brad Campbell wrote:
> On 01/06/11 17:41, Avi Kivity wrote:
>> On 06/01/2011 12:40 PM, Avi Kivity wrote:
>>>
>>> bridge and netfilter, IIRC this was also the problem last time.
>>>
>>> Do you have any ebtables loaded?
>
> Never heard of them, but making a cursory check just in case..
>
> brad@srv:/raid10/src/linux-2.6.39$ grep EBTABLE .config
> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
>>> Can you try building a kernel without ebtables? Without netfilter at 
>>> all?
>
> Well, without netfilter I can't get it to crash. The problem is 
> without netfilter I can't actually use it the way I use it to get it 
> to crash.
>
> I rebooted into a netfilter kernel, and did all the steps I'd used on 
> the no-netfilter kernel and it ticked along happily.
>
> So the result of the experiment is inconclusive. Having said that, the 
> backtraces certainly smell networky.
>
> To get it to crash, I have to start IE in the VM and https to the 
> public address of the machine, which is then redirected by netfilter 
> back into another of the VM's.
>
> I can https directly to the other VM's address, but that does not 
> cause it to crash, however without netfilter loaded I can't bounce off 
> the public IP. It's all rather confusing really.
>
> What next Sherlock?
>

Maybe the Sherlocks at netdev@ can tell.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[CaT]

On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
> I rebooted into a netfilter kernel, and did all the steps I'd used
> on the no-netfilter kernel and it ticked along happily.
> 
> So the result of the experiment is inconclusive. Having said that,
> the backtraces certainly smell networky.
> 
> To get it to crash, I have to start IE in the VM and https to the
> public address of the machine, which is then redirected by netfilter
> back into another of the VM's.
> 
> I can https directly to the other VM's address, but that does not
> cause it to crash, however without netfilter loaded I can't bounce
> off the public IP. It's all rather confusing really.
> 
> What next Sherlock?

I think you're hitting something I've seen. Can you try rewriting
your firewall rules so that it does not reference any bridge
interfaces at all. Instead, reference the real interface names
in their place. I'm betting it wont crash.

(netdev added to CC since we're aleady bouncing there)

-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 01/06/11 19:18, CaT wrote:
> On Wed, Jun 01, 2011 at 06:53:31PM +0800, Brad Campbell wrote:
>> I rebooted into a netfilter kernel, and did all the steps I'd used
>> on the no-netfilter kernel and it ticked along happily.
>>
>> So the result of the experiment is inconclusive. Having said that,
>> the backtraces certainly smell networky.
>>
>> To get it to crash, I have to start IE in the VM and https to the
>> public address of the machine, which is then redirected by netfilter
>> back into another of the VM's.
>>
>> I can https directly to the other VM's address, but that does not
>> cause it to crash, however without netfilter loaded I can't bounce
>> off the public IP. It's all rather confusing really.
>>
>> What next Sherlock?
>
> I think you're hitting something I've seen. Can you try rewriting
> your firewall rules so that it does not reference any bridge
> interfaces at all. Instead, reference the real interface names
> in their place. I'm betting it wont crash.
>

Unfortunately the only interface that is mentioned by name anywhere in 
my firewall is $DMZ (which is ppp0 and not part of any bridge).

All of the nat/dnat and other horrible hacks are based on IP addresses.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[CaT]

On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
> Unfortunately the only interface that is mentioned by name anywhere
> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
> 
> All of the nat/dnat and other horrible hacks are based on IP addresses.

Damn. Not referencing the bridge interfaces at all stopped our host from
going down in flames when we passed it a few packets. These are two
of the oopses we got from it. Whilst the kernel here is .35 we got the
same issue from a range of kernels. Seems related.

The oopses may be a bit weird. Copy and paste from an ipmi terminal.


slab error in cache_alloc_debugcheck_after(): cache `size-64': double 
free, or n
Pid: 2431, comm: kvm Tainted: G      D     
2.6.35.9-local.20110314-141930 #1
Call Trace:
<IRQ>  [<ffffffff810fb8bf>] ? __slab_error+0x1f/0x30
  [<ffffffff810fc22b>] ? cache_alloc_debugcheck_after+0x6b/0x1f0
  [<ffffffff81530a00>] ? br_nf_pre_routing_finish+0x0/0x370
  [<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
  [<ffffffff810fdd3d>] ? kmem_cache_alloc_notrace+0x7d/0xf0

  [<ffffffff8153106b>] ? br_nf_pre_routing+0x2fb/0x980
  [<ffffffff81466e66>] ? nf_iterate+0x66/0xb0
  [<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
  [<ffffffff81466f14>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff8152b9f0>] ? br_handle_frame_finish+0x0/0x1c0
  [<ffffffff8152bd3c>] ? br_handle_frame+0x18c/0x250
  [<ffffffff81445459>] ? __netif_receive_skb+0x169/0x2a0
  [<ffffffff81445673>] ? process_backlog+0xe3/0x1d0
  [<ffffffff81446347>] ? net_rx_action+0x87/0x1c0
  [<ffffffff810793f7>] ? __do_softirq+0xa7/0x1d0
  [<ffffffff81035b8c>] ? call_softirq+0x1c/0x30
<EOI>  [<ffffffff81037c6d>] ? do_softirq+0x4d/0x80
  [<ffffffff81446b4e>] ? netif_rx_ni+0x1e/0x30
  [<ffffffff8139541a>] ? tun_chr_aio_write+0x36a/0x510
  [<ffffffff813950b0>] ? tun_chr_aio_write+0x0/0x510
  [<ffffffff81102859>] ? do_sync_readv_writev+0xa9/0xf0
  [<ffffffff810973fb>] ? ktime_get+0x5b/0xe0
  [<ffffffff8104f958>] ? lapic_next_event+0x18/0x20
  [<ffffffff8109be18>] ? tick_dev_program_event+0x38/0x100
  [<ffffffff81102697>] ? rw_copy_check_uvector+0x77/0x130
  [<ffffffff81102f0c>] ? do_readv_writev+0xdc/0x200
  [<ffffffff8108dfec>] ? sys_timer_settime+0x13c/0x2e0
  [<ffffffff8110317e>] ? sys_writev+0x4e/0x90
  [<ffffffff81034d6b>] ? system_call_fastpath+0x16/0x1b
ffff8801e7621500: redzone 1:0xbf05bd0100000006, redzone 2:0x9f911029d74e35b

----------

Code: 40 01 00 00 4c 8b a4 24 48 01 00 00 4c 8b ac 24 50 01 00 00 4c 8b 
b4 24 5
RIP  [<ffffffff81652c67>] icmp_send+0x297/0x650
  RSP <ffff880001c036b8>
---[ end trace 9d3f7be7684ac91e ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 0, comm: swapper Tainted: G      D     
2.6.35.9-local.20110314-144920 #2
Call Trace:
<IRQ>  [<ffffffff8170eada>] ? panic+0x94/0x116
  [<ffffffff81711326>] ? _raw_spin_lock_irqsave+0x26/0x40
  [<ffffffff8103a05f>] ? oops_end+0xef/0xf0
  [<ffffffff81711a15>] ? general_protection+0x25/0x30
  [<ffffffff81652c2f>] ? icmp_send+0x25f/0x650
  [<ffffffff81652c67>] ? icmp_send+0x297/0x650
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816dbfa0>] ? br_nf_forward_finish+0x0/0x170
  [<ffffffff816dc461>] ? br_nf_forward_ip+0x201/0x3e0
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d6620>] ? br_forward_finish+0x0/0x60
  [<ffffffff816d66e9>] ? __br_forward+0x69/0xb0
  [<ffffffff816d741a>] ? br_handle_frame_finish+0x12a/0x280
  [<ffffffff816dcac8>] ? br_nf_pre_routing_finish+0x208/0x370
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816dc8c0>] ? br_nf_pre_routing_finish+0x0/0x370
  [<ffffffff816dc538>] ? br_nf_forward_ip+0x2d8/0x3e0
  [<ffffffff816dd3b5>] ? br_nf_pre_routing+0x785/0x980
  [<ffffffff815fd8e6>] ? nf_iterate+0x66/0xb0
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
  [<ffffffff815fd994>] ? nf_hook_slow+0x64/0xf0
  [<ffffffff816d72f0>] ? br_handle_frame_finish+0x0/0x280
  [<ffffffff816d76fc>] ? br_handle_frame+0x18c/0x250
  [<ffffffff815dec5b>] ? __netif_receive_skb+0x1cb/0x350
  [<ffffffff8103d115>] ? read_tsc+0x5/0x20
  [<ffffffff815dfa18>] ? netif_receive_skb+0x78/0x80
  [<ffffffff815e0217>] ? napi_gro_receive+0x27/0x40
  [<ffffffff815e01d8>] ? napi_skb_finish+0x38/0x50
  [<ffffffff8152586d>] ? bnx2_poll_work+0xd0d/0x13d0
  [<ffffffff8160c950>] ? ctnetlink_conntrack_event+0x210/0x7d0
  [<ffffffff81092029>] ? autoremove_wake_function+0x9/0x30
  [<ffffffff8109a71b>] ? ktime_get+0x5b/0xe0
  [<ffffffff81526051>] ? bnx2_poll+0x61/0x230
  [<ffffffff81051db8>] ? lapic_next_event+0x18/0x20
  [<ffffffff815dfbef>] ? net_rx_action+0x9f/0x200
  [<ffffffff8109636f>] ? __hrtimer_start_range_ns+0x22f/0x410
  [<ffffffff8107c35f>] ? __do_softirq+0xaf/0x1e0
  [<ffffffff810ab547>] ? handle_IRQ_event+0x47/0x160
  [<ffffffff81036d5c>] ? call_softirq+0x1c/0x30
  [<ffffffff81038c85>] ? do_softirq+0x65/0xa0
  [<ffffffff8107c235>] ? irq_exit+0x85/0x90
  [<ffffffff8103820b>] ? do_IRQ+0x6b/0xe0
  [<ffffffff817117d3>] ? ret_from_intr+0x0/0x11
<EOI>  [<ffffffff81269340>] ? intel_idle+0xf0/0x180
  [<ffffffff81269320>] ? intel_idle+0xd0/0x180
  [<ffffffff815b0b0f>] ? cpuidle_idle_call+0x9f/0x140
  [<ffffffff81035032>] ? cpu_idle+0x62/0xb0
  [<ffffffff81a40c77>] ? start_kernel+0x2ef/0x2fa
  [<ffffffff81a403e3>] ? x86_64_start_kernel+0xfb/0x10a



-- 
  "A search of his car uncovered pornography, a homemade sex aid, women's 
  stockings and a Jack Russell terrier."
    - http://www.dailytelegraph.com.au/news/wacky/indeed/story-e6frev20-1111118083480

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 02/06/11 07:03, CaT wrote:
> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>> Unfortunately the only interface that is mentioned by name anywhere
>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>
>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>
> Damn. Not referencing the bridge interfaces at all stopped our host from
> going down in flames when we passed it a few packets. These are two
> of the oopses we got from it. Whilst the kernel here is .35 we got the
> same issue from a range of kernels. Seems related.

Well, I tried sending an explanatory message to netdev, netfilter & cc'd 
to kvm, but it appears not to have made it to kvm or netfilter, and the 
cc to netdev has not elicited a response. My resend to netfilter seems 
to have dropped into the bit bucket also.

Is there anyone who can point me at the appropriate cage to rattle? I 
know it appears to be a netfilter issue, but I don't seem to be able to 
get a message to the list (and I am subscribed to it and have been 
getting mail for months) and I'm not sure who to pester. The other 
alternative is I just stop doing "that" and wait for it to bite someone 
else.

Cheers.
Brad

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Bernhard Held]

Am 03.06.2011 15:38, schrieb Brad Campbell:
> On 02/06/11 07:03, CaT wrote:
>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>> Unfortunately the only interface that is mentioned by name anywhere
>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>
>>> All of the nat/dnat and other horrible hacks are based on IP addresses.
>>
>> Damn. Not referencing the bridge interfaces at all stopped our host from
>> going down in flames when we passed it a few packets. These are two
>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>> same issue from a range of kernels. Seems related.
>
> Well, I tried sending an explanatory message to netdev, netfilter & cc'd to kvm,
> but it appears not to have made it to kvm or netfilter, and the cc to netdev has
> not elicited a response. My resend to netfilter seems to have dropped into the
> bit bucket also.
Just another reference 3.5 months ago:
http://www.spinics.net/lists/netfilter-devel/msg17239.html

Bernhard

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 03/06/11 23:50, Bernhard Held wrote:
>Am 03.06.2011 15:38, schrieb Brad Campbell:
>>On 02/06/11 07:03, CaT wrote:
>>>On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>Unfortunately the only interface that is mentioned by name anywhere
>>>>in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>
>>>>All of the nat/dnat and other horrible hacks are based on IP addresses.
>>>
>>>Damn. Not referencing the bridge interfaces at all stopped our host from
>>>going down in flames when we passed it a few packets. These are two
>>>of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>same issue from a range of kernels. Seems related.
>>
>>Well, I tried sending an explanatory message to netdev, netfilter &
>>cc'd to kvm,
>>but it appears not to have made it to kvm or netfilter, and the cc to
>>netdev has
>>not elicited a response. My resend to netfilter seems to have dropped
>>into the
>>bit bucket also.
>Just another reference 3.5 months ago:
>http://www.spinics.net/lists/netfilter-devel/msg17239.html

<waves hands around shouting "I have a reproducible test case for this 
and don't mind patching and crashing the machine to get it fixed">

Attempted to add netfilter-devel to the cc this time.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Bart De Schuymer]

Hi Brad,

This has probably nothing to do with ebtables, so please rmmod in case 
it's loaded.
A few questions I didn't directly see an answer to in the threads I 
scanned...
I'm assuming you actually use the bridging firewall functionality. So, 
what iptables modules do you use? Can you reduce your iptables rules to 
a core that triggers the bug?
Or does it get triggered even with an empty set of firewall rules?
Are you using a stock .35 kernel or is it patched?
Is this something I can trigger on a poor guy's laptop or does it 
require specialized hardware (I'm catching up on qemu/kvm...)?

cheers,
Bart

PS: I'm not sure if we should keep CC-ing everybody, netfilter-devel 
together with kvm should probably do fine.

Op 3/06/2011 18:07, Brad Campbell schreef:
> On 03/06/11 23:50, Bernhard Held wrote:
>> Am 03.06.2011 15:38, schrieb Brad Campbell:
>>> On 02/06/11 07:03, CaT wrote:
>>>> On Wed, Jun 01, 2011 at 07:52:33PM +0800, Brad Campbell wrote:
>>>>> Unfortunately the only interface that is mentioned by name anywhere
>>>>> in my firewall is $DMZ (which is ppp0 and not part of any bridge).
>>>>>
>>>>> All of the nat/dnat and other horrible hacks are based on IP 
>>>>> addresses.
>>>>
>>>> Damn. Not referencing the bridge interfaces at all stopped our host 
>>>> from
>>>> going down in flames when we passed it a few packets. These are two
>>>> of the oopses we got from it. Whilst the kernel here is .35 we got the
>>>> same issue from a range of kernels. Seems related.
>>>
>>> Well, I tried sending an explanatory message to netdev, netfilter &
>>> cc'd to kvm,
>>> but it appears not to have made it to kvm or netfilter, and the cc to
>>> netdev has
>>> not elicited a response. My resend to netfilter seems to have dropped
>>> into the
>>> bit bucket also.
>> Just another reference 3.5 months ago:
>> http://www.spinics.net/lists/netfilter-devel/msg17239.html
>
> <waves hands around shouting "I have a reproducible test case for this 
> and don't mind patching and crashing the machine to get it fixed">
>
> Attempted to add netfilter-devel to the cc this time.
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>


-- 
Bart De Schuymer
www.artinalgorithms.be

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le lundi 06 juin 2011 A  22:10 +0200, Bart De Schuymer a A(C)crit :
> Hi Brad,
> 
> This has probably nothing to do with ebtables, so please rmmod in case 
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I 
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So, 
> what iptables modules do you use? Can you reduce your iptables rules to 
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it 
> require specialized hardware (I'm catching up on qemu/kvm...)?
> 

Keep netdev, as this most probably is a networking bug.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 04:10, Bart De Schuymer wrote:
> Hi Brad,
>
> This has probably nothing to do with ebtables, so please rmmod in case
> it's loaded.
> A few questions I didn't directly see an answer to in the threads I
> scanned...
> I'm assuming you actually use the bridging firewall functionality. So,
> what iptables modules do you use? Can you reduce your iptables rules to
> a core that triggers the bug?
> Or does it get triggered even with an empty set of firewall rules?
> Are you using a stock .35 kernel or is it patched?
> Is this something I can trigger on a poor guy's laptop or does it
> require specialized hardware (I'm catching up on qemu/kvm...)?

Not specialised hardware as such, I've just not been able to reproduce 
it outside of this specific operating scenario.

I can't trigger it with empty firewall rules as it relies on a DNAT to 
occur. If I try it directly to the internal IP address (as I have to 
without netfilter loaded) then of course nothing fails.

It's a pain in the bum as a fault, but it's one I can easily reproduce 
as long as I use the same set of circumstances.

I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it 
on that then I'll attempt to pare down the IPTABLES rules to a bare minimum.

It is nothing to do with ebtables as I don't compile it. I'm not really 
sure about "bridging firewall" functionality. I just use a couple of 
hand coded bash scripts to set the tables up.

brad@srv:~$ lsmod
Module                  Size  Used by
xt_iprange              1637  1
xt_DSCP                 2077  2
xt_length               1216  1
xt_CLASSIFY             1091  26
sch_sfq                 6681  4
xt_CHECKSUM             1229  2 brad@srv:~$ lsmod
Module                  Size  Used by
xt_iprange              1637  1
xt_DSCP                 2077  2
xt_length               1216  1
xt_CLASSIFY             1091  26
sch_sfq                 6681  4
xt_CHECKSUM             1229  2
ipt_REJECT              2277  1
ipt_MASQUERADE          1759  7
ipt_REDIRECT            1133  1
xt_recent               8223  2
xt_state                1226  5
iptable_nat             3993  1
nf_nat                 16773  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11868  8 iptable_nat,nf_nat
nf_conntrack           60962  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1417  1 nf_conntrack_ipv4
xt_TCPMSS               2567  2
xt_tcpmss               1469  0
xt_tcpudp               2467  56
iptable_mangle          1487  1
pppoe                   9574  2
pppox                   2188  1 pppoe
iptable_filter          1442  1
ip_tables              16762  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               20462  17 
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic            24243  6 pppoe,pppox
slhc                    5293  1 ppp_generic
cls_u32                 6468  6
sch_htb                14432  2
deflate                 1937  0
zlib_deflate           21228  1 deflate
des_generic            16135  0
cbc                     2721  0
ecb                     1975  0
crypto_blkcipher       13645  2 cbc,ecb
sha1_generic            2095  0
md5                     4001  0
hmac                    2977  0
crypto_hash            14519  3 sha1_generic,md5,hmac
cryptomgr               2636  0
aead                    6137  1 cryptomgr
crypto_algapi          15289  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27372  0
fuse                   66747  1
w83627ehf              32052  0
hwmon_vid               2867  1 w83627ehf
vhost_net              16802  6
powernow_k8            12932  0
mperf                   1263  1 powernow_k8
kvm_amd                53431  24
kvm                   235155  1 kvm_amd
pl2303                 12732  1
xhci_hcd               62865  0
i2c_piix4               8391  0
k10temp                 3183  0
usbserial              34452  3 pl2303
usb_storage            37887  1
usb_libusual           10999  1 usb_storage
ohci_hcd               18105  0
ehci_hcd               33641  0
ahci                   20748  4
usbcore               130936  7 
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci                21202  1 ahci
sata_mv                26939  0
megaraid_sas           71659  14

Nat Table (external ip substituted for xxx.xxx.xxx.xxx)

Chain PREROUTING (policy ACCEPT 1761K packets, 152M bytes)
  pkts bytes target     prot opt in     out     source 
destination
     5   210 DNAT       udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1195 to:192.168.253.199
     6   252 DNAT       udp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       udp dpt:1195 to:192.168.253.199
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25001 to:192.168.253.199:465
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25000 to:192.168.253.199:993
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:25001 to:192.168.253.199:465
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:25000 to:192.168.253.199:993
     2   142 DNAT       47   --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           to:192.168.253.199
    18   880 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:1723 to:192.168.253.199
     0     0 DNAT       47   --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       to:192.168.253.199
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:1723 to:192.168.253.199
  2969  149K DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:443 to:192.168.253.198
    20  1280 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:443 to:192.168.253.198
     0     0 DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3101 to:192.168.253.197
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:3101 to:192.168.253.197
     0     0 DNAT       tcp  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       tcp dpt:4101 to:192.168.253.197
44528 2718K REDIRECT   tcp  --  !ppp0  *       0.0.0.0/0 
!192.168.0.0/16      tcp dpt:80 redir ports 8080
     0     0 DNAT       tcp  --  *      *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3724 to:192.168.2.107
  596K   33M DNAT       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpts:2001:2030 to:10.99.99.2
1420K  119M DNAT       udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpts:2001:2030 to:10.99.99.2
  7483  449K DNAT       all  --  !ppp0  *       0.0.0.0/0 
xxx.xxx.xxx.xxx       to:192.168.2.1


Mangle Table

Chain INPUT (policy ACCEPT 270K packets, 17M bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain OUTPUT (policy ACCEPT 170K packets, 12M bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain POSTROUTING (policy ACCEPT 2205K packets, 166M bytes)
  pkts bytes target     prot opt in     out     source 
destination
     0     0 MASQUERADE  all  --  *      *       0.0.0.0/0 
192.168.254.3
     6   360 ACCEPT     all  --  *      *       0.0.0.0/0 
xxx.xxx.xxx.xxx
20424 2120K MASQUERADE  all  --  *      ppp0    192.168.0.0/16 
!192.168.0.0/16
     0     0 MASQUERADE  all  --  *      ppp0    10.0.0.0/24 
0.0.0.0/0
     3   204 MASQUERADE  all  --  *      *       192.168.2.0/24 
10.8.0.0/24
1418K  128M MASQUERADE  all  --  *      *       10.99.99.0/24 
0.0.0.0/0
68248 4095K MASQUERADE  all  --  *      *       192.168.253.0/24 
10.8.0.0/16
13305 2405K MASQUERADE  all  --  *      *       192.168.253.0/24 
!192.168.0.0/16

Chain PREROUTING (policy ACCEPT 278M packets, 293G bytes)
  pkts bytes target     prot opt in     out     source 
destination
   169 55528 CHECKSUM   udp  --  br1    *       0.0.0.0/0 
0.0.0.0/0           udp dpt:67 CHECKSUM fill

Chain INPUT (policy ACCEPT 180M packets, 250G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
  pkts bytes target     prot opt in     out     source 
destination

Chain POSTROUTING (policy ACCEPT 253M packets, 223G bytes)
  pkts bytes target     prot opt in     out     source 
destination
   165 54182 CHECKSUM   udp  --  *      br1     0.0.0.0/0 
0.0.0.0/0           udp spt:67 CHECKSUM fill
    51  3712 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:53 CLASSIFY set 1:20
85274 6454K CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp dpt:53 CLASSIFY set 1:20
   187  257K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:81 CLASSIFY set 1:20
   25M 1180M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x3F/0x10 state ESTABLISHED length 40:100 
CLASSIFY set 1:15
  728K   67M CLASSIFY   icmp --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:15
   231 23484 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:2401 CLASSIFY set 1:15
65636 5610K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 CLASSIFY set 1:10
  2018  315K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:22 CLASSIFY set 1:10
    80 10092 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:3389 CLASSIFY set 1:10
26063 8910K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:8080 CLASSIFY set 1:15
  932K  131M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:80 CLASSIFY set 1:15
  3511  267K CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp dpt:123 CLASSIFY set 1:10
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:20 CLASSIFY set 1:15
     3   180 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:20 CLASSIFY set 1:15
94058   38M CLASSIFY   47   --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:10
1086K  183M CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1194 CLASSIFY set 1:10
1086K  183M TOS        udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1194 TOS set 0x10/0x3f
48817   10M CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1195 CLASSIFY set 1:10
48817   10M TOS        udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:1195 TOS set 0x10/0x3f
94058   38M CLASSIFY   47   --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           CLASSIFY set 1:15
   106  7207 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:1863 CLASSIFY set 1:15
  188K   34M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:443 CLASSIFY set 1:15
51541 3327K CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpts:6660:6669 CLASSIFY set 1:15
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spts:2021:2030 CLASSIFY set 1:15
    85  4944 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp dpt:19999 CLASSIFY set 1:15
  208K   86M CLASSIFY   udp  --  *      *       0.0.0.0/0 
0.0.0.0/0           source IP range 192.168.2.80-192.168.2.120 CLASSIFY 
set 1:10
     0     0 CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp spt:12345 CLASSIFY set 1:15
     1    80 CLASSIFY   udp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           udp spt:12345 CLASSIFY set 1:15


Default table

Chain INPUT (policy ACCEPT 176M packets, 247G bytes)
  pkts bytes target     prot opt in     out     source 
destination
     0     0 ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:4569
1187K  582M ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1194
     2   577 ACCEPT     udp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           udp dpt:1195
    28  1224 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:3389
   230 12372            tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 state NEW recent: SET name: DEFAULT side: 
source
     3   180 DROP       tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22 state NEW recent: UPDATE seconds: 300 
hit_count: 4 name: DEFAULT side: source
  1750  143K ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:22
     3   144 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:113
   120  6090 ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:81
36094   29M ACCEPT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp dpt:25
1456K 1706M ACCEPT     all  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           state RELATED,ESTABLISHED
31047 2334K REJECT     tcp  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0           tcp option=!2 reject-with tcp-reset
  552K   60M ACCEPT     all  --  !ppp0  *       0.0.0.0/0 
0.0.0.0/0           state NEW
13552 1207K ACCEPT     icmp --  ppp0   *       0.0.0.0/0 
0.0.0.0/0
  5712  392K DROP       all  --  ppp0   *       0.0.0.0/0 
0.0.0.0/0

Chain FORWARD (policy ACCEPT 98M packets, 44G bytes)
  pkts bytes target     prot opt in     out     source 
destination
1207K   68M TCPMSS     tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 155M packets, 180G bytes)
  pkts bytes target     prot opt in     out     source 
destination
31675 1895K TCPMSS     tcp  --  *      ppp0    0.0.0.0/0 
0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

lsmod

ipt_REJECT              2277  1
ipt_MASQUERADE          1759  7
ipt_REDIRECT            1133  1
xt_recent               8223  2
xt_state                1226  5
iptable_nat             3993  1
nf_nat                 16773  3 ipt_MASQUERADE,ipt_REDIRECT,iptable_nat
nf_conntrack_ipv4      11868  8 iptable_nat,nf_nat
nf_conntrack           60962  5 
ipt_MASQUERADE,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nf_defrag_ipv4          1417  1 nf_conntrack_ipv4
xt_TCPMSS               2567  2
xt_tcpmss               1469  0
xt_tcpudp               2467  56
iptable_mangle          1487  1
pppoe                   9574  2
pppox                   2188  1 pppoe
iptable_filter          1442  1
ip_tables              16762  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               20462  17 
xt_iprange,xt_DSCP,xt_length,xt_CLASSIFY,xt_CHECKSUM,ipt_REJECT,ipt_MASQUERADE,ipt_REDIRECT,xt_recent,xt_state,iptable_nat,xt_TCPMSS,xt_tcpmss,xt_tcpudp,iptable_mangle,iptable_filter,ip_tables
ppp_generic            24243  6 pppoe,pppox
slhc                    5293  1 ppp_generic
cls_u32                 6468  6
sch_htb                14432  2
deflate                 1937  0
zlib_deflate           21228  1 deflate
des_generic            16135  0
cbc                     2721  0
ecb                     1975  0
crypto_blkcipher       13645  2 cbc,ecb
sha1_generic            2095  0
md5                     4001  0
hmac                    2977  0
crypto_hash            14519  3 sha1_generic,md5,hmac
cryptomgr               2636  0
aead                    6137  1 cryptomgr
crypto_algapi          15289  9 
deflate,des_generic,cbc,ecb,crypto_blkcipher,hmac,crypto_hash,cryptomgr,aead
af_key                 27372  0
fuse                   66747  1
w83627ehf              32052  0
hwmon_vid               2867  1 w83627ehf
vhost_net              16802  6
powernow_k8            12932  0
mperf                   1263  1 powernow_k8
kvm_amd                53431  24
kvm                   235155  1 kvm_amd
pl2303                 12732  1
xhci_hcd               62865  0
i2c_piix4               8391  0
k10temp                 3183  0
usbserial              34452  3 pl2303
usb_storage            37887  1
usb_libusual           10999  1 usb_storage
ohci_hcd               18105  0
ehci_hcd               33641  0
ahci                   20748  4
usbcore               130936  7 
pl2303,xhci_hcd,usbserial,usb_storage,usb_libusual,ohci_hcd,ehci_hcd
libahci                21202  1 ahci
sata_mv                26939  0
megaraid_sas           71659  14

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Patrick McHardy]

On 07.06.2011 05:33, Brad Campbell wrote:
> On 07/06/11 04:10, Bart De Schuymer wrote:
>> Hi Brad,
>>
>> This has probably nothing to do with ebtables, so please rmmod in case
>> it's loaded.
>> A few questions I didn't directly see an answer to in the threads I
>> scanned...
>> I'm assuming you actually use the bridging firewall functionality. So,
>> what iptables modules do you use? Can you reduce your iptables rules to
>> a core that triggers the bug?
>> Or does it get triggered even with an empty set of firewall rules?
>> Are you using a stock .35 kernel or is it patched?
>> Is this something I can trigger on a poor guy's laptop or does it
>> require specialized hardware (I'm catching up on qemu/kvm...)?
> 
> Not specialised hardware as such, I've just not been able to reproduce
> it outside of this specific operating scenario.

The last similar problem we've had was related to the 32/64 bit compat
code. Are you running 32 bit userspace on a 64 bit kernel?

> I can't trigger it with empty firewall rules as it relies on a DNAT to
> occur. If I try it directly to the internal IP address (as I have to
> without netfilter loaded) then of course nothing fails.
> 
> It's a pain in the bum as a fault, but it's one I can easily reproduce
> as long as I use the same set of circumstances.
> 
> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
> on that then I'll attempt to pare down the IPTABLES rules to a bare
> minimum.
> 
> It is nothing to do with ebtables as I don't compile it. I'm not really
> sure about "bridging firewall" functionality. I just use a couple of
> hand coded bash scripts to set the tables up.

>From one of your previous mails:

> # CONFIG_BRIDGE_NF_EBTABLES is not set

How about CONFIG_BRIDGE_NETFILTER?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 21:30, Patrick McHardy wrote:
> On 07.06.2011 05:33, Brad Campbell wrote:
>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>> Hi Brad,
>>>
>>> This has probably nothing to do with ebtables, so please rmmod in case
>>> it's loaded.
>>> A few questions I didn't directly see an answer to in the threads I
>>> scanned...
>>> I'm assuming you actually use the bridging firewall functionality. So,
>>> what iptables modules do you use? Can you reduce your iptables rules to
>>> a core that triggers the bug?
>>> Or does it get triggered even with an empty set of firewall rules?
>>> Are you using a stock .35 kernel or is it patched?
>>> Is this something I can trigger on a poor guy's laptop or does it
>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>
>> Not specialised hardware as such, I've just not been able to reproduce
>> it outside of this specific operating scenario.
>
> The last similar problem we've had was related to the 32/64 bit compat
> code. Are you running 32 bit userspace on a 64 bit kernel?

No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.

Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is 
current git


>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>> occur. If I try it directly to the internal IP address (as I have to
>> without netfilter loaded) then of course nothing fails.
>>
>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>> as long as I use the same set of circumstances.
>>
>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>> minimum.
>>
>> It is nothing to do with ebtables as I don't compile it. I'm not really
>> sure about "bridging firewall" functionality. I just use a couple of
>> hand coded bash scripts to set the tables up.
>
>  From one of your previous mails:
>
>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>
> How about CONFIG_BRIDGE_NETFILTER?
>

It was compiled in.

With the following table set I was able to reproduce the problem on 
3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

root@srv:~# iptables-save
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*filter
:INPUT ACCEPT [978:107619]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1659:291870]
-A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
-A INPUT -i ppp0 -j DROP
COMMIT
# Completed on Tue Jun  7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*nat
:PREROUTING ACCEPT [813:49170]
:INPUT ACCEPT [91:7090]
:OUTPUT ACCEPT [267:20731]
:POSTROUTING ACCEPT [296:22281]
-A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443 
-j DNAT --to-destination 192.168.253.198
COMMIT
# Completed on Tue Jun  7 22:11:30 2011
# Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
*mangle
:PREROUTING ACCEPT [2729:274392]
:INPUT ACCEPT [2508:262976]
:FORWARD ACCEPT [142:7068]
:OUTPUT ACCEPT [1674:293701]
:POSTROUTING ACCEPT [2131:346411]
-A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 
1400:1536 -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Tue Jun  7 22:11:30 2011

I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access 
the address the way I was doing it, so that's a no-go for me.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Patrick McHardy]

On 07.06.2011 16:40, Brad Campbell wrote:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
> 
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit kernel.
> 
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is
> current git
> 
> 
>>> I can't trigger it with empty firewall rules as it relies on a DNAT to
>>> occur. If I try it directly to the internal IP address (as I have to
>>> without netfilter loaded) then of course nothing fails.
>>>
>>> It's a pain in the bum as a fault, but it's one I can easily reproduce
>>> as long as I use the same set of circumstances.
>>>
>>> I'll try using 3.0-rc2 (current git) tonight, and if I can reproduce it
>>> on that then I'll attempt to pare down the IPTABLES rules to a bare
>>> minimum.
>>>
>>> It is nothing to do with ebtables as I don't compile it. I'm not really
>>> sure about "bridging firewall" functionality. I just use a couple of
>>> hand coded bash scripts to set the tables up.
>>
>>  From one of your previous mails:
>>
>>> # CONFIG_BRIDGE_NF_EBTABLES is not set
>>
>> How about CONFIG_BRIDGE_NETFILTER?
>>
> 
> It was compiled in.
> 
> With the following table set I was able to reproduce the problem on
> 3.0-rc2. Replaced my IP with xxx.xxx.xxx.xxx, but otherwise unmodified

Which kernel was the last version without this problem?

> root@srv:~# iptables-save
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *filter
> :INPUT ACCEPT [978:107619]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1659:291870]
> -A INPUT -i ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT ! -i ppp0 -m state --state NEW -j ACCEPT
> -A INPUT -i ppp0 -j DROP
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *nat
> :PREROUTING ACCEPT [813:49170]
> :INPUT ACCEPT [91:7090]
> :OUTPUT ACCEPT [267:20731]
> :POSTROUTING ACCEPT [296:22281]
> -A PREROUTING -d xxx.xxx.xxx.xxx/32 ! -i ppp0 -p tcp -m tcp --dport 443
> -j DNAT --to-destination 192.168.253.198
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011
> # Generated by iptables-save v1.4.10 on Tue Jun  7 22:11:30 2011
> *mangle
> :PREROUTING ACCEPT [2729:274392]
> :INPUT ACCEPT [2508:262976]
> :FORWARD ACCEPT [142:7068]
> :OUTPUT ACCEPT [1674:293701]
> :POSTROUTING ACCEPT [2131:346411]
> -A FORWARD -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m tcpmss --mss
> 1400:1536 -j TCPMSS --clamp-mss-to-pmtu
> COMMIT
> # Completed on Tue Jun  7 22:11:30 2011

The main suspects would be NAT and TCPMSS. Did you also try whether
the crash occurs with only one of these these rules?

> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> the address the way I was doing it, so that's a no-go for me.

That's really weird since you're apparently not using any bridge
netfilter features. It shouldn't have any effect besides changing
at which point ip_tables is invoked. How are your network devices
configured (specifically any bridges)?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le mardi 07 juin 2011 A  17:35 +0200, Patrick McHardy a A(C)crit :

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?
> 
> > I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> > the address the way I was doing it, so that's a no-go for me.
> 
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?

Something in the kernel does 

u16 *ptr = addr (given by kmalloc())

ptr[-1] = 0;

Could be an off-one error in a memmove()/memcopy() or loop...

I cant see a network issue here.

I checked arch/x86/lib/memmove_64.S and it seems fine.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Patrick McHardy]

On 07.06.2011 20:31, Eric Dumazet wrote:
> Le mardi 07 juin 2011 a 17:35 +0200, Patrick McHardy a ecrit :
> 
>> The main suspects would be NAT and TCPMSS. Did you also try whether
>> the crash occurs with only one of these these rules?
>>
>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>> the address the way I was doing it, so that's a no-go for me.
>>
>> That's really weird since you're apparently not using any bridge
>> netfilter features. It shouldn't have any effect besides changing
>> at which point ip_tables is invoked. How are your network devices
>> configured (specifically any bridges)?
> 
> Something in the kernel does 
> 
> u16 *ptr = addr (given by kmalloc())
> 
> ptr[-1] = 0;
> 
> Could be an off-one error in a memmove()/memcopy() or loop...
> 
> I cant see a network issue here.

So far me neither, but netfilter appears to trigger the bug.

> I checked arch/x86/lib/memmove_64.S and it seems fine.

I was thinking it might be a missing skb_make_writable() combined
with vhost_net specifics in the netfilter code (TCPMSS and NAT are
both suspect), but was unable to find something. I also went
through the dst_metrics() conversion to see whether anything could
cause problems with the bridge fake_rttable, but also nothing
so far.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 08/06/11 06:57, Patrick McHardy wrote:
> On 07.06.2011 20:31, Eric Dumazet wrote:
>> Le mardi 07 juin 2011 a 17:35 +0200, Patrick McHardy a ecrit :
>>
>>> The main suspects would be NAT and TCPMSS. Did you also try whether
>>> the crash occurs with only one of these these rules?
>>>
>>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>>>> the address the way I was doing it, so that's a no-go for me.
>>>
>>> That's really weird since you're apparently not using any bridge
>>> netfilter features. It shouldn't have any effect besides changing
>>> at which point ip_tables is invoked. How are your network devices
>>> configured (specifically any bridges)?
>>
>> Something in the kernel does
>>
>> u16 *ptr = addr (given by kmalloc())
>>
>> ptr[-1] = 0;
>>
>> Could be an off-one error in a memmove()/memcopy() or loop...
>>
>> I cant see a network issue here.
>
> So far me neither, but netfilter appears to trigger the bug.

Would it help if I tried some older kernels? This issue only surfaced 
for me recently as I only installed the VM's in question about 12 weeks 
ago and have only just started really using them in anger. I could try 
reproducing it on progressively older kernels to see if I can find one 
that works and then bisect from there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le mercredi 08 juin 2011 A  08:18 +0800, Brad Campbell a A(C)crit :
> On 08/06/11 06:57, Patrick McHardy wrote:
> > On 07.06.2011 20:31, Eric Dumazet wrote:
> >> Le mardi 07 juin 2011 A  17:35 +0200, Patrick McHardy a A(C)crit :
> >>
> >>> The main suspects would be NAT and TCPMSS. Did you also try whether
> >>> the crash occurs with only one of these these rules?
> >>>
> >>>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
> >>>> the address the way I was doing it, so that's a no-go for me.
> >>>
> >>> That's really weird since you're apparently not using any bridge
> >>> netfilter features. It shouldn't have any effect besides changing
> >>> at which point ip_tables is invoked. How are your network devices
> >>> configured (specifically any bridges)?
> >>
> >> Something in the kernel does
> >>
> >> u16 *ptr = addr (given by kmalloc())
> >>
> >> ptr[-1] = 0;
> >>
> >> Could be an off-one error in a memmove()/memcopy() or loop...
> >>
> >> I cant see a network issue here.
> >
> > So far me neither, but netfilter appears to trigger the bug.
> 
> Would it help if I tried some older kernels? This issue only surfaced 
> for me recently as I only installed the VM's in question about 12 weeks 
> ago and have only just started really using them in anger. I could try 
> reproducing it on progressively older kernels to see if I can find one 
> that works and then bisect from there.

Well, a bisection definitely should help, but needs a lot of time in
your case.

Could you try following patch, because this is the 'usual suspect' I had
yesterday :

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 46cbd28..9f548f9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
 	}
 
+#if 0
 	if (fastpath &&
 	    size + sizeof(struct skb_shared_info) <= ksize(skb->head)) {
 		memmove(skb->head + size, skb_shinfo(skb),
@@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 		off = nhead;
 		goto adjust_others;
 	}
-
+#endif
 	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
 	if (!data)
 		goto nodata;


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 08/06/11 11:59, Eric Dumazet wrote:

> Well, a bisection definitely should help, but needs a lot of time in
> your case.

Yes. compile, test, crash, walk out to the other building to press 
reset, lather, rinse, repeat.

I need a reset button on the end of a 50M wire, or a hardware watchdog!

Actually it's not so bad. If I turn off slub debugging the kernel panics 
and reboots itself.

This.. :
[    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
[    2.913066] netconsole: device eth0 not up yet, forcing it
[    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
[    3.660118] Switching to clocksource tsc
[   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
rtl_nic/rtl8168e-1.fw (-2)
[   63.223513] r8169 0000:03:00.0: eth0: link down
[   63.223556] r8169 0000:03:00.0: eth0: link down

..is slowing down reboots considerably. 3.0-rc does _not_ like some 
timing hardware in my machine. Having said that, at least it does not 
randomly panic on SCSI like 2.6.39 does.

Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
it. I've replicated it with only the single DNAT rule.


> Could you try following patch, because this is the 'usual suspect' I had
> yesterday :
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 46cbd28..9f548f9 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
>   	}
>
> +#if 0
>   	if (fastpath&&
>   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
>   		memmove(skb->head + size, skb_shinfo(skb),
> @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
>   		off = nhead;
>   		goto adjust_others;
>   	}
> -
> +#endif
>   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
>   	if (!data)
>   		goto nodata;
>
>
>

Nope.. that's not it. <sigh> That might have changed the characteristic 
of the fault slightly, but unfortunately I got caught with a couple of 
fsck's, so I only got to test it 3 times tonight.

It's unfortunate that this is a production system, so I can only take it 
down between about 9pm and 1am. That would normally be pretty 
productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
panics at the wrong time.

I'm out of time tonight, but I'll have a crack at some bisection 
tomorrow night. Now I just have to go back far enough that it works, and 
be near enough not to have to futz around with /proc /sys or drivers.

I really, really, really appreciate you guys helping me with this. It 
has been driving me absolutely bonkers. If I'm ever in the same town as 
any of you, dinner and drinks are on me.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le jeudi 09 juin 2011 A  01:02 +0800, Brad Campbell a A(C)crit :
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> > Well, a bisection definitely should help, but needs a lot of time in
> > your case.
> 
> Yes. compile, test, crash, walk out to the other building to press 
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!
> 
> Actually it's not so bad. If I turn off slub debugging the kernel panics 
> and reboots itself.
> 
> This.. :
> [    2.913034] netconsole: remote ethernet address 00:16:cb:a7:dd:d1
> [    2.913066] netconsole: device eth0 not up yet, forcing it
> [    3.660062] Refined TSC clocksource calibration: 3213.422 MHz.
> [    3.660118] Switching to clocksource tsc
> [   63.200273] r8169 0000:03:00.0: eth0: unable to load firmware patch 
> rtl_nic/rtl8168e-1.fw (-2)
> [   63.223513] r8169 0000:03:00.0: eth0: link down
> [   63.223556] r8169 0000:03:00.0: eth0: link down
> 
> ..is slowing down reboots considerably. 3.0-rc does _not_ like some 
> timing hardware in my machine. Having said that, at least it does not 
> randomly panic on SCSI like 2.6.39 does.
> 
> Ok, I've ruled out TCPMSS. Found out where it was being set and neutered 
> it. I've replicated it with only the single DNAT rule.
> 
> 
> > Could you try following patch, because this is the 'usual suspect' I had
> > yesterday :
> >
> > diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> > index 46cbd28..9f548f9 100644
> > --- a/net/core/skbuff.c
> > +++ b/net/core/skbuff.c
> > @@ -792,6 +792,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		fastpath = atomic_read(&skb_shinfo(skb)->dataref) == delta;
> >   	}
> >
> > +#if 0
> >   	if (fastpath&&
> >   	size + sizeof(struct skb_shared_info)<= ksize(skb->head)) {
> >   		memmove(skb->head + size, skb_shinfo(skb),
> > @@ -802,7 +803,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
> >   		off = nhead;
> >   		goto adjust_others;
> >   	}
> > -
> > +#endif
> >   	data = kmalloc(size + sizeof(struct skb_shared_info), gfp_mask);
> >   	if (!data)
> >   		goto nodata;
> >
> >
> >
> 
> Nope.. that's not it. <sigh> That might have changed the characteristic 
> of the fault slightly, but unfortunately I got caught with a couple of 
> fsck's, so I only got to test it 3 times tonight.
> 
> It's unfortunate that this is a production system, so I can only take it 
> down between about 9pm and 1am. That would normally be pretty 
> productive, except that an fsck of a 14TB ext4 can take 30 minutes if it 
> panics at the wrong time.
> 
> I'm out of time tonight, but I'll have a crack at some bisection 
> tomorrow night. Now I just have to go back far enough that it works, and 
> be near enough not to have to futz around with /proc /sys or drivers.
> 
> I really, really, really appreciate you guys helping me with this. It 
> has been driving me absolutely bonkers. If I'm ever in the same town as 
> any of you, dinner and drinks are on me.

Hmm, I wonder if kmemcheck could help you, but its slow as hell, so not
appropriate for production :(



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Simon Horman]

On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
> On 08/06/11 11:59, Eric Dumazet wrote:
> 
> >Well, a bisection definitely should help, but needs a lot of time in
> >your case.
> 
> Yes. compile, test, crash, walk out to the other building to press
> reset, lather, rinse, repeat.
> 
> I need a reset button on the end of a 50M wire, or a hardware watchdog!

Not strictly on-topic, but in situations where I have machines
that either don't have lights-out facilities or have broken ones
I find that network controlled power switches to be very useful.

At one point I would have need an 8000km long wire to the reset switch :-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Mark Lord]

On 11-06-09 10:52 PM, Simon Horman wrote:
> On Thu, Jun 09, 2011 at 01:02:13AM +0800, Brad Campbell wrote:
>> On 08/06/11 11:59, Eric Dumazet wrote:
>>
>>> Well, a bisection definitely should help, but needs a lot of time in
>>> your case.
>>
>> Yes. compile, test, crash, walk out to the other building to press
>> reset, lather, rinse, repeat.
>>
>> I need a reset button on the end of a 50M wire, or a hardware watchdog!


Something many of us don't realize is that nearly all Intel chipsets
have a built-in hardware watchdog timer.  This includes chipset for
consumer desktop boards as well as the big iron server stuff.

It's the "i8xx_tco" driver in the kernel enables use of them:

    modprobe i8xx_tco

Cheers

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Henrique de Moraes Holschuh]

On Fri, 10 Jun 2011, Mark Lord wrote:
> Something many of us don't realize is that nearly all Intel chipsets
> have a built-in hardware watchdog timer.  This includes chipset for
> consumer desktop boards as well as the big iron server stuff.
> 
> It's the "i8xx_tco" driver in the kernel enables use of them:

That's the old module name, but yes, it is very useful in desktops and
laptops (when it works).   Server-class hardware will have a baseboard
management unit that can really power-cycle the system instead of just
rebooting.

And test it first before you depend on it triggering at a remote location,
as the firmware might cause the Intel chipset watchdog to actually hang the
box instead of causing a proper reboot (happens on the IBM thinkpad T43, for
example).

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/10/2011 05:52 AM, Simon Horman wrote:
> At one point I would have need an 8000km long wire to the reset switch :-)

Even more off-topic, there has been a case when a 200,000,000 km long 
wire to the reset button was needed.  IIRC they got away with a watchdog.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 07/06/11 23:35, Patrick McHardy wrote:

> The main suspects would be NAT and TCPMSS. Did you also try whether
> the crash occurs with only one of these these rules?

To be honest I'm actually having trouble finding where TCPMSS is 
actually set in that ruleset. This is a production machine so I can only 
take it down after about 9PM at night. I'll have another crack at it 
tonight.

>> I've just compiled out CONFIG_BRIDGE_NETFILTER and can no longer access
>> the address the way I was doing it, so that's a no-go for me.
>
> That's really weird since you're apparently not using any bridge
> netfilter features. It shouldn't have any effect besides changing
> at which point ip_tables is invoked. How are your network devices
> configured (specifically any bridges)?
>

I have one bridge with all my virtual machines on it.

In this particular instance the packets leave VM A destined for the IP 
address of ppp0 (the external interface). This is intercepted by the 
DNAT PREROUTING rule above and shunted back to VM B.

The VM's are on br1 and the external address is ppp0. Without 
CONFIG_BRIDGE_NETFILTER compiled in I can see the traffic entering and 
leaving VM B with tcpdump, but the packets never seem to get back to VM A.

VM A is XP 32 bit, VM B is Linux. I have some other Linux VM's, so I'll 
do some more testing tonight between those to see where the packets are 
going without CONFIG_BRIDGE_NETFILTER set.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Bart De Schuymer]

Op 7/06/2011 16:40, Brad Campbell schreef:
> On 07/06/11 21:30, Patrick McHardy wrote:
>> On 07.06.2011 05:33, Brad Campbell wrote:
>>> On 07/06/11 04:10, Bart De Schuymer wrote:
>>>> Hi Brad,
>>>>
>>>> This has probably nothing to do with ebtables, so please rmmod in case
>>>> it's loaded.
>>>> A few questions I didn't directly see an answer to in the threads I
>>>> scanned...
>>>> I'm assuming you actually use the bridging firewall functionality. So,
>>>> what iptables modules do you use? Can you reduce your iptables 
>>>> rules to
>>>> a core that triggers the bug?
>>>> Or does it get triggered even with an empty set of firewall rules?
>>>> Are you using a stock .35 kernel or is it patched?
>>>> Is this something I can trigger on a poor guy's laptop or does it
>>>> require specialized hardware (I'm catching up on qemu/kvm...)?
>>>
>>> Not specialised hardware as such, I've just not been able to reproduce
>>> it outside of this specific operating scenario.
>>
>> The last similar problem we've had was related to the 32/64 bit compat
>> code. Are you running 32 bit userspace on a 64 bit kernel?
>
> No, 32 bit Guest OS, but a completely 64 bit userspace on a 64 bit 
> kernel.
>
> Userspace is current Debian Stable. Kernel is Vanilla and qemu-kvm is 
> current git
>
If the bug is easily triggered with your guest os, then you could try to 
capture the traffic with wireshark (or something else) in a 
configuration that doesn't crash your system. Save the traffic in a pcap 
file. Then you can see if resending that traffic in the vulnerable 
configuration triggers the bug (I don't know if something in Windows 
exists, but tcpreplay should work for Linux). Once you have such a 
capture , chances are the bug is even easily reproducible by us (unless 
it's hardware-specific). Success isn't guaranteed, but I think it's 
worth a shot...

cheers,
Bart


-- 
Bart De Schuymer
www.artinalgorithms.be

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 08/06/11 02:04, Bart De Schuymer wrote:

> If the bug is easily triggered with your guest os, then you could try to
> capture the traffic with wireshark (or something else) in a
> configuration that doesn't crash your system. Save the traffic in a pcap
> file. Then you can see if resending that traffic in the vulnerable
> configuration triggers the bug (I don't know if something in Windows
> exists, but tcpreplay should work for Linux). Once you have such a
> capture , chances are the bug is even easily reproducible by us (unless
> it's hardware-specific). Success isn't guaranteed, but I think it's
> worth a shot...

The issue with this is I don't have a configuration that does not crash 
the system. This only happens under the specific circumstance that 
traffic from VM A is being DNAT'd to VM B. If I disable 
CONFIG_BRIDGE_NETFILTER, or I leave out the DNAT then I can't replicate 
the problem as I don't seem to be able to get the packets to go where I 
want them to go.

Let me try and explain it a little more clearly with made up IP 
addresses to illustrate the problem.

I have VM A (1.1.1.2) and VM B (1.1.1.3) on br1 (1.1.1.1)
I have public IP on ppp0 (2.2.2.2).

VM B can talk to VM A using its host address (1.1.1.2) and there is no 
problem.

The DNAT says anything destined for PPP0 that is on port 443 and coming 
from anywhere other than PPP0 (ie inside the network) is to be DNAT'd to 
1.1.1.3.

So VM B (1.1.1.3) tries to connect to ppp0 (2.2.2.2) on port 443, and 
this is redirected to VM B on 1.1.1.2.

Only under this specific circumstance does the problem occur. I can get 
VM B (1.1.1.3) to talk directly to VM A (1.1.1.2) all day long and there 
is no problem, it's only when VM B tries to talk to ppp0 that there is 
an issue (and it happens within seconds of the initial connection).

All these tests have been performed with VM B being a Windows XP guest. 
Tonight I'll try it with a Linux guest and see if I can make it happen. 
If that works I might be able to come up with some reproducible test 
case for you. I have a desktop machine that has Intel VT extensions, so 
I'll work toward making a portable test case.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/03/2011 04:38 PM, Brad Campbell wrote:
>
> Is there anyone who can point me at the appropriate cage to rattle? I 
> know it appears to be a netfilter issue, but I don't seem to be able 
> to get a message to the list (and I am subscribed to it and have been 
> getting mail for months) and I'm not sure who to pester. The other 
> alternative is I just stop doing "that" and wait for it to bite 
> someone else.

The mailing list might be set not to send your own mails back to you.  
Check the list archive.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Brad Campbell]

On 05/06/11 16:14, Avi Kivity wrote:
> On 06/03/2011 04:38 PM, Brad Campbell wrote:
>>
>> Is there anyone who can point me at the appropriate cage to rattle? I
>> know it appears to be a netfilter issue, but I don't seem to be able
>> to get a message to the list (and I am subscribed to it and have been
>> getting mail for months) and I'm not sure who to pester. The other
>> alternative is I just stop doing "that" and wait for it to bite
>> someone else.
>
> The mailing list might be set not to send your own mails back to you.
> Check the list archive.

Yep, I did that first..

Given the response to previous issues along the same line, it looks a 
bit like I just remember not to actually use the system in the way that 
triggers the bug and be happy that 99% of the time the kernel does not 
panic, but have that lovely feeling in the back of the skull that says 
"any time now, and without obvious reason the whole machine might just 
come crashing down"..

I guess it's still better than running Xen or Windows..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Avi Kivity]

On 06/05/2011 04:45 PM, Brad Campbell wrote:
>> The mailing list might be set not to send your own mails back to you.
>> Check the list archive.
>
>
> Yep, I did that first..
>
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way 
> that triggers the bug and be happy that 99% of the time the kernel 
> does not panic, but have that lovely feeling in the back of the skull 
> that says "any time now, and without obvious reason the whole machine 
> might just come crashing down"..
>
> I guess it's still better than running Xen or Windows..

Not at all.  Can some networking/netfilter expert look at this?

Please file a bug with all the relevant information in this thread.  If 
you can look for a previous version that worked, that might increase the 
chances of the bug being resolved faster.

-- 
error compiling committee.c: too many arguments to function

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: KVM induced panic on 2.6.38[2367] & 2.6.39

[Eric Dumazet]

Le dimanche 05 juin 2011 A  21:45 +0800, Brad Campbell a A(C)crit :
> On 05/06/11 16:14, Avi Kivity wrote:
> > On 06/03/2011 04:38 PM, Brad Campbell wrote:
> >>
> >> Is there anyone who can point me at the appropriate cage to rattle? I
> >> know it appears to be a netfilter issue, but I don't seem to be able
> >> to get a message to the list (and I am subscribed to it and have been
> >> getting mail for months) and I'm not sure who to pester. The other
> >> alternative is I just stop doing "that" and wait for it to bite
> >> someone else.
> >
> > The mailing list might be set not to send your own mails back to you.
> > Check the list archive.
> 
> Yep, I did that first..
> 
> Given the response to previous issues along the same line, it looks a 
> bit like I just remember not to actually use the system in the way that 
> triggers the bug and be happy that 99% of the time the kernel does not 
> panic, but have that lovely feeling in the back of the skull that says 
> "any time now, and without obvious reason the whole machine might just 
> come crashing down"..
> 
> I guess it's still better than running Xen or Windows..

Could you please try latest linux-2.6 tree ?

We fixed many networking bugs that could explain your crash.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>