BUG at kmem_cache

linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed

* BUG at kmem_cache_alloc
       [not found] <1089649229.4894208.1363925156257.JavaMail.root@redhat.com>
@ 2013-03-22  4:18 ` CAI Qian
  2013-03-22  9:35   ` David Rientjes
  0 siblings, 1 reply; 9+ messages in thread
From: CAI Qian @ 2013-03-22  4:18 UTC (permalink / raw)
  To: linux-mm; +Cc: stable, LKML, Oleg Nesterov

Starting to see those on 3.8.4 (never saw in 3.8.2) stable kernel on a few systems
during LTP run,

[11297.597242] BUG: unable to handle kernel paging request at 00000000fffffffe 
[11297.598022] IP: [] kmem_cache_alloc+0x68/0x1e0 
[11297.598022] PGD 7b9eb067 PUD 0  
[11297.598022] Oops: 0000 [#2] SMP  
[11297.598022] Modules linked in: cmtp kernelcapi bnep scsi_transport_iscsi rfcomm l2tp_ppp l2tp_netlink l2tp_core hidp ipt_ULOG af_key nfc rds pppoe pppox ppp_generic slhc af_802154 atm ip6table_filter ip6_tables iptable_filter ip_tables btrfs zlib_deflate vfat fat nfs_layout_nfsv41_files nfsv4 auth_rpcgss nfsv3 nfs_acl nfsv2 nfs lockd sunrpc fscache nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp sctp sg kvm_amd kvm virtio_balloon i2c_piix4 pcspkr xfs libcrc32c ata_generic pata_acpi cirrus drm_kms_helper ttm ata_piix virtio_net drm libata virtio_blk i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipt_REJECT] 
[11297.598022] CPU 1  
[11297.598022] Pid: 14134, comm: ltp-pan Tainted: G      D      3.8.4+ #1 Bochs Bochs 
[11297.598022] RIP: 0010:[]  [] kmem_cache_alloc+0x68/0x1e0 
[11297.598022] RSP: 0018:ffff8800447dbdd0  EFLAGS: 00010246 
[11297.598022] RAX: 0000000000000000 RBX: ffff88007c169970 RCX: 00000000018acdcd 
[11297.598022] RDX: 000000000006c104 RSI: 00000000000080d0 RDI: ffff88007d04ac00 
[11297.598022] RBP: ffff8800447dbe10 R08: 0000000000017620 R09: ffffffff810fe2e2 
[11297.598022] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fffffffe 
[11297.598022] R13: 00000000000080d0 R14: ffff88007d04ac00 R15: ffff88007d04ac00 
[11297.598022] FS:  00007f09c29b4740(0000) GS:ffff88007fd00000(0000) knlGS:00000000f74d86c0 
[11297.598022] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
[11297.598022] CR2: 00000000fffffffe CR3: 0000000037213000 CR4: 00000000000006e0 
[11297.598022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
[11297.598022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
[11297.598022] Process ltp-pan (pid: 14134, threadinfo ffff8800447da000, task ffff8800551ab2e0) 
[11297.598022] Stack: 
[11297.598022]  ffffffff810fe2e2 ffffffff8108cf0f 0000000001200011 ffff88007c169970 
[11297.598022]  0000000000000000 00007f09c29b4a10 0000000000000000 ffff88007c169970 
[11297.598022]  ffff8800447dbe30 ffffffff810fe2e2 0000000000000000 0000000001200011 
[11297.598022] Call Trace: 
[11297.598022]  [] ? __delayacct_tsk_init+0x22/0x40 
[11297.598022]  [] ? prepare_creds+0xdf/0x190 
[11297.598022]  [] __delayacct_tsk_init+0x22/0x40 
[11297.598022]  [] copy_process.part.25+0x31f/0x13f0 
[11297.598022]  [] do_fork+0xa9/0x350 
[11297.598022]  [] sys_clone+0x16/0x20 
[11297.598022]  [] stub_clone+0x69/0x90 
[11297.598022]  [] ? system_call_fastpath+0x16/0x1b 
[11297.598022] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 2b 01 00 00 49 63 46 20 4d 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 55 01 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7  
[11297.598022] RIP  [] kmem_cache_alloc+0x68/0x1e0 
[11297.598022]  RSP  
[11297.598022] CR2: 00000000fffffffe 
[11297.727799] ---[ end trace 037bde72f23b34d2 ]---

Never saw this in mainline but only something like this wondering could be related
(that kmem_cache_alloc also in the trace).

[12124.201919] INFO: task kworker/2:1:166 blocked for more than 120 seconds. 
[12124.242758] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[12124.289801] kworker/2:1     D ffff88081fc54440     0   166      2 0x00000000 
[12124.330784]  ffff88081361ba68 0000000000000046 ffff880813568000 ffff88081361bfd8 
[12124.373694]  ffff88081361bfd8 ffff88081361bfd8 ffff8808144fb2e0 ffff880813568000 
[12124.416896]  0000000000000000 ffff880813568000 ffff8808133f8930 0000000000000002 
[12124.458674] Call Trace: 
[12124.473291]  [] schedule+0x29/0x70 
[12124.502143]  [] rwsem_down_failed_common+0xda/0x230 
[12124.539311]  [] rwsem_down_write_failed+0x13/0x20 
[12124.575585]  [] call_rwsem_down_write_failed+0x13/0x20 
[12124.614129]  [] ? down_write+0x32/0x40 
[12124.644703]  [] xlog_cil_push+0x89/0x3c0 [xfs] 
[12124.680046]  [] ? up+0x32/0x50 
[12124.706083]  [] ? flush_work+0x113/0x170 
[12124.738078]  [] xlog_cil_force_lsn+0xf7/0x160 [xfs] 
[12124.776062]  [] ? xfs_trans_free_items+0x88/0xb0 [xfs] 
[12124.814503]  [] _xfs_log_force_lsn+0x5a/0x2e0 [xfs] 
[12124.851512]  [] xfs_trans_commit+0x263/0x270 [xfs] 
[12124.887996]  [] xfs_fs_log_dummy+0x61/0x90 [xfs] 
[12124.924015]  [] ? xfs_log_need_covered+0x93/0xc0 [xfs] 
[12124.963079]  [] xfs_log_worker+0x48/0x50 [xfs] 
[12124.997404]  [] process_one_work+0x174/0x3d0 
[12125.031408]  [] worker_thread+0x10f/0x390 
[12125.062936]  [] ? busy_worker_rebind_fn+0xb0/0xb0 
[12125.098924]  [] kthread+0xc0/0xd0 
[12125.126124]  [] ? kthread_create_on_node+0x120/0x120 
[12125.162995]  [] ret_from_fork+0x7c/0xb0 
[12125.193516]  [] ? kthread_create_on_node+0x120/0x120 
[12125.229431] INFO: task beah-beaker-bac:3331 blocked for more than 120 seconds. 
[12125.269795] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disabfd5bb59b0 0000000000000086 ffff881fbf770000 ffff881fd5bb5fd8 
[12129.329892]  ffff881fd5bb5fd8 ffff8Trace: 
[12134.346772]  [] schedule+0x29/0x70 
[12134.401771] 30 
[12136.376743]  [] ? kmem_cache_alloc+0x35/0x1e0 
[12136.411867]  [] rwsem_down_read_failed+0x15/0x17 
[12136.448141]  [] call_rwsem_down_read_failed+0x14/0x30 
[12136.487438]  [] ? kmem_alloc+0x67/0xf0 [xfs] 
[12136.521108]  [] ? down_read+0x24/0x2b 
[12136.549333]  [] xfs_log_commit_cil+0x1a6/0x4a0 [xfs] 
[12136.586227]  [] ? kmem_zone_alloc+0x67/0xf0 [xfs] 
[12136.621792]  [] xfs_trans_commit+0x134/0x270 [xfs] 
[12136.658163]  [] xfs_vn_update_time+0xf7/0x1a0 [xfs] 
[12136.694257]  [] update_time+0x23/0xc0 
[12136.722821]  [] ? mnt_clone_write+0x12/0x30 
[12136.755240]  [] file_update_time+0x98/0xf0 
[12136.785989]  [] xfs_file_aio_write_checks+0xdb/0xf0 [xfs] 
[12136.825592]  [] xfs_file_buffered_aio_write+0x7b/0x1a0 [xfs] 
[12136.868827]  [] xfs_file_aio_write+0xf9/0x160 [xfs] 
[12136.907037]  [] do_sync_write+0xa7/0xe0 
[12136.939287]  [] vfs_write+0xac/0x180 
[12136.969067]  [] sys_pwrite64+0x9a/0xb0 
[12137.000528]  [] system_call_fastpath+0x16/0x1b 
[12137.036022] INFO: task master:3497 blocked for more than 120 seconds. 
[12137.073730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 
[12137.117609] master          D ffff88101fd54440     0  3497      1 0x00000080 
[12137.155846]  ffff8808091a3a60 0000000000000082 ffff880813079970 ffff8808091a3fd8 
[12137.197460]  ffff8808091a3fd8 ffff8808091a3fd8 ffff8808145432e0 ffff880813079970 
[12137.239430]  ffff8808091a3a88 ffff880813079970 ffff8808133f8930 0000000000000001 
[12137.279786] Call Trace: 
[12137.293627]  [] schedule+0x29/0x70 
[12137.321390]  [] rwsem_down_failed_common+0xda/0x230 
[12137.357677]  [] ? __enqueue_entity+0x78/0x80 
[12137.390943]  [] ? kmem_cache_alloc+0x35/0x1e0 
[12137.426089]  [] rwsem_down_read_failed+0x15/0x17 
[12137.462395]  [] call_rwsem_down_read_failed+0x14/0x30 
[12137.502332]  [] ? kmem_alloc+0x67/0xf0 [xfs] 
[12137.537843]  [] ? down_read+0x24/

Any idea?

CAI Qian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: BUG at kmem_cache_alloc
  2013-03-22  4:18 ` BUG at kmem_cache_alloc CAI Qian
@ 2013-03-22  9:35   ` David Rientjes
  2013-03-25  5:37     ` CAI Qian
  0 siblings, 1 reply; 9+ messages in thread
From: David Rientjes @ 2013-03-22  9:35 UTC (permalink / raw)
  To: CAI Qian; +Cc: linux-mm, linux-kernel, Oleg Nesterov

On Fri, 22 Mar 2013, CAI Qian wrote:

> Starting to see those on 3.8.4 (never saw in 3.8.2) stable kernel on a few systems
> during LTP run,
> 
> [11297.597242] BUG: unable to handle kernel paging request at 00000000fffffffe 
> [11297.598022] IP: [] kmem_cache_alloc+0x68/0x1e0 

Is this repeatable?  Do you have CONFIG_SLAB or CONFIG_SLUB enabled?

> [11297.598022] PGD 7b9eb067 PUD 0  
> [11297.598022] Oops: 0000 [#2] SMP  
> [11297.598022] Modules linked in: cmtp kernelcapi bnep scsi_transport_iscsi rfcomm l2tp_ppp l2tp_netlink l2tp_core hidp ipt_ULOG af_key nfc rds pppoe pppox ppp_generic slhc af_802154 atm ip6table_filter ip6_tables iptable_filter ip_tables btrfs zlib_deflate vfat fat nfs_layout_nfsv41_files nfsv4 auth_rpcgss nfsv3 nfs_acl nfsv2 nfs lockd sunrpc fscache nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp sctp sg kvm_amd kvm virtio_balloon i2c_piix4 pcspkr xfs libcrc32c ata_generic pata_acpi cirrus drm_kms_helper ttm ata_piix virtio_net drm libata virtio_blk i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipt_REJECT] 
> [11297.598022] CPU 1  
> [11297.598022] Pid: 14134, comm: ltp-pan Tainted: G      D      3.8.4+ #1 Bochs Bochs 
> [11297.598022] RIP: 0010:[]  [] kmem_cache_alloc+0x68/0x1e0 
> [11297.598022] RSP: 0018:ffff8800447dbdd0  EFLAGS: 00010246 
> [11297.598022] RAX: 0000000000000000 RBX: ffff88007c169970 RCX: 00000000018acdcd 
> [11297.598022] RDX: 000000000006c104 RSI: 00000000000080d0 RDI: ffff88007d04ac00 
> [11297.598022] RBP: ffff8800447dbe10 R08: 0000000000017620 R09: ffffffff810fe2e2 
> [11297.598022] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fffffffe 
> [11297.598022] R13: 00000000000080d0 R14: ffff88007d04ac00 R15: ffff88007d04ac00 
> [11297.598022] FS:  00007f09c29b4740(0000) GS:ffff88007fd00000(0000) knlGS:00000000f74d86c0 
> [11297.598022] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
> [11297.598022] CR2: 00000000fffffffe CR3: 0000000037213000 CR4: 00000000000006e0 
> [11297.598022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
> [11297.598022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
> [11297.598022] Process ltp-pan (pid: 14134, threadinfo ffff8800447da000, task ffff8800551ab2e0) 
> [11297.598022] Stack: 
> [11297.598022]  ffffffff810fe2e2 ffffffff8108cf0f 0000000001200011 ffff88007c169970 
> [11297.598022]  0000000000000000 00007f09c29b4a10 0000000000000000 ffff88007c169970 
> [11297.598022]  ffff8800447dbe30 ffffffff810fe2e2 0000000000000000 0000000001200011 
> [11297.598022] Call Trace: 
> [11297.598022]  [] ? __delayacct_tsk_init+0x22/0x40 
> [11297.598022]  [] ? prepare_creds+0xdf/0x190 
> [11297.598022]  [] __delayacct_tsk_init+0x22/0x40 
> [11297.598022]  [] copy_process.part.25+0x31f/0x13f0 
> [11297.598022]  [] do_fork+0xa9/0x350 
> [11297.598022]  [] sys_clone+0x16/0x20 
> [11297.598022]  [] stub_clone+0x69/0x90 
> [11297.598022]  [] ? system_call_fastpath+0x16/0x1b 
> [11297.598022] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 2b 01 00 00 49 63 46 20 4d 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 55 01 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7  
> [11297.598022] RIP  [] kmem_cache_alloc+0x68/0x1e0 
> [11297.598022]  RSP  
> [11297.598022] CR2: 00000000fffffffe 
> [11297.727799] ---[ end trace 037bde72f23b34d2 ]---
> 
> Never saw this in mainline but only something like this wondering could be related
> (that kmem_cache_alloc also in the trace).
> 

These are unrelated.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: BUG at kmem_cache_alloc
  2013-03-22  9:35   ` David Rientjes
@ 2013-03-25  5:37     ` CAI Qian
  2013-03-25  7:42       ` CAI Qian
  2013-03-25 18:00       ` Christoph Lameter
  0 siblings, 2 replies; 9+ messages in thread
From: CAI Qian @ 2013-03-25  5:37 UTC (permalink / raw)
  To: David Rientjes; +Cc: linux-mm, linux-kernel, Oleg Nesterov



----- Original Message -----
> From: "David Rientjes" <rientjes@google.com>
> To: "CAI Qian" <caiqian@redhat.com>
> Cc: "linux-mm" kvack.org>, linux-kernel@vger.kernel.org, "Oleg Nesterov" <oleg@redhat.com>
> Sent: Friday, March 22, 2013 5:35:34 PM
> Subject: Re: BUG at kmem_cache_alloc
> 
> On Fri, 22 Mar 2013, CAI Qian wrote:
> 
> > Starting to see those on 3.8.4 (never saw in 3.8.2) stable kernel
> > on a few systems
> > during LTP run,
> > 
> > [11297.597242] BUG: unable to handle kernel paging request at
> > 00000000fffffffe
> > [11297.598022] IP: [] kmem_cache_alloc+0x68/0x1e0
> 
> Is this repeatable?  Do you have CONFIG_SLAB or CONFIG_SLUB enabled?
Saw it on 2 systems so far - one HP server and one KVM guest. Still
trying to reproduce. Used CONFIG_SLUB=y.
CAI Qian
> 
> > [11297.598022] PGD 7b9eb067 PUD 0
> > [11297.598022] Oops: 0000 [#2] SMP
> > [11297.598022] Modules linked in: cmtp kernelcapi bnep
> > scsi_transport_iscsi rfcomm l2tp_ppp l2tp_netlink l2tp_core hidp
> > ipt_ULOG af_key nfc rds pppoe pppox ppp_generic slhc af_802154 atm
> > ip6table_filter ip6_tables iptable_filter ip_tables btrfs
> > zlib_deflate vfat fat nfs_layout_nfsv41_files nfsv4 auth_rpcgss
> > nfsv3 nfs_acl nfsv2 nfs lockd sunrpc fscache nfnetlink_log
> > nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver
> > nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp sctp sg kvm_amd kvm
> > virtio_balloon i2c_piix4 pcspkr xfs libcrc32c ata_generic
> > pata_acpi cirrus drm_kms_helper ttm ata_piix virtio_net drm libata
> > virtio_blk i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
> > [last unloaded: ipt_REJECT]
> > [11297.598022] CPU 1
> > [11297.598022] Pid: 14134, comm: ltp-pan Tainted: G      D
> >      3.8.4+ #1 Bochs Bochs
> > [11297.598022] RIP: 0010:[]  [] kmem_cache_alloc+0x68/0x1e0
> > [11297.598022] RSP: 0018:ffff8800447dbdd0  EFLAGS: 00010246
> > [11297.598022] RAX: 0000000000000000 RBX: ffff88007c169970 RCX:
> > 00000000018acdcd
> > [11297.598022] RDX: 000000000006c104 RSI: 00000000000080d0 RDI:
> > ffff88007d04ac00
> > [11297.598022] RBP: ffff8800447dbe10 R08: 0000000000017620 R09:
> > ffffffff810fe2e2
> > [11297.598022] R10: 0000000000000000 R11: 0000000000000000 R12:
> > 00000000fffffffe
> > [11297.598022] R13: 00000000000080d0 R14: ffff88007d04ac00 R15:
> > ffff88007d04ac00
> > [11297.598022] FS:  00007f09c29b4740(0000)
> > GS:ffff88007fd00000(0000) knlGS:00000000f74d86c0
> > [11297.598022] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [11297.598022] CR2: 00000000fffffffe CR3: 0000000037213000 CR4:
> > 00000000000006e0
> > [11297.598022] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [11297.598022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> > 0000000000000400
> > [11297.598022] Process ltp-pan (pid: 14134, threadinfo
> > ffff8800447da000, task ffff8800551ab2e0)
> > [11297.598022] Stack:
> > [11297.598022]  ffffffff810fe2e2 ffffffff8108cf0f 0000000001200011
> > ffff88007c169970
> > [11297.598022]  0000000000000000 00007f09c29b4a10 0000000000000000
> > ffff88007c169970
> > [11297.598022]  ffff8800447dbe30 ffffffff810fe2e2 0000000000000000
> > 0000000001200011
> > [11297.598022] Call Trace:
> > [11297.598022]  [] ? __delayacct_tsk_init+0x22/0x40
> > [11297.598022]  [] ? prepare_creds+0xdf/0x190
> > [11297.598022]  [] __delayacct_tsk_init+0x22/0x40
> > [11297.598022]  [] copy_process.part.25+0x31f/0x13f0
> > [11297.598022]  [] do_fork+0xa9/0x350
> > [11297.598022]  [] sys_clone+0x16/0x20
> > [11297.598022]  [] stub_clone+0x69/0x90
> > [11297.598022]  [] ? system_call_fastpath+0x16/0x1b
> > [11297.598022] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00
> > 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 2b 01 00 00 49 63 46 20 4d
> > 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 55 01 00 00 48 8d 4a 01 4c
> > 89 e0 65 49 0f c7
> > [11297.598022] RIP  [] kmem_cache_alloc+0x68/0x1e0
> > [11297.598022]  RSP
> > [11297.598022] CR2: 00000000fffffffe
> > [11297.727799] ---[ end trace 037bde72f23b34d2 ]---
> > 
> > Never saw this in mainline but only something like this wondering
> > could be related
> > (that kmem_cache_alloc also in the trace).
> > 
> 
> These are unrelated.
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: href=mailto:"dont@kvack.org"> email@kvack.org 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: BUG at kmem_cache_alloc
  2013-03-25  5:37     ` CAI Qian
@ 2013-03-25  7:42       ` CAI Qian
  2013-03-25 18:00       ` Christoph Lameter
  1 sibling, 0 replies; 9+ messages in thread
From: CAI Qian @ 2013-03-25  7:42 UTC (permalink / raw)
  To: David Rientjes; +Cc: linux-mm, linux-kernel, Oleg Nesterov



----- Original Message -----
> From: "CAI Qian" <caiqian@redhat.com>
> To: "David Rientjes" <rientjes@google.com>
> Cc: "linux-mm" <linux-mm@kvack.org>, linux-kernel@vger.kernel.org, "Oleg Nesterov" <oleg@redhat.com>
> Sent: Monday, March 25, 2013 1:37:50 PM
> Subject: Re: BUG at kmem_cache_alloc
> 
> 
> 
> ----- Original Message -----
> > From: "David Rientjes" <rientjes@google.com>
> > To: "CAI Qian" <caiqian@redhat.com>
> > Cc: "linux-mm" kvack.org>, linux-kernel@vger.kernel.org, "Oleg
> > Nesterov" <oleg@redhat.com>
> > Sent: Friday, March 22, 2013 5:35:34 PM
> > Subject: Re: BUG at kmem_cache_alloc
> > 
> > On Fri, 22 Mar 2013, CAI Qian wrote:
> > 
> > > Starting to see those on 3.8.4 (never saw in 3.8.2) stable kernel
> > > on a few systems
> > > during LTP run,
> > > 
> > > [11297.597242] BUG: unable to handle kernel paging request at
> > > 00000000fffffffe
> > > [11297.598022] IP: [] kmem_cache_alloc+0x68/0x1e0
> > 
> > Is this repeatable?  Do you have CONFIG_SLAB or CONFIG_SLUB
> > enabled?
> Saw it on 2 systems so far - one HP server and one KVM guest. Still
This happened again during trinity run. Bisecting is in-progress...
CAI Qian

[ 9236.418633] BUG: unable to handle kernel paging request at 0000000000010000 
[ 9236.420033] IP: [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9236.421405] PGD 147308067 PUD 149456067 PMD 0  
[ 9236.422476] Oops: 0000 [#1] SMP  
[ 9236.423182] Modules linked in: tun(F+) cmtp(F) kernelcapi(F) hidp(F) rfcomm(F) bnep(F) l2tp_ppp(F) l2tp_netlink(F) l2tp_core(F) ipt_ULOG(F) scsi_transport_iscsi(F) af_802154(F) rds(F) af_key(F) pppoe(F) pppox(F) ppp_generic(F) slhc(F) nfc(F) atm(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) btrfs(F) zlib_deflate(F) vfat(F) fat(F) nfs_layout_nfsv41_files(F) nfsv4(F) auth_rpcgss(F) nfsv3(F) nfs_acl(F) nfsv2(F) nfs(F) lockd(F) sunrpc(F) fscache(F) nfnetlink_log(F) nfnetlink(F) bluetooth(F) rfkill(F) arc4(F) md4(F) nls_utf8(F) cifs(F) dns_resolver(F) nf_tproxy_core(F) nls_koi8_u(F) nls_cp932(F) ts_kmp(F) sctp(F) fuse(F) sg(F) kvm_amd(F) kvm(F) amd64_edac_mod(F) edac_mce_amd(F) bnx2x(F) serio_raw(F) edac_core(F) netxen_nic(F) mdio(F) k10temp(F) microcode(F) i2c_piix4(F) ipmi_si(F) ipmi_msghandler(F) shpchp(F) hpwdt(F) hpilo(F) xfs(F) libcrc32c(F) sd_mod(F) crc_t10dif(F) radeon(F) i2c_algo_bit(F) drm_kms_helper(F) sata_svw(F) ttm(F) libata(F) drm(F) i2c_core(F) dm_mirror(F) dm_region_hash(F) dm_log(F) dm_mod(F) [last unloaded: ipt_REJECT] 
[ 9236.444373] CPU 3  
[ 9236.444885] Pid: 3495, comm: modprobe Tainted: GF       W    3.8.4 #1 HP ProLiant BL495c G5   
[ 9236.446808] RIP: 0010:[<ffffffff8118a008>]  [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9236.448757] RSP: 0018:ffff8801206c3c88  EFLAGS: 00010246 
[ 9236.449946] RAX: 0000000000000000 RBX: 000000000000000f RCX: 0000000000000124 
[ 9236.451531] RDX: 000000000002c644 RSI: 00000000000080d0 RDI: ffff88014b040c00 
[ 9236.453123] RBP: ffff8801206c3cd8 R08: 0000000000017690 R09: ffffffff81215c94 
[ 9236.454735] R10: 0000000000004380 R11: 0000000000000001 R12: 0000000000010000 
[ 9236.456373] R13: 00000000000080d0 R14: ffff88014b040c00 R15: ffff88014b040c00 
[ 9236.457993] FS:  00007f92ce2d5740(0000) GS:ffff88014fd80000(0000) knlGS:00000000f75546c0 
[ 9236.459803] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
[ 9236.461081] CR2: 0000000000010000 CR3: 00000001360c9000 CR4: 00000000000007e0 
[ 9236.462682] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
[ 9236.464279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
[ 9236.465875] Process modprobe (pid: 3495, threadinfo ffff8801206c2000, task ffff8801463065c0) 
[ 9236.467747] Stack: 
[ 9236.468260]  ffff8801206c3ca8 ffffffff812156c5 ffffffff81215c94 ffffffff812156c5 
[ 9236.469903]  ffff8801495405ae 000000000000000f 0000000000000002 ffff8800a2862d10 
[ 9236.471569]  0000000000000000 0000000000008124 ffff8801206c3d28 ffffffff81215c94 
[ 9236.473315] Call Trace: 
[ 9236.473862]  [<ffffffff812156c5>] ? sysfs_link_sibling+0xb5/0xe0 
[ 9236.475216]  [<ffffffff81215c94>] ? sysfs_new_dirent+0x54/0x110 
[ 9236.476571]  [<ffffffff812156c5>] ? sysfs_link_sibling+0xb5/0xe0 
[ 9236.477970]  [<ffffffff81215c94>] sysfs_new_dirent+0x54/0x110 
[ 9236.479253]  [<ffffffff81215ebc>] ? sysfs_add_one+0x2c/0x100 
[ 9236.480548]  [<ffffffff8121517b>] sysfs_add_file_mode+0x6b/0xe0 
[ 9236.481889]  [<ffffffff81217f70>] internal_create_group+0xd0/0x230 
[ 9236.483288]  [<ffffffff81218103>] sysfs_create_group+0x13/0x20 
[ 9236.484613]  [<ffffffff810c90dc>] load_module+0x138c/0x1660 
[ 9236.485865]  [<ffffffff8131abe0>] ? ddebug_proc_open+0xc0/0xc0 
[ 9236.487174]  [<ffffffff810c9487>] sys_init_module+0xd7/0x120 
[ 9236.488498]  [<ffffffff8161f7d9>] system_call_fastpath+0x16/0x1b 
[ 9236.489797] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 1c 01 00 00 49 63 46 20 4d 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 46 01 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7  
[ 9236.494164] RIP  [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9236.495550]  RSP <ffff8801206c3c88> 
[ 9236.496287] CR2: 0000000000010000 
[ 9236.513347] ---[ end trace 3567090873e2c5de ]--- 
[watchdog] 9407744 iterations. [F:8626700 S:781126] 
[ 9237.335465] BUG: unable to handle kernel paging request at 0000000000010000 
[ 9237.336924] IP: [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9237.338280] PGD 1489d8067 PUD 1489da067 PMD 0  
[ 9237.339197] Oops: 0000 [#2] SMP  
[ 9237.339945] Modules linked in: tun(F+) cmtp(F) kernelcapi(F) hidp(F) rfcomm(F) bnep(F) l2tp_ppp(F) l2tp_netlink(F) l2tp_core(F) ipt_ULOG(F) scsi_transport_iscsi(F) af_802154(F) rds(F) af_key(F) pppoe(F) pppox(F) ppp_generic(F) slhc(F) nfc(F) atm(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) btrfs(F) zlib_deflate(F) vfat(F) fat(F) nfs_layout_nfsv41_files(F) nfsv4(F) auth_rpcgss(F) nfsv3(F) nfs_acl(F) nfsv2(F) nfs(F) lockd(F) sunrpc(F) fscache(F) nfnetlink_log(F) nfnetlink(F) bluetooth(F) rfkill(F) arc4(F) md4(F) nls_utf8(F) cifs(F) dns_resolver(F) nf_tproxy_core(F) nls_koi8_u(F) nls_cp932(F) ts_kmp(F) sctp(F) fuse(F) sg(F) kvm_amd(F) kvm(F) amd64_edac_mod(F) edac_mce_amd(F) bnx2x(F) serio_raw(F) edac_core(F) netxen_nic(F) mdio(F) k10temp(F) microcode(F) i2c_piix4(F) ipmi_si(F) ipmi_msghandler(F) shpchp(F) hpwdt(F) hpilo(F) xfs(F) libcrc32c(F) sd_mod(F) crc_t10dif(F) radeon(F) i2c_algo_bit(F) drm_kms_helper(F) sata_svw(F) ttm(F) libata(F) drm(F) i2c_core(F) dm_mirror(F) dm_region_hash(F) dm_log(F) dm_mod(F) [last unloaded: ipt_REJECT] 
[ 9237.361067] CPU 3  
[ 9237.361512] Pid: 4191, comm: rhts-test-runne Tainted: GF     D W    3.8.4 #1 HP ProLiant BL495c G5   
[ 9237.363861] RIP: 0010:[<ffffffff8118a008>]  [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9237.365823] RSP: 0018:ffff8801208d9dc0  EFLAGS: 00010246 
[ 9237.367069] RAX: 0000000000000000 RBX: ffff8801463065c0 RCX: 0000000000000000 
[ 9237.368381] RDX: 000000000002c644 RSI: 00000000000080d0 RDI: ffff88014b040c00 
[ 9237.369678] RBP: ffff8801208d9e10 R08: 0000000000017690 R09: ffffffff810fe6e2 
[ 9237.370967] R10: 0000000000000000 R11: ffffffffffffffe2 R12: 0000000000010000 
[ 9237.372213] R13: 00000000000080d0 R14: ffff88014b040c00 R15: ffff88014b040c00 
[ 9237.373498] FS:  00007fc53f905740(0000) GS:ffff88014fd80000(0000) knlGS:00000000f75546c0 
[ 9237.374947] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
[ 9237.376071] CR2: 0000000000010000 CR3: 000000010817d000 CR4: 00000000000007e0 
[ 9237.377370] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
[ 9237.378652] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
[ 9237.379950] Process rhts-test-runne (pid: 4191, threadinfo ffff8801208d8000, task ffff880146898000) 
[ 9237.381520] Stack: 
[ 9237.381913]  ffff8801208d9dd0 ffffffff8129aea6 ffffffff810fe6e2 ffffffff8108d29f 
[ 9237.383499]  0000000001200011 ffff8801463065c0 0000000000000000 00007fc53f905a10 
[ 9237.385169]  0000000000000000 ffff8801463065c0 ffff8801208d9e30 ffffffff810fe6e2 
[ 9237.386871] Call Trace: 
[ 9237.387484]  [<ffffffff8129aea6>] ? security_prepare_creds+0x16/0x20 
[ 9237.388905]  [<ffffffff810fe6e2>] ? __delayacct_tsk_init+0x22/0x40 
[ 9237.390303]  [<ffffffff8108d29f>] ? prepare_creds+0xdf/0x190 
[ 9237.391592]  [<ffffffff810fe6e2>] __delayacct_tsk_init+0x22/0x40 
[ 9237.392936]  [<ffffffff810605df>] copy_process.part.25+0x31f/0x13f0 
[ 9237.394297]  [<ffffffff8129abd6>] ? security_file_alloc+0x16/0x20 
[ 9237.395677]  [<ffffffff811be652>] ? __alloc_fd+0x42/0x110 
[ 9237.396905]  [<ffffffff810617a9>] do_fork+0xa9/0x350 
[ 9237.398102]  [<ffffffff811be750>] ? get_unused_fd_flags+0x30/0x40 
[ 9237.399466]  [<ffffffff811be78e>] ? __fd_install+0x2e/0x60 
[ 9237.400731]  [<ffffffff81061ad6>] sys_clone+0x16/0x20 
[ 9237.401866]  [<ffffffff8161fb39>] stub_clone+0x69/0x90 
[ 9237.403050]  [<ffffffff8161f7d9>] ? system_call_fastpath+0x16/0x1b 
[ 9237.404441] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 1c 01 00 00 49 63 46 20 4d 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 46 01 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7  
[ 9237.408602] RIP  [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9237.410080]  RSP <ffff8801208d9dc0> 
[ 9237.410862] CR2: 0000000000010000 
[ 9237.411687] ---[ end trace 3567090873e2c5df ]--- 
2013-03-25 03:10:52,681 rhts_task task_exited: INFO task_exited([Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ProcessTerminated'>: A process has ended with a probable error condition: process ended with exit code 137. 
]) 
2013-03-25 03:10:52,710 rhts_task on_exit: INFO quitting... 
2013-03-25 03:10:52,716 rhts_task task_ended: INFO task_ended([Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ProcessTerminated'>: A process has ended with a probable error condition: process ended with exit code 137. 
]) 
[ 9237.541330] BUG: unable to handle kernel paging request at 0000000000010000 
[ 9237.542719] IP: [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9237.544095] PGD 133477067 PUD 148e5b067 PMD 0  
[ 9237.545166] Oops: 0000 [#3] SMP  
[ 9237.545868] Modules linked in: tun(F+) cmtp(F) kernelcapi(F) hidp(F                                                                                                                                ppoe(F) pppox(F) ppp_generic(F) slhc(F) nfc(F) atm(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) btrfs(F) zlib_deflate(F) vfat(F) fat(F) nfs_layout_nfsv41_files(F) nfsv4(F) auth_rpcgss(F) nfsv3(F) nfs_acl(F) nfsv2(F) nfs(F) lockd(F) sunrpc(F) fscache(F) nfnetlink_log(F) nfnetlink(F) bluetooth(F) rfkill(F) arc4(F) md4(F) nls_utf8(F) cifs(F) dns_resolver(F) nf_tproxy_core(F) nls_koi8_u(F) nls_cp932(F) ts_kmp(F) sctp(F) fuse(F) sg(F) kvm_amd(F) kvm(F) amd64_edac_mod(F) edac_mce_amd(F) bnx2x(F) serio_raw(F) edac_core(F) netxen_nic(F) mdio(F) k10temp(F) microcode(F) i2c_piix4(F) ipmi_si(F) ipmi_msghandler(F) shpchp(F) hpwdt(F) hpilo(F) xfs(F) libcrc32c(F) sd_mod(F) crc_t10dif(F) radeon(F) i2c_algo_bit(F) drm_kms_helper(F) sata_svw(F) ttm(F) libata(F) drm(F) i2c_core(F) dm_mirror(F) dm_region_hash(F) dm_log(F) dm_mod(F) [last unloaded: ipt_REJECT] 
[ 9237.567229] CPU 3  
[ 9237.567676] Pid: 3508, comm: kworker/u:0 Tainted: GF     D W    3.8.4 #1 HP ProLiant BL495c G5   
[ 9237.569270] RIP: 0010:[<ffffffff8118a008>]  [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9237.570846] RSP: 0000:ffff880146cc3d98  EFLAGS: 00010246 
[ 9237.571992] RAX: 0000000000000000 RBX: ffff880146304c50 RCX: 0000000000000000 
[ 9237.573580] RDX: 000000000002c644 RSI: 00000000000080d0 RDI: ffff88014b040c00 
[ 9237.575195] RBP: ffff880146cc3de8 R08: 0000000000017690 R09: ffffffff810fe6e2 
[ 9237.576854] R10: ffff88014b001308 R11: ffffffffffffffe2 R12: 0000000000010000 
[ 9237.578440] R13: 00000000000080d0 R14: ffff88014b040c00 R15: ffff88014b040c00 
[ 9237.580063] FS:  00007f11cf46f740(0000) GS:ffff88014fd80000(0000) knlGS:00000000f75546c0 
[ 9237.581853] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
[ 9237.583140] CR2: 0000000000010000 CR3: 0000000141ad5000 CR4: 00000000000007e0 
[ 9237.584740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
[ 9237.586319] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
[ 9237.587939] Process kworker/u:0 (pid: 3508, threadinfo ffff880146cc2000, task ffff8801322ab2e0) 
[ 9237.589905] Stack: 
[ 9237.590333]  ffff880146cc3da8 ffffffff8129aea6 ffffffff810fe6e2 ffffffff8108d29f 
[ 9237.592093]  0000000000800111 ffff880146304c50 0000000000000000 0000000000000000 
[ 9237.593748]  0000000000000000 ffff880146304c50 ffff880146cc3e08 ffffffff810fe6e2 
[ 9237.595407] Call Trace: 
[ 9237.596021]  [<ffffffff8129aea6>] ? security_prepare_creds+0x16/0x20 
[ 9237.597139]  [<ffffffff810fe6e2>] ? __delayacct_tsk_init+0x22/0x40 
[ 9237.598256]  [<ffffffff8108d29f>] ? prepare_creds+0xdf/0x190 
[ 9237.599344]  [<ffffffff810fe6e2>] __delayacct_tsk_init+0x22/0x40 
[ 9237.600433]  [<ffffffff810605df>] copy_process.part.25+0x31f/0x13f0 
[ 9237.601573]  [<ffffffff8107cf40>] ? proc_cap_handler+0x1b0/0x1b0 
[ 9237.602623]  [<ffffffff810617a9>] do_fork+0xa9/0x350 
[ 9237.603512]  [<ffffffff81061a76>] kernel_thread+0x26/0x30 
[ 9237.604495]  [<ffffffff8107c738>] wait_for_helper+0x68/0xa0 
[ 9237.605516]  [<ffffffff81096387>] ? schedule_tail+0x27/0xb0 
[ 9237.606526]  [<ffffffff8107c6d0>] ? __call_usermodehelper+0xb0/0xb0 
[ 9237.607674]  [<ffffffff8161f72c>] ret_from_fork+0x7c/0xb0 
[ 9237.608706]  [<ffffffff8107c6d0>] ? __call_usermodehelper+0xb0/0xb0 
[ 9237.619523] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 1c 01 00 00 49 63 46 20 4d 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 46 01 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7  
[ 9237.648837] RIP  [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9237.650087]  RSP <ffff880146cc3d98> 
[ 9237.650832] CR2: 0000000000010000 
[ 9237.666224] ---[ end trace 3567090873e2c5e0 ]--- 
[watchdog] 9465269 iterations. [F:8684132 S:781220] 
[ 9238.441169] BUG: unable to handle kernel paging request at 0000000000010000 
[ 9238.443191] IP: [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9238.444509] PGD 129a98067 PUD 1363ef067 PMD 0  
[ 9238.445574] Oops: 0000 [#4] SMP  
[ 9238.446286] Modules linked in: tun(F+) cmtp(F) kernelcapi(F) hidp(F) rfcomm(F) bnep(F) l2tp_ppp(F) l2tp_netlink(F) l2tp_core(F) ipt_ULOG(F) scsi_transport_iscsi(F) af_802154(F) rds(F) af_key(F) pppoe(F) pppox(F) ppp_generic(F) slhc(F) nfc(F) atm(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) btrfs(F) zlib_deflate(F) vfat(F) fat(F) nfs_layout_nfsv41_files(F) nfsv4(F) auth_rpcgss(F) nfsv3(F) nfs_acl(F) nfsv2(F) nfs(F) lockd(F) sunrpc(F) fscache(F) nfnetlink_log(F) nfnetlink(F) bluetooth(F) rfkill(F) arc4(F) md4(F) nls_utf8(F) cifs(F) dns_resolver(F) nf_tproxy_core(F) nls_koi8_u(F) nls_cp932(F) ts_kmp(F) sctp(F) fuse(F) sg(F) kvm_amd(F) kvm(F) amd64_edac_mod(F) edac_mce_amd(F) bnx2x(F) serio_raw(F) edac_core(F) netxen_nic(F) mdio(F) k10temp(F) microcode(F) i2c_piix4(F) ipmi_si(F) ipmi_msghandler(F) shpchp(F) hpwdt(F) hpilo(F) xfs(F) libcrc32c(F) sd_mod(F) crc_t10dif(F) radeon(F) i2c_algo_bit(F) drm_kms_helper(F) sata_svw(F) ttm(F) libata(F) drm(F) i2c_core(F) dm_mirror(F) dm_region_hash(F) dm_log(F) dm_mod(F) [last unloaded: ipt_REJECT] 
[ 9238.467980] CPU 3  
[ 9238.468382] Pid: 27728, comm: kworker/u:0 Tainted: GF     D W    3.8.4 #1 HP ProLiant BL495c G5   
[ 9238.470307] RIP: 0010:[<ffffffff8118a008>]  [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9238.471832] RSP: 0018:ffff8800a326bc38  EFLAGS: 00010246 
[ 9238.472807] RAX: 0000000000000000 RBX: ffff880146301970 RCX: 0000000000000000 
[ 9238.474105] RDX: 000000000002c644 RSI: 00000000000080d0 RDI: ffff88014b040c00 
[ 9238.475447] RBP: ffff8800a326bc88 R08: 0000000000017690 R09: ffffffff810fe6e2 
[ 9238.476730] R10: ffffffffffffffff R11: ffffffffffffffe2 R12: 0000000000010000 
[ 9238.478034] R13: 00000000000080d0 R14: ffff88014b040c00 R15: ffff88014b040c00 
[ 9238.479330] FS:  00007f00036a3740(0000) GS:ffff88014fd80000(0000) knlGS:00000000f75546c0 
[ 9238.480773] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
[ 9238.481823] CR2: 0000000000010000 CR3: 0000000135563000 CR4: 00000000000007e0 
[ 9238.483124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
[ 9238.484411] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
[ 9238.485734] Process kworker/u:0 (pid: 27728, threadinfo ffff8800a326a000, task ffff8801321ee5c0) 
[ 9238.487310] Stack: 
[ 9238.487722]  ffff8800a326bc48 ffffffff8129aea6 ffffffff810fe6e2 ffffffff8108d29f 
[ 9238.489162]  0000000000800711 ffff880146301970 0000000000000000 0000000000000000 
[ 9238.490800]  0000000000000000 ffff880146301970 ffff8800a326bca8 ffffffff810fe6e2 
[ 9238.492409] Call Trace: 
[ 9238.492985]  [<ffffffff8129aea6>] ? security_prepare_creds+0x16/0x20 
[ 9238.494412]  [<ffffffff810fe6e2>] ? __delayacct_tsk_init+0x22/0x40 
[ 9238.495854]  [<ffffffff8108d29f>] ? prepare_creds+0xdf/0x190 
[ 9238.497133]  [<ffffffff810fe6e2>] __delayacct_tsk_init+0x22/0x40 
[ 9238.498488]  [<ffffffff810605df>] copy_process.part.25+0x31f/0x13f0 
[ 9238.499895]  [<ffffffff8107c6d0>] ? __call_usermodehelper+0xb0/0xb0 
[ 9238.501307]  [<ffffffff810617a9>] do_fork+0xa9/0x350 
[ 9238.502432]  [<ffffffff8101358e>] ? __switch_to+0x13e/0x4a0 
[ 9238.503698]  [<ffffffff8110ad65>] ? tracing_is_on+0x15/0x30 
[ 9238.504962]  [<ffffffff81061a76>] kernel_thread+0x26/0x30 
[ 9238.506246]  [<ffffffff8107c69c>] __call_usermodehelper+0x7c/0xb0 
[ 9238.507595]  [<ffffffff8107e5b4>] process_one_work+0x164/0x490 
[ 9238.508912]  [<ffffffff81080c3e>] worker_thread+0x15e/0x450 
[ 9238.510173]  [<ffffffff81080ae0>] ? busy_worker_rebind_fn+0x110/0x110 
[ 9238.511605]  [<ffffffff81085f80>] kthread+0xc0/0xd0 
[ 9238.512719]  [<ffffffff81085ec0>] ? kthread_create_on_node+0x120/0x120 
[ 9238.514200]  [<ffffffff8161f72c>] ret_from_fork+0x7c/0xb0 
[ 9238.515423]  [<ffffffff81085ec0>] ? kthread_create_on_node+0x120/0x120 
[ 9238.516909] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 1c 01 00 00 49 63 46 20 4d 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 46 01 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7  
[ 9238.521164] RIP  [<ffffffff8118a008>] kmem_cache_alloc+0x68/0x200 
[ 9238.522547]  RSP <ffff8800a326bc38> 
[ 9238.523367] CR2: 0000000000010000 
[ 9238.524218] ---[ end trace 3567090873e2c5e1 ]--- 
[ 9238.525360] BUG: unable to handle kernel paging request at ffffffffffffffd8 
[ 9238.526891] IP: [<ffffffff81086370>] kthread_data+0x10/0x20 
[ 9238.528114] PGD 18f8067 PUD 18f9067 PMD 0  
[ 9238.529096] Oops: 0000 [#5] SMP  
[ 9238.529780] Modules linked in: tun(F+) cmtp(F) kernelcapi(F) hidp(F) rfcomm(F) bnep(F) l2tp_ppp(F) l2tp_netlink(F) l2tp_core(F) ipt_ULOG(F) scsi_transport_iscsi(F) af_802154(F) rds(F) af_key(F) pppoe(F) pppox(F) ppp_generic(F) slhc(F) nfc(F) atm(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) btrfs(F) zlib_deflate(F) vfat(F) fat(F) nfs_layout_nfsv41_files(F) nfsv4(F) auth_rpcgss(F) nfsv3(F) nfs_acl(F) nfsv2(F) nfs(F) lockd(F) sunrpc(F) fscache(F) nfnetlink_log(F) nfnetlink(F) bluetooth(F) rfkill(F) arc4(F) md4(F) nls_utf8(F) cifs(F) dns_resolver(F) nf_tproxy_core(F) nls_koi8_u(F) nls_cp932(F) ts_kmp(F) sctp(F) fuse(F) sg(F) kvm_amd(F) kvm(F) amd64_edac_mod(F) edac_mce_amd(F) bnx2x(F) serio_raw(F) edac_core(F) netxen_nic(F) mdio(F) k10temp(F) microcode(F) i2c_piix4(F) ipmi_si(F) ipmi_msghandler(F) shpchp(F) hpwdt(F) hpilo(F) xfs(F) libcrc32c(F) sd_mod(F) crc_t10dif(F) radeon(F) i2c_algo_bit(F) drm_kms_helper(F) sata_svw(F) ttm(F) libata(F) drm(F) i2c_core(F) dm_mirror(F) dm_region_hash(F) dm_log(F) dm_mod(F) [last unloaded: ipt_REJECT] 
[ 9238.549662] CPU 3  
[ 9238.549997] Pid: 27728, comm: kworker/u:0 Tainted: GF     D W    3.8.4 #1 HP ProLiant BL495c G5   
[ 9238.551600] RIP: 0010:[<ffffffff81086370>]  [<ffffffff81086370>] kthread_data+0x10/0x20 
[ 9238.553155] RSP: 0018:ffff8800a326b828  EFLAGS: 00010092 
[ 9238.554136] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 000000000000000d 
[ 9238.555426] RDX: 0000000000000004 RSI: 0000000000000003 RDI: ffff8801321ee5c0 
[ 9238.556724] RBP: ffff8800a326b828 R08: ffff8801321ee630 R09: 000000000000010b 
[ 9238.558025] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88014fd94180 
[ 9238.559317] R13: 0000000000000003 R14: ffff8801321ee5b0 R15: ffff8801321ee5c0 
[ 9238.560598] FS:  00007f00036a3740(0000) GS:ffff88014fd80000(0000) knlGS:00000000f75546c0 
[ 9238.562074] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
[ 9238.563162] CR2: ffffffffffffffd8 CR3: 0000000135563000 CR4: 00000000000007e0 
[ 9238.564448] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
[ 9238.565729] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
[ 9238.567030] Process kworker/u:0 (pid: 27728, threadinfo ffff8800a326a000, task ffff8801321ee5c0) 
[ 9238.568600] Stack: 
[ 9238.569023]  ffff8800a326b848 ffffffff81081775 ffff8800a326b848 ffff8801321eeb90 
[ 9238.570483]  ffff8800a326b8b8 ffffffff81615af2 ffff8801321ee5c0 ffff8800a326bfd8 
[ 9238.572187]  ffff8800a326bfd8 ffff8800a326bfd8 ffff8801321ee5c0 ffff8801321ee5c0 
[ 9238.583863] Call Trace: 
[ 9238.584293]  [<ffffffff81081775>] wq_worker_sleeping+0x15/0xc0 
[ 9238.590261]  [<ffffffff81615af2>] __schedule+0x5c2/0x7b0 
[ 9238.591244]  [<ffffffff81616009>] schedule+0x29/0x70 
[ 9238.602045]  [<ffffffff810684df>] do_exit+0x6af/0x9f0 
[ 9238.603146]  [<ffffffff816183ae>] oops_end+0x9e/0xe0 
[ 9238.604304]  [<ffffffff8160acbd>] no_context+0x253/0x27e 
[ 9238.614916]  [<ffffffff810b4e6c>] ? ktime_get_ts+0x4c/0xf0 
[ 9238.616189]  [<ffffffff8160aea8>] __bad_area_nosemaphore+0x1c0/0x1df 
[ 9238.627046]  [<ffffffff81310811>] ? list_del+0x11/0x40 
[ 9238.628185]  [<ffffffff8160aeda>] bad_area_nosemaphore+0x13/0x15 
[ 9238.638969]  [<ffffffff8161b04e>] __do_page_fault+0x38e/0x4d0 
[ 9238.644921]  [<ffffffff8119cef1>] ? mem_cgroup_bad_page_check+0x21/0x30 
[ 9238.651155]  [<ffffffff812fbe59>] ? cpumask_next_and+0x29/0x50 
[ 9238.662141]  [<ffffffff8161b19e>] do_page_fault+0xe/0x10 
[ 9238.663318]  [<ffffffff816177d8>] page_fault+0x28/0x30 
[ 9238.668961]  [<ffffffff810fe6e2>] ? __delayacct_tsk_init+0x22/0x40 
[ 9238.680192]  [<ffffffff8118a008>] ? kmem_cache_alloc+0x68/0x200 
[ 9238.691111]  [<ffffffff81189fd5>] ? kmem_cache_alloc+0x35/0x200 
[ 9238.692393]  [<ffffffff8129aea6>] ? security_prepare_creds+0x16/0x20 
[ 9238.703313]  [<ffffffff810fe6e2>] ? __delayacct_tsk_init+0x22/0x40 
[ 9238.704694]  [<ffffffff8108d29f>] ? prepare_creds+0xdf/0x190 
[ 9238.715769]  [<ffffffff810fe6e2>] __delayacct_tsk_init+0x22/0x40 
[ 9238.716857]  [<ffffffff810605df>] copy_process.part.25+0x31f/0x13f0 
[ 9238.727647]  [<ffffffff8107c6d0>] ? __call_usermodehelper+0xb0/0xb0 
[ 9238.729076]  [<ffffffff810617a9>] do_fork+0xa9/0x350 
[ 9238.739650]  [<ffffffff8101358e>] ? __switch_to+0x13e/0x4a0 
[ 9238.740659]  [<ffffffff8110ad65>] ? tracing_is_on+0x15/0x30 
[ 9238.751661]  [<ffffffff81061a76>] kernel_thread+0x26/0x30 
[ 9238.752639]  [<ffffffff8107c69c>] __call_usermodehelper+0x7c/0xb0 
[ 9238.763691]  [<ffffffff8107e5b4>] process_one_work+0x164/0x490 
[ 9238.764993]  [<ffffffff81080c3e>] worker_thread+0x15e/0x450 
[ 9238.775749]  [<ffffffff81080ae0>] ? busy_worker_rebind_fn+0x110/0x110 
[ 9238.786726]  [<ffffffff81085f80>] kthread+0xc0/0xd0 
[ 9238.787811]  [<ffffffff81085ec0>] ? kthread_create_on_node+0x120/0x120 
[ 9238.798777]  [<ffffffff8161f72c>] ret_from_fork+0x7c/0xb0 
[ 9238.799887]  [<ffffffff81085ec0>] ? kthread_create_on_node+0x120/0x120 
[ 9238.810874] Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 87 78 05 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90  
[ 9238.824739] RIP  [<ffffffff81086370>] kthread_data+0x10/0x20 
[ 9238.835318]  RSP <ffff8800a326b828> 
[ 9238.836050] CR2: ffffffffffffffd8 
[ 9238.836704] ---[ end trace 3567090873e2c5e2 ]--- 
[ 9238.847352] Fixing recursive fault but reboot is needed! 
[ 9251.888205] bnx2x 0000:03:00.0 ksdev0: MDC/MDIO access timeout 
[ 9251.911020] bnx2x 0000:03:00.0 ksdev0: NIC Link is Down 
[ 9270.036495] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0 
[ 9271.140360] Shutting down cpus with NMI 
[ 9271.141082] drm_kms_helper: panic occurred, switching back to text console  
> trying to reproduce. Used CONFIG_SLUB=y.
> CAI Qian
> > 
> > > [11297.598022] PGD 7b9eb067 PUD 0
> > > [11297.598022] Oops: 0000 [#2] SMP
> > > [11297.598022] Modules linked in: cmtp kernelcapi bnep
> > > scsi_transport_iscsi rfcomm l2tp_ppp l2tp_netlink l2tp_core hidp
> > > ipt_ULOG af_key nfc rds pppoe pppox ppp_generic slhc af_802154
> > > atm
> > > ip6table_filter ip6_tables iptable_filter ip_tables btrfs
> > > zlib_deflate vfat fat nfs_layout_nfsv41_files nfsv4 auth_rpcgss
> > > nfsv3 nfs_acl nfsv2 nfs lockd sunrpc fscache nfnetlink_log
> > > nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver
> > > nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp sctp sg kvm_amd kvm
> > > virtio_balloon i2c_piix4 pcspkr xfs libcrc32c ata_generic
> > > pata_acpi cirrus drm_kms_helper ttm ata_piix virtio_net drm
> > > libata
> > > virtio_blk i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
> > > [last unloaded: ipt_REJECT]
> > > [11297.598022] CPU 1
> > > [11297.598022] Pid: 14134, comm: ltp-pan Tainted: G      D
> > >      3.8.4+ #1 Bochs Bochs
> > > [11297.598022] RIP: 0010:[]  [] kmem_cache_alloc+0x68/0x1e0
> > > [11297.598022] RSP: 0018:ffff8800447dbdd0  EFLAGS: 00010246
> > > [11297.598022] RAX: 0000000000000000 RBX: ffff88007c169970 RCX:
> > > 00000000018acdcd
> > > [11297.598022] RDX: 000000000006c104 RSI: 00000000000080d0 RDI:
> > > ffff88007d04ac00
> > > [11297.598022] RBP: ffff8800447dbe10 R08: 0000000000017620 R09:
> > > ffffffff810fe2e2
> > > [11297.598022] R10: 0000000000000000 R11: 0000000000000000 R12:
> > > 00000000fffffffe
> > > [11297.598022] R13: 00000000000080d0 R14: ffff88007d04ac00 R15:
> > > ffff88007d04ac00
> > > [11297.598022] FS:  00007f09c29b4740(0000)
> > > GS:ffff88007fd00000(0000) knlGS:00000000f74d86c0
> > > [11297.598022] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > [11297.598022] CR2: 00000000fffffffe CR3: 0000000037213000 CR4:
> > > 00000000000006e0
> > > [11297.598022] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [11297.598022] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> > > 0000000000000400
> > > [11297.598022] Process ltp-pan (pid: 14134, threadinfo
> > > ffff8800447da000, task ffff8800551ab2e0)
> > > [11297.598022] Stack:
> > > [11297.598022]  ffffffff810fe2e2 ffffffff8108cf0f
> > > 0000000001200011
> > > ffff88007c169970
> > > [11297.598022]  0000000000000000 00007f09c29b4a10
> > > 0000000000000000
> > > ffff88007c169970
> > > [11297.598022]  ffff8800447dbe30 ffffffff810fe2e2
> > > 0000000000000000
> > > 0000000001200011
> > > [11297.598022] Call Trace:
> > > [11297.598022]  [] ? __delayacct_tsk_init+0x22/0x40
> > > [11297.598022]  [] ? prepare_creds+0xdf/0x190
> > > [11297.598022]  [] __delayacct_tsk_init+0x22/0x40
> > > [11297.598022]  [] copy_process.part.25+0x31f/0x13f0
> > > [11297.598022]  [] do_fork+0xa9/0x350
> > > [11297.598022]  [] sys_clone+0x16/0x20
> > > [11297.598022]  [] stub_clone+0x69/0x90
> > > [11297.598022]  [] ? system_call_fastpath+0x16/0x1b
> > > [11297.598022] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00
> > > 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 2b 01 00 00 49 63 46 20 4d
> > > 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 55 01 00 00 48 8d 4a 01 4c
> > > 89 e0 65 49 0f c7
> > > [11297.598022] RIP  [] kmem_cache_alloc+0x68/0x1e0
> > > [11297.598022]  RSP
> > > [11297.598022] CR2: 00000000fffffffe
> > > [11297.727799] ---[ end trace 037bde72f23b34d2 ]---
> > > 
> > > Never saw this in mainline but only something like this wondering
> > > could be related
> > > (that kmem_cache_alloc also in the trace).
> > > 
> > 
> > These are unrelated.
> > 
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org.  For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: href=mailto:"dont@kvack.org"> email@kvack.org
> > 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: BUG at kmem_cache_alloc
  2013-03-25  5:37     ` CAI Qian
  2013-03-25  7:42       ` CAI Qian
@ 2013-03-25 18:00       ` Christoph Lameter
  2013-03-26  9:32         ` CAI Qian
  1 sibling, 1 reply; 9+ messages in thread
From: Christoph Lameter @ 2013-03-25 18:00 UTC (permalink / raw)
  To: CAI Qian; +Cc: David Rientjes, linux-mm, linux-kernel, Oleg Nesterov


Please enable CONFIG_SLUB_DEBUG_ON or run the kernel with slub_debug on
the command line to get detailed diagnostics as to what causes this.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: BUG at kmem_cache_alloc
  2013-03-25 18:00       ` Christoph Lameter
@ 2013-03-26  9:32         ` CAI Qian
  2013-03-26 19:53           ` Dave Jones
  0 siblings, 1 reply; 9+ messages in thread
From: CAI Qian @ 2013-03-26  9:32 UTC (permalink / raw)
  To: Christoph Lameter
  Cc: David Rientjes, linux-mm, linux-kernel, Oleg Nesterov, Dave Jones



----- Original Message -----
> From: "Christoph Lameter" <cl@linux.com>
> To: "CAI Qian" <caiqian@redhat.com>
> Cc: "David Rientjes" <rientjes@google.com>, "linux-mm" <linux-mm@kvack.org>, linux-kernel@vger.kernel.org, "Oleg
> Nesterov" <oleg@redhat.com>
> Sent: Tuesday, March 26, 2013 2:00:16 AM
> Subject: Re: BUG at kmem_cache_alloc
> 
> 
> Please enable CONFIG_SLUB_DEBUG_ON or run the kernel with slub_debug
> on
> the command line to get detailed diagnostics as to what causes this.
> 
Still running and will update ASAP. One thing I noticed was that trinity
threw out this error before the kernel crash.

[19380] Random reseed: 644697889 
trinity(19380): Randomness reseeded to 0x266d4f21 
trinity: trinity(19380) Randomness reseeded to 0x266d4f21 
[19380] Random reseed: 1927643389 
trinity(19380): Randomness reseeded to 0x72e580fd 
trinity: trinity(19380) Randomness reseeded to 0x72e580fd 
[watchdog] 9381710 iterations. [F:8140812 S:1240290] 
[watchdog] 9383499 iterations. [F:8142333 S:1240558] 
 
Session terminated, killing shell...      
BUG!:       
CHILD (pid:28825) GOT REPARENTED! parent pid:19380. Watchdog pid:19379 
      
BUG!:       
Last syscalls: 
[0]  pid:28515 call:settimeofday callno:10356 
[1]  pid:28822 call:setgid callno:322 
[2]  pid:28581 call:init_module callno:3622 
[3]  pid:28825 call:readlinkat callno:403 
child 28581 exiting 
child 28515 exiting 
 ...killed. 

Then, some tests in LTP called epoll triggered it eventually.

[ 9788.955733] BUG: unable to handle kernel paging request at 00000000fffffff7 
[ 9788.956687] IP: [<ffffffff811876a8>] kmem_cache_alloc+0x68/0x1e0 
[ 9788.956687] PGD bebd3067 PUD 0  
[ 9788.956687] Oops: 0000 [#1] SMP  
[ 9788.956687] Modules linked in: l2tp_ppp l2tp_netlink l2tp_core tun cmtp kernelcapi bnep fuse rfcomm hidp ipt_ULOG rds af_key pppoe pppox ppp_generic slhc af_802154 nfc atm ip6table_filter ip6_tables iptable_filter ip_tables btrfs zlib_deflate vfat fat nfs_layout_nfsv41_files nfsv4 auth_rpcgss nfsv3 nfs_acl nfsv2 nfs lockd sunrpc fscache nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp sctp sg i5000_edac coretemp edac_core kvm_intel iTCO_wdt iTCO_vendor_support kvm lpc_ich ipmi_si ipmi_msghandler i5k_amb mfd_core hpilo hpwdt shpchp serio_raw microcode pcspkr xfs sd_mod crc_t10dif sr_mod cdrom ata_generic hpsa pata_acpi radeon i2c_algo_bit drm_kms_helper ttm drm ata_piix libata i2c_core bnx2 bnx2x cciss 3w_9xxx libcrc32c dm_mirror dm_region_hash dm_log dm_mod iscsi_tcp be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libiscsi_tcp qla4xxx libiscsi scsi_transport_iscsi iscsi_ibft iscsi_boot_sysfs [last unloaded: ipt_REJECT] 
[ 9788.956687] CPU 0  
[ 9788.956687] Pid: 25412, comm: epoll-ltp Tainted: G        W I  3.8.4+ #1 HP ProLiant DL380 G5 
[ 9788.956687] RIP: 0010:[<ffffffff811876a8>]  [<ffffffff811876a8>] kmem_cache_alloc+0x68/0x1e0 
[ 9788.956687] RSP: 0018:ffff8800bbee9dd0  EFLAGS: 00010246 
[ 9788.956687] RAX: 0000000000000000 RBX: ffff8801a356e5c0 RCX: 0000000000000000 
[ 9788.956687] RDX: 0000000000036b0a RSI: 00000000000080d0 RDI: ffff8801a906ad00 
[ 9788.956687] RBP: ffff8800bbee9e10 R08: 00000000000176b0 R09: ffffffff810fe2e2 
[ 9788.956687] R10: 0000000000000016 R11: ffffffffffffffdc R12: 00000000fffffff7 
[ 9788.956687] R13: 00000000000080d0 R14: ffff8801a906ad00 R15: ffff8801a906ad00 
[ 9788.956687] FS:  00007fbf42058740(0000) GS:ffff8801afc00000(0000) knlGS:0000000000000000 
[ 9788.956687] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b 
[ 9788.956687] CR2: 00000000fffffff7 CR3: 00000000aece6000 CR4: 00000000000407f0 
[ 9788.956687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
[ 9788.956687] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
[ 9788.956687] Process epoll-ltp (pid: 25412, threadinfo ffff8800bbee8000, task ffff8801a6524c50) 
[ 9788.956687] Stack: 
[ 9788.956687]  ffffffff810fe2e2 ffffffff8108cf0f 0000000001200011 ffff8801a356e5c0 
[ 9788.956687]  0000000000000000 00007fbf42058a10 0000000000000000 ffff8801a356e5c0 
[ 9788.956687]  ffff8800bbee9e30 ffffffff810fe2e2 0000000000000000 0000000001200011 
[ 9788.956687] Call Trace: 
[ 9788.956687]  [<ffffffff810fe2e2>] ? __delayacct_tsk_init+0x22/0x40 
[ 9788.956687]  [<ffffffff8108cf0f>] ? prepare_creds+0xdf/0x190 
[ 9788.956687]  [<ffffffff810fe2e2>] __delayacct_tsk_init+0x22/0x40 
[ 9788.956687]  [<ffffffff8106027f>] copy_process.part.25+0x31f/0x13f0 
[ 9788.956687]  [<ffffffff8106765b>] ? do_wait+0x12b/0x250 
[ 9788.956687]  [<ffffffff81097f3e>] ? wake_up_new_task+0xfe/0x160 
[ 9788.956687]  [<ffffffff81061449>] do_fork+0xa9/0x350 
[ 9788.956687]  [<ffffffff81068810>] ? sys_wait4+0x80/0xf0 
[ 9788.956687]  [<ffffffff81061776>] sys_clone+0x16/0x20 
[ 9788.956687]  [<ffffffff8161a7f9>] stub_clone+0x69/0x90 
[ 9788.956687]  [<ffffffff8161a499>] ? system_call_fastpath+0x16/0x1b 
[ 9788.956687] Code: 90 4d 89 fe 4d 8b 06 65 4c 03 04 25 c8 db 00 00 49 8b 50 08 4d 8b 20 4d 85 e4 0f 84 2b 01 00 00 49 63 46 20 4d 8b 06 41 f6 c0 0f <49> 8b 1c 04 0f 85 55 01 00 00 48 8d 4a 01 4c 89 e0 65 49 0f c7  
[ 9788.956687] RIP  [<ffffffff811876a8>] kmem_cache_alloc+0x68/0x1e0 
[ 9788.956687]  RSP <ffff8800bbee9dd0> 
[ 9788.956687] CR2: 00000000fffffff7 
[ 9789.029177] ---[ end trace 001669df502cd1ce ]---

CAI Qian

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: BUG at kmem_cache_alloc
  2013-03-26  9:32         ` CAI Qian
@ 2013-03-26 19:53           ` Dave Jones
  2013-03-28  7:49             ` CAI Qian
  0 siblings, 1 reply; 9+ messages in thread
From: Dave Jones @ 2013-03-26 19:53 UTC (permalink / raw)
  To: CAI Qian
  Cc: Christoph Lameter, David Rientjes, linux-mm, linux-kernel,
	Oleg Nesterov

On Tue, Mar 26, 2013 at 05:32:27AM -0400, CAI Qian wrote:

 > Still running and will update ASAP. One thing I noticed was that trinity
 > threw out this error before the kernel crash.
 > 
 > BUG!:       
 > CHILD (pid:28825) GOT REPARENTED! parent pid:19380. Watchdog pid:19379 
 >       
 > BUG!:       
 > Last syscalls: 
 > [0]  pid:28515 call:settimeofday callno:10356 
 > [1]  pid:28822 call:setgid callno:322 
 > [2]  pid:28581 call:init_module callno:3622 
 > [3]  pid:28825 call:readlinkat callno:403 
 > child 28581 exiting 
 > child 28515 exiting 
 >  ...killed. 

When this happens, it usually means that the parent segfaulted.
I've been trying to reproduce a few reports of this for a while
without success.  If you get time, running trinity inside gdb should
be enough to get a useful backtrace.

(Or run with -D, and collect coredumps [there will a lot], and match the
 core to the pid of the process we're interested in)

	Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: BUG at kmem_cache_alloc
  2013-03-26 19:53           ` Dave Jones
@ 2013-03-28  7:49             ` CAI Qian
  2013-03-28 14:27               ` Dave Jones
  0 siblings, 1 reply; 9+ messages in thread
From: CAI Qian @ 2013-03-28  7:49 UTC (permalink / raw)
  To: Dave Jones
  Cc: Christoph Lameter, David Rientjes, linux-mm, linux-kernel,
	Oleg Nesterov



----- Original Message -----
> From: "Dave Jones" <davej@redhat.com>
> To: "CAI Qian" <caiqian@redhat.com>
> Cc: "Christoph Lameter" <cl@linux.com>, "David Rientjes" <rientjes@google.com>, "linux-mm" <linux-mm@kvack.org>,
> linux-kernel@vger.kernel.org, "Oleg Nesterov" <oleg@redhat.com>
> Sent: Wednesday, March 27, 2013 3:53:44 AM
> Subject: Re: BUG at kmem_cache_alloc
> 
> On Tue, Mar 26, 2013 at 05:32:27AM -0400, CAI Qian wrote:
> 
>  > Still running and will update ASAP. One thing I noticed was that
>  > trinity
>  > threw out this error before the kernel crash.
>  > 
>  > BUG!:
>  > CHILD (pid:28825) GOT REPARENTED! parent pid:19380. Watchdog
>  > pid:19379
>  >       
>  > BUG!:
>  > Last syscalls:
>  > [0]  pid:28515 call:settimeofday callno:10356
>  > [1]  pid:28822 call:setgid callno:322
>  > [2]  pid:28581 call:init_module callno:3622
>  > [3]  pid:28825 call:readlinkat callno:403
>  > child 28581 exiting
>  > child 28515 exiting
>  >  ...killed.
> 
> When this happens, it usually means that the parent segfaulted.
> I've been trying to reproduce a few reports of this for a while
> without success.  If you get time, running trinity inside gdb should
> be enough to get a useful backtrace.
> 
> (Or run with -D, and collect coredumps [there will a lot], and match
> the
>  core to the pid of the process we're interested in)
> 
> 	Dave
> 
While reproducing this, it triggered something else with SLUB_DEBUG_ON.
CAI Qian

[87295.499233] general protection fault: 0000 [#1] SMP 
[87295.500228] Modules linked in: binfmt_misc fuse tun cmtp kernelcapi rfcomm bnep hidp scsi_transport_iscsi nfnetlink ipt_ULOG nfc bluetooth rfkill af_key atm lockd sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg kvm_amd kvm microcode amd64_edac_mod edac_mce_amd pcspkr serio_raw edac_core k10temp bnx2x netxen_nic mdio i2c_piix4 i2c_core hpilo shpchp ipmi_si ipmi_msghandler hpwdt xfs libcrc32c sd_mod crc_t10dif sata_svw libata dm_mirror dm_region_hash dm_log dm_mod
[87295.515752] CPU 1 
[87295.516184] Pid: 23211, comm: trinity-main Tainted: G        W    3.8.4 #4 HP ProLiant BL495c G5  
[87295.517810] RIP: 0010:[<ffffffff812e0b43>]  [<ffffffff812e0b43>] rb_next+0x23/0x50
[87295.519254] RSP: 0018:ffff880127f5de58  EFLAGS: 00010202
[87295.520398] RAX: 6b6b6b6b6b6b6b6b RBX: 0000000000000000 RCX: ffff88014181d9c8
[87295.521996] RDX: 6b6b6b6b6b6b6b6b RSI: ffff88014181a6e0 RDI: ffff88014181d9e0
[87295.523606] RBP: ffff880127f5de58 R08: 0000000000003d7b R09: 0000000000000008
[87295.525201] R10: ffffffff81197360 R11: 0000000000000246 R12: ffff8801314f3180
[87295.526793] R13: 0000000000000000 R14: 000000000000000f R15: ffff88014181d9c8
[87295.528465] FS:  00007f94bbc0f740(0000) GS:ffff88014fc80000(0000) knlGS:0000000000000000
[87295.530271] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[87295.531578] CR2: 0000000001f53008 CR3: 00000001129f5000 CR4: 00000000000007e0
[87295.533210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[87295.534797] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[87295.536402] Process trinity-main (pid: 23211, threadinfo ffff880127f5c000, task ffff8801418e98a0)
[87295.538368] Stack:
[87295.538793]  ffff880127f5ded8 ffffffff811f8220 0000000000000008 0000000000003d7b
[87295.540579]  ffff880127f50001 ffff8801314f3190 0000000000020000 ffffffff81197360
[87295.542313]  ffff880127f5df40 ffff88014181a6e0 ffff880127f5ded8 ffff8801314f3180
[87295.543959] Call Trace:
[87295.544513]  [<ffffffff811f8220>] sysfs_readdir+0x150/0x280
[87295.545774]  [<ffffffff81197360>] ? fillonedir+0x100/0x100
[87295.547004]  [<ffffffff81197360>] ? fillonedir+0x100/0x100
[87295.548268]  [<ffffffff81197238>] vfs_readdir+0xb8/0xe0
[87295.549446]  [<ffffffff811a159b>] ? set_close_on_exec+0x3b/0x70
[87295.550832]  [<ffffffff8119758f>] sys_getdents+0x8f/0x110
[87295.552068]  [<ffffffff815e6419>] system_call_fastpath+0x16/0x1b
[87295.553433] Code: 48 89 70 10 eb a9 66 90 55 48 8b 17 48 89 e5 48 39 d7 74 3b 48 8b 47 08 48 85 c0 75 0e eb 1f 66 0f 1f 84 00 00 00 00 00 48 89 d0 <48> 8b 50 10 48 85 d2 75 f4 5d c3 66 90 48 8b 10 48 89 c7 48 89 
[87295.557829] RIP  [<ffffffff812e0b43>] rb_next+0x23/0x50
[87295.558960]  RSP <ffff880127f5de58>
[87295.560213] ---[ end trace d5f25cc963b1f1d9 ]---
[watchdog] Triggering periodic reseed.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: BUG at kmem_cache_alloc
  2013-03-28  7:49             ` CAI Qian
@ 2013-03-28 14:27               ` Dave Jones
  0 siblings, 0 replies; 9+ messages in thread
From: Dave Jones @ 2013-03-28 14:27 UTC (permalink / raw)
  To: CAI Qian
  Cc: Christoph Lameter, David Rientjes, linux-mm, linux-kernel,
	Oleg Nesterov

On Thu, Mar 28, 2013 at 03:49:37AM -0400, CAI Qian wrote:
 
 > While reproducing this, it triggered something else with SLUB_DEBUG_ON.
 > CAI Qian
 > 
 > [87295.499233] general protection fault: 0000 [#1] SMP 
 > [87295.500228] Modules linked in: binfmt_misc fuse tun cmtp kernelcapi rfcomm bnep hidp scsi_transport_iscsi nfnetlink ipt_ULOG nfc bluetooth rfkill af_key atm lockd sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg kvm_amd kvm microcode amd64_edac_mod edac_mce_amd pcspkr serio_raw edac_core k10temp bnx2x netxen_nic mdio i2c_piix4 i2c_core hpilo shpchp ipmi_si ipmi_msghandler hpwdt xfs libcrc32c sd_mod crc_t10dif sata_svw libata dm_mirror dm_region_hash dm_log dm_mod
 > [87295.515752] CPU 1 
 > [87295.516184] Pid: 23211, comm: trinity-main Tainted: G        W    3.8.4 #4 HP ProLiant BL495c G5  
 > [87295.517810] RIP: 0010:[<ffffffff812e0b43>]  [<ffffffff812e0b43>] rb_next+0x23/0x50
 > [87295.519254] RSP: 0018:ffff880127f5de58  EFLAGS: 00010202
 > [87295.520398] RAX: 6b6b6b6b6b6b6b6b RBX: 0000000000000000 RCX: ffff88014181d9c8
 > [87295.521996] RDX: 6b6b6b6b6b6b6b6b RSI: ffff88014181a6e0 RDI: ffff88014181d9e0
 > [87295.523606] RBP: ffff880127f5de58 R08: 0000000000003d7b R09: 0000000000000008
 > [87295.525201] R10: ffffffff81197360 R11: 0000000000000246 R12: ffff8801314f3180
 > [87295.526793] R13: 0000000000000000 R14: 000000000000000f R15: ffff88014181d9c8
 > [87295.528465] FS:  00007f94bbc0f740(0000) GS:ffff88014fc80000(0000) knlGS:0000000000000000
 > [87295.530271] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 > [87295.531578] CR2: 0000000001f53008 CR3: 00000001129f5000 CR4: 00000000000007e0
 > [87295.533210] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 > [87295.534797] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 > [87295.536402] Process trinity-main (pid: 23211, threadinfo ffff880127f5c000, task ffff8801418e98a0)
 > [87295.538368] Stack:
 > [87295.538793]  ffff880127f5ded8 ffffffff811f8220 0000000000000008 0000000000003d7b
 > [87295.540579]  ffff880127f50001 ffff8801314f3190 0000000000020000 ffffffff81197360
 > [87295.542313]  ffff880127f5df40 ffff88014181a6e0 ffff880127f5ded8 ffff8801314f3180
 > [87295.543959] Call Trace:
 > [87295.544513]  [<ffffffff811f8220>] sysfs_readdir+0x150/0x280
 > [87295.545774]  [<ffffffff81197360>] ? fillonedir+0x100/0x100
 > [87295.547004]  [<ffffffff81197360>] ? fillonedir+0x100/0x100
 > [87295.548268]  [<ffffffff81197238>] vfs_readdir+0xb8/0xe0
 > [87295.549446]  [<ffffffff811a159b>] ? set_close_on_exec+0x3b/0x70
 > [87295.550832]  [<ffffffff8119758f>] sys_getdents+0x8f/0x110
 > [87295.552068]  [<ffffffff815e6419>] system_call_fastpath+0x16/0x1b
 > [87295.553433] Code: 48 89 70 10 eb a9 66 90 55 48 8b 17 48 89 e5 48 39 d7 74 3b 48 8b 47 08 48 85 c0 75 0e eb 1f 66 0f 1f 84 00 00 00 00 00 48 89 d0 <48> 8b 50 10 48 85 d2 75 f4 5d c3 66 90 48 8b 10 48 89 c7 48 89 
 > [87295.557829] RIP  [<ffffffff812e0b43>] rb_next+0x23/0x50
 > [87295.558960]  RSP <ffff880127f5de58>
 > [87295.560213] ---[ end trace d5f25cc963b1f1d9 ]---
 > [watchdog] Triggering periodic reseed.

That's fixed by the patch below from Ming Lei.


diff --git a/fs/sysfs/dir.c b/fs/sysfs/dir.c
index 2fbdff6..014ed97 100644
--- a/fs/sysfs/dir.c
+++ b/fs/sysfs/dir.c
@@ -280,6 +280,11 @@ void release_sysfs_dirent(struct sysfs_dirent * sd)
 	 * sd->s_parent won't change beneath us.
 	 */
 	parent_sd = sd->s_parent;
+	if(!(sd->s_flags & SYSFS_FLAG_REMOVED)) {
+		printk("%s-%d sysfs_dirent use after free: %s-%s\n",
+			__func__, __LINE__, parent_sd->s_name, sd->s_name);
+		dump_stack();
+	}
 
 	if (sysfs_type(sd) == SYSFS_KOBJ_LINK)
 		sysfs_put(sd->s_symlink.target_sd);
@@ -962,6 +967,12 @@ static struct sysfs_dirent *sysfs_dir_pos(const void *ns,
 		int valid = !(pos->s_flags & SYSFS_FLAG_REMOVED) &&
 			pos->s_parent == parent_sd &&
 			hash == pos->s_hash;
+
+		if ((atomic_read(&pos->s_count) == 1)) {
+			printk("%s-%d sysfs_dirent use after free: %s(%s)-%s, %lld-%u\n",
+				__func__, __LINE__, parent_sd->s_name, pos->s_parent->s_name,
+				pos->s_name, hash, pos->s_hash);
+		}
 		sysfs_put(pos);
 		if (!valid)
 			pos = NULL;
@@ -1020,6 +1031,8 @@ static int sysfs_readdir(struct file * filp, void * dirent, filldir_t filldir)
 		ino = parent_sd->s_ino;
 		if (filldir(dirent, ".", 1, filp->f_pos, ino, DT_DIR) == 0)
 			filp->f_pos++;
+		else
+			return 0;
 	}
 	if (filp->f_pos == 1) {
 		if (parent_sd->s_parent)
@@ -1028,6 +1041,8 @@ static int sysfs_readdir(struct file * filp, void * dirent, filldir_t filldir)
 			ino = parent_sd->s_ino;
 		if (filldir(dirent, "..", 2, filp->f_pos, ino, DT_DIR) == 0)
 			filp->f_pos++;
+		else
+			return 0;
 	}
 	mutex_lock(&sysfs_mutex);
 	for (pos = sysfs_dir_pos(ns, parent_sd, filp->f_pos, pos);
@@ -1058,10 +1073,21 @@ static int sysfs_readdir(struct file * filp, void * dirent, filldir_t filldir)
 	return 0;
 }
 
+static loff_t sysfs_dir_llseek(struct file *file, loff_t offset, int whence)
+{
+	struct inode *inode = file_inode(file);
+	loff_t ret;
+
+	mutex_lock(&inode->i_mutex);
+	ret = generic_file_llseek(file, offset, whence);
+	mutex_unlock(&inode->i_mutex);
+
+	return ret;
+}
 
 const struct file_operations sysfs_dir_operations = {
 	.read		= generic_read_dir,
 	.readdir	= sysfs_readdir,
 	.release	= sysfs_dir_release,
-	.llseek		= generic_file_llseek,
+	.llseek		= sysfs_dir_llseek,
 };

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

^ permalink raw reply related	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2013-03-28 14:27 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <1089649229.4894208.1363925156257.JavaMail.root@redhat.com>
2013-03-22  4:18 ` BUG at kmem_cache_alloc CAI Qian
2013-03-22  9:35   ` David Rientjes
2013-03-25  5:37     ` CAI Qian
2013-03-25  7:42       ` CAI Qian
2013-03-25 18:00       ` Christoph Lameter
2013-03-26  9:32         ` CAI Qian
2013-03-26 19:53           ` Dave Jones
2013-03-28  7:49             ` CAI Qian
2013-03-28 14:27               ` Dave Jones

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).