* Use-after-free in hacked 2.6.38.8 kernel.
@ 2011-06-23 22:29 Ben Greear
2011-06-24 15:29 ` Ben Greear
0 siblings, 1 reply; 2+ messages in thread
From: Ben Greear @ 2011-06-23 22:29 UTC (permalink / raw)
To: linux-nfs
2.6.38.8 kernel, with our NFS bind-source-IP patches and some other
stuff, including a tainting module (though that module isn't
active in this test).
I'm also running the patch I posted a few days ago that explicitly
un-links the xpt_ready list:
diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index ab86b79..178716f 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -901,6 +901,7 @@ void svc_delete_xprt(struct svc_xprt *xprt)
spin_lock_bh(&serv->sv_lock);
if (!test_and_set_bit(XPT_DETACHED, &xprt->xpt_flags))
list_del_init(&xprt->xpt_list);
+ list_del_init(&xprt->xpt_ready);
/*
* We used to delete the transport from whichever list
* it's sk_xprt.xpt_ready node was on, but we don't actually
Test is to create 200 unique mounts (using unique srcaddr)
and mount/run-file-io-traffic/unmount them every 15 seconds.
It hit this bug after about 5 hours.
I'm going to try to figure this out, but any help is appreciated!
=============================================================================
BUG kmalloc-64: Poison overwritten
-----------------------------------------------------------------------------
INFO: 0xffff8800c6da9dd0-0xffff8800c6da9e03. First byte 0x48 instead of 0x6b
INFO: Allocated in nfs_get_lock_context+0xa4/0x179 [nfs] age=60 cpu=2 pid=9218
INFO: Freed in nfs_put_lock_context+0x3f/0x44 [nfs] age=70 cpu=0 pid=8543
INFO: Slab 0xffffea0002b7fcf8 objects=30 used=26 fp=0xffff8800c6da9dd0
flags=0x200000000000c1
INFO: Object 0xffff8800c6da9dd0 @offset=3536 fp=0xffff8800c6da9d48
Bytes b4 0xffff8800c6da9dc0: fe b7 0f 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a
??......ZZZZZZZZ
Object 0xffff8800c6da9dd0: 48 90 b9 b3 00 88 ff ff 6b 6b 6b 6b 6b 6b 6b 6b
H.??..??kkkkkkkk
Object 0xffff8800c6da9de0: 06 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
......kkkkkkkkkk
Object 0xffff8800c6da9df0: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b
........kkkkkkkk
Object 0xffff8800c6da9e00: f3 ff ff ff 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5
????kkkkkkkkkkk?
Redzone 0xffff8800c6da9e10: bb bb bb bb bb bb bb bb
????????
Padding 0xffff8800c6da9e50: 5a 5a 5a 5a 5a 5a 5a 5a
ZZZZZZZZ
Pid: 9019, comm: btserver Tainted: P 2.6.38.8+ #9
Call Trace:
[<ffffffff81100aee>] ? print_trailer+0x12e/0x137
[<ffffffff81100fb7>] ? check_bytes_and_report+0xb9/0xfd
[<ffffffffa030ddbb>] ? nfs_get_lock_context+0x94/0x179 [nfs]
[<ffffffff811010b0>] ? check_object+0xb5/0x192
[<ffffffffa030ddcb>] ? nfs_get_lock_context+0xa4/0x179 [nfs]
[<ffffffff811014d1>] ? alloc_debug_processing+0x79/0xf2
[<ffffffff81102bff>] ? __slab_alloc+0x337/0x375
[<ffffffffa030ddcb>] ? nfs_get_lock_context+0xa4/0x179 [nfs]
[<ffffffffa030dd4f>] ? nfs_get_lock_context+0x28/0x179 [nfs]
[<ffffffffa030ddcb>] ? nfs_get_lock_context+0xa4/0x179 [nfs]
[<ffffffff81103d87>] ? kmem_cache_alloc_trace+0x76/0xef
[<ffffffff81465d62>] ? sub_preempt_count+0x92/0xa6
[<ffffffffa030ddcb>] ? nfs_get_lock_context+0xa4/0x179 [nfs]
[<ffffffffa0313c32>] ? nfs_file_direct_write+0x1ab/0x752 [nfs]
[<ffffffff81122b25>] ? pollwake+0x0/0x4f
[<ffffffff810423db>] ? get_parent_ip+0x11/0x41
[<ffffffff811026f5>] ? __slab_free+0x86/0xf1
[<ffffffff811429cf>] ? fsnotify_put_event+0x63/0x67
[<ffffffff81077d44>] ? trace_hardirqs_on+0xd/0xf
[<ffffffffa030bd9b>] ? nfs_file_write+0x5d/0x169 [nfs]
[<ffffffff811134c8>] ? do_sync_write+0xc6/0x103
[<ffffffff811df2b4>] ? security_file_permission+0x29/0x2e
[<ffffffff81113e58>] ? vfs_write+0xa9/0x105
[<ffffffff811145f5>] ? fget_light+0x35/0x94
[<ffffffff81113f6d>] ? sys_write+0x45/0x6c
[<ffffffff8100aa92>] ? system_call_fastpath+0x16/0x1b
FIX kmalloc-64: Restoring 0xffff8800c6da9dd0-0xffff8800c6da9e03=0x6b
FIX kmalloc-64: Marking all objects used
=============================================================================
BUG kmalloc-64: Redzone overwritten
-----------------------------------------------------------------------------
INFO: 0xffff8800c6da9e10-0xffff8800c6da9e17. First byte 0xbb instead of 0xcc
INFO: Allocated in nfs_get_lock_context+0xa4/0x179 [nfs] age=173 cpu=2 pid=9218
INFO: Freed in nfs_put_lock_context+0x3f/0x44 [nfs] age=172 cpu=0 pid=8543
INFO: Slab 0xffffea0002b7fcf8 objects=30 used=30 fp=0x (null)
flags=0x20000000000081
INFO: Object 0xffff8800c6da9dd0 @offset=3536 fp=0xffff8800c6da9d48
Bytes b4 0xffff8800c6da9dc0: fe b7 0f 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a
??......ZZZZZZZZ
Object 0xffff8800c6da9dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
Object 0xffff8800c6da9de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
Object 0xffff8800c6da9df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
Object 0xffff8800c6da9e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5
kkkkkkkkkkkkkkk?
Redzone 0xffff8800c6da9e10: bb bb bb bb bb bb bb bb
????????
Padding 0xffff8800c6da9e50: 5a 5a 5a 5a 5a 5a 5a 5a
ZZZZZZZZ
Pid: 13574, comm: mount.nfs Tainted: P 2.6.38.8+ #9
Call Trace:
[<ffffffff81100aee>] ? print_trailer+0x12e/0x137
[<ffffffff81100fb7>] ? check_bytes_and_report+0xb9/0xfd
[<ffffffffa028a9cc>] ? rpcb_create_local+0x6a/0x112 [sunrpc]
[<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
[<ffffffff81101044>] ? check_object+0x49/0x192
[<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
[<ffffffff81101efd>] ? free_debug_processing+0x7a/0x18e
[<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
[<ffffffff8110274b>] ? __slab_free+0xdc/0xf1
[<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
[<ffffffff811031ad>] ? kfree+0x12e/0x166
[<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
[<ffffffffa0281eae>] ? rpc_release_calldata+0x12/0x14 [sunrpc]
[<ffffffffa0282080>] ? rpc_free_task+0x59/0x61 [sunrpc]
[<ffffffffa028210a>] ? rpc_final_put_task+0x82/0x8a [sunrpc]
[<ffffffffa028213d>] ? rpc_do_put_task+0x2b/0x32 [sunrpc]
[<ffffffffa028215e>] ? rpc_put_task+0xb/0xd [sunrpc]
[<ffffffffa028a8dd>] ? rpcb_getport_async+0x564/0x5a5 [sunrpc]
[<ffffffff810423db>] ? get_parent_ip+0x11/0x41
[<ffffffffa027b349>] ? call_bind+0x70/0x75 [sunrpc]
[<ffffffffa0282911>] ? __rpc_execute+0x78/0x24b [sunrpc]
[<ffffffff8106750e>] ? wake_up_bit+0x20/0x25
[<ffffffffa0282b21>] ? rpc_execute+0x3d/0x42 [sunrpc]
[<ffffffffa027ca9f>] ? rpc_run_task+0xe3/0xef [sunrpc]
[<ffffffffa027cb89>] ? rpc_call_sync+0x3f/0x60 [sunrpc]
[<ffffffffa027cbec>] ? rpc_ping+0x42/0x58 [sunrpc]
[<ffffffff8146275b>] ? _raw_spin_unlock+0x45/0x52
[<ffffffffa027d4d5>] ? rpc_create+0x493/0x50e [sunrpc]
[<ffffffffa0307077>] ? nfs_get_client+0x50/0x536 [nfs]
[<ffffffffa030698e>] ? nfs_create_rpc_client+0xb1/0xf6 [nfs]
[<ffffffffa0307f92>] ? nfs_create_server+0x170/0x48e [nfs]
[<ffffffff81077d44>] ? trace_hardirqs_on+0xd/0xf
[<ffffffffa0312486>] ? nfs_get_sb+0x4e8/0x742 [nfs]
[<ffffffff81115eb7>] ? vfs_kern_mount+0xea/0x1f6
[<ffffffff81116021>] ? do_kern_mount+0x48/0xd8
[<ffffffff8112da55>] ? do_mount+0x708/0x770
[<ffffffff810f9723>] ? alloc_pages_current+0xaa/0xcd
[<ffffffff8112db40>] ? sys_mount+0x83/0xbd
[<ffffffff8100aa92>] ? system_call_fastpath+0x16/0x1b
FIX kmalloc-64: Restoring 0xffff8800c6da9e10-0xffff8800c6da9e17=0xcc
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: Use-after-free in hacked 2.6.38.8 kernel.
2011-06-23 22:29 Use-after-free in hacked 2.6.38.8 kernel Ben Greear
@ 2011-06-24 15:29 ` Ben Greear
0 siblings, 0 replies; 2+ messages in thread
From: Ben Greear @ 2011-06-24 15:29 UTC (permalink / raw)
To: linux-nfs
On 06/23/2011 03:29 PM, Ben Greear wrote:
> 2.6.38.8 kernel, with our NFS bind-source-IP patches and some other
> stuff, including a tainting module (though that module isn't
> active in this test).
And, another one. Different place this time though:
[root@simech2 ~]# general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
last sysfs file: /sys/devices/virtual/bdi/0:51/uevent
CPU 0
Modules linked in:
=============================================================================
BUG kmalloc-64: Poison overwritten
-----------------------------------------------------------------------------
INFO: 0xffff880097422de4-0xffff880097422de5. First byte 0x1 instead of 0x6b
INFO: Allocated in rpcb_getport_async+0x39c/0x5a5 [sunrpc] age=22 cpu=1 pid=23678
INFO: Freed in rpcb_map_release+0x3f/0x44 [sunrpc] age=20 cpu=0 pid=18587
INFO: Slab 0xffffea0002116770 objects=30 used=7 fp=0xffff880097422dd0 flags=0x200000000000c1
INFO: Object 0xffff880097422dd0 @offset=3536 fp=0xffff8800974224c8
Bytes b4 0xffff880097422dc0: ae db 4e 00 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ®ÛN.....ZZZZZZZZ
Object 0xffff880097422dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object 0xffff880097422de0: 6b 6b 6b 6b 01 08 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkk..kkkkkkkkkk
Object 0xffff880097422df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object 0xffff880097422e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
Redzone 0xffff880097422e10: bb bb bb bb bb bb bb bb »»»»»»»»
Padding 0xffff880097422e50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
Pid: 2259, comm: gnuserver Tainted: P 2.6.38.8+ #9
Call Trace:
[<ffffffff81100aee>] ? print_trailer+0x12e/0x137
[<ffffffff81100fb7>] ? check_bytes_and_report+0xb9/0xfd
[<ffffffff81129cbb>] ? alloc_fdtable+0xb2/0xda
[<ffffffff811010b0>] ? check_object+0xb5/0x192
[<ffffffff81129c6b>] ? alloc_fdtable+0x62/0xda
[<ffffffff811014d1>] ? alloc_debug_processing+0x79/0xf2
[<ffffffff81102bff>] ? __slab_alloc+0x337/0x375
[<ffffffff81129c6b>] ? alloc_fdtable+0x62/0xda
[<ffffffff81129d92>] ? dup_fd+0xaf/0x35b
[<ffffffff81129c6b>] ? alloc_fdtable+0x62/0xda
[<ffffffff81103d87>] ? kmem_cache_alloc_trace+0x76/0xef
[<ffffffff81129c6b>] ? alloc_fdtable+0x62/0xda
[<ffffffff81129e61>] ? dup_fd+0x17e/0x35b
[<ffffffff8104957b>] ? copy_process+0x714/0x12a5
[<ffffffff8105a25f>] ? sigprocmask+0x2f/0xc6
[<ffffffff8104a217>] ? do_fork+0x10b/0x2ed
[<ffffffff810e143d>] ? might_fault+0x63/0xb3
[<ffffffff8111c993>] ? path_put+0x1d/0x22
[<ffffffff81011fed>] ? sys_clone+0x23/0x25
[<ffffffff8100ae33>] ? stub_clone+0x13/0x20
[<ffffffff8100aa92>] ? system_call_fastpath+0x16/0x1b
FIX kmalloc-64: Restoring 0xffff880097422de4-0xffff880097422de5=0x6b
IX kmalloc-64: Marking all objects used
xt_TPROXY nf_tproxy_core xt_socket nf_defrag_ipv6 xt_connlimit 8021q garp macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse
ip6table_filter ip6_tables ebtable_nat ebtables stp llc nfs lockd fscache nfs_acl auth_rpcgss sunrpc ipv6 kvm_intel kvm uinput i5k_amb i5000_edac iTCO_wdt
ioatdma i2c_i801 shpchp iTCO_vendor_support edac_core pcspkr serio_raw e1000e microcode dca floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core
[last unloaded: ipt_addrtype]
Pid: 18587, comm: kworker/0:1 Tainted: P 2.6.38.8+ #9 Supermicro X7DBU/X7DBU
RIP: 0010:[<ffffffffa0294f5b>] [<ffffffffa0294f5b>] rpcb_getport_done+0x47/0xab [sunrpc]
RSP: 0018:ffff8800c7133d20 EFLAGS: 00010246
RAX: ffffffffa0294f14 RBX: 0000000000000000 RCX: 0000000000000088
RDX: ffff880097422dd0 RSI: 000000006b6b0801 RDI: ffff880124668c80
RBP: ffff8800c7133d40 R08: ffff880097422dd0 R09: 0000000000000000
R10: ffff8800c7133d20 R11: ffff8800c7133c40 R12: ffff880097422dd0
R13: 6b6b6b6b6b6b6b6b R14: ffff880124668c80 R15: ffffffffa028db26
FS: 0000000000000000(0000) GS:ffff8800cfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000002666940 CR3: 00000000a73b6000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/0:1 (pid: 18587, threadinfo ffff8800c7132000, task ffff8800c4fea0a0)
Stack:
ffff880124668c80 ffff880124668cf0 0000000000000001 0000000000000000
ffff8800c7133d60 ffffffffa028d520 0000000000000000 ffff880124668c80
ffff8800c7133db0 ffffffffa028d911 ffff8800ca597080 0000000000000000
Call Trace:
[<ffffffffa028d520>] rpc_exit_task+0x27/0x55 [sunrpc]
[<ffffffffa028d911>] __rpc_execute+0x78/0x24b [sunrpc]
[<ffffffffa028db26>] ? rpc_async_schedule+0x0/0x12 [sunrpc]
[<ffffffffa028db36>] rpc_async_schedule+0x10/0x12 [sunrpc]
[<ffffffff81061613>] process_one_work+0x259/0x41b
[<ffffffff8106153b>] ? process_one_work+0x181/0x41b
[<ffffffff81063a51>] worker_thread+0x133/0x217
[<ffffffff8106391e>] ? worker_thread+0x0/0x217
[<ffffffff81066fc0>] kthread+0x7d/0x85
[<ffffffff8100b924>] kernel_thread_helper+0x4/0x10
[<ffffffff81463098>] ? restore_args+0x0/0x30
[<ffffffff81066f43>] ? kthread+0x0/0x85
[<ffffffff8100b920>] ? kernel_thread_helper+0x0/0x10
Code: fb 74 05 83 fb a3 75 0e 41 ff 85 f4 05 00 00 bb a3 ff ff ff eb 04 85 db 79 0e 49 8b 45 08 31 f6 4c 89 ef ff 50 20 eb 32 8b 76 14 <49> 8b 45 08 66 85 f6 75
0f 31 f6 4c 89 ef bb f3 ff ff ff ff 50
RIP [<ffffffffa0294f5b>] rpcb_getport_done+0x47/0xab [sunrpc]
RSP <ffff8800c7133d20>
---[ end trace 56d19572836bccfa ]---
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2011-06-24 15:29 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-06-23 22:29 Use-after-free in hacked 2.6.38.8 kernel Ben Greear
2011-06-24 15:29 ` Ben Greear
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.