* Potential out-of-bounds access in ip6_finish_output2
@ 2013-09-17 5:13 Dmitry Vyukov
2013-09-17 22:48 ` Hannes Frederic Sowa
0 siblings, 1 reply; 6+ messages in thread
From: Dmitry Vyukov @ 2013-09-17 5:13 UTC (permalink / raw)
To: yoshfuji, hannes, netdev, Paul Turner, Andrey Konovalov,
Kostya Serebryany, Tom Herbert
Hi,
I am working on AddressSanitizer -- a tool that detects use-after-free
and out-of-bounds bugs
(https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel).
I've got a dozen of reports in ip6_finish_output2. Below are 2 of
them. They are always followed by kernel crash. Unfortunately I don't
have a reproducer because I am using trinity fuzzer. I would
appreciate if somebody familiar with the code look at sources and
maybe spot the bug.
The reports are obtained on revision 6a7492a4b2e05051a44458d7187023e22d580666.
[ 977.765485] ERROR: AddressSanitizer: heap-buffer-overflow on
address ffff8800521e8730
[ 977.767205] ffff8800521e8730 is located 16 bytes to the left of
512-byte region [ffff8800521e8740, ffff8800521e8940)
[ 977.769399] Accessed by thread T11464:
[ 977.770274] #0 ffffffff810dd2a6 (asan_report_error+0x306/0x410)
[ 977.771570] #1 ffffffff810dc6a0 (asan_check_region+0x30/0x40)
[ 977.772883] #2 ffffffff810dc9ff (asan_memcpy+0x1f/0x60)
[ 977.774033] #3 ffffffffa0003b1c (ip6_finish_output2+0x54c/0x840 [ipv6])
[ 977.775451] #4 ffffffffa00088dc (ip6_fragment+0xe2c/0x1520 [ipv6])
[ 977.776710] #5 ffffffffa00090f7 (ip6_finish_output+0x127/0x190 [ipv6])
[ 977.777649] #6 ffffffffa00091e1 (ip6_output+0x81/0x140 [ipv6])
[ 977.778503] #7 ffffffffa000630c (ip6_local_out+0x4c/0x60 [ipv6])
[ 977.779379] #8 ffffffffa0006afd
(ip6_push_pending_frames+0x7dd/0xac0 [ipv6])
[ 977.780391] #9 ffffffffa00319de (rawv6_sendmsg+0x12ae/0x15c0 [ipv6])
[ 977.781295] #10 ffffffff818bb498 (inet_sendmsg+0x108/0x160)
[ 977.782094] #11 ffffffff817d0016 (sock_aio_write+0x296/0x2e0)
[ 977.782885] #12 ffffffff8129dcb1 (do_sync_write+0x111/0x170)
[ 977.783699] #13 ffffffff8129e9fd (vfs_write+0x2dd/0x300)
[ 977.784468] #14 ffffffff8129f9a0 (SyS_write+0x80/0xe0)
[ 977.785214] #15 ffffffff81928582 (system_call_fastpath+0x16/0x1b)
[ 977.786066]
[ 977.786284] Allocated by thread T11464:
[ 977.786858] #0 ffffffff810dc768 (asan_slab_alloc+0x48/0xc0)
[ 977.787661] #1 ffffffff81283d89 (kmem_cache_alloc_node_trace+0x99/0x4f0)
[ 977.788860] #2 ffffffff81284211 (__kmalloc_node_track_caller+0x31/0x40)
[ 977.790359] #3 ffffffff817ded6a (__kmalloc_reserve.isra.27+0x4a/0xb0)
[ 977.791800] #4 ffffffff817e0201 (__alloc_skb+0x91/0x280)
[ 977.792985] #5 ffffffff817d807a (sock_wmalloc+0x6a/0xe0)
[ 977.794183] #6 ffffffffa0005ea6 (ip6_append_data+0x1906/0x1c20 [ipv6])
[ 977.795597] #7 ffffffffa0030dd7 (rawv6_sendmsg+0x6a7/0x15c0 [ipv6])
[ 977.796831] #8 ffffffff818bb498 (inet_sendmsg+0x108/0x160)
[ 977.798035] #9 ffffffff817d0016 (sock_aio_write+0x296/0x2e0)
[ 977.799260] #10 ffffffff8129dcb1 (do_sync_write+0x111/0x170)
[ 977.800495] #11 ffffffff8129e9fd (vfs_write+0x2dd/0x300)
[ 977.801709] #12 ffffffff8129f9a0 (SyS_write+0x80/0xe0)
[ 977.802882] #13 ffffffff81928582 (system_call_fastpath+0x16/0x1b)
[ 977.804209]
[ 977.804529] Shadow bytes around the buggy address:
[ 977.805588] ffff8800521e8480: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 977.807192] ffff8800521e8500: 00 00 00 00 00 00 00 fb fb fb fb fb
fb fb fb fb
[ 977.808655] ffff8800521e8580: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 977.810122] ffff8800521e8600: fa fa fa fa fa fa fa fa fa fa fa fa
fa fa fa fa
[ 977.811776] ffff8800521e8680: fa fa fa fa fa fa fa fa fa fa fa fa
fa fa fa fa
[ 977.813128] =>ffff8800521e8700: fa fa fa fa fa fa[fa]fa 00 00 00 00
00 00 00 00
[ 977.814463] ffff8800521e8780: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 977.815625] ffff8800521e8800: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 977.816685] ffff8800521e8880: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 977.817814] ffff8800521e8900: 00 00 00 00 00 00 00 00 fa fa fa fa
fa fa fa fa
[ 977.818907] ffff8800521e8980: fa fa fa fa fa fa fa fa fa fa fa fa
fa fa fa fa
[ 977.819917] Shadow byte legend (one shadow byte represents 8
application bytes):
[ 977.820929] Addressable: 00
[ 977.821479] Partially addressable: 01 02 03 04 05 06 07
[ 977.822251] Heap redzone: fa
[ 977.822841] Heap kmalloc redzone: fb
[ 977.823414] Freed heap region: fd
[ 977.823955] Shadow gap: fe
[ 977.824512] =========================================================================
[ 977.825607] skbuff: skb_under_panic: text:ffffffffa0003b35 len:125
put:14 head:ffff8800521e8740 data:ffff8800521e8732 tail:0x6f end:0xc0
dev:lo
[ 977.827336] ------------[ cut here ]------------
[ 977.828000] kernel BUG at net/core/skbuff.c:126!
[ 977.828270] invalid opcode: 0000 [#1] SMP
[ 977.828270] Modules linked in: snd_seq_dummy snd_seq_oss
snd_seq_midi_event snd_seq snd_seq_device tun 8021q snd_pcm_oss
snd_pcm snd_page_alloc snd_timer snd_mixer_oss snd sr_mod cdrom loop
bridge stp llc st ipt_ULOG nfnetlink iptable_mangle tg3 ptp pps_core
i2c_piix4 i2c_core msr cpuid e1000 ipv6
[ 977.828270] CPU: 1 PID: 11464 Comm: trinity-child28 Not tainted
3.11.0-smp-DEV #8
[ 977.828270] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 977.828270] task: ffff880053321280 ti: ffff880049194000 task.ti:
ffff880049194000
[ 977.828270] RIP: 0010:[<ffffffff81913878>] [<ffffffff81913878>]
skb_panic+0xd5/0xd7
[ 977.828270] RSP: 0018:ffff8800491957a0 EFLAGS: 00010286
[ 977.828270] RAX: 0000000000000083 RBX: ffff8800485be6c0 RCX: 0000000000000000
[ 977.828270] RDX: ffff880000000000 RSI: 0000000000000008 RDI: ffffffff81c44cd8
[ 977.828270] RBP: ffff880049195808 R08: 000000000000006f R09: 0000000000000000
[ 977.828270] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88005bf8b400
[ 977.828270] R13: ffff8800521e8732 R14: 000000000000006f R15: 00000000000000c0
[ 977.828270] FS: 0000000001642880(0063) GS:ffff88005fd00000(0000)
knlGS:0000000000000000
[ 977.828270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 977.828270] CR2: 0000000000000009 CR3: 0000000049eef000 CR4: 00000000000006e0
[ 977.828270] Stack:
[ 977.828270] ffff8800521e8732 000000000000006f 00000000000000c0
ffff88005bf8b400
[ 977.828270] 0000000e485be6c0 ffffffffa0003b35 ffffffff81aa3940
ffff8800521e8740
[ 977.828270] ffff8800485be6c0 ffff8800521e8732 000000000000000e
ffff8800485be720
[ 977.828270] Call Trace:
[ 977.828270] [<ffffffffa0003b35>] ? ip6_finish_output2+0x565/0x840 [ipv6]
[ 977.828270] [<ffffffff817ddb59>] skb_push+0xa9/0xb0
[ 977.828270] [<ffffffffa0003b35>] ip6_finish_output2+0x565/0x840 [ipv6]
[ 977.828270] [<ffffffffa00088dc>] ip6_fragment+0xe2c/0x1520 [ipv6]
[ 977.828270] [<ffffffffa00035d0>] ?
ip6_flush_pending_frames+0x1d0/0x1d0 [ipv6]
[ 977.828270] [<ffffffffa00090f7>] ip6_finish_output+0x127/0x190 [ipv6]
[ 977.828270] [<ffffffffa00091e1>] ip6_output+0x81/0x140 [ipv6]
[ 977.828270] [<ffffffffa000630c>] ip6_local_out+0x4c/0x60 [ipv6]
[ 977.828270] [<ffffffff810dc689>] ? asan_check_region+0x19/0x40
[ 977.828270] [<ffffffffa0006afd>] ip6_push_pending_frames+0x7dd/0xac0 [ipv6]
[ 977.828270] [<ffffffffa00319de>] rawv6_sendmsg+0x12ae/0x15c0 [ipv6]
[ 977.828270] [<ffffffff810dc689>] ? asan_check_region+0x19/0x40
[ 977.828270] [<ffffffff818bb498>] inet_sendmsg+0x108/0x160
[ 977.828270] [<ffffffff817d0016>] sock_aio_write+0x296/0x2e0
[ 977.828270] [<ffffffff8129dcb1>] do_sync_write+0x111/0x170
[ 977.828270] [<ffffffff8129e9fd>] vfs_write+0x2dd/0x300
[ 977.828270] [<ffffffff8129f9a0>] SyS_write+0x80/0xe0
[ 977.828270] [<ffffffff81928582>] system_call_fastpath+0x16/0x1b
[ 977.828270] Code: c7 f0 a2 ba 81 44 8b 45 bc 48 8b 55 c0 31 c0 48
8b 75 c8 4c 89 64 24 18 4c 89 7c 24 10 4c 89 74 24 08 4c 89 2c 24 e8
7d 73 ff ff <0f> 0b 55 48 89 e5 48 8b 7d 08 e8 39 9b 7c ff 0f 0b 55 48
89 e5
[ 977.828270] RIP [<ffffffff81913878>] skb_panic+0xd5/0xd7
[ 977.828270] RSP <ffff8800491957a0>
[ 977.871681] ---[ end trace 20970757dd5daf11 ]---
[ 521.772929] ERROR: AddressSanitizer: heap-buffer-overflow on
address ffff88004965fbe8
[ 521.774073] ffff88004965fbe8 is located 24 bytes to the left of
512-byte region [ffff88004965fc00, ffff88004965fe00)
[ 521.775741] Accessed by thread T2167:
[ 521.776475] #0 ffffffff810dd2a6 (asan_report_error+0x306/0x410)
[ 521.777728] #1 ffffffff810dc6a0 (asan_check_region+0x30/0x40)
[ 521.778966] #2 ffffffff810dc9ff (asan_memcpy+0x1f/0x60)
[ 521.780145] #3 ffffffffa0003b1c (ip6_finish_output2+0x54c/0x840 [ipv6])
[ 521.781570] #4 ffffffffa00088dc (ip6_fragment+0xe2c/0x1520 [ipv6])
[ 521.782912] #5 ffffffffa00090f7 (ip6_finish_output+0x127/0x190 [ipv6])
[ 521.784032] #6 ffffffffa00091e1 (ip6_output+0x81/0x140 [ipv6])
[ 521.785157] #7 ffffffffa000630c (ip6_local_out+0x4c/0x60 [ipv6])
[ 521.786460] #8 ffffffffa0006afd
(ip6_push_pending_frames+0x7dd/0xac0 [ipv6])
[ 521.787977] #9 ffffffffa00319de (rawv6_sendmsg+0x12ae/0x15c0 [ipv6])
[ 521.789366] #10 ffffffff818bb498 (inet_sendmsg+0x108/0x160)
[ 521.790597] #11 ffffffff817d0016 (sock_aio_write+0x296/0x2e0)
[ 521.791826] #12 ffffffff8129dcb1 (do_sync_write+0x111/0x170)
[ 521.792975] #13 ffffffff8129e9fd (vfs_write+0x2dd/0x300)
[ 521.793821] #14 ffffffff8129f9a0 (SyS_write+0x80/0xe0)
[ 521.794684] #15 ffffffff81928582 (system_call_fastpath+0x16/0x1b)
[ 521.795640]
[ 521.795878] Allocated by thread T6026:
[ 521.796474] #0 ffffffff810dc768 (asan_slab_alloc+0x48/0xc0)
[ 521.797360] #1 ffffffff81283d89 (kmem_cache_alloc_node_trace+0x99/0x4f0)
[ 521.798365] #2 ffffffff81284211 (__kmalloc_node_track_caller+0x31/0x40)
[ 521.799406] #3 ffffffff817ded6a (__kmalloc_reserve.isra.27+0x4a/0xb0)
[ 521.800436] #4 ffffffff817e0201 (__alloc_skb+0x91/0x280)
[ 521.801328] #5 ffffffff817d807a (sock_wmalloc+0x6a/0xe0)
[ 521.802170] #6 ffffffffa0005ea6 (ip6_append_data+0x1906/0x1c20 [ipv6])
[ 521.803073] #7 ffffffffa0030dd7 (rawv6_sendmsg+0x6a7/0x15c0 [ipv6])
[ 521.804068] #8 ffffffff818bb498 (inet_sendmsg+0x108/0x160)
[ 521.804919] #9 ffffffff817d18e3 (sock_sendmsg+0x133/0x170)
[ 521.805760] #10 ffffffff817d2009 (SYSC_sendto+0x1e9/0x2d0)
[ 521.806618] #11 ffffffff817d2cc9 (SyS_sendto+0x49/0x70)
[ 521.807598] #12 ffffffff81928582 (system_call_fastpath+0x16/0x1b)
[ 521.808826]
[ 521.809188] Shadow bytes around the buggy address:
[ 521.810231] ffff88004965f900: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 521.811752] ffff88004965f980: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 521.813253] ffff88004965fa00: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 521.814743] ffff88004965fa80: 00 00 00 00 00 00 00 00 fa fa fa fa
fa fa fa fa
[ 521.816052] ffff88004965fb00: fa fa fa fa fa fa fa fa fa fa fa fa
fa fa fa fa
[ 521.817113] =>ffff88004965fb80: fa fa fa fa fa fa fa fa fa fa fa fa
fa[fa]fa fa
[ 521.818149] ffff88004965fc00: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 521.819224] ffff88004965fc80: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 521.820280] ffff88004965fd00: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 521.821357] ffff88004965fd80: 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00
[ 521.822398] ffff88004965fe00: fa fa fa fa fa fa fa fa fa fa fa fa
fa fa fa fa
[ 521.823392] Shadow byte legend (one shadow byte represents 8
application bytes):
[ 521.824388] Addressable: 00
[ 521.824901] Partially addressable: 01 02 03 04 05 06 07
[ 521.825667] Heap redzone: fa
[ 521.826260] Heap kmalloc redzone: fb
[ 521.826802] Freed heap region: fd
[ 521.827347] Shadow gap: fe
[ 521.827884] =========================================================================
[ 521.828976] skbuff: skb_under_panic: text:ffffffffa0003b35 len:133
put:14 head:ffff88004965fc00 data:ffff88004965fbea tail:0x6f end:0xc0
dev:lo
[ 521.830736] ------------[ cut here ]------------
[ 521.831372] kernel BUG at net/core/skbuff.c:126!
Dec 31 18[:5 4: 035 21.831680] invalid opcode: 0000 [#1] SMP
[ 521.831680] Modules linked in: snd_mixer_oss snd sr_mod cdrom loop
tun 8021qasa n3b krerinedl:g [e 5s21t.8p28976] slkblc st ipt_ULOG
nfnetlink iptable_mangle tg3 ptp pps_core i2c_piix4 i2c_core msr cpuid
e1000 ipv6
[ 521.831680] CPU: 1 PID: 2167 Comm: trinity-child52 Not tainted
3.11.0-smp-DEV #8
[ 521.831680] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
uff:[ s kb _u5nd2er1_p.831680] task: ffff88004b720be0 ti:
ffff88004fc54000 task.ti: ffff88004fc54000
[ 521.831680] RIP: 0010:[<ffffffff81913878>] anic :
[te<xtf:ffffffffffff81913878>] skb_panic+0xd5/0xd7
[ 521.831680] RSP: 0018:ffff88004fc557a0 EFLAGS: 00010286
fffa[00 03 b355 2le1n:.831680] RAX: 0000000000000083 RBX:
ffff88004a919d80 RCX: 0000000000000000
[ 521.831680] RDX: ffff880000000000 RSI: 0000000000000008 RDI: ffffffff81c44cd8
133 [pu t: 145 h2ea1d:.831680] RBP: ffff88004fc55808 R08:
000000000000006f R09: 0000000000000000
[fff f8 8050429615f.c08031680] R10: 0000000000000000 R11:
0000000007f70a60 R12: ffff88005bf89400
[ 521.831680] R13: ffff88004965fbea R14: 000000000000006f R15: 00000000000000c0
d[at a: ff5ff8280104.9831680] FS: 0000000001a48880(0063)
GS:ffff88005fd00000(0000) knlGS:0000000000000000
[ 521.831680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 521.831680] CR2: 0000000000000000 CR3: 0000000013404000 CR4: 00000000000006e0
65fbe[a ta il52:01x.6f831680] Stack:
[ 521.831680] ffff88004965fbea 000000000000006f en d:00x0c00
d0ev0:l0o00000000c0 ffff88005bf89400
[ 521.831680] 0000000e4a919d80 ffffffffa0003b35 ffffffff81aa3940
ffff88004965fc00
[ 521.831680] ffff88004a919d80 ffff88004965fbea 000000000000000e
ffff88004a919de0
[ 521.831680] Call Trace:
[ 521.831680] [<ffffffffa0003b35>] ? ip6_finish_output2+0x565/0x840 [ipv6]
[ 521.831680] [<ffffffff817ddb59>] skb_push+0xa9/0xb0
[ 521.831680] [<ffffffffa0003b35>] ip6_finish_output2+0x565/0x840 [ipv6]
[ 521.831680] [<ffffffffa00088dc>] ip6_fragment+0xe2c/0x1520 [ipv6]
[ 521.831680] [<ffffffffa00035d0>] ?
ip6_flush_pending_frames+0x1d0/0x1d0 [ipv6]
[ 521.831680] [<ffffffff810dcd19>] ? asan_region_is_poisoned+0x89/0x1a0
[ 521.831680] [<ffffffffa00090f7>] ip6_finish_output+0x127/0x190 [ipv6]
[ 521.831680] [<ffffffffa00091e1>] ip6_output+0x81/0x140 [ipv6]
[ 521.831680] [<ffffffffa000630c>] ip6_local_out+0x4c/0x60 [ipv6]
[ 521.831680] [<ffffffff810dc689>] ? asan_check_region+0x19/0x40
[ 521.831680] [<ffffffffa0006afd>] ip6_push_pending_frames+0x7dd/0xac0 [ipv6]
[ 521.831680] [<ffffffffa00319de>] rawv6_sendmsg+0x12ae/0x15c0 [ipv6]
[ 521.831680] [<ffffffff810dc689>] ? asan_check_region+0x19/0x40
[ 521.831680] [<ffffffff818bb498>] inet_sendmsg+0x108/0x160
[ 521.831680] [<ffffffff817d0016>] sock_aio_write+0x296/0x2e0
[ 521.831680] [<ffffffff8129dcb1>] do_sync_write+0x111/0x170
[ 521.831680] [<ffffffff8129e9fd>] vfs_write+0x2dd/0x300
[ 521.831680] [<ffffffff8129f9a0>] SyS_write+0x80/0xe0
[ 521.831680] [<ffffffff81928582>] system_call_fastpath+0x16/0x1b
[ 521.831680] Code: c7 f0 a2 ba 81 44 8b 45 bc 48 8b 55 c0 31 c0 48
8b 75 c8 4c 89 64 24 18 4c 89 7c 24 10 4c 89 74 24 08 4c 89 2c 24 e8
7d 73 ff ff <0f> 0b 55 48 89 e5 48 8b 7d 08 e8 39 9b 7c ff 0f 0b 55 48
89 e5
[ 521.831680] RIP [<ffffffff81913878>] skb_panic+0xd5/0xd7
[ 521.831680] RSP <ffff88004fc557a0>
[ 521.876810] ---[ end trace 4037fd48810bceeb ]---
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potential out-of-bounds access in ip6_finish_output2
2013-09-17 5:13 Potential out-of-bounds access in ip6_finish_output2 Dmitry Vyukov
@ 2013-09-17 22:48 ` Hannes Frederic Sowa
2013-09-21 15:14 ` Hannes Frederic Sowa
0 siblings, 1 reply; 6+ messages in thread
From: Hannes Frederic Sowa @ 2013-09-17 22:48 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: yoshfuji, netdev, Paul Turner, Andrey Konovalov,
Kostya Serebryany, Tom Herbert
On Mon, Sep 16, 2013 at 10:13:10PM -0700, Dmitry Vyukov wrote:
> Hi,
>
> I am working on AddressSanitizer -- a tool that detects use-after-free
> and out-of-bounds bugs
> (https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel).
>
> I've got a dozen of reports in ip6_finish_output2. Below are 2 of
> them. They are always followed by kernel crash. Unfortunately I don't
> have a reproducer because I am using trinity fuzzer. I would
> appreciate if somebody familiar with the code look at sources and
> maybe spot the bug.
Thanks for the report!
I tried reproducing the bug and hit some other bugs nearby. I'll try to fix
them, but this could take some time.
Greetings,
Hannes
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potential out-of-bounds access in ip6_finish_output2
2013-09-17 22:48 ` Hannes Frederic Sowa
@ 2013-09-21 15:14 ` Hannes Frederic Sowa
2013-09-21 18:08 ` Dmitry Vyukov
0 siblings, 1 reply; 6+ messages in thread
From: Hannes Frederic Sowa @ 2013-09-21 15:14 UTC (permalink / raw)
To: Dmitry Vyukov, yoshfuji, netdev, Paul Turner, Andrey Konovalov,
Kostya Serebryany, Tom Herbert
On Wed, Sep 18, 2013 at 12:48:51AM +0200, Hannes Frederic Sowa wrote:
> On Mon, Sep 16, 2013 at 10:13:10PM -0700, Dmitry Vyukov wrote:
> > I am working on AddressSanitizer -- a tool that detects use-after-free
> > and out-of-bounds bugs
> > (https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel).
> >
> > I've got a dozen of reports in ip6_finish_output2. Below are 2 of
> > them. They are always followed by kernel crash. Unfortunately I don't
> > have a reproducer because I am using trinity fuzzer. I would
> > appreciate if somebody familiar with the code look at sources and
> > maybe spot the bug.
>
> Thanks for the report!
>
> I tried reproducing the bug and hit some other bugs nearby. I'll try to fix
> them, but this could take some time.
I fixed the first bug I encountered with trinity here:
http://patchwork.ozlabs.org/patch/276835/
The main cause of this bug has nothing to do with raw sockets, so I first
thought they are not related. But I left my machine run trinity while
I was sleeping and did not see any other splats (I added some manually
range checks in ip6_append_data). So maybe your bug happend because the
premature exit in the dontfrag check without resetting cork->length. Maybe
you could give this patch a try? I'll have a second look later today.
Thanks,
Hannes
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potential out-of-bounds access in ip6_finish_output2
2013-09-21 15:14 ` Hannes Frederic Sowa
@ 2013-09-21 18:08 ` Dmitry Vyukov
0 siblings, 0 replies; 6+ messages in thread
From: Dmitry Vyukov @ 2013-09-21 18:08 UTC (permalink / raw)
To: Dmitry Vyukov, yoshfuji, netdev, Paul Turner, Andrey Konovalov,
Kostya Serebryany, Tom Herbert
On Sat, Sep 21, 2013 at 8:14 AM, Hannes Frederic Sowa
<hannes@stressinduktion.org> wrote:
> On Wed, Sep 18, 2013 at 12:48:51AM +0200, Hannes Frederic Sowa wrote:
>> On Mon, Sep 16, 2013 at 10:13:10PM -0700, Dmitry Vyukov wrote:
>> > I am working on AddressSanitizer -- a tool that detects use-after-free
>> > and out-of-bounds bugs
>> > (https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel).
>> >
>> > I've got a dozen of reports in ip6_finish_output2. Below are 2 of
>> > them. They are always followed by kernel crash. Unfortunately I don't
>> > have a reproducer because I am using trinity fuzzer. I would
>> > appreciate if somebody familiar with the code look at sources and
>> > maybe spot the bug.
>>
>> Thanks for the report!
>>
>> I tried reproducing the bug and hit some other bugs nearby. I'll try to fix
>> them, but this could take some time.
>
> I fixed the first bug I encountered with trinity here:
> http://patchwork.ozlabs.org/patch/276835/
>
> The main cause of this bug has nothing to do with raw sockets, so I first
> thought they are not related. But I left my machine run trinity while
> I was sleeping and did not see any other splats (I added some manually
> range checks in ip6_append_data). So maybe your bug happend because the
> premature exit in the dontfrag check without resetting cork->length. Maybe
> you could give this patch a try? I'll have a second look later today.
Hi Hannes!
Testing now with your patch.
I've seen this report very few times, so it will be difficult to say
if it's not happening any more or just not triggered.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Potential out-of-bounds access in ip6_finish_output2
@ 2014-04-21 13:22 Alexey Preobrazhensky
2014-04-22 12:00 ` Hannes Frederic Sowa
0 siblings, 1 reply; 6+ messages in thread
From: Alexey Preobrazhensky @ 2014-04-21 13:22 UTC (permalink / raw)
To: netdev, Hannes Frederic Sowa, Kostya Serebryany, Dmitry Vyukov,
yoshfuji, maze, edumazet, brutus
Hi,
I’m working on AddressSanitizer[1] -- a tool that detects
use-after-free and out-of-bounds bugs in kernel.
I’ve encountered a heap-buffer-overflow in ip6_finish_output2 in linux
kernel 3.14 (revision 455c6fdbd219161bd09b1165f11699d6d73de11c). A
similar problem was reported earlier[2] and resulted in a patch[3],
but we’ve hit this report again, so it seems the issue weren’t fixed,
or there is another issue. The offending code in
include/net/neighbour.h:401 is:
do {
seq = read_seqbegin(&hh->hh_lock);
hh_len = hh->hh_len;
if (likely(hh_len <= HH_DATA_MOD)) {
/* this is inlined by gcc */
/* 401: */ memcpy(skb->data - HH_DATA_MOD, hh->hh_data, HH_DATA_MOD);
} else {
int hh_alen = HH_DATA_ALIGN(hh_len);
memcpy(skb->data - hh_alen, hh->hh_data, hh_alen);
}
} while (read_seqretry(&hh->hh_lock, seq));
This heap-buffer-overflow was triggered under trinity syscall fuzzer,
so there is no reproducer. The report is followed by crash (included).
It would be great if someone familiar with the code took time to look
into this report.
Thanks,
Alexey
[1] https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel
[2] http://www.spinics.net/lists/netdev/msg250734.html
[3] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=2811ebac2521ceac84f2bdae402455baa6a7fb47
==================================================================
AddressSanitizer: heap-buffer-overflow in ip6_finish_output2
Write of size 16 by thread T15020:
[< inlined >] ip6_finish_output2+0x51e/0x840 [ipv6]
neigh_hh_output ./include/net/neighbour.h:401
[< inlined >] ip6_finish_output2+0x51e/0x840 [ipv6]
dst_neigh_output ./include/net/dst.h:409
[<ffffffffa0003a9e>] ip6_finish_output2+0x51e/0x840 [ipv6]
./net/ipv6/ip6_output.c:113
[<ffffffffa0008efd>] ip6_fragment+0x12ed/0x14e0 [ipv6]
./net/ipv6/ip6_output.c:664
[<ffffffffa0009247>] ip6_finish_output+0x157/0x1b0 [ipv6]
./net/ipv6/ip6_output.c:130
[< inlined >] ip6_output+0x77/0x130 [ipv6] NF_HOOK_COND
./include/linux/netfilter.h:186
[<ffffffffa0009317>] ip6_output+0x77/0x130 [ipv6] ./net/ipv6/ip6_output.c:146
[<ffffffff8185c4fc>] ip6_local_out+0x3c/0x50 ??:0
[<ffffffffa0006b9a>] ip6_push_pending_frames+0x86a/0xa90 [ipv6]
./net/ipv6/ip6_output.c:1573
[< inlined >] rawv6_sendmsg+0x1323/0x1680 [ipv6]
rawv6_push_pending_frames ./net/ipv6/raw.c:608
[<ffffffffa0033eb3>] rawv6_sendmsg+0x1323/0x1680 [ipv6] ./net/ipv6/raw.c:906
[<ffffffff81829a3a>] inet_sendmsg+0x10a/0x160 ./net/ipv4/af_inet.c:740
[< inlined >] sock_aio_write+0x263/0x290 do_sock_write ./net/socket.c:633
[<ffffffff81739003>] sock_aio_write+0x263/0x290 ./net/socket.c:973
[<ffffffff812b1ad9>] do_sync_write+0xd9/0x120 ??:0
[<ffffffff812b27fd>] vfs_write+0x2cd/0x2f0 ??:0
[<ffffffff812b3742>] SyS_write+0x62/0xd0 ??:0
[<ffffffff8188e492>] system_call_fastpath+0x16/0x1b
./arch/x86/kernel/entry_64.S:629
Allocated by thread T15020:
[<ffffffff81295a86>] __kmalloc_node_track_caller+0x36/0x60 ./mm/slab.c:3567
[< inlined >] __alloc_skb+0x8f/0x290 __kmalloc_reserve
./net/core/skbuff.c:132
[<ffffffff817487ef>] __alloc_skb+0x8f/0x290 ./net/core/skbuff.c:225
[< inlined >] sock_wmalloc+0x5d/0x100 alloc_skb
./include/linux/skbuff.h:668
[<ffffffff81742aed>] sock_wmalloc+0x5d/0x100 ./net/core/sock.c:1649
[<ffffffffa000614d>] ip6_append_data+0x1c1d/0x1e00 [ipv6]
./net/ipv6/ip6_output.c:1359
[<ffffffffa0033ac5>] rawv6_sendmsg+0xf35/0x1680 [ipv6] ./net/ipv6/raw.c:899
[<ffffffff81829a3a>] inet_sendmsg+0x10a/0x160 ./net/ipv4/af_inet.c:740
[< inlined >] sock_aio_write+0x263/0x290 do_sock_write ./net/socket.c:633
[<ffffffff81739003>] sock_aio_write+0x263/0x290 ./net/socket.c:973
[<ffffffff812b1ad9>] do_sync_write+0xd9/0x120 ??:0
[<ffffffff812b27fd>] vfs_write+0x2cd/0x2f0 ??:0
[<ffffffff812b3742>] SyS_write+0x62/0xd0 ??:0
[<ffffffff8188e492>] system_call_fastpath+0x16/0x1b
./arch/x86/kernel/entry_64.S:629
The buggy address ffff880035ab9bb8 is located 8 bytes to the left
of 448-byte region [ffff880035ab9bc0, ffff880035ab9d80)
Memory state around the buggy address:
ffff880035ab9600: rrrrrrrr ........ ........ ........
ffff880035ab9700: ........ .......r rrrrrrrr rrrrrrrr
ffff880035ab9800: rrrrrrrr rrrrrrrr rrrrrrrr rrrrrrrr
ffff880035ab9900: ........ ........ ........ ........
ffff880035ab9a00: .......r rrrrrrrr rrrrrrrr rrrrrrrr
>ffff880035ab9b00: rrrrrrrr rrrrrrrr rrrrrrrr ........
^
ffff880035ab9c00: ........ ........ ........ ........
ffff880035ab9d00: ........ ........ rrrrrrrr rrrrrrrr
ffff880035ab9e00: rrrrrrrr rrrrrrrr rrrrrrrr rrrrrrrr
ffff880035ab9f00: rrrrrrrr rrrrrrrr rrrrrrrr rrrrrrrr
ffff880035aba000: rrrrrrrr r....... .rrrrrrr rrrrrrrr
Legend:
f - 8 freed bytes
r - 8 redzone bytes
. - 8 allocated bytes
x=1..7 - x allocated bytes + (8-x) redzone bytes
==================================================================
skbuff: skb_under_panic: text:ffffffffa0003abb len:102 put:14
head:ffff880035ab9bc0 data:ffff880035ab9bba tail:0x60 end:0x80 dev:lo
------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:99!
invalid opcode: 0000 [#1] SMP
Modules linked in: sr_mod cdrom loop 8021q bridge stp llc st ipt_ULOG
nfnetlink i2c_piix4 i2c_core msr cpuid e1000 ipv6
CPU: 0 PID: 15020 Comm: trinity-child39 Not tainted 3.14.0-smp-DEV #1
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
task: ffff8800355d69c0 ti: ffff880026d70000 task.ti: ffff880026d70000
RIP: 0010:[<ffffffff8174709c>] [<ffffffff8174709c>] skb_panic+0xcc/0xd0
RSP: 0018:ffff880026d71820 EFLAGS: 00010282
RAX: 0000000000000083 RBX: ffff88002bc8f680 RCX: ffff88003fc0d308
RDX: ffff880036400600 RSI: ffff88003fc0d308 RDI: ffff88003fc0d300
RBP: ffff880026d71888 R08: 000000000000000a R09: 0000000000000006
R10: 0000000000000011 R11: 0000000000000000 R12: ffff8800345aaa80
R13: ffff880035ab9bba R14: 0000000000000060 R15: 0000000000000080
FS: 000000000092e880(0063) GS:ffff88003fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f027d76c000 CR3: 00000000316af000 CR4: 00000000000006f0
Stack:
ffff880035ab9bba 0000000000000060 0000000000000080 ffff8800345aaa80
0000000e35ab9bb8 ffffffffa0003abb ffffffff81a9e440 ffff880035ab9bc0
ffff88002bc8f680 ffff880035ab9bba 000000000000000e ffff88002bc8f6e0
Call Trace:
[<ffffffffa0003abb>] ? ip6_finish_output2+0x53b/0x840 [ipv6]
[<ffffffff8174713e>] skb_push+0x9e/0xa0 ./net/core/skbuff.c:109
[< inlined >] ip6_finish_output2+0x53b/0x840 [ipv6]
neigh_hh_output ./include/net/neighbour.h:409
[< inlined >] ip6_finish_output2+0x53b/0x840 [ipv6]
dst_neigh_output ./include/net/dst.h:409
[<ffffffffa0003abb>] ip6_finish_output2+0x53b/0x840 [ipv6]
./net/ipv6/ip6_output.c:113
[< inlined >] ? asan_memcpy+0x5a2/0x630 check_memory_region
./arch/x86/mm/asan/asan.c:256
[<ffffffff810e47c2>] ? asan_memcpy+0x5a2/0x630 ./arch/x86/mm/asan/asan.c:547
[<ffffffffa0008efd>] ip6_fragment+0x12ed/0x14e0 [ipv6]
./net/ipv6/ip6_output.c:664
[<ffffffffa0003580>] ? ip6_flush_pending_frames+0x1c0/0x1c0 [ipv6]
[< inlined >] ? kmem_cache_alloc_node_trace+0x9c/0x640
slab_alloc_node ./mm/slab.c:3257
[<ffffffff812954ac>] ? kmem_cache_alloc_node_trace+0x9c/0x640 ./mm/slab.c:3548
[<ffffffffa0009247>] ip6_finish_output+0x157/0x1b0 [ipv6]
./net/ipv6/ip6_output.c:130
[< inlined >] ip6_output+0x77/0x130 [ipv6] NF_HOOK_COND
./include/linux/netfilter.h:186
[<ffffffffa0009317>] ip6_output+0x77/0x130 [ipv6] ./net/ipv6/ip6_output.c:146
[<ffffffff8185c4fc>] ip6_local_out+0x3c/0x50 ??:0
[<ffffffffa0006b9a>] ip6_push_pending_frames+0x86a/0xa90 [ipv6]
./net/ipv6/ip6_output.c:1573
[< inlined >] rawv6_sendmsg+0x1323/0x1680 [ipv6]
rawv6_push_pending_frames ./net/ipv6/raw.c:608
[<ffffffffa0033eb3>] rawv6_sendmsg+0x1323/0x1680 [ipv6] ./net/ipv6/raw.c:906
[<ffffffff81223e27>] ? __generic_file_aio_write+0x367/0x560 ./mm/filemap.c:2524
[<ffffffff81829a3a>] inet_sendmsg+0x10a/0x160 ./net/ipv4/af_inet.c:740
[< inlined >] sock_aio_write+0x263/0x290 do_sock_write ./net/socket.c:633
[<ffffffff81739003>] sock_aio_write+0x263/0x290 ./net/socket.c:973
[<ffffffff812b1ad9>] do_sync_write+0xd9/0x120 ??:0
[<ffffffff812b27fd>] vfs_write+0x2cd/0x2f0 ??:0
[<ffffffff812b3742>] SyS_write+0x62/0xd0 ??:0
[<ffffffff8188e492>] system_call_fastpath+0x16/0x1b
./arch/x86/kernel/entry_64.S:629
Code: c7 50 7e b9 81 44 8b 45 bc 48 8b 55 c0 31 c0 48 8b 75 c8 4c 89
64 24 18 4c 89 7c 24 10 4c 89 74 24 08 4c 89 2c 24 e8 c5 f3 12 00 <0f>
0b 66 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 4c 8d b7 c8
RIP [<ffffffff8174709c>] skb_panic+0xcc/0xd0
RSP <ffff880026d71820>
---[ end trace 3c86b5021571cac8 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range:
0xffffffff80000000-0xffffffff9fffffff)
Rebooting in 10 seconds..
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Potential out-of-bounds access in ip6_finish_output2
2014-04-21 13:22 Alexey Preobrazhensky
@ 2014-04-22 12:00 ` Hannes Frederic Sowa
0 siblings, 0 replies; 6+ messages in thread
From: Hannes Frederic Sowa @ 2014-04-22 12:00 UTC (permalink / raw)
To: Alexey Preobrazhensky
Cc: netdev, Kostya Serebryany, Dmitry Vyukov, yoshfuji, maze,
edumazet, brutus
On Mon, Apr 21, 2014 at 05:22:05PM +0400, Alexey Preobrazhensky wrote:
> Hi,
>
> I’m working on AddressSanitizer[1] -- a tool that detects
> use-after-free and out-of-bounds bugs in kernel.
>
> I’ve encountered a heap-buffer-overflow in ip6_finish_output2 in linux
> kernel 3.14 (revision 455c6fdbd219161bd09b1165f11699d6d73de11c). A
> similar problem was reported earlier[2] and resulted in a patch[3],
> but we’ve hit this report again, so it seems the issue weren’t fixed,
> or there is another issue. The offending code in
> include/net/neighbour.h:401 is:
>
> do {
> seq = read_seqbegin(&hh->hh_lock);
> hh_len = hh->hh_len;
> if (likely(hh_len <= HH_DATA_MOD)) {
> /* this is inlined by gcc */
> /* 401: */ memcpy(skb->data - HH_DATA_MOD, hh->hh_data, HH_DATA_MOD);
> } else {
> int hh_alen = HH_DATA_ALIGN(hh_len);
>
> memcpy(skb->data - hh_alen, hh->hh_data, hh_alen);
> }
> } while (read_seqretry(&hh->hh_lock, seq));
>
> This heap-buffer-overflow was triggered under trinity syscall fuzzer,
> so there is no reproducer. The report is followed by crash (included).
>
> It would be great if someone familiar with the code took time to look
> into this report.
Thanks for the report!
By only looking at the report I doubt it is the same problem. I still have a
lot of mails to catch up but hope that I can audit the code in the next days.
Thanks,
Hannes
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-04-22 12:00 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-09-17 5:13 Potential out-of-bounds access in ip6_finish_output2 Dmitry Vyukov
2013-09-17 22:48 ` Hannes Frederic Sowa
2013-09-21 15:14 ` Hannes Frederic Sowa
2013-09-21 18:08 ` Dmitry Vyukov
-- strict thread matches above, loose matches on Subject: below --
2014-04-21 13:22 Alexey Preobrazhensky
2014-04-22 12:00 ` Hannes Frederic Sowa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).