* Potential use-after-free in ____call_usermodehelper @ 2013-08-21 16:35 Dmitry Vyukov 2013-08-23 15:48 ` Dmitry Vyukov 0 siblings, 1 reply; 7+ messages in thread From: Dmitry Vyukov @ 2013-08-21 16:35 UTC (permalink / raw) To: LKML Cc: Andrey Konovalov, Kostya Serebryany, Alexander Potapenko, Evgeniy Stepanov Hi, I'm working on a memory error detector AddressSanitizer for Linux kernel (https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel), it can detect use-after-free and buffer-overflow errors. Currently the tool is in very early stage and it can contain bugs. Here is one of the reports produced during testing: [ 196.951434] ERROR: AddressSanitizer: heap-use-after-free on address ffff880008a632c4 [ 196.952135] Stack trace: [ 196.952380] [<ffffffff810dd1f5>] asan_report_error+0x85/0x2c0 [ 196.952890] [<ffffffff810dc700>] asan_check_region+0x30/0x40 [ 196.953466] [<ffffffff810dd553>] __tsan_write4+0x13/0x20 [ 196.953987] [<ffffffff8110a76a>] ____call_usermodehelper+0x21a/0x240 [ 196.954651] [<ffffffff8110a7cc>] call_helper+0x3c/0x50 [ 196.955155] [<ffffffff81924b5c>] ret_from_fork+0x7c/0xb0 [ 196.955686] [<ffffffffffffffff>] 0xffffffffffffffff [ 196.956230] Free stack trace: [ 196.956532] [<ffffffff810dc831>] asan_slab_free+0x61/0xb0 [ 196.957052] [<ffffffff8128070a>] kfree+0x9a/0x240 [ 196.957558] [<ffffffff8110a165>] call_usermodehelper_freeinfo+0x35/0x40 [ 196.958308] [<ffffffff8110aabe>] call_usermodehelper_exec+0xae/0x1d0 [ 196.958920] [<ffffffff8110af81>] call_usermodehelper+0x61/0x90 [ 196.959490] [<ffffffff814d7e4e>] kobject_uevent_env+0x5be/0x5f0 [ 196.960161] [<ffffffff814d7ea3>] kobject_uevent+0x23/0x40 [ 196.960706] [<ffffffff814d63ad>] kobject_release+0xad/0xc0 [ 196.961274] [<ffffffff814d618a>] kobject_put+0x3a/0x80 [ 196.961889] [<ffffffff8181af6c>] net_rx_queue_update_kobjects+0x12c/0x170 [ 196.962701] [<ffffffff8181b1b2>] netdev_unregister_kobject+0x62/0xa0 [ 196.963475] [<ffffffff817f342b>] rollback_registered_many+0x27b/0x340 [ 196.964175] [<ffffffff817f35d5>] unregister_netdevice_many+0x35/0xe0 [ 196.964836] [<ffffffff817f43f7>] default_device_exit_batch+0x107/0x180 [ 196.965568] [<ffffffff817ebb1c>] ops_exit_list.isra.3+0x8c/0xa0 I've looked at the sources, but I can't say that I fully understand them. The report looks valid, though. I see several potential issues in the code. 1. When wait=UMH_WAIT_EXEC and do_execve() fails, ____call_usermodehelper() writes sub_info->retval=retval to freed memory. This is the use-after-free reported by the tool. 2. When wait=UMH_NO_WAIT, __call_usermodehelper() starts child thread and instantly frees subprocess_info. The child thread reads subprocess_info. Looks like another use-after-free. 3. UMH_WAIT_EXEC does not actually wait for exec, it only waits for starting the child thread that will do exec. I don't know whether it's a problem with the code or with the name. The kernel version is 3.11-rc4 (last commit: b7bc9e7d808ba55729bd263b0210cda36965be32). Please help to confirm these issues, and advice what to do next with them. TIA ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential use-after-free in ____call_usermodehelper 2013-08-21 16:35 Potential use-after-free in ____call_usermodehelper Dmitry Vyukov @ 2013-08-23 15:48 ` Dmitry Vyukov 2013-09-03 13:49 ` Dmitry Vyukov 0 siblings, 1 reply; 7+ messages in thread From: Dmitry Vyukov @ 2013-08-23 15:48 UTC (permalink / raw) To: LKML Cc: Andrey Konovalov, Kostya Serebryany, Alexander Potapenko, Evgeniy Stepanov On Wed, Aug 21, 2013 at 8:35 PM, Dmitry Vyukov <dvyukov@google.com> wrote: > Hi, > > I'm working on a memory error detector AddressSanitizer for Linux > kernel (https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel), > it can detect use-after-free and buffer-overflow errors. Currently the > tool is in very early stage and it can contain bugs. > > Here is one of the reports produced during testing: > > [ 196.951434] ERROR: AddressSanitizer: heap-use-after-free on address > ffff880008a632c4 > [ 196.952135] Stack trace: > [ 196.952380] [<ffffffff810dd1f5>] asan_report_error+0x85/0x2c0 > [ 196.952890] [<ffffffff810dc700>] asan_check_region+0x30/0x40 > [ 196.953466] [<ffffffff810dd553>] __tsan_write4+0x13/0x20 > [ 196.953987] [<ffffffff8110a76a>] ____call_usermodehelper+0x21a/0x240 > [ 196.954651] [<ffffffff8110a7cc>] call_helper+0x3c/0x50 > [ 196.955155] [<ffffffff81924b5c>] ret_from_fork+0x7c/0xb0 > [ 196.955686] [<ffffffffffffffff>] 0xffffffffffffffff > [ 196.956230] Free stack trace: > [ 196.956532] [<ffffffff810dc831>] asan_slab_free+0x61/0xb0 > [ 196.957052] [<ffffffff8128070a>] kfree+0x9a/0x240 > [ 196.957558] [<ffffffff8110a165>] call_usermodehelper_freeinfo+0x35/0x40 > [ 196.958308] [<ffffffff8110aabe>] call_usermodehelper_exec+0xae/0x1d0 > [ 196.958920] [<ffffffff8110af81>] call_usermodehelper+0x61/0x90 > [ 196.959490] [<ffffffff814d7e4e>] kobject_uevent_env+0x5be/0x5f0 > [ 196.960161] [<ffffffff814d7ea3>] kobject_uevent+0x23/0x40 > [ 196.960706] [<ffffffff814d63ad>] kobject_release+0xad/0xc0 > [ 196.961274] [<ffffffff814d618a>] kobject_put+0x3a/0x80 > [ 196.961889] [<ffffffff8181af6c>] net_rx_queue_update_kobjects+0x12c/0x170 > [ 196.962701] [<ffffffff8181b1b2>] netdev_unregister_kobject+0x62/0xa0 > [ 196.963475] [<ffffffff817f342b>] rollback_registered_many+0x27b/0x340 > [ 196.964175] [<ffffffff817f35d5>] unregister_netdevice_many+0x35/0xe0 > [ 196.964836] [<ffffffff817f43f7>] default_device_exit_batch+0x107/0x180 > [ 196.965568] [<ffffffff817ebb1c>] ops_exit_list.isra.3+0x8c/0xa0 > > > I've looked at the sources, but I can't say that I fully understand > them. The report looks valid, though. I see several potential issues > in the code. > > 1. When wait=UMH_WAIT_EXEC and do_execve() fails, > ____call_usermodehelper() writes sub_info->retval=retval to freed > memory. This is the use-after-free reported by the tool. > > 2. When wait=UMH_NO_WAIT, __call_usermodehelper() starts child thread > and instantly frees subprocess_info. The child thread reads > subprocess_info. Looks like another use-after-free. > > 3. UMH_WAIT_EXEC does not actually wait for exec, it only waits for > starting the child thread that will do exec. I don't know whether it's > a problem with the code or with the name. > > The kernel version is 3.11-rc4 (last commit: > b7bc9e7d808ba55729bd263b0210cda36965be32). > > Please help to confirm these issues, and advice what to do next with them. > > TIA Here is more detailed report with allocation stack and thread ids: [ 957.069245] ERROR: AddressSanitizer: heap-use-after-free on address ffff8800148452c4 [ 957.070438] Accessed by thread T22400: [ 957.071134] #0 ffffffff810dd260 (asan_report_error+0xb0/0x250) [ 957.071947] #1 ffffffff810dc6d0 (asan_check_region+0x30/0x40) [ 957.072766] #2 ffffffff810dd523 (__tsan_write4+0x13/0x20) [ 957.073559] #3 ffffffff8110a77a (____call_usermodehelper+0x21a/0x240) [ 957.074488] #4 ffffffff8110a7dc (call_helper+0x3c/0x50) [ 957.075266] #5 ffffffff8192569c (ret_from_fork+0x7c/0xb0) [ 957.076006] Allocated by thread T506: [ 957.076662] #0 ffffffff810dc794 (asan_slab_alloc+0x44/0xa0) [ 957.077475] #1 ffffffff812821bc (__kmalloc+0xbc/0x500) [ 957.078239] #2 ffffffff8110a1c4 (call_usermodehelper_setup+0x44/0x120) [ 957.079173] #3 ffffffff8110af82 (call_usermodehelper+0x52/0x90) [ 957.081469] #4 ffffffff814d831e (kobject_uevent_env+0x5be/0x5f0) [ 957.082329] #5 ffffffff814d8373 (kobject_uevent+0x23/0x40) [ 957.083116] #6 ffffffff814d687d (kobject_release+0xad/0xc0) [ 957.085582] #7 ffffffff814d665a (kobject_put+0x3a/0x80) [ 957.086348] #8 ffffffff8181b873 (netdev_queue_update_kobjects+0x133/0x1a0) [ 957.087324] #9 ffffffff8181b94f (netdev_unregister_kobject+0x6f/0xa0) [ 957.088232] #10 ffffffff817f3b9b (rollback_registered_many+0x27b/0x340) [ 957.089159] #11 ffffffff817f3d45 (unregister_netdevice_many+0x35/0xe0) [ 957.090091] #12 ffffffff817f4b67 (default_device_exit_batch+0x107/0x180) [ 957.091010] #13 ffffffff817ec27c (ops_exit_list.isra.3+0x8c/0xa0) [ 957.091899] #14 ffffffff817ed0c9 (cleanup_net+0x229/0x390) [ 957.092699] #15 ffffffff811113af (process_one_work+0x2cf/0x750) [ 957.093555] Freed by thread T506: [ 957.094185] #0 ffffffff810dc867 (asan_slab_free+0x77/0xb0) [ 957.094958] #1 ffffffff8128081a (kfree+0x9a/0x240) [ 957.095659] #2 ffffffff8110a175 (call_usermodehelper_freeinfo+0x35/0x40) [ 957.096608] #3 ffffffff8110aace (call_usermodehelper_exec+0xae/0x1d0) [ 957.097520] #4 ffffffff8110af91 (call_usermodehelper+0x61/0x90) [ 957.098377] #5 ffffffff814d831e (kobject_uevent_env+0x5be/0x5f0) [ 957.099235] #6 ffffffff814d8373 (kobject_uevent+0x23/0x40) [ 957.099984] #7 ffffffff814d687d (kobject_release+0xad/0xc0) [ 957.100781] #8 ffffffff814d665a (kobject_put+0x3a/0x80) [ 957.101541] #9 ffffffff8181b873 (netdev_queue_update_kobjects+0x133/0x1a0) [ 957.102515] #10 ffffffff8181b94f (netdev_unregister_kobject+0x6f/0xa0) [ 957.103460] #11 ffffffff817f3b9b (rollback_registered_many+0x27b/0x340) [ 957.104419] #12 ffffffff817f3d45 (unregister_netdevice_many+0x35/0xe0) [ 957.105352] #13 ffffffff817f4b67 (default_device_exit_batch+0x107/0x180) [ 957.106342] #14 ffffffff817ec27c (ops_exit_list.isra.3+0x8c/0xa0) [ 957.107182] #15 ffffffff817ed0c9 (cleanup_net+0x229/0x390) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential use-after-free in ____call_usermodehelper 2013-08-23 15:48 ` Dmitry Vyukov @ 2013-09-03 13:49 ` Dmitry Vyukov 2013-09-03 20:27 ` Dan Aloni 0 siblings, 1 reply; 7+ messages in thread From: Dmitry Vyukov @ 2013-09-03 13:49 UTC (permalink / raw) To: LKML Cc: Andrey Konovalov, Kostya Serebryany, Alexander Potapenko, Evgeniy Stepanov Hi, Is anybody reading this? Is it a correct place to post such things? Maybe there is a more appropriate place? I am hitting basically thousands of this use-after-free bug. Here is a more detailed report with line numbers: [ 107.292348] ERROR: AddressSanitizer: heap-use-after-free on address ffff8800320f7b84 [ 107.293494] ffff8800320f7b84 is located 68 bytes inside of 96-byte region [ffff8800320f7b40, ffff8800320f7ba0) [ 107.294992] Accessed by thread T5540: [ 107.295685] inlined in describe_heap_address ./arch/x86/mm/asan/report.c:164 [ 107.295685] #0 ffffffff810dd277 in asan_report_error ./arch/x86/mm/asan/report.c:278 [ 107.296731] #1 ffffffff810dc6a0 in asan_check_region ./arch/x86/mm/asan/asan.c:37 [ 107.297759] #2 ffffffff810dd4a3 in __tsan_write4 ??:0 [ 107.298718] #3 ffffffff8110aa1a in ____call_usermodehelper ./kernel/kmod.c:250 [ 107.299819] #4 ffffffff8110aa7c in call_helper ./kernel/kmod.c:258 [ 107.300772] #5 ffffffff8192841c in ret_from_fork ./arch/x86/kernel/entry_64.S:570 [ 107.301691] [ 107.301905] Freed by thread T5457: [ 107.302531] #0 ffffffff810dc839 in asan_slab_free ./arch/x86/mm/asan/asan.c:130 [ 107.303454] inlined in __cache_free ./mm/slab.c:3591 [ 107.303454] #1 ffffffff8128194a in kfree ./mm/slab.c:3831 [ 107.304294] #2 ffffffff8110a415 in call_usermodehelper_freeinfo ./kernel/kmod.c:266 [ 107.305368] #3 ffffffff8110ad6e in call_usermodehelper_exec ././arch/x86/include/asm/atomic.h:123 [ 107.306410] #4 ffffffff8110b231 in call_usermodehelper ./kernel/kmod.c:642 [ 107.307387] #5 ffffffff814d861b in kobject_uevent_env ./lib/kobject_uevent.c:311 [ 107.308393] #6 ffffffff814d8673 in kobject_uevent ./lib/kobject_uevent.c:333 [ 107.309302] inlined in netdev_queue_add_kobject ./net/core/net-sysfs.c:1071 [ 107.309302] #7 ffffffff8181bd1a in netdev_queue_update_kobjects ./net/core/net-sysfs.c:1088 [ 107.310410] inlined in register_queue_kobjects ./net/core/net-sysfs.c:1132 [ 107.310410] #8 ffffffff8181bf5d in netdev_register_kobject ./net/core/net-sysfs.c:1291 [ 107.311449] #9 ffffffff817fdaf8 in register_netdevice ./net/core/dev.c:5425 [ 107.312436] #10 ffffffff817fdca3 in register_netdev ./net/core/dev.c:5536 [ 107.313392] #11 ffffffff816bd643 in loopback_net_init loopback.c:0 [ 107.314363] #12 ffffffff817ec8a4 in ops_init ./net/core/net_namespace.c:107 [ 107.315236] #13 ffffffff817ecac3 in setup_net ./net/core/net_namespace.c:168 [ 107.315947] #14 ffffffff817ed4e4 in copy_net_ns ./net/core/net_namespace.c:255 [ 107.316858] #15 ffffffff81123980 in create_new_namespaces ./kernel/nsproxy.c:94 [ 107.318259] Allocated by thread T5457: [ 107.318764] #0 ffffffff810dc768 in asan_slab_alloc ./arch/x86/mm/asan/asan.c:91 [ 107.319691] inlined in slab_alloc ./mm/slab.c:3475 [ 107.319691] inlined in __do_kmalloc ./mm/slab.c:3749 [ 107.319691] #1 ffffffff812832ec in __kmalloc ./mm/slab.c:3763 [ 107.320564] #2 ffffffff8110a464 in call_usermodehelper_setup ./kernel/kmod.c:541 [ 107.321617] #3 ffffffff8110b222 in call_usermodehelper ./kernel/kmod.c:639 [ 107.322603] #4 ffffffff814d861b in kobject_uevent_env ./lib/kobject_uevent.c:311 [ 107.323607] #5 ffffffff814d8673 in kobject_uevent ./lib/kobject_uevent.c:333 [ 107.324526] inlined in netdev_queue_add_kobject ./net/core/net-sysfs.c:1071 [ 107.324526] #6 ffffffff8181bd1a in netdev_queue_update_kobjects ./net/core/net-sysfs.c:1088 [ 107.325621] inlined in register_queue_kobjects ./net/core/net-sysfs.c:1132 [ 107.325621] #7 ffffffff8181bf5d in netdev_register_kobject ./net/core/net-sysfs.c:1291 [ 107.326662] #8 ffffffff817fdaf8 in register_netdevice ./net/core/dev.c:5425 [ 107.327645] #9 ffffffff817fdca3 in register_netdev ./net/core/dev.c:5536 [ 107.328576] #10 ffffffff816bd643 in loopback_net_init loopback.c:0 [ 107.329548] #11 ffffffff817ec8a4 in ops_init ./net/core/net_namespace.c:107 [ 107.330469] #12 ffffffff817ecac3 in setup_net ./net/core/net_namespace.c:168 [ 107.331359] #13 ffffffff817ed4e4 in copy_net_ns ./net/core/net_namespace.c:255 [ 107.332268] #14 ffffffff81123980 in create_new_namespaces ./kernel/nsproxy.c:94 [ 107.333295] #15 ffffffff81123ccc in unshare_nsproxy_namespaces ./kernel/nsproxy.c:201 [ 107.334346] [ 107.334557] Shadow bytes around the buggy address: [ 107.335472] ffff8800320f7900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [ 107.336699] ffff8800320f7980: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa [ 107.337904] ffff8800320f7a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [ 107.339209] ffff8800320f7a80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [ 107.340423] ffff8800320f7b00: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd [ 107.341649] =>ffff8800320f7b80:[fd]fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa [ 107.342859] ffff8800320f7c00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [ 107.344129] ffff8800320f7c80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [ 107.345341] ffff8800320f7d00: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa [ 107.346557] ffff8800320f7d80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [ 107.347771] ffff8800320f7e00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [ 107.348953] Shadow byte legend (one shadow byte represents 8 application bytes): [ 107.350093] Addressable: 00 [ 107.350603] Partially addressable: 01 02 03 04 05 06 07 [ 107.351485] Heap redzone: fa [ 107.351994] Heap kmalloc redzone: fb [ 107.352677] Freed heap region: fd [ 107.353355] Shadow gap: fe On Fri, Aug 23, 2013 at 7:48 PM, Dmitry Vyukov <dvyukov@google.com> wrote: > On Wed, Aug 21, 2013 at 8:35 PM, Dmitry Vyukov <dvyukov@google.com> wrote: >> Hi, >> >> I'm working on a memory error detector AddressSanitizer for Linux >> kernel (https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel), >> it can detect use-after-free and buffer-overflow errors. Currently the >> tool is in very early stage and it can contain bugs. >> >> Here is one of the reports produced during testing: >> >> [ 196.951434] ERROR: AddressSanitizer: heap-use-after-free on address >> ffff880008a632c4 >> [ 196.952135] Stack trace: >> [ 196.952380] [<ffffffff810dd1f5>] asan_report_error+0x85/0x2c0 >> [ 196.952890] [<ffffffff810dc700>] asan_check_region+0x30/0x40 >> [ 196.953466] [<ffffffff810dd553>] __tsan_write4+0x13/0x20 >> [ 196.953987] [<ffffffff8110a76a>] ____call_usermodehelper+0x21a/0x240 >> [ 196.954651] [<ffffffff8110a7cc>] call_helper+0x3c/0x50 >> [ 196.955155] [<ffffffff81924b5c>] ret_from_fork+0x7c/0xb0 >> [ 196.955686] [<ffffffffffffffff>] 0xffffffffffffffff >> [ 196.956230] Free stack trace: >> [ 196.956532] [<ffffffff810dc831>] asan_slab_free+0x61/0xb0 >> [ 196.957052] [<ffffffff8128070a>] kfree+0x9a/0x240 >> [ 196.957558] [<ffffffff8110a165>] call_usermodehelper_freeinfo+0x35/0x40 >> [ 196.958308] [<ffffffff8110aabe>] call_usermodehelper_exec+0xae/0x1d0 >> [ 196.958920] [<ffffffff8110af81>] call_usermodehelper+0x61/0x90 >> [ 196.959490] [<ffffffff814d7e4e>] kobject_uevent_env+0x5be/0x5f0 >> [ 196.960161] [<ffffffff814d7ea3>] kobject_uevent+0x23/0x40 >> [ 196.960706] [<ffffffff814d63ad>] kobject_release+0xad/0xc0 >> [ 196.961274] [<ffffffff814d618a>] kobject_put+0x3a/0x80 >> [ 196.961889] [<ffffffff8181af6c>] net_rx_queue_update_kobjects+0x12c/0x170 >> [ 196.962701] [<ffffffff8181b1b2>] netdev_unregister_kobject+0x62/0xa0 >> [ 196.963475] [<ffffffff817f342b>] rollback_registered_many+0x27b/0x340 >> [ 196.964175] [<ffffffff817f35d5>] unregister_netdevice_many+0x35/0xe0 >> [ 196.964836] [<ffffffff817f43f7>] default_device_exit_batch+0x107/0x180 >> [ 196.965568] [<ffffffff817ebb1c>] ops_exit_list.isra.3+0x8c/0xa0 >> >> >> I've looked at the sources, but I can't say that I fully understand >> them. The report looks valid, though. I see several potential issues >> in the code. >> >> 1. When wait=UMH_WAIT_EXEC and do_execve() fails, >> ____call_usermodehelper() writes sub_info->retval=retval to freed >> memory. This is the use-after-free reported by the tool. >> >> 2. When wait=UMH_NO_WAIT, __call_usermodehelper() starts child thread >> and instantly frees subprocess_info. The child thread reads >> subprocess_info. Looks like another use-after-free. >> >> 3. UMH_WAIT_EXEC does not actually wait for exec, it only waits for >> starting the child thread that will do exec. I don't know whether it's >> a problem with the code or with the name. >> >> The kernel version is 3.11-rc4 (last commit: >> b7bc9e7d808ba55729bd263b0210cda36965be32). >> >> Please help to confirm these issues, and advice what to do next with them. >> >> TIA > > > > Here is more detailed report with allocation stack and thread ids: > > [ 957.069245] ERROR: AddressSanitizer: heap-use-after-free on address > ffff8800148452c4 > [ 957.070438] Accessed by thread T22400: > [ 957.071134] #0 ffffffff810dd260 (asan_report_error+0xb0/0x250) > [ 957.071947] #1 ffffffff810dc6d0 (asan_check_region+0x30/0x40) > [ 957.072766] #2 ffffffff810dd523 (__tsan_write4+0x13/0x20) > [ 957.073559] #3 ffffffff8110a77a (____call_usermodehelper+0x21a/0x240) > [ 957.074488] #4 ffffffff8110a7dc (call_helper+0x3c/0x50) > [ 957.075266] #5 ffffffff8192569c (ret_from_fork+0x7c/0xb0) > > [ 957.076006] Allocated by thread T506: > [ 957.076662] #0 ffffffff810dc794 (asan_slab_alloc+0x44/0xa0) > [ 957.077475] #1 ffffffff812821bc (__kmalloc+0xbc/0x500) > [ 957.078239] #2 ffffffff8110a1c4 (call_usermodehelper_setup+0x44/0x120) > [ 957.079173] #3 ffffffff8110af82 (call_usermodehelper+0x52/0x90) > [ 957.081469] #4 ffffffff814d831e (kobject_uevent_env+0x5be/0x5f0) > [ 957.082329] #5 ffffffff814d8373 (kobject_uevent+0x23/0x40) > [ 957.083116] #6 ffffffff814d687d (kobject_release+0xad/0xc0) > [ 957.085582] #7 ffffffff814d665a (kobject_put+0x3a/0x80) > [ 957.086348] #8 ffffffff8181b873 (netdev_queue_update_kobjects+0x133/0x1a0) > [ 957.087324] #9 ffffffff8181b94f (netdev_unregister_kobject+0x6f/0xa0) > [ 957.088232] #10 ffffffff817f3b9b (rollback_registered_many+0x27b/0x340) > [ 957.089159] #11 ffffffff817f3d45 (unregister_netdevice_many+0x35/0xe0) > [ 957.090091] #12 ffffffff817f4b67 (default_device_exit_batch+0x107/0x180) > [ 957.091010] #13 ffffffff817ec27c (ops_exit_list.isra.3+0x8c/0xa0) > [ 957.091899] #14 ffffffff817ed0c9 (cleanup_net+0x229/0x390) > [ 957.092699] #15 ffffffff811113af (process_one_work+0x2cf/0x750) > > [ 957.093555] Freed by thread T506: > [ 957.094185] #0 ffffffff810dc867 (asan_slab_free+0x77/0xb0) > [ 957.094958] #1 ffffffff8128081a (kfree+0x9a/0x240) > [ 957.095659] #2 ffffffff8110a175 (call_usermodehelper_freeinfo+0x35/0x40) > [ 957.096608] #3 ffffffff8110aace (call_usermodehelper_exec+0xae/0x1d0) > [ 957.097520] #4 ffffffff8110af91 (call_usermodehelper+0x61/0x90) > [ 957.098377] #5 ffffffff814d831e (kobject_uevent_env+0x5be/0x5f0) > [ 957.099235] #6 ffffffff814d8373 (kobject_uevent+0x23/0x40) > [ 957.099984] #7 ffffffff814d687d (kobject_release+0xad/0xc0) > [ 957.100781] #8 ffffffff814d665a (kobject_put+0x3a/0x80) > [ 957.101541] #9 ffffffff8181b873 (netdev_queue_update_kobjects+0x133/0x1a0) > [ 957.102515] #10 ffffffff8181b94f (netdev_unregister_kobject+0x6f/0xa0) > [ 957.103460] #11 ffffffff817f3b9b (rollback_registered_many+0x27b/0x340) > [ 957.104419] #12 ffffffff817f3d45 (unregister_netdevice_many+0x35/0xe0) > [ 957.105352] #13 ffffffff817f4b67 (default_device_exit_batch+0x107/0x180) > [ 957.106342] #14 ffffffff817ec27c (ops_exit_list.isra.3+0x8c/0xa0) > [ 957.107182] #15 ffffffff817ed0c9 (cleanup_net+0x229/0x390) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential use-after-free in ____call_usermodehelper 2013-09-03 13:49 ` Dmitry Vyukov @ 2013-09-03 20:27 ` Dan Aloni 2013-09-03 20:49 ` Dmitry Vyukov 0 siblings, 1 reply; 7+ messages in thread From: Dan Aloni @ 2013-09-03 20:27 UTC (permalink / raw) To: Dmitry Vyukov Cc: LKML, Andrey Konovalov, Kostya Serebryany, Alexander Potapenko, Evgeniy Stepanov On Tue, Sep 03, 2013 at 05:49:03PM +0400, Dmitry Vyukov wrote: > Is anybody reading this? Is it a correct place to post such things? > Maybe there is a more appropriate place? This is the correct place, and people are reading this. However, the bug is not an obvious one, and you mentioned that this bug reproduces on a tree with significant deviation from vanilla in core kernel code (i.e. memory allocations) with potential bugs, and it also relies on a gcc feature not used by the kernel yet. Because of that it went down in priority compared to other kernel bugs that are currently being investigated. Your work and contribution is appreciated nonetheless, but your bug report needs to pertain closer to the work the core kernel hackers are doing. Despite that, I did look into your report, and I'll address your analysis: > >> I've looked at the sources, but I can't say that I fully understand > >> them. The report looks valid, though. I see several potential issues > >> in the code. > >> > >> 1. When wait=UMH_WAIT_EXEC and do_execve() fails, > >> ____call_usermodehelper() writes sub_info->retval=retval to freed > >> memory. This is the use-after-free reported by the tool. The 'pid = kernel_thread(call_helper, sub_info, CLONE_VFORK | SIGCHLD);' call is designed to block until either do_execve() succeeds or the kernel thread exits - that happens after the store that triggers the issue, so the use-after-free cannot occur in that case. This is thanks to the CLONE_VFORK flag, and I doubt it is broken. > >> 2. When wait=UMH_NO_WAIT, __call_usermodehelper() starts child thread > >> and instantly frees subprocess_info. The child thread reads > >> subprocess_info. Looks like another use-after-free. Same, kernel_thread() would be blocking because of CLONE_VFORK. > >> 3. UMH_WAIT_EXEC does not actually wait for exec, it only waits for > >> starting the child thread that will do exec. I don't know whether it's > >> a problem with the code or with the name. Same. > >> > >> The kernel version is 3.11-rc4 (last commit: > >> b7bc9e7d808ba55729bd263b0210cda36965be32). > >> > >> Please help to confirm these issues, and advice what to do next with them. Theory aside, I tried in practice to confirm the suggested issues and was unsuccessful, CLONE_VFORK seems to work as expected. Note that /sys/kernel/uevent_helper is empty in recent distros, so I had to put a non-existing executable path in there, for the relevant code path to run. Concerning the original reporting, my suggestion is that you take the patch below or a similar idea and try to reproduce the assert it adds with the *vanilla* kernel. i.e, take the unmodified b7bc9e7d808ba5 version (or better - v3.11), and apply it. Also, you must remove the standard memory debugging options from the config of the kernel you that build, so that kfree would not zero out the memory (I think this is the default behavior in non-debug kernel, for performance). diff --git a/kernel/kmod.c b/kernel/kmod.c index 8241906..d10eab6 100644 --- a/kernel/kmod.c +++ b/kernel/kmod.c @@ -244,6 +244,7 @@ static int ____call_usermodehelper(void *data) /* Exec failed? */ fail: + BUG_ON(sub_info->retval == 0x12345678); sub_info->retval = retval; do_exit(0); } @@ -259,6 +260,7 @@ static void call_usermodehelper_freeinfo(struct subprocess_info *info) { if (info->cleanup) (*info->cleanup)(info); + info->retval = 0x12345678; kfree(info); } This should help to prove it or not, and if it does prove it would be appealing to more eyes. Please try the same approach with similar future issues. -- Dan Aloni ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: Potential use-after-free in ____call_usermodehelper 2013-09-03 20:27 ` Dan Aloni @ 2013-09-03 20:49 ` Dmitry Vyukov 2013-09-03 21:27 ` Lucas De Marchi 2013-09-04 4:55 ` Dan Aloni 0 siblings, 2 replies; 7+ messages in thread From: Dmitry Vyukov @ 2013-09-03 20:49 UTC (permalink / raw) To: Dan Aloni Cc: LKML, Andrey Konovalov, Kostya Serebryany, Alexander Potapenko, Evgeniy Stepanov On Wed, Sep 4, 2013 at 12:27 AM, Dan Aloni <alonid@stratoscale.com> wrote: > On Tue, Sep 03, 2013 at 05:49:03PM +0400, Dmitry Vyukov wrote: >> Is anybody reading this? Is it a correct place to post such things? >> Maybe there is a more appropriate place? > > This is the correct place, and people are reading this. However, the > bug is not an obvious one, and you mentioned that this bug reproduces > on a tree with significant deviation from vanilla in core kernel code > (i.e. memory allocations) with potential bugs, and it also relies on a > gcc feature not used by the kernel yet. > > Because of that it went down in priority compared to other kernel bugs > that are currently being investigated. Thanks Dan! I am new here, so I was thinking maybe I am doing something wrong. > Your work and contribution is appreciated nonetheless, but your bug > report needs to pertain closer to the work the core kernel hackers > are doing. What exactly do you mean? > Despite that, I did look into your report, and I'll address your analysis: > >> >> I've looked at the sources, but I can't say that I fully understand >> >> them. The report looks valid, though. I see several potential issues >> >> in the code. >> >> >> >> 1. When wait=UMH_WAIT_EXEC and do_execve() fails, >> >> ____call_usermodehelper() writes sub_info->retval=retval to freed >> >> memory. This is the use-after-free reported by the tool. > > The 'pid = kernel_thread(call_helper, sub_info, CLONE_VFORK | SIGCHLD);' > call is designed to block until either do_execve() succeeds or the kernel > thread exits - that happens after the store that triggers the issue, > so the use-after-free cannot occur in that case. > > This is thanks to the CLONE_VFORK flag, and I doubt it is broken. Aha! I've missed VFORK. It's all like in userspace, right? I will redo my analysis. >> >> 2. When wait=UMH_NO_WAIT, __call_usermodehelper() starts child thread >> >> and instantly frees subprocess_info. The child thread reads >> >> subprocess_info. Looks like another use-after-free. > > Same, kernel_thread() would be blocking because of CLONE_VFORK. > >> >> 3. UMH_WAIT_EXEC does not actually wait for exec, it only waits for >> >> starting the child thread that will do exec. I don't know whether it's >> >> a problem with the code or with the name. > > Same. > >> >> >> >> The kernel version is 3.11-rc4 (last commit: >> >> b7bc9e7d808ba55729bd263b0210cda36965be32). >> >> >> >> Please help to confirm these issues, and advice what to do next with them. > > Theory aside, I tried in practice to confirm the suggested issues and was > unsuccessful, CLONE_VFORK seems to work as expected. Note that > /sys/kernel/uevent_helper is empty in recent distros, so I had to put a > non-existing executable path in there, for the relevant code path to run. > > Concerning the original reporting, my suggestion is that you take the > patch below or a similar idea and try to reproduce the assert it adds > with the *vanilla* kernel. i.e, take the unmodified b7bc9e7d808ba5 version > (or better - v3.11), and apply it. Also, you must remove the standard > memory debugging options from the config of the kernel you that build, so > that kfree would not zero out the memory (I think this is the default > behavior in non-debug kernel, for performance). > > diff --git a/kernel/kmod.c b/kernel/kmod.c > index 8241906..d10eab6 100644 > --- a/kernel/kmod.c > +++ b/kernel/kmod.c > @@ -244,6 +244,7 @@ static int ____call_usermodehelper(void *data) > > /* Exec failed? */ > fail: > + BUG_ON(sub_info->retval == 0x12345678); > sub_info->retval = retval; > do_exit(0); > } > @@ -259,6 +260,7 @@ static void call_usermodehelper_freeinfo(struct subprocess_info *info) > { > if (info->cleanup) > (*info->cleanup)(info); > + info->retval = 0x12345678; > kfree(info); > } > > This should help to prove it or not, and if it does prove it would be > appealing to more eyes. Please try the same approach with similar future > issues. OK, I will try to do it. It may be possible for this "bug", because it's triggered quite often. It may not be possible for all similar issues, because I've seen some of them only once. Btw, is it better to report such things here or file on bugzilla.kernel.org? Thanks again! ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential use-after-free in ____call_usermodehelper 2013-09-03 20:49 ` Dmitry Vyukov @ 2013-09-03 21:27 ` Lucas De Marchi 2013-09-04 4:55 ` Dan Aloni 1 sibling, 0 replies; 7+ messages in thread From: Lucas De Marchi @ 2013-09-03 21:27 UTC (permalink / raw) To: Dmitry Vyukov Cc: Dan Aloni, LKML, Andrey Konovalov, Kostya Serebryany, Alexander Potapenko, Evgeniy Stepanov, oleg On Tue, Sep 3, 2013 at 5:49 PM, Dmitry Vyukov <dvyukov@google.com> wrote: > On Wed, Sep 4, 2013 at 12:27 AM, Dan Aloni <alonid@stratoscale.com> wrote: >> On Tue, Sep 03, 2013 at 05:49:03PM +0400, Dmitry Vyukov wrote: >>> Is anybody reading this? Is it a correct place to post such things? >>> Maybe there is a more appropriate place? >> >> This is the correct place, and people are reading this. However, the >> bug is not an obvious one, and you mentioned that this bug reproduces >> on a tree with significant deviation from vanilla in core kernel code >> (i.e. memory allocations) with potential bugs, and it also relies on a >> gcc feature not used by the kernel yet. >> >> Because of that it went down in priority compared to other kernel bugs >> that are currently being investigated. > > Thanks Dan! > > I am new here, so I was thinking maybe I am doing something wrong. Another thing is.... try to put in CC who touched this function lately. I remember touching it some time ago and also Oleg (adding him). > > >> Your work and contribution is appreciated nonetheless, but your bug >> report needs to pertain closer to the work the core kernel hackers >> are doing. > > What exactly do you mean? > > >> Despite that, I did look into your report, and I'll address your analysis: >> >>> >> I've looked at the sources, but I can't say that I fully understand >>> >> them. The report looks valid, though. I see several potential issues >>> >> in the code. >>> >> >>> >> 1. When wait=UMH_WAIT_EXEC and do_execve() fails, >>> >> ____call_usermodehelper() writes sub_info->retval=retval to freed >>> >> memory. This is the use-after-free reported by the tool. >> >> The 'pid = kernel_thread(call_helper, sub_info, CLONE_VFORK | SIGCHLD);' >> call is designed to block until either do_execve() succeeds or the kernel >> thread exits - that happens after the store that triggers the issue, >> so the use-after-free cannot occur in that case. >> >> This is thanks to the CLONE_VFORK flag, and I doubt it is broken. > > Aha! I've missed VFORK. It's all like in userspace, right? I will redo > my analysis. yes... the VFORK here is what you were missing. > > >>> >> 2. When wait=UMH_NO_WAIT, __call_usermodehelper() starts child thread >>> >> and instantly frees subprocess_info. The child thread reads >>> >> subprocess_info. Looks like another use-after-free. >> >> Same, kernel_thread() would be blocking because of CLONE_VFORK. >> >>> >> 3. UMH_WAIT_EXEC does not actually wait for exec, it only waits for >>> >> starting the child thread that will do exec. I don't know whether it's >>> >> a problem with the code or with the name. >> >> Same. Agreed Lucas De Marchi ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential use-after-free in ____call_usermodehelper 2013-09-03 20:49 ` Dmitry Vyukov 2013-09-03 21:27 ` Lucas De Marchi @ 2013-09-04 4:55 ` Dan Aloni 1 sibling, 0 replies; 7+ messages in thread From: Dan Aloni @ 2013-09-04 4:55 UTC (permalink / raw) To: Dmitry Vyukov Cc: LKML, Andrey Konovalov, Kostya Serebryany, Alexander Potapenko, Evgeniy Stepanov On Wed, Sep 04, 2013 at 12:49:58AM +0400, Dmitry Vyukov wrote: > On Wed, Sep 4, 2013 at 12:27 AM, Dan Aloni <alonid@stratoscale.com> wrote: > > Your work and contribution is appreciated nonetheless, but your bug > > report needs to pertain closer to the work the core kernel hackers > > are doing. > > What exactly do you mean? The kernel hackers are more likely to look into the issue when it is based on their development trees or stable tree. i.e the minimal 'git diff' from one of their versions is the smallest. > Btw, is it better to report such things here or file on bugzilla.kernel.org? See the REPORTING-BUGS file in the source tree. -- Dan Aloni ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-09-04 4:55 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-08-21 16:35 Potential use-after-free in ____call_usermodehelper Dmitry Vyukov 2013-08-23 15:48 ` Dmitry Vyukov 2013-09-03 13:49 ` Dmitry Vyukov 2013-09-03 20:27 ` Dan Aloni 2013-09-03 20:49 ` Dmitry Vyukov 2013-09-03 21:27 ` Lucas De Marchi 2013-09-04 4:55 ` Dan Aloni
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox