From mboxrd@z Thu Jan 1 00:00:00 1970 From: CAI Qian Subject: Re: possible memory leak in ipc Date: Fri, 26 Aug 2016 11:25:56 -0400 (EDT) Message-ID: <364510660.2709914.1472225156558.JavaMail.zimbra@redhat.com> References: <544270328.2536943.1472154674747.JavaMail.zimbra@redhat.com> <1474737839.2538727.1472155133819.JavaMail.zimbra@redhat.com> <2107409749.2541400.1472156272168.JavaMail.zimbra@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: Thomas Graf , Herbert Xu , Eric Dumazet , Network Development To: Linus Torvalds Return-path: Received: from mx5-phx2.redhat.com ([209.132.183.37]:42356 "EHLO mx5-phx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750715AbcHZP1d (ORCPT ); Fri, 26 Aug 2016 11:27:33 -0400 In-Reply-To: Sender: netdev-owner@vger.kernel.org List-ID: ----- Original Message ----- > From: "Linus Torvalds" > To: "CAI Qian" , "Thomas Graf" , "Herbert Xu" > Cc: "Eric Dumazet" , "Network Development" > Sent: Thursday, August 25, 2016 6:20:03 PM > Subject: Re: possible memory leak in ipc > > On Thu, Aug 25, 2016 at 1:17 PM, CAI Qian wrote: > > I am unsure if it is really a memleak (could be a security issue due to > > eventually OOM and DoS) or just a soft lockup with in kmemlock code with > > false alarm. > > Hmm. The reported leaks look like > > unreferenced object 0xffffc90004857000 (size 4608): > comm "kworker/16:0", pid 110, jiffies 4294705908 (age 883.925s) > hex dump (first 32 bytes): > c0 05 3d 5e 08 88 ff ff ff ff ff ff 00 00 dc 6e ..=^...........n > ff ff ff ff ff ff ff ff 28 c7 46 83 ff ff ff ff ........(.F..... > backtrace: > [] kmemleak_alloc+0x4a/0xa0 > [] __vmalloc_node_range+0x1de/0x2f0 > [] vmalloc+0x54/0x60 > [] alloc_bucket_locks.isra.7+0xd4/0xf0 > [] bucket_table_alloc+0x58/0x100 > [] rht_deferred_worker+0x10e/0x890 > [] process_one_work+0x218/0x750 > [] worker_thread+0x125/0x4a0 > [] kthread+0x101/0x120 > [] ret_from_fork+0x1f/0x40 > [] 0xffffffffffffffff > > which would indicate that it's a rhashtable resize event where we > perhaps haven't free'd the old hash table when we create a new one. > > The actually freeing of the old one is done RCU-deferred from > rhashtable_rehash_table(), but that itself is also deferred by a > worker thread (rht_deferred_worker). > > I'm not seeing anything wrong in the logic, but let's bring in Thomas > Graf and Herbert Xu. > > Hmm. The size (4608) is always the same and doesn't change, so maybe > it's not actually a rehash events per se - it's somebody creating a > rhashtable, but perhaps not freeing it? > > Sadly, all but one of the traces are that kthread one, and the one > that isn't that might give an idea about what code triggers this is: > > unreferenced object 0xffffc900048b6000 (size 4608): > comm "modprobe", pid 2485, jiffies 4294727633 (age 862.590s) > hex dump (first 32 bytes): > 00 9c 49 21 00 ea ff ff 00 d5 59 21 00 ea ff ff ..I!......Y!.... > 00 a5 7d 21 00 ea ff ff c0 da 74 21 00 ea ff ff ..}!......t!.... > backtrace: > [] kmemleak_alloc+0x4a/0xa0 > [] __vmalloc_node_range+0x1de/0x2f0 > [] vmalloc+0x54/0x60 > [] alloc_bucket_locks.isra.7+0xd4/0xf0 > [] bucket_table_alloc+0x58/0x100 > [] rhashtable_init+0x1ed/0x390 > [] 0xffffffffa05b201b > [] do_one_initcall+0x50/0x190 > [] do_init_module+0x60/0x1f3 > [] load_module+0x1487/0x1ca0 > [] SYSC_finit_module+0xa6/0xf0 > [] SyS_finit_module+0xe/0x10 > [] do_syscall_64+0x6c/0x1e0 > [] return_from_SYSCALL_64+0x0/0x7a > [] 0xffffffffffffffff > > so it comes from some module init code, but since the module hasn't > fully initialized, the kallsym code doesn't find the symbol name > either. Annoying. > > Maybe the above just makes one of the rhashtable people go "Oh, that's > obvious". > > Linus > FYI, this also happened while compiling gcc. $ make -j 56 $ cat /sys/kernel/debug/kmemleak unreferenced object 0xffffc9000485a000 (size 4608): comm "kworker/7:1", pid 368, jiffies 4294835499 (age 1033.075s) hex dump (first 32 bytes): 5f 65 76 65 6e 74 5f 69 74 65 6d 2e 68 00 05 00 _event_item.h... 00 76 6d 73 74 61 74 2e 68 00 05 00 00 70 6c 69 .vmstat.h....pli backtrace: [] kmemleak_alloc+0x4a/0xa0 [] __vmalloc_node_range+0x378/0x700 [] vmalloc+0x54/0x60 [] alloc_bucket_locks.isra.7+0x188/0x220 [] bucket_table_alloc+0xac/0x290 [] rht_deferred_worker+0x8c5/0x1890 [] process_one_work+0x731/0x16c0 [] worker_thread+0xdc/0xf10 [] kthread+0x223/0x2f0 [] ret_from_fork+0x1f/0x40 [] 0xffffffffffffffff unreferenced object 0xffffc90004861000 (size 4608): comm "kworker/10:1", pid 380, jiffies 4294843418 (age 1025.169s) hex dump (first 32 bytes): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace: [] kmemleak_alloc+0x4a/0xa0 [] __vmalloc_node_range+0x378/0x700 [] vmalloc+0x54/0x60 [] alloc_bucket_locks.isra.7+0x188/0x220 [] bucket_table_alloc+0xac/0x290 [] rht_deferred_worker+0x15d4/0x1890 [] process_one_work+0x731/0x16c0 [] worker_thread+0xdc/0xf10 [] kthread+0x223/0x2f0 [] ret_from_fork+0x1f/0x40 [] 0xffffffffffffffff unreferenced object 0xffffc90004864000 (size 4608): comm "kworker/27:0", pid 176, jiffies 4294866756 (age 1002.065s) hex dump (first 32 bytes): 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 00 15 ......8.|....... 00 00 48 8b 85 f0 fe ff ff 45 85 e4 8b 40 4c 74 ..H......E...@Lt backtrace: [] kmemleak_alloc+0x4a/0xa0 [] __vmalloc_node_range+0x378/0x700 [] vmalloc+0x54/0x60 [] alloc_bucket_locks.isra.7+0x188/0x220 [] bucket_table_alloc+0xac/0x290 [] rht_deferred_worker+0x8c5/0x1890 [] process_one_work+0x731/0x16c0 [] worker_thread+0xdc/0xf10 [] kthread+0x223/0x2f0 [] ret_from_fork+0x1f/0x40 [] 0xffffffffffffffff unreferenced object 0xffffc9000486a000 (size 4608): comm "kworker/3:1", pid 365, jiffies 4294867142 (age 1001.782s) hex dump (first 32 bytes): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 10 10 d2 2a 04 88 ff ff 10 10 d2 2a 04 88 ff ff ...*.......*.... backtrace: [] kmemleak_alloc+0x4a/0xa0 [] __vmalloc_node_range+0x378/0x700 [] vmalloc+0x54/0x60 [] alloc_bucket_locks.isra.7+0x188/0x220 [] bucket_table_alloc+0xac/0x290 [] rht_deferred_worker+0x8c5/0x1890 [] process_one_work+0x731/0x16c0 [] worker_thread+0xdc/0xf10 [] kthread+0x223/0x2f0 [] ret_from_fork+0x1f/0x40 [] 0xffffffffffffffff unreferenced object 0xffffc90004a71000 (size 4608): comm "kworker/16:1", pid 376, jiffies 4294891121 (age 977.869s) hex dump (first 32 bytes): 00 00 00 00 00 00 00 00 00 00 00 00 fe 01 00 00 ................ 00 98 c3 1c 00 ea ff ff 40 9a c3 1c 00 ea ff ff ........@....... backtrace: [] kmemleak_alloc+0x4a/0xa0 [] __vmalloc_node_range+0x378/0x700 [] vmalloc+0x54/0x60 [] alloc_bucket_locks.isra.7+0x188/0x220 [] bucket_table_alloc+0xac/0x290 [] rht_deferred_worker+0x15d4/0x1890 [] process_one_work+0x731/0x16c0 [] worker_thread+0xdc/0xf10 [] kthread+0x223/0x2f0 [] ret_from_fork+0x1f/0x40 [] 0xffffffffffffffff