From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f45.google.com (mail-pb0-f45.google.com [209.85.160.45]) by kanga.kvack.org (Postfix) with ESMTP id 3056F6B0035 for ; Mon, 20 Jan 2014 05:37:41 -0500 (EST) Received: by mail-pb0-f45.google.com with SMTP id un15so3157212pbc.18 for ; Mon, 20 Jan 2014 02:37:40 -0800 (PST) Received: from szxga03-in.huawei.com (szxga03-in.huawei.com. [119.145.14.66]) by mx.google.com with ESMTPS id ot3si925436pac.137.2014.01.20.02.37.35 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 20 Jan 2014 02:37:39 -0800 (PST) Message-ID: <52DCFC33.80008@huawei.com> Date: Mon, 20 Jan 2014 18:36:35 +0800 From: Jianguo Wu MIME-Version: 1.0 Subject: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton , Johannes Weiner , Rik van Riel , David Rientjes , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" When OOM happen, will dump buddy free areas info, hugetlb pages info, memory state of all eligible tasks, per-cpu memory info. But do not dump slab/vmalloc info, sometime, it's not enough to figure out the reason OOM happened. So, my questions are: 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file, but usually we do not monitor the logs and check proc file immediately when OOM happened. 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped? 3. Without these info, usually how to figure out OOM reason? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f47.google.com (mail-yh0-f47.google.com [209.85.213.47]) by kanga.kvack.org (Postfix) with ESMTP id C9C6D6B0035 for ; Tue, 21 Jan 2014 00:34:48 -0500 (EST) Received: by mail-yh0-f47.google.com with SMTP id c41so771204yho.34 for ; Mon, 20 Jan 2014 21:34:48 -0800 (PST) Received: from mail-yk0-x230.google.com (mail-yk0-x230.google.com [2607:f8b0:4002:c07::230]) by mx.google.com with ESMTPS id 21si1947098yhx.156.2014.01.20.21.34.47 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 20 Jan 2014 21:34:47 -0800 (PST) Received: by mail-yk0-f176.google.com with SMTP id 131so5453512ykp.7 for ; Mon, 20 Jan 2014 21:34:47 -0800 (PST) Date: Mon, 20 Jan 2014 21:34:44 -0800 (PST) From: David Rientjes Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? In-Reply-To: <52DCFC33.80008@huawei.com> Message-ID: References: <52DCFC33.80008@huawei.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Jianguo Wu Cc: Andrew Morton , Johannes Weiner , Rik van Riel , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" On Mon, 20 Jan 2014, Jianguo Wu wrote: > When OOM happen, will dump buddy free areas info, hugetlb pages info, > memory state of all eligible tasks, per-cpu memory info. > But do not dump slab/vmalloc info, sometime, it's not enough to figure out the > reason OOM happened. > > So, my questions are: > 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file, > but usually we do not monitor the logs and check proc file immediately when OOM happened. > The problem is that slabinfo becomes excessively verbose and dumping it all to the kernel log often times causes important messages to be lost. This is why we control things like the tasklist dump with a VM sysctl. It would be possible to dump, say, the top ten slab caches with the highest memory usage, but it will only be helpful for slab leaks. Typically there are better debugging tools available than analyzing the kernel log; if you see unusually high slab memory in the meminfo dump, you can enable it. > 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped? > Also very verbose and would cause important messages to be lost, we try to avoid spamming the kernel log with all of this information as much as possible. > 3. Without these info, usually how to figure out OOM reason? > Analyze the memory usage in the meminfo and determine what is unusually high; if it's mostly anonymous memory, you can usually correlate it back to a high rss for a process in the tasklist that you didn't suspect to be using that much memory, for example. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f53.google.com (mail-yh0-f53.google.com [209.85.213.53]) by kanga.kvack.org (Postfix) with ESMTP id B918F6B0035 for ; Tue, 21 Jan 2014 07:40:40 -0500 (EST) Received: by mail-yh0-f53.google.com with SMTP id b20so2421345yha.26 for ; Tue, 21 Jan 2014 04:40:40 -0800 (PST) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com. [119.145.14.64]) by mx.google.com with ESMTPS id t26si5606262yhl.105.2014.01.21.04.40.21 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 21 Jan 2014 04:40:39 -0800 (PST) Message-ID: <52DE6AA0.1000801@huawei.com> Date: Tue, 21 Jan 2014 20:40:00 +0800 From: Jianguo Wu MIME-Version: 1.0 Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? References: <52DCFC33.80008@huawei.com> In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: David Rientjes Cc: Andrew Morton , Johannes Weiner , Rik van Riel , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" On 2014/1/21 13:34, David Rientjes wrote: > On Mon, 20 Jan 2014, Jianguo Wu wrote: > >> When OOM happen, will dump buddy free areas info, hugetlb pages info, >> memory state of all eligible tasks, per-cpu memory info. >> But do not dump slab/vmalloc info, sometime, it's not enough to figure out the >> reason OOM happened. >> >> So, my questions are: >> 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file, >> but usually we do not monitor the logs and check proc file immediately when OOM happened. >> > Hi David, Thank you for your patience to answer! > The problem is that slabinfo becomes excessively verbose and dumping it > all to the kernel log often times causes important messages to be lost. > This is why we control things like the tasklist dump with a VM sysctl. It > would be possible to dump, say, the top ten slab caches with the highest > memory usage, but it will only be helpful for slab leaks. Typically there > are better debugging tools available than analyzing the kernel log; if you > see unusually high slab memory in the meminfo dump, you can enable it. > But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys? Thanks. >> 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped? >> > > Also very verbose and would cause important messages to be lost, we try to > avoid spamming the kernel log with all of this information as much as > possible. > >> 3. Without these info, usually how to figure out OOM reason? >> > > Analyze the memory usage in the meminfo and determine what is unusually > high; if it's mostly anonymous memory, you can usually correlate it back > to a high rss for a process in the tasklist that you didn't suspect to be > using that much memory, for example. > > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f42.google.com (mail-yh0-f42.google.com [209.85.213.42]) by kanga.kvack.org (Postfix) with ESMTP id 2F1806B0062 for ; Tue, 21 Jan 2014 15:41:47 -0500 (EST) Received: by mail-yh0-f42.google.com with SMTP id a41so1247404yho.29 for ; Tue, 21 Jan 2014 12:41:46 -0800 (PST) Received: from mail-yh0-x234.google.com (mail-yh0-x234.google.com [2607:f8b0:4002:c01::234]) by mx.google.com with ESMTPS id v21si7416893yhm.73.2014.01.21.12.41.44 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 21 Jan 2014 12:41:45 -0800 (PST) Received: by mail-yh0-f52.google.com with SMTP id v1so2985635yhn.39 for ; Tue, 21 Jan 2014 12:41:44 -0800 (PST) Date: Tue, 21 Jan 2014 12:41:41 -0800 (PST) From: David Rientjes Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? In-Reply-To: <52DE6AA0.1000801@huawei.com> Message-ID: References: <52DCFC33.80008@huawei.com> <52DE6AA0.1000801@huawei.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Jianguo Wu Cc: Andrew Morton , Johannes Weiner , Rik van Riel , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue, 21 Jan 2014, Jianguo Wu wrote: > > The problem is that slabinfo becomes excessively verbose and dumping it > > all to the kernel log often times causes important messages to be lost. > > This is why we control things like the tasklist dump with a VM sysctl. It > > would be possible to dump, say, the top ten slab caches with the highest > > memory usage, but it will only be helpful for slab leaks. Typically there > > are better debugging tools available than analyzing the kernel log; if you > > see unusually high slab memory in the meminfo dump, you can enable it. > > > > But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc > is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys? > You could, but it's a tradeoff between how much to dump to a general resource such as the kernel log and how many sysctls we add that control every possible thing. Slab leaks would definitely be a minority of oom conditions and you should normally be able to reproduce them by running the same workload; just use slabtop(1) or manually inspect /proc/slabinfo while such a workload is running for indicators. I don't think we want to add the information by default, though, nor do we want to add sysctls to control the behavior (you'd still need to reproduce the issue after enabling it). We are currently discussing userspace oom handlers, though, that would allow you to run a process that would be notified and allowed to allocate a small amount of memory on oom conditions. It would then be trivial to dump any information you feel pertinent in userspace prior to killing something. I like to inspect heap profiles for memory hogs while debugging our malloc() issues, for example, and you could look more closely at kernel memory. I'll cc you on future discussions of that feature. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f52.google.com (mail-pb0-f52.google.com [209.85.160.52]) by kanga.kvack.org (Postfix) with ESMTP id E9A666B0031 for ; Mon, 10 Feb 2014 23:07:42 -0500 (EST) Received: by mail-pb0-f52.google.com with SMTP id jt11so7174432pbb.39 for ; Mon, 10 Feb 2014 20:07:42 -0800 (PST) Received: from szxga03-in.huawei.com (szxga03-in.huawei.com. [119.145.14.66]) by mx.google.com with ESMTPS id zk9si17487985pac.260.2014.02.10.20.07.18 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 10 Feb 2014 20:07:42 -0800 (PST) Message-ID: <52F9A1D8.7040301@huawei.com> Date: Tue, 11 Feb 2014 12:06:48 +0800 From: Jianguo Wu MIME-Version: 1.0 Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? References: <52DCFC33.80008@huawei.com> <52DE6AA0.1000801@huawei.com> In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: David Rientjes Cc: Andrew Morton , Johannes Weiner , Rik van Riel , linux-mm@kvack.org, linux-kernel@vger.kernel.org On 2014/1/22 4:41, David Rientjes wrote: > On Tue, 21 Jan 2014, Jianguo Wu wrote: > >>> The problem is that slabinfo becomes excessively verbose and dumping it >>> all to the kernel log often times causes important messages to be lost. >>> This is why we control things like the tasklist dump with a VM sysctl. It >>> would be possible to dump, say, the top ten slab caches with the highest >>> memory usage, but it will only be helpful for slab leaks. Typically there >>> are better debugging tools available than analyzing the kernel log; if you >>> see unusually high slab memory in the meminfo dump, you can enable it. >>> >> >> But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc >> is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys? >> > > You could, but it's a tradeoff between how much to dump to a general > resource such as the kernel log and how many sysctls we add that control > every possible thing. Slab leaks would definitely be a minority of oom > conditions and you should normally be able to reproduce them by running > the same workload; just use slabtop(1) or manually inspect /proc/slabinfo > while such a workload is running for indicators. I don't think we want to > add the information by default, though, nor do we want to add sysctls to > control the behavior (you'd still need to reproduce the issue after > enabling it). > > We are currently discussing userspace oom handlers, though, that would > allow you to run a process that would be notified and allowed to allocate > a small amount of memory on oom conditions. It would then be trivial to > dump any information you feel pertinent in userspace prior to killing > something. I like to inspect heap profiles for memory hogs while > debugging our malloc() issues, for example, and you could look more > closely at kernel memory. > > I'll cc you on future discussions of that feature. > Hi David, Thanks for your kindly explanation, do you have any specific plans on this? Thanks, Jianguo Wu. > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f45.google.com (mail-pa0-f45.google.com [209.85.220.45]) by kanga.kvack.org (Postfix) with ESMTP id 10C546B0037 for ; Tue, 11 Feb 2014 19:28:03 -0500 (EST) Received: by mail-pa0-f45.google.com with SMTP id lf10so8400225pab.32 for ; Tue, 11 Feb 2014 16:28:03 -0800 (PST) Received: from mail-pa0-x231.google.com (mail-pa0-x231.google.com [2607:f8b0:400e:c03::231]) by mx.google.com with ESMTPS id wm3si20599197pab.223.2014.02.11.16.28.02 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 11 Feb 2014 16:28:03 -0800 (PST) Received: by mail-pa0-f49.google.com with SMTP id hz1so8382191pad.22 for ; Tue, 11 Feb 2014 16:28:02 -0800 (PST) Date: Tue, 11 Feb 2014 16:28:00 -0800 (PST) From: David Rientjes Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? In-Reply-To: <52F9A1D8.7040301@huawei.com> Message-ID: References: <52DCFC33.80008@huawei.com> <52DE6AA0.1000801@huawei.com> <52F9A1D8.7040301@huawei.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Jianguo Wu Cc: Andrew Morton , Johannes Weiner , Rik van Riel , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue, 11 Feb 2014, Jianguo Wu wrote: > Thanks for your kindly explanation, do you have any specific plans on this? > We're going to be discussing it at the LSF/mm conference at the end of March. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753254AbaATKjz (ORCPT ); Mon, 20 Jan 2014 05:39:55 -0500 Received: from szxga03-in.huawei.com ([119.145.14.66]:6535 "EHLO szxga03-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751797AbaATKjv (ORCPT ); Mon, 20 Jan 2014 05:39:51 -0500 Message-ID: <52DCFC33.80008@huawei.com> Date: Mon, 20 Jan 2014 18:36:35 +0800 From: Jianguo Wu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Andrew Morton , Johannes Weiner , Rik van Riel , David Rientjes , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" Subject: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.177.24.155] X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When OOM happen, will dump buddy free areas info, hugetlb pages info, memory state of all eligible tasks, per-cpu memory info. But do not dump slab/vmalloc info, sometime, it's not enough to figure out the reason OOM happened. So, my questions are: 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file, but usually we do not monitor the logs and check proc file immediately when OOM happened. 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped? 3. Without these info, usually how to figure out OOM reason? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751710AbaAUFey (ORCPT ); Tue, 21 Jan 2014 00:34:54 -0500 Received: from mail-yk0-f171.google.com ([209.85.160.171]:51237 "EHLO mail-yk0-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750740AbaAUFer (ORCPT ); Tue, 21 Jan 2014 00:34:47 -0500 Date: Mon, 20 Jan 2014 21:34:44 -0800 (PST) From: David Rientjes X-X-Sender: rientjes@chino.kir.corp.google.com To: Jianguo Wu cc: Andrew Morton , Johannes Weiner , Rik van Riel , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? In-Reply-To: <52DCFC33.80008@huawei.com> Message-ID: References: <52DCFC33.80008@huawei.com> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 20 Jan 2014, Jianguo Wu wrote: > When OOM happen, will dump buddy free areas info, hugetlb pages info, > memory state of all eligible tasks, per-cpu memory info. > But do not dump slab/vmalloc info, sometime, it's not enough to figure out the > reason OOM happened. > > So, my questions are: > 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file, > but usually we do not monitor the logs and check proc file immediately when OOM happened. > The problem is that slabinfo becomes excessively verbose and dumping it all to the kernel log often times causes important messages to be lost. This is why we control things like the tasklist dump with a VM sysctl. It would be possible to dump, say, the top ten slab caches with the highest memory usage, but it will only be helpful for slab leaks. Typically there are better debugging tools available than analyzing the kernel log; if you see unusually high slab memory in the meminfo dump, you can enable it. > 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped? > Also very verbose and would cause important messages to be lost, we try to avoid spamming the kernel log with all of this information as much as possible. > 3. Without these info, usually how to figure out OOM reason? > Analyze the memory usage in the meminfo and determine what is unusually high; if it's mostly anonymous memory, you can usually correlate it back to a high rss for a process in the tasklist that you didn't suspect to be using that much memory, for example. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754597AbaAUMk4 (ORCPT ); Tue, 21 Jan 2014 07:40:56 -0500 Received: from szxga01-in.huawei.com ([119.145.14.64]:17504 "EHLO szxga01-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754289AbaAUMkx (ORCPT ); Tue, 21 Jan 2014 07:40:53 -0500 Message-ID: <52DE6AA0.1000801@huawei.com> Date: Tue, 21 Jan 2014 20:40:00 +0800 From: Jianguo Wu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: David Rientjes CC: Andrew Morton , Johannes Weiner , Rik van Riel , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? References: <52DCFC33.80008@huawei.com> In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.177.24.155] X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2014/1/21 13:34, David Rientjes wrote: > On Mon, 20 Jan 2014, Jianguo Wu wrote: > >> When OOM happen, will dump buddy free areas info, hugetlb pages info, >> memory state of all eligible tasks, per-cpu memory info. >> But do not dump slab/vmalloc info, sometime, it's not enough to figure out the >> reason OOM happened. >> >> So, my questions are: >> 1. Should dump slab/vmalloc info when OOM happen? Though we can get these from proc file, >> but usually we do not monitor the logs and check proc file immediately when OOM happened. >> > Hi David, Thank you for your patience to answer! > The problem is that slabinfo becomes excessively verbose and dumping it > all to the kernel log often times causes important messages to be lost. > This is why we control things like the tasklist dump with a VM sysctl. It > would be possible to dump, say, the top ten slab caches with the highest > memory usage, but it will only be helpful for slab leaks. Typically there > are better debugging tools available than analyzing the kernel log; if you > see unusually high slab memory in the meminfo dump, you can enable it. > But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys? Thanks. >> 2. /proc/$pid/smaps and pagecache info also helpful when OOM, should also be dumped? >> > > Also very verbose and would cause important messages to be lost, we try to > avoid spamming the kernel log with all of this information as much as > possible. > >> 3. Without these info, usually how to figure out OOM reason? >> > > Analyze the memory usage in the meminfo and determine what is unusually > high; if it's mostly anonymous memory, you can usually correlate it back > to a high rss for a process in the tasklist that you didn't suspect to be > using that much memory, for example. > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753140AbaAUUlr (ORCPT ); Tue, 21 Jan 2014 15:41:47 -0500 Received: from mail-yk0-f170.google.com ([209.85.160.170]:59986 "EHLO mail-yk0-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751873AbaAUUlq (ORCPT ); Tue, 21 Jan 2014 15:41:46 -0500 Date: Tue, 21 Jan 2014 12:41:41 -0800 (PST) From: David Rientjes X-X-Sender: rientjes@chino.kir.corp.google.com To: Jianguo Wu cc: Andrew Morton , Johannes Weiner , Rik van Riel , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? In-Reply-To: <52DE6AA0.1000801@huawei.com> Message-ID: References: <52DCFC33.80008@huawei.com> <52DE6AA0.1000801@huawei.com> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 21 Jan 2014, Jianguo Wu wrote: > > The problem is that slabinfo becomes excessively verbose and dumping it > > all to the kernel log often times causes important messages to be lost. > > This is why we control things like the tasklist dump with a VM sysctl. It > > would be possible to dump, say, the top ten slab caches with the highest > > memory usage, but it will only be helpful for slab leaks. Typically there > > are better debugging tools available than analyzing the kernel log; if you > > see unusually high slab memory in the meminfo dump, you can enable it. > > > > But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc > is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys? > You could, but it's a tradeoff between how much to dump to a general resource such as the kernel log and how many sysctls we add that control every possible thing. Slab leaks would definitely be a minority of oom conditions and you should normally be able to reproduce them by running the same workload; just use slabtop(1) or manually inspect /proc/slabinfo while such a workload is running for indicators. I don't think we want to add the information by default, though, nor do we want to add sysctls to control the behavior (you'd still need to reproduce the issue after enabling it). We are currently discussing userspace oom handlers, though, that would allow you to run a process that would be notified and allowed to allocate a small amount of memory on oom conditions. It would then be trivial to dump any information you feel pertinent in userspace prior to killing something. I like to inspect heap profiles for memory hogs while debugging our malloc() issues, for example, and you could look more closely at kernel memory. I'll cc you on future discussions of that feature. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752379AbaBKEIF (ORCPT ); Mon, 10 Feb 2014 23:08:05 -0500 Received: from szxga03-in.huawei.com ([119.145.14.66]:41201 "EHLO szxga03-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752186AbaBKEIB (ORCPT ); Mon, 10 Feb 2014 23:08:01 -0500 Message-ID: <52F9A1D8.7040301@huawei.com> Date: Tue, 11 Feb 2014 12:06:48 +0800 From: Jianguo Wu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: David Rientjes CC: Andrew Morton , Johannes Weiner , Rik van Riel , , Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? References: <52DCFC33.80008@huawei.com> <52DE6AA0.1000801@huawei.com> In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.177.24.155] X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2014/1/22 4:41, David Rientjes wrote: > On Tue, 21 Jan 2014, Jianguo Wu wrote: > >>> The problem is that slabinfo becomes excessively verbose and dumping it >>> all to the kernel log often times causes important messages to be lost. >>> This is why we control things like the tasklist dump with a VM sysctl. It >>> would be possible to dump, say, the top ten slab caches with the highest >>> memory usage, but it will only be helpful for slab leaks. Typically there >>> are better debugging tools available than analyzing the kernel log; if you >>> see unusually high slab memory in the meminfo dump, you can enable it. >>> >> >> But, when OOM has happened, we can only use kernel log, slab/vmalloc info from proc >> is stale. Maybe we can dump slab/vmalloc with a VM sysctl, and only top 10/20 entrys? >> > > You could, but it's a tradeoff between how much to dump to a general > resource such as the kernel log and how many sysctls we add that control > every possible thing. Slab leaks would definitely be a minority of oom > conditions and you should normally be able to reproduce them by running > the same workload; just use slabtop(1) or manually inspect /proc/slabinfo > while such a workload is running for indicators. I don't think we want to > add the information by default, though, nor do we want to add sysctls to > control the behavior (you'd still need to reproduce the issue after > enabling it). > > We are currently discussing userspace oom handlers, though, that would > allow you to run a process that would be notified and allowed to allocate > a small amount of memory on oom conditions. It would then be trivial to > dump any information you feel pertinent in userspace prior to killing > something. I like to inspect heap profiles for memory hogs while > debugging our malloc() issues, for example, and you could look more > closely at kernel memory. > > I'll cc you on future discussions of that feature. > Hi David, Thanks for your kindly explanation, do you have any specific plans on this? Thanks, Jianguo Wu. > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752549AbaBLA2F (ORCPT ); Tue, 11 Feb 2014 19:28:05 -0500 Received: from mail-pa0-f44.google.com ([209.85.220.44]:65044 "EHLO mail-pa0-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752504AbaBLA2D (ORCPT ); Tue, 11 Feb 2014 19:28:03 -0500 Date: Tue, 11 Feb 2014 16:28:00 -0800 (PST) From: David Rientjes X-X-Sender: rientjes@chino.kir.corp.google.com To: Jianguo Wu cc: Andrew Morton , Johannes Weiner , Rik van Riel , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [question] how to figure out OOM reason? should dump slab/vmalloc info when OOM? In-Reply-To: <52F9A1D8.7040301@huawei.com> Message-ID: References: <52DCFC33.80008@huawei.com> <52DE6AA0.1000801@huawei.com> <52F9A1D8.7040301@huawei.com> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 11 Feb 2014, Jianguo Wu wrote: > Thanks for your kindly explanation, do you have any specific plans on this? > We're going to be discussing it at the LSF/mm conference at the end of March.