From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f49.google.com (mail-pa0-f49.google.com [209.85.220.49]) by kanga.kvack.org (Postfix) with ESMTP id 9A63182F64 for ; Wed, 14 Oct 2015 09:41:54 -0400 (EDT) Received: by payp3 with SMTP id p3so7100078pay.1 for ; Wed, 14 Oct 2015 06:41:54 -0700 (PDT) Received: from mailout3.samsung.com (mailout3.samsung.com. [203.254.224.33]) by mx.google.com with ESMTPS id v9si13382183pbs.198.2015.10.14.06.41.53 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 14 Oct 2015 06:41:53 -0700 (PDT) Received: from epcpsbgr1.samsung.com (u141.gpu120.samsung.co.kr [203.254.230.141]) by mailout3.samsung.com (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014)) with ESMTP id <0NW702117Q1RLW80@mailout3.samsung.com> for linux-mm@kvack.org; Wed, 14 Oct 2015 22:41:51 +0900 (KST) From: PINTU KUMAR References: <1444656800-29915-1-git-send-email-pintu.k@samsung.com> <1444660139-30125-1-git-send-email-pintu.k@samsung.com> In-reply-to: Subject: RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter Date: Wed, 14 Oct 2015 19:11:05 +0530 Message-id: <081301d10686$370d2e10$a5278a30$@samsung.com> MIME-version: 1.0 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7bit Content-language: en-us Sender: owner-linux-mm@kvack.org List-ID: To: 'David Rientjes' Cc: akpm@linux-foundation.org, minchan@kernel.org, dave@stgolabs.net, mhocko@suse.cz, koct9i@gmail.com, hannes@cmpxchg.org, penguin-kernel@i-love.sakura.ne.jp, bywxiaobai@163.com, mgorman@suse.de, vbabka@suse.cz, js1304@gmail.com, kirill.shutemov@linux.intel.com, alexander.h.duyck@redhat.com, sasha.levin@oracle.com, cl@linux.com, fengguang.wu@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, cpgs@samsung.com, pintu_agarwal@yahoo.com, pintu.ping@gmail.com, vishnu.ps@samsung.com, rohit.kr@samsung.com, c.rajkumar@samsung.com Hi, Thank you very much for your review and comments. > -----Original Message----- > From: David Rientjes [mailto:rientjes@google.com] > Sent: Wednesday, October 14, 2015 8:36 AM > To: Pintu Kumar > Cc: akpm@linux-foundation.org; minchan@kernel.org; dave@stgolabs.net; > mhocko@suse.cz; koct9i@gmail.com; hannes@cmpxchg.org; penguin-kernel@i- > love.sakura.ne.jp; bywxiaobai@163.com; mgorman@suse.de; vbabka@suse.cz; > js1304@gmail.com; kirill.shutemov@linux.intel.com; > alexander.h.duyck@redhat.com; sasha.levin@oracle.com; cl@linux.com; > fengguang.wu@intel.com; linux-kernel@vger.kernel.org; linux-mm@kvack.org; > cpgs@samsung.com; pintu_agarwal@yahoo.com; pintu.ping@gmail.com; > vishnu.ps@samsung.com; rohit.kr@samsung.com; c.rajkumar@samsung.com; > sreenathd@samsung.com > Subject: Re: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat > counter > > On Mon, 12 Oct 2015, Pintu Kumar wrote: > > > This patch maintains the number of oom victims kill count in > > /proc/vmstat. > > Currently, we are dependent upon kernel logs when the kernel OOM occurs. > > But kernel OOM can went passed unnoticed by the developer as it can > > silently kill some background applications/services. > > In some small embedded system, it might be possible that OOM is > > captured in the logs but it was over-written due to ring-buffer. > > Thus this interface can quickly help the user in analyzing, whether > > there were any OOM kill happened in the past, or whether the system > > have ever entered the oom kill stage till date. > > > > Thus, it can be beneficial under following cases: > > 1. User can monitor kernel oom kill scenario without looking into the > > kernel logs. > > I'm not sure how helpful that would be since we don't know anything about the > oom kill itself, only that at some point during the uptime there were oom kills. > Not sure about others. For me it was very helpful during sluggish and long duration ageing tests. With this, I don't have to look into the logs manually. I just monitor this count in a script. The moment I get nr_oom_victims > 1, I know that kernel OOM would have happened and I need to take the log dump. So, then I do: dmesg >> oom_logs.txt Or, even stop the tests for further tuning. > > 2. It can help in tuning the watermark level in the system. > > I disagree with this one, because we can encounter oom kills due to > fragmentation rather than low memory conditions for high-order allocations. > The amount of free memory may be substantially higher than all zone > watermarks. > AFAIK, kernel oom happens only for lower-order (PAGE_ALLOC_COSTLY_ORDER). For higher-order we get page allocation failure. > > 3. It can help in tuning the low memory killer behavior in user space. > > Same reason as above. > > > 4. It can be helpful on a logless system or if klogd logging > > (/var/log/messages) are disabled. > > > > This would be similar to point (1) above, and I question how helpful it would be. > I notice that all oom kills (system, cpuset, mempolicy, and > memcg) are treated equally in this case and there's no way to differentiate them. > That would lead me to believe that you are targeting this change for systems > that don't use mempolicies or cgroups. That's fine, but I doubt it will be helpful > for anybody else. > No, we are not targeting any specific category. Our goal is simple, track and report kernel oom kill as soon as it occurs. > > A snapshot of the result of 3 days of over night test is shown below: > > System: ARM Cortex A7, 1GB RAM, 8GB EMMC > > Linux: 3.10.xx > > Category: reference smart phone device > > Loglevel: 7 > > Conditions: Fully loaded, BT/WiFi/GPS ON > > Tests: auto launching of ~30+ apps using test scripts, in a loop for > > 3 days. > > At the end of tests, check: > > $ cat /proc/vmstat > > nr_oom_victims 6 > > > > As we noticed, there were around 6 oom kill victims. > > > > The OOM is bad for any system. So, this counter can help in quickly > > tuning the OOM behavior of the system, without depending on the logs. > > > > NACK to the patch since it isn't justified. > > We've long had a desire to have a better oom reporting mechanism rather than > just the kernel log. It seems like you're feeling the same pain. I think it would be > better to have an eventfd notifier for system oom conditions so we can track > kernel oom kills (and conditions) in userspace. I have a patch for that, and it > works quite well when userspace is mlocked with a buffer in memory. > Ok, this would be interesting. Can you point me to the patches? I will quickly check if it is useful for us. > If you are only interested in a strict count of system oom kills, this could then > easily be implemented without adding vmstat counters. > We are interested only to know when kernel OOM occurs and not even the oom victim count. So that we can tune something is user space to avoid or delay it as far as possible. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753802AbbJNNl4 (ORCPT ); Wed, 14 Oct 2015 09:41:56 -0400 Received: from mailout3.samsung.com ([203.254.224.33]:40856 "EHLO mailout3.samsung.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753094AbbJNNlx (ORCPT ); Wed, 14 Oct 2015 09:41:53 -0400 X-AuditID: cbfee68d-f79ae6d00000149a-66-561e5b9fc91f From: PINTU KUMAR To: "'David Rientjes'" Cc: akpm@linux-foundation.org, minchan@kernel.org, dave@stgolabs.net, mhocko@suse.cz, koct9i@gmail.com, hannes@cmpxchg.org, penguin-kernel@i-love.sakura.ne.jp, bywxiaobai@163.com, mgorman@suse.de, vbabka@suse.cz, js1304@gmail.com, kirill.shutemov@linux.intel.com, alexander.h.duyck@redhat.com, sasha.levin@oracle.com, cl@linux.com, fengguang.wu@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, cpgs@samsung.com, pintu_agarwal@yahoo.com, pintu.ping@gmail.com, vishnu.ps@samsung.com, rohit.kr@samsung.com, c.rajkumar@samsung.com References: <1444656800-29915-1-git-send-email-pintu.k@samsung.com> <1444660139-30125-1-git-send-email-pintu.k@samsung.com> In-reply-to: Subject: RE: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat counter Date: Wed, 14 Oct 2015 19:11:05 +0530 Message-id: <081301d10686$370d2e10$a5278a30$@samsung.com> MIME-version: 1.0 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-index: AQJMXYG34mF3dXtZVCEjGRrPxnXARgFDhPPaANqCrXidYvdsgA== Content-language: en-us X-Brightmail-Tracker: H4sIAAAAAAAAA02Se0hTYRjG+85lOxuNjlPrUzFrEeLAypr2jS5If9SB7oRBBdVaBy0v2c4U KyjTrFSUnLea0wZGrG0mTqm00rlMitQuVqazZVAOWzaytKR1cR4D/3u+733e9/e88FK49CUZ Sh1J07KaNFWKTCAmrEGKk9FX9y3cvaIoV44M9VYB8v2pJFCxIRIVlpViqG/iM0Ajjihk6bcC 5HXX48hi24qu3K8mUb/bQKAb+UMk6m0xCJDL+pdEpV+GAfqU+4NE18e9QlR17guJ+kYqCDQx 6hSi89caMOTOzSNQbYcTR1VniwEqK34L4kOYWlMZyTz47MWZZv1bIWO0ZTDNdWaMqb03gjE2 c76AsY3phMyjy78I5sPLSoypebyT+fpxgGC8ra8ETHGTGTDlVWeYLmOHcEfgXvHaw2zKkUxW s3z9QXGS+fc4kd4WmXUz/zaeDdoiCoCIgrQC5rQ3kLyeD5+56gUFQExJaROAhX9b8P+mb/qn OF/QA1jiy8P4xyiAA00eUAAoSkBHwk67xN8QRMvhu8la4Nc4bSLg4BsV728D8IW3D/MXRPQW 2GrpnjYF0gnQNTgo9GuCXgoLPZ7pfwmthLnlXpLXAfBnqYvgh8ph460cktcRsNE6OpN0EbzT zfcG0RtgZ1PPjH8B1A29F/pDQPquCOZ0vMd5GA0nSh2EfwFIh0ObfWZOCGw3vSEuAaifhdbP QutnofWzEEZAmEEwm65O5w4lamKWcapULiMtcZn6WKoNTB3bkz/DRXeA077GAWgKyOZK7HXh u6WkKpM7keoAsVOJSvDQYPWxqftM0x6IWRW3EsUqYletXK2Mky2QLA6d3CWlE1VaNpll01nN AU1GCss5AEaJQrOBers7P67DklWg3HMqYrNWhvbnZDQFVpXTr0OCOae9WbrJoVPKvvp6FMNR eUueh1Gnw+c1blTZnz8sD/Od/85WKwyvskaT5sRtQy0XEoZ+uDip+WxdUWWyJtqji1nU2zsW f/TwukxfQGPNzXF5xTqtx7kCOy42quHHoequi1dkBJekipHjGk71D3gPzOlnAwAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFlrIJsWRmVeSWpSXmKPExsVy+t9jQd350XJhBo1dshZz1q9hs/jzbzqL Rd8cdYvuKZOZLK5/e8No8fKQpsXqm2sYLd4/X89ssXqTr8XMvXNZLW4+n8NisbLzAavF5V1z 2CzurfnPajH53TNGi1fN31ktln19z24xu+Udq8X1l9NYLL69vc1u0bZkI5PF8+ZWFovFR24z W8xu7GO0mNJ3l9FB0mPxiimsHoffvGf22DnrLrvHgk2lHjvXrmLyWLznJZPHplWdbB6bPk1i 9zgx4zeLx5Mr05k85p0M9Pj49BaLx/t9V9k8+rasYvSYOrve48yCI+wBwlENjDYZqYkpqUUK qXnJ+SmZeem2St7B8c7xpmYGhrqGlhbmSgp5ibmptkouPgG6bpk5wDBRUihLzCkFCgUkFhcr 6dthmhAa4qZrAdMYoesbEgTXY2SABhLWMGas+vuVpWC/esW6zu3MDYz75bsYOTkkBEwkPs86 zwxhi0lcuLeerYuRi0NIYBajxMQ/rUwQzltGiVtbXjN2MXJwsAmoSxw7wAvSICKgJXH/52JG EJtZYAWLxJ0biRD1+xklLr2/zgSS4BTwkdi3+ixYkbBAqMS9O3fYQWwWAVWJ7tevweK8ApYS zVPfs0LYghI/Jt9jgRiqJbF5WxMrhC0vsXnNW6hLFSR2nIXoFRFwkji25RxUvbjEpAcP2Scw Cs1CMmoWklGzkIyahaRlASPLKkaJ1ILkguKk9FzDvNRyveLE3OLSvHS95PzcTYzg5PlMagfj wV3uhxgFOBiVeHgzVsuGCbEmlhVX5h5ilOBgVhLh/WMgFybEm5JYWZValB9fVJqTWnyI0RTo 2YnMUqLJ+cDEnlcSb2hsYm5qbGppYmFiZqkkznvjEEOYkEB6YklqdmpqQWoRTB8TB6dUA6P2 hF/NqoJfmiZ8YXdY83NZkt3LuTXvr19u+8cW4qQcN8Ew20VPUdzP2V+OJXH63jMub3Kyp+zt KDD6GXMgX1WxOepotdHltR4TU51t3h5e8T55X+nKt4ncMn0vf63onCN9X/dFzhHxgpP5lfd/ GnNHLld2y5IULuGftWqFzv9z72eULmi+sF2JpTgj0VCLuag4EQDM8USbtAMAAA== DLP-Filter: Pass X-MTR: 20000000000000000@CPGS X-CFilter-Loop: Reflected Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, Thank you very much for your review and comments. > -----Original Message----- > From: David Rientjes [mailto:rientjes@google.com] > Sent: Wednesday, October 14, 2015 8:36 AM > To: Pintu Kumar > Cc: akpm@linux-foundation.org; minchan@kernel.org; dave@stgolabs.net; > mhocko@suse.cz; koct9i@gmail.com; hannes@cmpxchg.org; penguin-kernel@i- > love.sakura.ne.jp; bywxiaobai@163.com; mgorman@suse.de; vbabka@suse.cz; > js1304@gmail.com; kirill.shutemov@linux.intel.com; > alexander.h.duyck@redhat.com; sasha.levin@oracle.com; cl@linux.com; > fengguang.wu@intel.com; linux-kernel@vger.kernel.org; linux-mm@kvack.org; > cpgs@samsung.com; pintu_agarwal@yahoo.com; pintu.ping@gmail.com; > vishnu.ps@samsung.com; rohit.kr@samsung.com; c.rajkumar@samsung.com; > sreenathd@samsung.com > Subject: Re: [RESEND PATCH 1/1] mm: vmstat: Add OOM victims count in vmstat > counter > > On Mon, 12 Oct 2015, Pintu Kumar wrote: > > > This patch maintains the number of oom victims kill count in > > /proc/vmstat. > > Currently, we are dependent upon kernel logs when the kernel OOM occurs. > > But kernel OOM can went passed unnoticed by the developer as it can > > silently kill some background applications/services. > > In some small embedded system, it might be possible that OOM is > > captured in the logs but it was over-written due to ring-buffer. > > Thus this interface can quickly help the user in analyzing, whether > > there were any OOM kill happened in the past, or whether the system > > have ever entered the oom kill stage till date. > > > > Thus, it can be beneficial under following cases: > > 1. User can monitor kernel oom kill scenario without looking into the > > kernel logs. > > I'm not sure how helpful that would be since we don't know anything about the > oom kill itself, only that at some point during the uptime there were oom kills. > Not sure about others. For me it was very helpful during sluggish and long duration ageing tests. With this, I don't have to look into the logs manually. I just monitor this count in a script. The moment I get nr_oom_victims > 1, I know that kernel OOM would have happened and I need to take the log dump. So, then I do: dmesg >> oom_logs.txt Or, even stop the tests for further tuning. > > 2. It can help in tuning the watermark level in the system. > > I disagree with this one, because we can encounter oom kills due to > fragmentation rather than low memory conditions for high-order allocations. > The amount of free memory may be substantially higher than all zone > watermarks. > AFAIK, kernel oom happens only for lower-order (PAGE_ALLOC_COSTLY_ORDER). For higher-order we get page allocation failure. > > 3. It can help in tuning the low memory killer behavior in user space. > > Same reason as above. > > > 4. It can be helpful on a logless system or if klogd logging > > (/var/log/messages) are disabled. > > > > This would be similar to point (1) above, and I question how helpful it would be. > I notice that all oom kills (system, cpuset, mempolicy, and > memcg) are treated equally in this case and there's no way to differentiate them. > That would lead me to believe that you are targeting this change for systems > that don't use mempolicies or cgroups. That's fine, but I doubt it will be helpful > for anybody else. > No, we are not targeting any specific category. Our goal is simple, track and report kernel oom kill as soon as it occurs. > > A snapshot of the result of 3 days of over night test is shown below: > > System: ARM Cortex A7, 1GB RAM, 8GB EMMC > > Linux: 3.10.xx > > Category: reference smart phone device > > Loglevel: 7 > > Conditions: Fully loaded, BT/WiFi/GPS ON > > Tests: auto launching of ~30+ apps using test scripts, in a loop for > > 3 days. > > At the end of tests, check: > > $ cat /proc/vmstat > > nr_oom_victims 6 > > > > As we noticed, there were around 6 oom kill victims. > > > > The OOM is bad for any system. So, this counter can help in quickly > > tuning the OOM behavior of the system, without depending on the logs. > > > > NACK to the patch since it isn't justified. > > We've long had a desire to have a better oom reporting mechanism rather than > just the kernel log. It seems like you're feeling the same pain. I think it would be > better to have an eventfd notifier for system oom conditions so we can track > kernel oom kills (and conditions) in userspace. I have a patch for that, and it > works quite well when userspace is mlocked with a buffer in memory. > Ok, this would be interesting. Can you point me to the patches? I will quickly check if it is useful for us. > If you are only interested in a strict count of system oom kills, this could then > easily be implemented without adding vmstat counters. > We are interested only to know when kernel OOM occurs and not even the oom victim count. So that we can tune something is user space to avoid or delay it as far as possible.