From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759901Ab3BHEQ7 (ORCPT ); Thu, 7 Feb 2013 23:16:59 -0500 Received: from fgwmail5.fujitsu.co.jp ([192.51.44.35]:46296 "EHLO fgwmail5.fujitsu.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759840Ab3BHEQz (ORCPT ); Thu, 7 Feb 2013 23:16:55 -0500 X-SecurityPolicyCheck: OK by SHieldMailChecker v1.8.4 Message-ID: <51147C1B.1000402@jp.fujitsu.com> Date: Fri, 08 Feb 2013 13:16:27 +0900 From: Kamezawa Hiroyuki User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: Michal Hocko CC: azurIt , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups mailinglist , Johannes Weiner Subject: Re: [PATCH for 3.2.34] memcg: do not trigger OOM from add_to_page_cache_locked References: <20121230020947.AA002F34@pobox.sk> <20121230110815.GA12940@dhcp22.suse.cz> <20130125160723.FAE73567@pobox.sk> <20130125163130.GF4721@dhcp22.suse.cz> <20130205134937.GA22804@dhcp22.suse.cz> <20130205154947.CD6411E2@pobox.sk> <20130205160934.GB22804@dhcp22.suse.cz> <20130206021721.1AE9E3C7@pobox.sk> <20130206140119.GD10254@dhcp22.suse.cz> <51138999.3090006@jp.fujitsu.com> <20130207123140.GA15820@dhcp22.suse.cz> In-Reply-To: <20130207123140.GA15820@dhcp22.suse.cz> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org (2013/02/07 21:31), Michal Hocko wrote: > On Thu 07-02-13 20:01:45, KAMEZAWA Hiroyuki wrote: >> (2013/02/06 23:01), Michal Hocko wrote: >>> On Wed 06-02-13 02:17:21, azurIt wrote: >>>>> 5-memcg-fix-1.patch is not complete. It doesn't contain the folloup I >>>>> mentioned in a follow up email. Here is the full patch: >>>> >>>> >>>> Here is the log where OOM, again, killed MySQL server [search for "(mysqld)"]: >>>> http://www.watchdog.sk/lkml/oom_mysqld6 >>> >>> [...] >>> WARNING: at mm/memcontrol.c:2409 T.1149+0x2d9/0x610() >>> Hardware name: S5000VSA >>> gfp_mask:4304 nr_pages:1 oom:0 ret:2 >>> Pid: 3545, comm: apache2 Tainted: G W 3.2.37-grsec #1 >>> Call Trace: >>> [] warn_slowpath_common+0x7a/0xb0 >>> [] warn_slowpath_fmt+0x46/0x50 >>> [] ? mem_cgroup_margin+0x73/0xa0 >>> [] T.1149+0x2d9/0x610 >>> [] ? blk_finish_plug+0x18/0x50 >>> [] mem_cgroup_cache_charge+0xc4/0xf0 >>> [] add_to_page_cache_locked+0x4f/0x140 >>> [] add_to_page_cache_lru+0x22/0x50 >>> [] filemap_fault+0x252/0x4f0 >>> [] __do_fault+0x78/0x5a0 >>> [] handle_pte_fault+0x84/0x940 >>> [] ? vma_prio_tree_insert+0x30/0x50 >>> [] ? vma_link+0x88/0xe0 >>> [] handle_mm_fault+0x138/0x260 >>> [] do_page_fault+0x13d/0x460 >>> [] ? do_mmap_pgoff+0x3dc/0x430 >>> [] page_fault+0x1f/0x30 >>> ---[ end trace 8817670349022007 ]--- >>> apache2 invoked oom-killer: gfp_mask=0x0, order=0, oom_adj=0, oom_score_adj=0 >>> apache2 cpuset=uid mems_allowed=0 >>> Pid: 3545, comm: apache2 Tainted: G W 3.2.37-grsec #1 >>> Call Trace: >>> [] dump_header+0x7e/0x1e0 >>> [] ? find_lock_task_mm+0x2f/0x70 >>> [] oom_kill_process+0x85/0x2a0 >>> [] out_of_memory+0xe5/0x200 >>> [] pagefault_out_of_memory+0xbd/0x110 >>> [] mm_fault_error+0xb6/0x1a0 >>> [] do_page_fault+0x3ee/0x460 >>> [] ? do_mmap_pgoff+0x3dc/0x430 >>> [] page_fault+0x1f/0x30 >>> >>> The first trace comes from the debugging WARN and it clearly points to >>> a file fault path. __do_fault pre-charges a page in case we need to >>> do CoW (copy-on-write) for the returned page. This one falls back to >>> memcg OOM and never returns ENOMEM as I have mentioned earlier. >>> However, the fs fault handler (filemap_fault here) can fallback to >>> page_cache_read if the readahead (do_sync_mmap_readahead) fails >>> to get page to the page cache. And we can see this happening in >>> the first trace. page_cache_read then calls add_to_page_cache_lru >>> and eventually gets to add_to_page_cache_locked which calls >>> mem_cgroup_cache_charge_no_oom so we will get ENOMEM if oom should >>> happen. This ENOMEM gets to the fault handler and kaboom. >>> >> >> Hmm. do we need to increase the "limit" virtually at memcg oom until >> the oom-killed process dies ? It may be doable by increasing stock->cache >> of each cpu....I think kernel can offer extra virtual charge up to >> oom-killed process's memory usage..... > > If we can guarantee that the overflow charges do not exceed the memory > usage of the killed process then this would work. The question is, how > do we find out how much we can overflow. immigrate_on_move will play > some role as well as the amount of the shared memory. I am afraid this > would get too complex. Nevertheless the idea is nice. > Yes, that's the problem. If we don't do in correct way, resouce usage undeflow can happen. I guess we can count it per task_struct at charging page-faulted anon pages. _Or_ in other consideration, for example, we do charge 1MB per thread regardless of its memory usage. And use it as a security at OOM-killing. Implemtation will be easy but explanation may be difficult.. Thanks, -Kame Thanks, -Kame