From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sha Zhengju Subject: Re: [PATCH] oom, memcg: handle sysctl oom_kill_allocating_task while memcg oom happening Date: Thu, 18 Oct 2012 21:51:57 +0800 Message-ID: <5080097D.5020501@gmail.com> References: <1350382328-28977-1-git-send-email-handai.szj@taobao.com> <20121016133439.GI13991@dhcp22.suse.cz> <20121018115640.GB24295@dhcp22.suse.cz> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=gljtvi3l1KhJlFefNmCou1+VhYKgPeWRoW6LEFIoyZQ=; b=tyiqaNol3wDoXIlkrEi/Tng6vo3fu+ij6u7EWLytYaNAj54nmz+KxeA/mdHSpBODPr SdBorSitNqMOWW9QJfnBJySizWq3iBeb9Q5f871VdN3pzlIJwBAfbGBFpNu1RaQ285HR QshYMZ7kstgt4z1MwTD116Mt+riOnmIg6YMBdony9GOznXFGqmfX9skmwzBbNx2khtND q8DcGN9UQk1WJyFGZd0eubvVCvzb7RORsKtuei4yAAZfsj+QaNDGSRZGAlYGYD+u7d+O 9czjbWBUiqMrb5xwjQvS8YhF+vswHH7p+PTQ47UHAj+5vXbtqRi3gPRGksUwoYb6qBUV 2IGQ== In-Reply-To: <20121018115640.GB24295-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org> Sender: cgroups-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-ID: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: Michal Hocko Cc: "linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org" , "cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "kamezawa.hiroyu-+CUm20s59erQFUHtdCDX3A@public.gmane.org" , "akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Sha Zhengju , David Rientjes On 10/18/2012 07:56 PM, Michal Hocko wrote: > On Wed 17-10-12 01:14:48, Sha Zhengju wrote: >> On Tuesday, October 16, 2012, Michal Hocko wrote: > [...] >>> Could you be more specific about the motivation for this patch? Is it >>> "let's be consistent with the global oom" or you have a real use case >>> for this knob. >>> >> In our environment(rhel6), we encounter a memcg oom 'deadlock' >> problem. Simply speaking, suppose process A is selected to be killed >> by memcg oom killer, but A is uninterruptible sleeping on a page >> lock. What's worse, the exact page lock is holding by another memcg >> process B which is trapped in mem_croup_oom_lock(proves to be a >> livelock). > Hmm, this is strange. How can you get down that road with the page lock > held? Is it possible this is related to the issue fixed by: 1d65f86d > (mm: preallocate page before lock_page() at filemap COW)? No, it has nothing with the cow page. By checking stack of the process A selected to be killed(uninterruptible sleeping), it was stuck at: __do_fault->filemap_fault->__lock_page_or_retry->wait_on_page_bit--(D state). The person B holding the exactly page lock is on the following path: __do_fault->filemap_fault->__do_page_cache_readahead->..->mpage_readpages ->add_to_page_cache_locked ---- >(in memcg oom and cannot exit) In mpage_readpages, B tends to read a dozen of pages in: for each of page will do locking, charging, and then send out a big bio. And A is waiting for one of the pages and stuck. As I said, 37b23e05 has made pagefault killable by changing uninterruptible sleeping to killable sleeping. So A can be woke up to exit successfully and free the memory which can in turn help B pass memcg charging period. (By the way, it seems commit 37b23e05 and 7d9fdac need to be backported to --stable tree to deliver RHEL users. ;-) ) >> Then A can not exit successfully to free the memory and both of them >> can not moving on. >> Indeed, we should dig into these locks to find the solution and >> in fact the 37b23e05 (x86, mm: make pagefault killable) and >> 7d9fdac(Memcg: make oom_lock 0 and 1 based other than counter) have >> already solved the problem, but if oom_killing_allocating_task is >> memcg aware, enabling this suicide oom behavior will be a simpler >> workaround. What's more, enabling the sysctl can avoid other potential >> oom problems to some extent. > As I said, I am not against this but I really want to see a valid use > case first. So far I haven't seen any because what you mention above is > a clear bug which should be fixed. I can imagine the huge number of > tasks in the group could be a problem as well but I would like to see > what are those problems first. > In view of consistent with global oom and performance benefit, I suggest we may as well open it in memcg oom as there's no obvious harm. As refer to the bug I mentioned, obviously the key solution is the above two patchset, but considing other *potential* memcg oom bugs, the sysctl may be a role of temporary workaround to some extent... but it's just a workaround. Thanks, Sha >>> The primary motivation for oom_kill_allocating_tas AFAIU was to reduce >>> search over huge tasklists and reduce task_lock holding times. I am not >>> sure whether the original concern is still valid since 6b0c81b (mm, >>> oom: reduce dependency on tasklist_lock) as the tasklist_lock usage has >>> been reduced conciderably in favor of RCU read locks is taken but maybe >>> even that can be too disruptive? >>> David? >> >> On the other hand, from the semantic meaning of oom_kill_allocating_task, >> it implies to allow suicide-like oom, which has no obvious relationship >> with performance problems(such as huge task lists or task_lock holding >> time). > I guess that suicide-like oom in fact means "kill the poor soul that > happened to charge the last". I do not see any use case for this from > top of my head (appart from the performance benefits of course). > >> So make the sysctl be consistent with global oom will be better or set >> an individual option for memcg oom just as panic_on_oom does.