From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f200.google.com (mail-pf0-f200.google.com [209.85.192.200]) by kanga.kvack.org (Postfix) with ESMTP id A2F9F6B0261 for ; Tue, 9 Aug 2016 03:02:54 -0400 (EDT) Received: by mail-pf0-f200.google.com with SMTP id o124so9083700pfg.1 for ; Tue, 09 Aug 2016 00:02:54 -0700 (PDT) Received: from mail-pa0-x241.google.com (mail-pa0-x241.google.com. [2607:f8b0:400e:c03::241]) by mx.google.com with ESMTPS id h21si41307709pfk.107.2016.08.09.00.02.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Aug 2016 00:02:53 -0700 (PDT) Received: by mail-pa0-x241.google.com with SMTP id hh10so462088pac.1 for ; Tue, 09 Aug 2016 00:02:53 -0700 (PDT) Subject: Re: [RFC][PATCH] cgroup_threadgroup_rwsem - affects scalability and OOM References: <4717ef90-ca86-4a34-c63a-94b8b4bfaaec@gmail.com> <20160809062900.GD4906@mtj.duckdns.org> From: Balbir Singh Message-ID: <0a7ffe43-c0c6-85df-9bc2-d00fc837e284@gmail.com> Date: Tue, 9 Aug 2016 17:02:47 +1000 MIME-Version: 1.0 In-Reply-To: <20160809062900.GD4906@mtj.duckdns.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Tejun Heo Cc: cgroups@vger.kernel.org, Oleg Nesterov , Andrew Morton , "linux-mm@kvack.org" On 09/08/16 16:29, Tejun Heo wrote: > Hello, Balbir. > > On Tue, Aug 09, 2016 at 02:19:01PM +1000, Balbir Singh wrote: >> >> cgroup_threadgroup_rwsem is acquired in read mode during process exit and fork. >> It is also grabbed in write mode during __cgroups_proc_write >> >> I've recently run into a scenario with lots of memory pressure and OOM >> and I am beginning to see >> >> systemd >> >> __switch_to+0x1f8/0x350 >> __schedule+0x30c/0x990 >> schedule+0x48/0xc0 >> percpu_down_write+0x114/0x170 >> __cgroup_procs_write.isra.12+0xb8/0x3c0 >> cgroup_file_write+0x74/0x1a0 >> kernfs_fop_write+0x188/0x200 >> __vfs_write+0x6c/0xe0 >> vfs_write+0xc0/0x230 >> SyS_write+0x6c/0x110 >> system_call+0x38/0xb4 >> >> This thread is waiting on the reader of cgroup_threadgroup_rwsem to exit. >> The reader itself is under memory pressure and has gone into reclaim after >> fork. There are times the reader also ends up waiting on oom_lock as well. >> > ... >> copy_page_range+0x4ec/0x950 >> copy_process.isra.5+0x15a0/0x1870 >> _do_fork+0xa8/0x4b0 >> ppc_clone+0x8/0xc > > Yeah, we definitely don't wanna be holding the rwsem during the actual > fork. > > ... >> There are other theoretical issues with this semaphore >> >> systemd can do >> >> 1. cgroup_mutex (cgroup_kn_lock_live) >> 2. cgroup_threadgroup_rwsem (W) (__cgroup_procs_write) >> >> and other threads can go >> >> 1. cgroup_threadgroup_rwsem (R) (copy_process) >> 2. mem_cgroup_iter (as a part of reclaim) (cgroup_mutex -- rcu lock or cgroup_mutex) > > Hmm? Where does mem_cgroup_iter grab cgroup_mutex? cgroup_mutex nests > outside cgroup_threadgroup_rwsem or most other mutexes for that matter > and isn't exposed from cgroup core. > I based my theory on the code mem_cgroup_iter -> css_next_descendant_pre which asserts cgroup_assert_mutex_or_rcu_locked(), although you are right, we hold RCU lock while calling css_* routines. >> However, I've not examined them in too much detail or looked at lockdep >> wait chains for those paths. >> >> I am sure there is a good reason for placing cgroup_threadgroup_rwsem >> where it is today and I might be missing something. I am also surprised > > I could be missing something too but the positioning is largely > historic. > >> no-one else has run into it so far. > > Maybe it might matter that much on a system which is already heavily > thrasing, but yeah, we definitely want to tighten down the reader > sections so that it doesn't get in the way of making forward progress. > It seems to cause my system to thrash quite badly. >> Comments? > > The change looks good to me on the first glance but I'll think more > about it tomorrow. > > Thanks! > Thanks for the review. Balbir Singh. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org