From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailgw2.hygon.cn (unknown [101.204.27.37]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1797140DFC7; Mon, 20 Apr 2026 02:10:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=101.204.27.37 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776651061; cv=none; b=VQsDpwkxDHrwrOOtqGxyXYMFm8Jdedc+wIz6bJAbDETWVz1n7o6uYTeM1h5HGoEFjWPBzyJmDGPRg2IkTtq7Tr7A8mjuA2Godd88Lb7aBPAiGls0G8A0sUKXRhchGQD94ToD7m94Kg1RXUk0kku4yaRXFTzDFpTK83jC6TUHN2E= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776651061; c=relaxed/simple; bh=XZMDY39SIKnjgr9AwOz4guH12QSe5xlaKf0Q6+yWohc=; h=Date:From:To:CC:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=qeXaFwLQNaepgAnoSOwzn1tIGaAqquX4ZisCGD4MlGIz59T97GtwiNlg6CS2hf++rjXLtQh6TaqhV6bI5HneZbNK4e2b8Noo1BYO+qVmrxZxHghFxBRnzJo15WMImPC4DcEHXqKwRoqZ951OrTdWdi3mjfNaGJXdF+aU2FYP078= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hygon.cn; spf=pass smtp.mailfrom=hygon.cn; arc=none smtp.client-ip=101.204.27.37 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hygon.cn Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=hygon.cn Received: from maildlp1.hygon.cn (unknown [127.0.0.1]) by mailgw2.hygon.cn (Postfix) with ESMTP id 4fzTTx2xMWz1YQpmD; Mon, 20 Apr 2026 10:10:29 +0800 (CST) Received: from maildlp1.hygon.cn (unknown [172.23.18.60]) by mailgw2.hygon.cn (Postfix) with ESMTP id 4fzTTw3BqFz1YQpmD; Mon, 20 Apr 2026 10:10:28 +0800 (CST) Received: from cncheex04.Hygon.cn (unknown [172.23.18.114]) by maildlp1.hygon.cn (Postfix) with ESMTPS id 29C42B53C; Mon, 20 Apr 2026 10:10:25 +0800 (CST) Received: from SH-HV00110.Hygon.cn (172.19.26.208) by cncheex04.Hygon.cn (172.23.18.114) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.36; Mon, 20 Apr 2026 10:10:17 +0800 Date: Mon, 20 Apr 2026 10:10:19 +0800 From: Huang Shijie To: Mateusz Guzik CC: , , , , , , , , , , , , , , , Subject: Re: [PATCH 0/3] mm: split the file's i_mmap tree for NUMA Message-ID: References: <20260413062042.804-1-huangsj@hygon.cn> <76pfiwabdgsej6q2yxfh3efuqvsyg7mt7rvl5itzzjyhdrto5r@53viaxsackzv> Precedence: bulk X-Mailing-List: linux-parisc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <76pfiwabdgsej6q2yxfh3efuqvsyg7mt7rvl5itzzjyhdrto5r@53viaxsackzv> X-ClientProxiedBy: cncheex05.Hygon.cn (172.23.18.115) To cncheex04.Hygon.cn (172.23.18.114) On Mon, Apr 13, 2026 at 05:33:21PM +0200, Mateusz Guzik wrote: > On Mon, Apr 13, 2026 at 02:20:39PM +0800, Huang Shijie wrote: > > In NUMA, there are maybe many NUMA nodes and many CPUs. > > For example, a Hygon's server has 12 NUMA nodes, and 384 CPUs. > > In the UnixBench tests, there is a test "execl" which tests > > the execve system call. > > > > When we test our server with "./Run -c 384 execl", > > the test result is not good enough. The i_mmap locks contended heavily on > > "libc.so" and "ld.so". For example, the i_mmap tree for "libc.so" can have > > over 6000 VMAs, all the VMAs can be in different NUMA mode. > > The insert/remove operations do not run quickly enough. > > > > patch 1 & patch 2 are try to hide the direct access of i_mmap. > > patch 3 splits the i_mmap into sibling trees, and we can get better > > performance with this patch set: > > we can get 77% performance improvement(10 times average) > > > > To my reading you kept the lock as-is and only distributed the protected > state. > > While I don't doubt the improvement, I'm confident should you take a > look at the profile you are going to find this still does not scale with > rwsem being one of the problems (there are other global locks, some of > which have experimental patches for). > > Apart from that this does nothing to help high core systems which are > all one node, which imo puts another question mark on this specific > proposal. > > Of course one may question whether a RB tree is the right choice here, > it may be the lock-protected cost can go way down with merely a better > data structure. > > Regardless of that, for actual scalability, there will be no way around > decentralazing locking around this and partitioning per some core count > (not just by numa awareness). > > Decentralizing locking is definitely possible, but I have not looked > into specifics of how problematic it is. Best case scenario it will > merely with separate locks. Worst case scenario something needs a fully > stabilized state for traversal, in that case another rw lock can be > slapped around this, creating locking order read lock -> per-subset > write lock -- this will suffer scalability due to the read locking, but > it will still scale drastically better as apart from that there will be > no serialization. In this setting the problematic consumer will write > lock the new thing to stabilize the state. > I thought over again. I can change this patch set to support the non-NUMA case by: 1.) Still use one rw lock. 2.) For NUMA, keep the patch set as it is. 3.) For non-NUMA case, split the i_mmap tree to several subtrees. For example, if a machine has 192 CPUs, split the 32 CPUs as a tree. So extend the patch set to support both the NUMA and non-NUMA machines. Thanks Huang Shijie