From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21CF1C7EE23 for ; Mon, 27 Feb 2023 15:08:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A693C6B0073; Mon, 27 Feb 2023 10:08:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A18DC6B0074; Mon, 27 Feb 2023 10:08:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90A2A6B0075; Mon, 27 Feb 2023 10:08:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 81D9C6B0073 for ; Mon, 27 Feb 2023 10:08:50 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 438AD1605D8 for ; Mon, 27 Feb 2023 15:08:50 +0000 (UTC) X-FDA: 80513404020.18.E5BC000 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf27.hostedemail.com (Postfix) with ESMTP id 85C014001A for ; Mon, 27 Feb 2023 15:08:47 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=P8QNMHyR; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf27.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677510527; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QmakQy/pwgBbnzNgqQ7Yw6qWolMn/rnbrByInKd1TUk=; b=ocLw0VXr8RZBeh74WThOsgf1WKLkEVMaXQAoJ7wcrVGi2OqGv4VbvaR3k/ksyhhWsvNnn3 +gqKu4e9z+aPeLsd/57zbJgV3ANK22RxF48wbqLRS0Rai3rHTFOlXPsNGWOzpMq2hClgpq PVP8hm9nTBOyP0lcCvBfaoPdGOxWdVY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=P8QNMHyR; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf27.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677510527; a=rsa-sha256; cv=none; b=k1dW9RjcWrip5LC8VZcT67mAE532Pa10oZ0clQLciipMQDpAaMnffB+yY5k2nd6WlAH83e P96wnzF221dUPBU3ZEsodUHrc6gVXtA+9iY0nywuGs1Y5JA/EsFDRGoEVbe6Yyy98+ognz EFcQAfFp0/fvMzP4AMGsxeSR9DYAGk4= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 86B8E60EA1; Mon, 27 Feb 2023 15:08:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5E830C433A4; Mon, 27 Feb 2023 15:08:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1677510525; bh=A4GC9qpKEzKIujfpTB17KrXszHgOinHLYyEqMxiUgU8=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=P8QNMHyRQ7hOWsOHfSzhLk0DE9NdtWxKjjPdoxktGf7ITel8dmVeNWswbLmKiLTjZ WgAysQDQg6GFG3uGUrL+/A82VN7NkFK/lwQuwQfzf6264Zt0EpiK7PxcO8SYM8+0Yr 8hur4UE4aHyKZGrUmePTc6fUPF8hQwgj6QfhrwsA2TjUiUpV+cP2gE7Ghr+3Io/+sS oF9YSBH/w6Pge8T1oX8PU9sO+gqsHgFZSeSblHffo7uHn/1snvFht3b6TDUwoybV4k J9CosbYx8byXhidNCW0yocu5d5nnhU2ya/YOFrf4PdhBbf8YzZ+JMlD/bish1ngoMj 3qGgvUPk2rK8A== Date: Mon, 27 Feb 2023 17:08:30 +0200 From: Mike Rapoport To: Qi Zheng Cc: Andrew Morton , tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@i-love.sakura.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v3 0/8] make slab shrink lockless Message-ID: References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> <20230226115100.7e12bda7931dd65dbabcebe3@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 85C014001A X-Stat-Signature: aa1xm7g3wyfmhkq8tw9h77yhsjdnwsgr X-HE-Tag: 1677510527-399535 X-HE-Meta: U2FsdGVkX18DlufkdQXmcXo/sak+yostgRTWaiUdo+YU4JF7Veok6aTydupr8hhi4YFN5BVvRnV1FvgPMo6frNfC1SH1fb9O8L2/1eRqH4nQHSUHt5HVWrNY4gzfaMR9TZSm19DnlbU1Uds6sdZIiAuDwX5XV7pfFkt92A4+zhhkAVKXENujhtMfs7V+i7/8sYPoOaFE2yjNuJAa/T2M88FtXsNLdOXhx6PIpoTgMzXuhvG3CgBozGB7IzIHuJkcmciaC+FtP4R3DcR+sRltdHzBpiF6cJL7bnN3gyKRMsHP2mAXtXQSodP8+lG73Ly06Oz1TRqW1ekDcmp8k8mCsJZSH3ncRlB4yR6Mri4nlHBXIP9rNFLSoqpWlbWt+oUL1uT7nhBBlAfWmGZ+EwXJHcmnDxoV4NMIjEaLhkf7QELmONIPkXSDe3HFLeLFKLr/0oOXKFL7YsbFkLCQ5qtWqH0mDxaE6Sn9zWRsnExApdJw+DDhYB8/kgSxdUBfWsrAufAMTyxVr0bwTgNN8mP+7KLyJfqH5fp7yJaDeFYTXhkrkMLJu4r0LAro5jQ1LPAfq2tY/yV6/22KU6E2QCII3kJrNhoFvpYqXGD02UrA89jCQJakLXNY9KAcI/t6fr9KLKSwjv6K1ido/MWRO9ZEzRwUcgsAv2lJvTxiuLQQl+XYLXKnjIFI+AF766dDQIqAG70d5a6LIzmlgi9EZzDbklpck+YhtQ3aLLIAPogRAJU1B7kPLYYm8+LlDsl226S7qtB0//2weAsmgEudixotaQ4cPbKb9iPBpLXF7J+Q8PDe9T6sfUlvhBFuDkl2gf02GrG7uny1C9tepq1+wCUdX2MrVdq8WE87ljNLEJ7jfCm0WIkgZZYkeq01k7SpsKwnLIlDoKVbddOVMRoiPGZVokRgCQtpFq6Rtbne0hBsxqJhQsUwGt7JdJJoPJP6gYfGTymGSPFhWhgIxEXNEto gDZKVt5X pk4UTGBWYD8ZPlfKkWO1EGzV+Ah9fyjyilOVxCqZJxRGhLUFt2NCTw8vqTc8njcWy4oKh2uKLTpYrfUlNN3FFnM3coRn+sup4B9cegTkZMueTjdhgLp+wJmYDDhWE8bMzHGm03hoNTRn3j6hF5WqIdcIGIWMi3neIbqny8iwtY2u0y9rLOA/6sGgFVdqIimbTVpHYHnN90ISbVJL90c2Wklltjx8BNSp2GQ515a2dTSNRHmoEaWtlMYuFnJ3Mwz2WSVP3POPr7XtyXonYdYs/tJDPXvgFZrpJ4qxnzjaD5FbaFtcPRRgV+apH636jjaNVh/04puklj8Y3Ltvy6FV8F1z12QeMBB1g8rwm6GsCsxz27b8CfBXJEobGeQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, On Mon, Feb 27, 2023 at 09:31:51PM +0800, Qi Zheng wrote: > > > On 2023/2/27 03:51, Andrew Morton wrote: > > On Sun, 26 Feb 2023 22:46:47 +0800 Qi Zheng wrote: > > > > > Hi all, > > > > > > This patch series aims to make slab shrink lockless. > > > > What an awesome changelog. > > > > > 2. Survey > > > ========= > > > > Especially this part. > > > > Looking through all the prior efforts and at this patchset I am not > > immediately seeing any statements about the overall effect upon > > real-world workloads. For a good example, does this patchset > > measurably improve throughput or energy consumption on your servers? > > Hi Andrew, > > I re-tested with the following physical machines: > > Architecture: x86_64 > CPU(s): 96 > On-line CPU(s) list: 0-95 > Model name: Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz > > I found that the reason for the hotspot I described in cover letter is > wrong. The reason for the down_read_trylock() hotspot is not because of > the failure to trylock, but simply because of the atomic operation > (cmpxchg). And this will lead to a significant reduction in IPC (insn > per cycle). ... > Then we can use the following perf command to view hotspots: > > perf top -U -F 999 > > 1) Before applying this patchset: > > 32.31% [kernel] [k] down_read_trylock > 19.40% [kernel] [k] pv_native_safe_halt > 16.24% [kernel] [k] up_read > 15.70% [kernel] [k] shrink_slab > 4.69% [kernel] [k] _find_next_bit > 2.62% [kernel] [k] shrink_node > 1.78% [kernel] [k] shrink_lruvec > 0.76% [kernel] [k] do_shrink_slab > > 2) After applying this patchset: > > 27.83% [kernel] [k] _find_next_bit > 16.97% [kernel] [k] shrink_slab > 15.82% [kernel] [k] pv_native_safe_halt > 9.58% [kernel] [k] shrink_node > 8.31% [kernel] [k] shrink_lruvec > 5.64% [kernel] [k] do_shrink_slab > 3.88% [kernel] [k] mem_cgroup_iter > > 2. At the same time, we use the following perf command to capture IPC > information: > > perf stat -e cycles,instructions -G test -a --repeat 5 -- sleep 10 > > 1) Before applying this patchset: > > Performance counter stats for 'system wide' (5 runs): > > 454187219766 cycles test ( > +- 1.84% ) > 78896433101 instructions test # 0.17 insn per > cycle ( +- 0.44% ) > > 10.0020430 +- 0.0000366 seconds time elapsed ( +- 0.00% ) > > 2) After applying this patchset: > > Performance counter stats for 'system wide' (5 runs): > > 841954709443 cycles test ( > +- 15.80% ) (98.69%) > 527258677936 instructions test # 0.63 insn per > cycle ( +- 15.11% ) (98.68%) > > 10.01064 +- 0.00831 seconds time elapsed ( +- 0.08% ) > > We can see that IPC drops very seriously when calling > down_read_trylock() at high frequency. After using SRCU, > the IPC is at a normal level. The results you present do show improvement in IPC for an artificial test script. But more interesting would be to see how a real world workloads benefit from your changes. > Thanks, > Qi -- Sincerely yours, Mike.