From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9932ECAAD5 for ; Tue, 6 Sep 2022 18:44:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4453C6B0074; Tue, 6 Sep 2022 14:44:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F3D78D0005; Tue, 6 Sep 2022 14:44:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E3B38D0003; Tue, 6 Sep 2022 14:44:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1E3086B0074 for ; Tue, 6 Sep 2022 14:44:50 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id EFC981A0748 for ; Tue, 6 Sep 2022 18:44:49 +0000 (UTC) X-FDA: 79882537098.06.5AD3363 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf06.hostedemail.com (Postfix) with ESMTP id 02683180069 for ; Tue, 6 Sep 2022 18:44:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662489889; x=1694025889; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=Y/lcPs3orVnls9Hb6iT8Kn+P8uLpV38epnOs8myLRow=; b=mpSVeXbCbYRmbxgKgXWJ5H5H1vxTu0miIHygs7uL5C6wwElig/MH6rx/ L0qKKhzy8J9AI7U5fCq2N5TGVuqHo+aFJNQysmQHH8yQEgitkYRY2Fg6R UaqNWUEHafapmSwogKuuYrJ6J5NV1xuAY0bDiqwbAODeGMklbNmIUwpXl nOaO0ZVxtS9L6tjtsy3qFmMxMgMS5Vhm28oI1f7QtQJuXn0Xs3faOxfvB hw82w6va9ZkeIT/452vxdbRrH3sA0/Qa4gO2/Yv+a19ub+lzAIvfhwluH gc2i4UU9GSISZ0sKU8CEUZzq1PPyC2Q21dN2yVEFK93gE0F87GNpUSVw5 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10462"; a="279689247" X-IronPort-AV: E=Sophos;i="5.93,294,1654585200"; d="scan'208";a="279689247" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Sep 2022 11:44:47 -0700 X-IronPort-AV: E=Sophos;i="5.93,294,1654585200"; d="scan'208";a="739998118" Received: from schen9-mobl.amr.corp.intel.com ([10.252.133.221]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Sep 2022 11:44:47 -0700 Message-ID: <048517e7f95aa8460cd47a169f3dfbd8e9b70d5c.camel@linux.intel.com> Subject: Re: [PATCH] ipc/msg.c: mitigate the lock contention with percpu counter From: Tim Chen To: Shakeel Butt , Jiebin Sun Cc: Andrew Morton , vasily.averin@linux.dev, Dennis Zhou , Tejun Heo , Christoph Lameter , "Eric W. Biederman" , Alexey Gladkov , Manfred Spraul , alexander.mikhalitsyn@virtuozzo.com, Linux MM , LKML , "Chen, Tim C" , Feng Tang , Huang Ying , tianyou.li@intel.com, wangyang.guo@intel.com Date: Tue, 06 Sep 2022 11:44:46 -0700 In-Reply-To: References: <20220902152243.479592-1-jiebin.sun@intel.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.34.4 (3.34.4-1.fc31) MIME-Version: 1.0 Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662489889; a=rsa-sha256; cv=none; b=vA5UsIgVmIrVEnjXsCPgM4hY7Hti60+BEE5f5i8bPfBSBD20ji5mxtflNzSzPlu/d4G3Dg w+Vpv260PUQdKwOq9eAbu4q756vPDGcG3k0KTAUw3Wslz8M0hnS/GzIx4vFARijxSlVisf d+wyrnh9h3Ljun207l1Pb9BOg3m64+4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=mpSVeXbC; spf=none (imf06.hostedemail.com: domain of tim.c.chen@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=tim.c.chen@linux.intel.com; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662489889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UFFpiQm8DZ52EoF/l54UiNY24ule6a2BUT7NZrdmsTs=; b=Hb/wNZh8vGmrfGUSRnbjKvvPX0xKKCei5ZZReMRhxYz8INq5dg56rchuYpK307XYynFkyS /peXQLvdgUPsVhHADxvhIVDXGyV88PhB/yTAiZCLkvBhrVpoJcK2sQFZsDlIeLpjSXHMlv QaRSed3JbgipnBM21/JtUJAPhQX+7/o= X-Stat-Signature: jnp4uui1hxmmqke8y7539xu95mddj4ie X-Rspamd-Queue-Id: 02683180069 Authentication-Results: imf06.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=mpSVeXbC; spf=none (imf06.hostedemail.com: domain of tim.c.chen@linux.intel.com has no SPF policy when checking 134.134.136.126) smtp.mailfrom=tim.c.chen@linux.intel.com; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none) X-Rspamd-Server: rspam11 X-Rspam-User: X-HE-Tag: 1662489888-3814 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 2022-09-02 at 09:27 -0700, Shakeel Butt wrote: > On Fri, Sep 2, 2022 at 12:04 AM Jiebin Sun wrote: > > The msg_bytes and msg_hdrs atomic counters are frequently > > updated when IPC msg queue is in heavy use, causing heavy > > cache bounce and overhead. Change them to percpu_counters > > greatly improve the performance. Since there is one unique > > ipc namespace, additional memory cost is minimal. Reading > > of the count done in msgctl call, which is infrequent. So > > the need to sum up the counts in each CPU is infrequent. > > > > Apply the patch and test the pts/stress-ng-1.4.0 > > -- system v message passing (160 threads). > > > > Score gain: 3.38x > > > > CPU: ICX 8380 x 2 sockets > > Core number: 40 x 2 physical cores > > Benchmark: pts/stress-ng-1.4.0 > > -- system v message passing (160 threads) > > > > Signed-off-by: Jiebin Sun > [...] > > +void percpu_counter_add_local(struct percpu_counter *fbc, s64 amount) > > +{ > > + this_cpu_add(*fbc->counters, amount); > > +} > > +EXPORT_SYMBOL(percpu_counter_add_local); > > Why not percpu_counter_add()? This may drift the fbc->count more than > batch*nr_cpus. I am assuming that is not the issue for you as you > always do an expensive sum in the slow path. As Andrew asked, this > should be a separate patch. In the IPC case, the read is always done with the accurate read using percpu_counter_sum() gathering all the counts and never with percpu_counter_read() that only read global count. So Jiebin was not worry about accuracy. However, the counter is s64 and the local per cpu counter is S32. So the counter size has shrunk if we only keep the count in local per cpu counter, which can overflow a lot sooner and is not okay. Jiebin, can you try to use percpu_counter_add_batch, but using a large batch size. That should achieve what you want without needing to create a percpu_counter_add_local() function, and also the overflow problem. Tim