From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F29ABC77B7F for ; Fri, 27 Jun 2025 09:02:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7962D6B00B3; Fri, 27 Jun 2025 05:02:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 76D5F6B00BD; Fri, 27 Jun 2025 05:02:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6AAB96B00BE; Fri, 27 Jun 2025 05:02:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5B23C6B00B3 for ; Fri, 27 Jun 2025 05:02:49 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0B0AE16065F for ; Fri, 27 Jun 2025 09:02:49 +0000 (UTC) X-FDA: 83600590458.19.8AE04F7 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf13.hostedemail.com (Postfix) with ESMTP id F2EFC2001B for ; Fri, 27 Jun 2025 09:02:43 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; spf=pass (imf13.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751014967; a=rsa-sha256; cv=none; b=BNqJSuss0Vj5ObZMHGMwRFHhKCGsNwbKX4Tr+f1e32QbVlI0No5i44xmduyONKTrZX4F4x uIYIiphq6dcgpHAyLNo1os/1XYBpBJ30Vmo2wKara4aw1Q+OrBTgHv5K3HOEq619ePRvNb yumITqL1mObkdMiyTDHnFm9h1Oz0m18= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751014967; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V2DCtqSuaxNrq50sP/8KrgV/eUF17iN6G5HkeXJ/Azo=; b=su2RgAbufMCz2tYHx6X0UyYfIwDoKVCeZ4ueF/PdwpboQR9z27swQuJ+oG5SkdI+vlFfn7 wk42a1ZCm14CSaYhp74o89t3SlVDtynfry1KPb64DvWMTzRzx2OM8/CkH0OBZJMQfuj/8P /LgNjF9/whmBi3I4lLHvicuYa0ZSpro= Received: from mail.maildlp.com (unknown [172.19.93.142]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTPS id 4bT8hc3GbhzYQtLT for ; Fri, 27 Jun 2025 17:02:40 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.252]) by mail.maildlp.com (Postfix) with ESMTP id 5A11F1A10F6 for ; Fri, 27 Jun 2025 17:02:39 +0800 (CST) Received: from [10.67.109.79] (unknown [10.67.109.79]) by APP3 (Coremail) with SMTP id _Ch0CgCH2MItXl5oDsw0Qg--.9912S2; Fri, 27 Jun 2025 17:02:39 +0800 (CST) Message-ID: <5f622eec-a039-4e82-9f37-3cad1692f268@huaweicloud.com> Date: Fri, 27 Jun 2025 17:02:37 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC 00/28] Eliminate Dying Memory Cgroup To: Kairui Song , Roman Gushchin Cc: Muchun Song , Muchun Song , hannes@cmpxchg.org, mhocko@kernel.org, shakeel.butt@linux.dev, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, hamzamahfooz@linux.microsoft.com, apais@linux.microsoft.com, yuzhao@google.com References: <20250415024532.26632-1-songmuchun@bytedance.com> Content-Language: en-US From: Chen Ridong In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-CM-TRANSID:_Ch0CgCH2MItXl5oDsw0Qg--.9912S2 X-Coremail-Antispam: 1UD129KBjvJXoW7Ww4xtw47CFWUWryUuFyxAFb_yoW8Ar4fpF WFqF42kFs5ur95A3y0y34kWFyUta97Wr15Ar1rJrn8tw1DZF1FyF47CrWF9as29r1xZ3yI vrWDur4UK3WDZa7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUv0b4IE77IF4wAFF20E14v26ryj6rWUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rwA2F7IY1VAKz4 vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7Cj xVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x 0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG 6I80ewAv7VC0I7IYx2IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFV Cjc4AY6r1j6r4UM4x0Y48IcVAKI48JM4IIrI8v6xkF7I0E8cxan2IY04v7MxkF7I0En4kS 14v26r4a6rW5MxAIw28IcxkI7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I 8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVW8ZVWr XwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x 0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_ Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7IU0 bAw3UUUUU== X-CM-SenderInfo: hfkh02xlgr0w46kxt4xhlfz01xgou0bp/ X-Rspam-User: X-Stat-Signature: zj7dedru3xj5415k5sq3gbsfdz1zso1o X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: F2EFC2001B X-HE-Tag: 1751014963-625470 X-HE-Meta: U2FsdGVkX18LIL7Ka+E5n1SRkKT1TjFkqFoVoaRBMhDUJMmRIbGZUK7rtotZyEG/hDvjKrzSOxKP/ptVukOfEMFNcpXpWVksoxyheWI6SM3Va4QZV4b01SHNHnAkRDB4z2/33Es0uh0GkbNebxYv5f9ueydQu+pDB76FnMUS2W4W+WfM3x90D8mDIZM8wULbdfMuIS7mTPVbLctjWnB5l+b90bVuzPl1WNn5cospCqMExP4R9iwBe3dmYMMCGwRyl0X77SzPSPrm2ZFodvi2uVx64S7SOm0CLXz5zKe1T1FVVg0Pfxts0RhDOcaoLhy01Xuo+tQslBUTDbWVJGrp2ufPclhLwwYVa4ueyWrQI2uKuzWJ/jLL+HYPlvcjUUehXIFO9kVeLC/i6yEsauccl9mANE8mbNveiq/H73b+Mv3R2p2CXPqutMwtdnoihclfSkSE+edOvOc2iRGQ6nAJeBS0b10BmuNisP729c3MIUgDXke2QK9/AV0tvzyksDPalLDgcBLkgmwihBUp5es1611Jt/AekawucRUMhWJqYCm9bmsgr6vC07ATKoNIFIFHA/Nskq2oaHAw2AZbd2P9+QW4HUdibJ0JoTkQzSRYUsVy1WGe1onNeCIiPDermMtxkrhIpJT9jUeg94Q7IgWoET/CJS2x/hrEfw092aTslfRCaU62rngO1H08KWqnAIBIan1Lc3YPUY0uxAD37vW9or42R7hbXzBkGJnkza12wnrFarBW+kCLaeC7/x1y2B1LiIMe6JcBsrPScy4ABnINCXEybOGPxbg7PNciFnPiUVY0EfOY+sdtWyBSfh8+HjatCSjS2UymiccAJRxPLigwwe925lxm/IFPoCBYfIXBH0eE2lQePscQEtlIPXowL+xGnhGf8zPGrkhDoGqVrB7ScM+BLKGt1tEJpO0tsUYjFDpzR5VfNGIltNXVPZBYgN7hHZagOej6rr9QQuzpIYc xuzoojdn M8JsEnQRKXKzchzGxzkQuUu8jmHi+TySrDL6fb00JYvIaAz+ardIj8svxMl4D8yXb7zZlB169M6eYuYZ9mLsOiPulduCgt1VlLlbFXMjTz7Iy1xhFPOqNjpIlJxURK8W407LMVXV85BYJ7vtzYyY1N1Nq6w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2025/4/28 11:43, Kairui Song wrote: > On Fri, Apr 18, 2025 at 5:45 AM Roman Gushchin wrote: >> >> On Fri, Apr 18, 2025 at 02:22:12AM +0800, Kairui Song wrote: >>> >>> We currently have some workloads running with `nokmem` due to objcg >>> performance issues. I know there are efforts to improve them, but so >>> far it's still not painless to have. So I'm a bit worried about >>> this... >> >> Do you mind sharing more details here? >> >> Thanks! > > Hi, > > Sorry for the late response, I was busy with another series and other works. > > It's not hard to observe such slow down, for example a simple redis > test can expose it: > > Without nokmem: > redis-benchmark -h 127.0.0.1 -q -t set,get -n 80000 -c 1 > SET: 16393.44 requests per second, p50=0.055 msec > GET: 16956.34 requests per second, p50=0.055 msec > > With nokmem: > redis-benchmark -h 127.0.0.1 -q -t set,get -n 80000 -c 1 > SET: 17263.70 requests per second, p50=0.055 msec > GET: 17410.23 requests per second, p50=0.055 msec > > And I'm testing with latest kernel: > uname -a > Linux localhost 6.15.0-rc2+ #1594 SMP PREEMPT_DYNAMIC Sun Apr 27 > 15:13:27 CST 2025 x86_64 GNU/Linux > > This is just an example. For redis, it can be a workaround by using > things like redis pipeline, but not all workloads can be adjusted > that flexibly. > > And the slowdown could be amplified in some cases. Hi Kairui, We've also encountered this issue in our Redis scenario. May I confirm whether your testing is based on cgroup v1 or v2? In our environment using cgroup v1, we've identified memcg_account_kmem as the critical performance bottleneck function - which, as you know, is specific to the v1 implementation. Best regards, Ridong