From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B49BC4332F for ; Tue, 20 Dec 2022 23:28:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7009B8E0002; Tue, 20 Dec 2022 18:28:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B0F58E0001; Tue, 20 Dec 2022 18:28:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5797E8E0002; Tue, 20 Dec 2022 18:28:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 472268E0001 for ; Tue, 20 Dec 2022 18:28:25 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0DAC914016E for ; Tue, 20 Dec 2022 23:28:25 +0000 (UTC) X-FDA: 80264275770.05.3EB9B8B Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) by imf17.hostedemail.com (Postfix) with ESMTP id 6D4AC4000B for ; Tue, 20 Dec 2022 23:28:23 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZHCUC4sY; spf=pass (imf17.hostedemail.com: domain of shakeelb@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671578903; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K8qBWuU9lWrXOPNMkI4xvvAbeO7Vw8VxOj8wpST/GOY=; b=789AyE5OxqhRUBMc+mb3DsT6qTDZAYqfzQP1f/gggZRxXYOKGZog1Zw5qsMh9vSztlzyvm fz5P8zATxj5KCXWmG/2DfpEywpRCcloWtUx0l+Lsaw+CsG1uXr69ThQj4aSizUu7QAy07l MdlH0k7ydoXdpuZ9vYH9fypXWMO/6YU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZHCUC4sY; spf=pass (imf17.hostedemail.com: domain of shakeelb@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671578903; a=rsa-sha256; cv=none; b=rdstq237psOuCKYl1ZJ5QlXivV0fSMe7mX+tkbw8FVHheTmIdDG8txIRI3oEkvIgAFb9HE EQaZJGqQihT8HVORluc9uh0ljbntWw4R/5TyJD7HIQ1UrkRBlx4uEt38TlDwgQRZbK/Lvl 6ih3XrBUCux8QX8YpaDvvKwvMsbC2fI= Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-45f4aef92daso5307847b3.0 for ; Tue, 20 Dec 2022 15:28:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=K8qBWuU9lWrXOPNMkI4xvvAbeO7Vw8VxOj8wpST/GOY=; b=ZHCUC4sYVijQClbfGy9lCOCGsTxidOH/HGkPOjbsE3ThDWMvFmTmX2PuoejRKKkg0Z V5MpGwB2PS28tvdKSV3Ar93uYkMTVuu73YFwoBDlnNO8uwVa5F+WUYhU+pYHhEnnYyhn 8FWweQ0WcaKZRexuf3rSABxIY66m8E4L266zqla1Qrzpsv1VBF636+o6PffptRRSLcb3 XHdiGNGl/r4vjwjDV7w+So60AcPtg9mFP1bVcVt+Nsr9LtqYAqKkTCidJPV7XLTxiBfT sepGgpIPVmjMVxo/FpwcOLjbeWxE8BcCmn+cTBVy21EeW1ZttCW0Wtf+dr4VxeINxIfX JDxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=K8qBWuU9lWrXOPNMkI4xvvAbeO7Vw8VxOj8wpST/GOY=; b=10Hgy0+a5fcoe3NFZ92N8UkfU14quvR9GCZ4bbozXfT8HqXuHFttj5zi8jQIJECqlF 2k+bgiItp2JhO3BOlclfs4F+iDStoyQjvX9+mv7dQQSoLP7H2QY1gqelGaoK2tVEj+zF QEMR6Oaa2kUhMvlJei2F8C4ZEo/GPpt6rKXnicnLXH2x7h3aqZbKDCCEj5wf7isxW4Sh nt6n0ltO5peWUAgpjfIetO94V+gjQcLqzSasKDIPTxucWyL4pDMX5/VmokM3LJdr0228 s4GJ81P4KFqLWxrAReG61W145Asgh2z3+3p4FriPw0ainXV2zE/r7RVY5g1OdA7YaNld +Ehg== X-Gm-Message-State: AFqh2kqCUZWy6k+o55aNozHXP090YlG4Tau+eUsKkEHphTuh3IfE5qGG KNFt7qFvBx+hAoDOC0aopJQ7dVXn/QpGyuDLrvwftQ== X-Google-Smtp-Source: AMrXdXuIHLK4fsh2qBj6ouNMe+aKaVwdL679K0vPCdyCPv4vWhJXZZeZmAaaxGX7Uhgx8q+63Gl6cprdLjrf+BSBoUc= X-Received: by 2002:a81:8643:0:b0:3d9:dd22:c785 with SMTP id w64-20020a818643000000b003d9dd22c785mr2444880ywf.486.1671578902467; Tue, 20 Dec 2022 15:28:22 -0800 (PST) MIME-Version: 1.0 References: <20221220184813.1908318-1-roman.gushchin@linux.dev> In-Reply-To: From: Shakeel Butt Date: Tue, 20 Dec 2022 15:28:11 -0800 Message-ID: Subject: Re: [PATCH RFC] ipc/mqueue: introduce msg cache To: Roman Gushchin Cc: Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Johannes Weiner , Michal Hocko , Muchun Song , Andrew Morton , Waiman Long , Sven Luther Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6D4AC4000B X-Stat-Signature: kqd4xsqsofotjw36buirwkgd1hyz856t X-Rspam-User: X-HE-Tag: 1671578903-405363 X-HE-Meta: U2FsdGVkX18VcrFnvXbclV4hQHSZX8pVvzSqxOtzqOvf9+T1TuGbcd5FnZKk2EW2LB8a6x5Knp6KYVP+3OfawNWL94I7FhaSU/3wSMmF1f/c86N+/IOz317YVqfYnEzXefeRbWgHFDiEBm956lo5XSge0RBddBB6+aUoAT/clAvxtRIA1y6AfX4Mh/bLSCriG67dDK2bkmM6WoXhpj/nffzR0FXZNLMak/KQrxKfnRaRA+EL8RRi8imMUHuaA4UR8Uv/XS3qpku0ay8A/pbR0cyZA+nxXPBxqLNxscAHbyWi21buNp7wXbOLdg6uzsnu7oyltELYASE64k1rZBl9KCcxKmGx4pC5YWSIquqGIHTZ5cEnh41absRPkjpcLIp/ns6sqDo36Vjzruq+SCTe1HdVQgpcDiPSWb3fODq4At/YKLuL+d8W5zDWzgaKoetM/FFgGdaSBD8Ty5TSkniGvRRvFZjmb9eJWz5cUKDsnbCLG5p1lbzfqsy7qLezp9sPpVu+R6+MnwrMQRTrJV5zpubg9tgGEkbW4CKz3Qr9BaKHEhPRx6NU/W64HAIzJb7FqfaEAPHk4YS9NOm52GQAyqs+1DoxSRvt3WhGftawkCd7KaToO9HejsE8TL0i6XfT3M9ecVKyYcOcL3m7tmKqCW5UP6UzqS9nRe5oAe9NdU3zUHtKhpYAJYOXZ76+2ZNATPz026q54gvQe0GINzmQF3cJTIMYtl3BTckySMc2V3lNllWVJzLlUILLC5Kb7Ow5ObUavOd5RLc3him5KWPd+5GT5kShHWAiFMZv4dk478XbfH7AXSEUkIZbIhXzDAiL4biYTRVImqDGqhPhEYJ3VFdrZ89EPdmR2CimVpnOWQ1uYOI3v4+l4VNSrhtwLYkyzw8sAFpOMUAqd5BF85S7sm8MtW+Y/MZYqUuPNjhY8ZJFUMmwL1DmGHeR534hIo5EG9opXe3CsZtK9i1AA8U MZ/a5wX3 krBAbBWfmb5fdebg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Dec 20, 2022 at 12:59 PM Roman Gushchin wrote: > > On Tue, Dec 20, 2022 at 11:53:25AM -0800, Shakeel Butt wrote: > > +Vlastimil > > > > On Tue, Dec 20, 2022 at 10:48 AM Roman Gushchin > > wrote: > > > > > > Sven Luther reported a regression in the posix message queues > > > performance caused by switching to the per-object tracking of > > > slab objects introduced by patch series ending with the > > > commit 10befea91b61 ("mm: memcg/slab: use a single set of kmem_caches for all > > > allocations"). > > > > > > To mitigate the regression cache allocated mqueue messages on a small > > > percpu cache instead of releasing and re-allocating them every time. > > > > > > > Seems fine with me but I am wondering what is stopping us to do this > > caching in the slab layer for all accounted allocations? Does this > > only make sense for specific scenarios/use-cases? > > It's far from trivial, unfortunately. Here we have an mqueue object to associate > a percpu cache with and the hit rate is expected to be high, assuming the mqueue > will be used to pass a lot of messages. > > With a generic slab cache we return to the necessity of managing > the per-cgroup x per-slab-cache x per-cpu free list (or some other data structure), > which is already far from trivial, based on the previous experience. It can > easily lead to a significant memory waste, which will fully compensate all perf > wins. > > So probably we need some heuristics to allocate caches only for really hot slab > caches and use some sort of a hash map (keyed by cgroup and slab cache) to > access freelists. What I miss to commit more time to this project (aside from not > having it), is the lack of real workloads which will noticeably win from this work. > > Sven provided a good example and benchmark to reproduce the regression, so it > was easy to justify the work. > Thanks for the explanation. I think we should add this to the commit message as well. I do think we should have a general framework for such caching as there are other users (e.g. io_uring) doing the same and some future users can take advantage as well e.g. I think this type of caching will be helpful for filelock_cache as well. Anyways that can be done in future.