From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F6BD3E47B for ; Sun, 26 Jan 2025 07:02:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737874933; cv=none; b=kWj5D1q6x/MXQ99Bov6i8zJYudoViBuR50+xt8sJgKw4eAAX0KUvQbQqa511Gk8ei2AGvVBCWWCVoSo1889PXvdkgxUvD/dAWjJrXkXxfnZtPN5nLFeBk55X7t7t5Y/uaMjiU1zCPsYeddFzWU5T24v32nucX5qXZVKW5KH3qlc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737874933; c=relaxed/simple; bh=6+W9OlcX6swcvu3hEbd+y/dcoKj5qfSsjiZKAd1qXo8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DcPhCdEb2sfSD4nlacEK+GbSAXjNpcrWfLmHmBwL9rWR1nDCs+XgE8pYHqOowA8D50dXRypRT8YUWNe8iJPVlv7i5W3BSPhRCIA+SpGEvns4ct9N/g92OiRGjWBlFnWwbqI4RfVxR9yGFo8kcZiRDEq7U99eyiS29V4zPs5Zfys= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=a2LBNXNE; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="a2LBNXNE" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ef114d8346so6659298a91.0 for ; Sat, 25 Jan 2025 23:02:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737874931; x=1738479731; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u+wqmnI/8fK6TKpLWm3RbIHTrNperIZqHqFOsh7k2n8=; b=a2LBNXNEcnNZumuiLnHXcqSRPNqZsTqkeTJsikm8EFVi1p0KxB+Ry2szx6N4QKD5rv nGtozu50/FZKJ6atX+LSORfJOQ/2vtsi7vfhCLBP3dWzs2NaPpquM3G0cLZInYtXIbqg blqBYySOSQwHvkoHVbiTWxo8BSo1dm9ir4cR1QV8DAv4RhH+KjYg7x3gODHcTgHMigTm rnlm52QIL8SBrYgIlPTWqudM38X+aWWSp35zfM163X71vsVwrItSthlxKlf1ALzgUx8s X5DFD7053iurGUb9DskbH+DdDVgmT134NjMqzo7OO5c8S+JQ0ziA/1NGo6yvR+H5BTx+ +wIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737874931; x=1738479731; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u+wqmnI/8fK6TKpLWm3RbIHTrNperIZqHqFOsh7k2n8=; b=LlZ3srWyd3p0XdLu3MW33LOh87pBRVmBkItw6+hyQvx5KZWpq77K7PRlhA49jRjy2n y8X5xc4T7ZehGr19C3yxpGK/P+j2y+9F29lmCMlhQ3YfjU1Hl+IsyiNhBp9aFhjjXfJ8 NvemnIGSwH0jpfcaU2Y+hm5Q08RHcINErhF0kxOOC6zY6D8FlgCFgkOlX4Rd/SXUo/ht 7IMvao2PZCnfSf6W5G5EKgt3CVpwWUbl0ITyYKrm3VG/IQY0Ih8urOzynmRJhdkucKo6 48Hi102vcW00x71h9unk9GE9n0vLsX95KC6GUxbRCnpL5iqow/KaFVF4gQXJE0ggW7cu fbNA== X-Forwarded-Encrypted: i=1; AJvYcCXbbH//MRa7dxRbdrG8sJZ9x5HxERBO8Bzl+gRWcLsw0p0CnUc6hzttBwIs4uVX+hw6lABVOAgjkGfyYe0=@vger.kernel.org X-Gm-Message-State: AOJu0YyaFwazRAn5iu+7f6FMLp6vj2YB5IyjCDkFfyEz76qiyNcsOvPX B2EijW9TgIkzXlWZWAs+JpOmtBaa7nj0P5L/oE5MdyKtAKcI+6eNX07uE+LqHrhlH8P1OtAwi3E GWA== X-Google-Smtp-Source: AGHT+IHTGE6/Vf5/+TRu1bIQ3qHUzvpK90Z3vXbQx7gL95QZe8ucUUi3LcdMyv8Ds5MGltY+8AwhAlKYMaY= X-Received: from pjbsw13.prod.google.com ([2002:a17:90b:2c8d:b0:2ea:6b84:3849]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3a0e:b0:2ee:b4bf:2d06 with SMTP id 98e67ed59e1d1-2f782cb9fe6mr49139163a91.19.1737874931632; Sat, 25 Jan 2025 23:02:11 -0800 (PST) Date: Sat, 25 Jan 2025 23:02:05 -0800 In-Reply-To: <20250126070206.381302-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250126070206.381302-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250126070206.381302-2-surenb@google.com> Subject: [PATCH 2/3] alloc_tag: uninline code gated by mem_alloc_profiling_key in slab allocator From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" When a sizable code section is protected by a disabled static key, that code gets into the instruction cache even though it's not executed and consumes the cache, increasing cache misses. This can be remedied by moving such code into a separate uninlined function. The improvement however comes at the expense of the configuration when this static key gets enabled since there is now an additional function call. The default state of the mem_alloc_profiling_key is controlled by CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. Apply this optimization only if CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n, improving the performance of the default configuration. When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y the functions are inlined and performance does not change. On a Pixel6 phone, slab allocation profiling overhead measured with CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and profiling disabled: baseline modified Big 3.31% 0.17% Medium 3.79% 0.57% Little 6.68% 1.28% When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and memory allocation profiling gets enabled, the difference in performance before and after this change stays within noise levels. On x86 this patch does not make noticeable difference because the overhead with mem_alloc_profiling_key disabled is much lower (under 1%) to start with, so any improvement is less visible and hard to distinguish from the noise. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 6 +++++ mm/slub.c | 46 ++++++++++++++++++++++++--------------- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index a946e0203e6d..c5de2a0c1780 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -116,6 +116,12 @@ DECLARE_PER_CPU(struct alloc_tag_counters, _shared_alloc_tag); DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, mem_alloc_profiling_key); +#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT +#define inline_if_mem_alloc_prof inline +#else +#define inline_if_mem_alloc_prof noinline +#endif + static inline bool mem_alloc_profiling_enabled(void) { return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, diff --git a/mm/slub.c b/mm/slub.c index 996691c137eb..3107d43dfddc 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2000,7 +2000,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, return 0; } -static inline void free_slab_obj_exts(struct slab *slab) +static inline_if_mem_alloc_prof void free_slab_obj_exts(struct slab *slab) { struct slabobj_ext *obj_exts; @@ -2077,33 +2077,35 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) return slab_obj_exts(slab) + obj_to_index(s, slab, p); } -static inline void -alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) +static inline_if_mem_alloc_prof void +__alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) { - if (need_slab_obj_ext()) { - struct slabobj_ext *obj_exts; + struct slabobj_ext *obj_exts; - obj_exts = prepare_slab_obj_exts_hook(s, flags, object); - /* - * Currently obj_exts is used only for allocation profiling. - * If other users appear then mem_alloc_profiling_enabled() - * check should be added before alloc_tag_add(). - */ - if (likely(obj_exts)) - alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); - } + obj_exts = prepare_slab_obj_exts_hook(s, flags, object); + /* + * Currently obj_exts is used only for allocation profiling. + * If other users appear then mem_alloc_profiling_enabled() + * check should be added before alloc_tag_add(). + */ + if (likely(obj_exts)) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); } static inline void -alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, +alloc_tagging_slab_alloc_hook(struct kmem_cache *s, void *object, gfp_t flags) +{ + if (need_slab_obj_ext()) + __alloc_tagging_slab_alloc_hook(s, object, flags); +} + +static inline_if_mem_alloc_prof void +__alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { struct slabobj_ext *obj_exts; int i; - if (!mem_alloc_profiling_enabled()) - return; - /* slab->obj_exts might not be NULL if it was created for MEMCG accounting. */ if (s->flags & (SLAB_NO_OBJ_EXT | SLAB_NOLEAKTRACE)) return; @@ -2119,6 +2121,14 @@ alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, } } +static inline void +alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) +{ + if (mem_alloc_profiling_enabled()) + __alloc_tagging_slab_free_hook(s, slab, p, objects); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void -- 2.48.1.262.g85cc9f2d1e-goog