From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9314C19DF81 for ; Sun, 26 Jan 2025 07:02:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737874938; cv=none; b=ee5uGHR9pgzbLivPwx70a4GFJc0ePXUAkmFEgun5PKdUNVubgojfpy8VddaQ+QE6rh/Riy4MqLE2op3kHyuvVU8x2MZMX6I7lz7brX26pgMpBFr/SSfAlDhKU6B1pFI7msNikM5BbbwPMW0914GnxdevS0Z4q+PrN7qjDqLsJL0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737874938; c=relaxed/simple; bh=lH23q4YxPo1tev8kDHNhwI/G4e+HazGWPW3g5YGkPKQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=iYMlNAAh14Mqexylk88OYqxtEaiE+NifHD0muMRJuZ1NIYrC9RTH1ewvmWjzgyxqviXU82zVNQQlUkCp1hxglexXUF8oxnwbTD7Pnb2VM7LTXv6VXyxAWAFO1qecj6mJ6oSImSAoKAvIKumIp2R/TBzN0lJEhKH8TtctybZ040Y= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=SjeN5Ank; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SjeN5Ank" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2164fad3792so55334255ad.0 for ; Sat, 25 Jan 2025 23:02:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737874934; x=1738479734; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VuMsPHTSMtWsNhCdVIn1M3R177GHzQc3VDuucKTMIUU=; b=SjeN5Ankrmu1ScSr9pGxMOiLHtyQaH9dFRomK+FrPZd43KhQiQSbP3yEAoyFlR1tyZ zGQsKZi3lXp63MlILzg735gGTJWLJ1MgpNMjTzCzYiuVgOqca1V9FrPnyVb99ah5z2cN S1IeXV6Kj9NLqw6MgfsCs2t9g5SQ/vnAirh1HzPcRz7eHaW4qFSfmQ1E+MRGhKUtAdpU 9xLT54qs8pPSULBCv7NkPa90yw6Ldg2DzXrxybG0n5DzigxWtQdMuXVwcLyW78BOc9/m c7hQAS5IgwpcjQWT+y9L1KODhuwRLdVOsCIpV3numHlBz0/GaV+qVipUINrv85s9TNZG DPew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737874934; x=1738479734; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VuMsPHTSMtWsNhCdVIn1M3R177GHzQc3VDuucKTMIUU=; b=c3tlZyMbFSQewN51zeEo7n32abfdPKFVkKQ2tZZGhc+Vaw+0ZFlP4+pdhGpoyP0ihD D2YiYWeFDVMEg7/w6nFQuVbuvjGVDTfdwhBqj8hpm5zqKHjENPjUK/HBHlkXsrlE7X8d fKcxDpTQTG+Osh8o9Ukj2jMnP3XG7kDedJxabxUxxVzN2nVv2GKnYfE32S/xU4XKK4ZT VjoRG1OPHmXzQQhUt/+lJSKpD/gwAzxvSfGXO2+MVFIqLXDCGOGf2B5/KAUNh1RdRLS5 zR0E+bjsAWsbJnF0owB9Ra7X7p2C129d7Why0yqdzU/dqdqGFJyigTTQqHEZGc7dGufw eowA== X-Forwarded-Encrypted: i=1; AJvYcCVngODE72oSYQZRVjsU+eR1469uoNiz9a81A9IbdF7oMICuRLYUfRdvuWK/Sowkw/twuHA1y1ddOEPzmcw=@vger.kernel.org X-Gm-Message-State: AOJu0YzcupRBpBcUZsvz3wPWWgArpdYnxLwNMvzWwiH9sQaDuEo1uLub XytjW9/Pn8nXKUu2SsG4diXy0xaWjUSlpOL1s/uOgIC0rZppNiwmW+i5eHAYcTnZ2Ab7QPq211k +pQ== X-Google-Smtp-Source: AGHT+IHy+6ALp0tq0HhC4NNxguhtqx7IJQwGFlemwAnliRSj9IqTbeoXX2L4PWoV+tmh3gWwLoTTF0wsrrU= X-Received: from pgbdh19.prod.google.com ([2002:a05:6a02:b93:b0:7fd:50ab:dc45]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:81a3:b0:1e8:bff6:8356 with SMTP id adf61e73a8af0-1eb214cb158mr41901012637.20.1737874933839; Sat, 25 Jan 2025 23:02:13 -0800 (PST) Date: Sat, 25 Jan 2025 23:02:06 -0800 In-Reply-To: <20250126070206.381302-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250126070206.381302-1-surenb@google.com> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog Message-ID: <20250126070206.381302-3-surenb@google.com> Subject: [PATCH 3/3] alloc_tag: uninline code gated by mem_alloc_profiling_key in page allocator From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, vbabka@suse.cz, yuzhao@google.com, minchan@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, 00107082@163.com, quic_zhenhuah@quicinc.com, surenb@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" When a sizable code section is protected by a disabled static key, that code gets into the instruction cache even though it's not executed and consumes the cache, increasing cache misses. This can be remedied by moving such code into a separate uninlined function. The improvement however comes at the expense of the configuration when this static key gets enabled since there is now an additional function call. The default state of the mem_alloc_profiling_key is controlled by CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. Apply this optimization only if CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n, improving the performance of the default configuration. When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y the functions are inlined and performance does not change. On a Pixel6 phone, page allocation profiling overhead measured with CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and profiling disabled: baseline modified Big 4.93% 1.53% Medium 4.39% 1.41% Little 1.02% 0.36% When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n and memory allocation profiling gets enabled, the difference in performance before and after this change stays within noise levels. On x86 this patch does not make noticeable difference because the overhead with mem_alloc_profiling_key disabled is much lower (under 1%) to start with, so any improvement is less visible and hard to distinguish from the noise. Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 60 +++------------------------- mm/page_alloc.c | 78 +++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 55 deletions(-) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 4a82b6b4820e..c74077977830 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -162,47 +162,13 @@ static inline void update_page_tag_ref(union pgtag_ref_handle handle, union code } } -static inline void clear_page_tag_ref(struct page *page) -{ - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - set_codetag_empty(&ref); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } -} - -static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, - unsigned int nr) -{ - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } -} +/* Should be called only if mem_alloc_profiling_enabled() */ +void __clear_page_tag_ref(struct page *page); -static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) +static inline void clear_page_tag_ref(struct page *page) { - if (mem_alloc_profiling_enabled()) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(page, &ref, &handle)) { - alloc_tag_sub(&ref, PAGE_SIZE * nr); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } + if (mem_alloc_profiling_enabled()) + __clear_page_tag_ref(page); } /* Should be called only if mem_alloc_profiling_enabled() */ @@ -222,18 +188,6 @@ static inline struct alloc_tag *__pgalloc_tag_get(struct page *page) return tag; } -static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) -{ - struct alloc_tag *tag; - - if (!mem_alloc_profiling_enabled()) - return; - - tag = __pgalloc_tag_get(page); - if (tag) - this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); -} - void pgalloc_tag_split(struct folio *folio, int old_order, int new_order); void pgalloc_tag_swap(struct folio *new, struct folio *old); @@ -242,10 +196,6 @@ void __init alloc_tag_sec_init(void); #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void clear_page_tag_ref(struct page *page) {} -static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, - unsigned int nr) {} -static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} -static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) {} static inline void alloc_tag_sec_init(void) {} static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) {} static inline void pgalloc_tag_swap(struct folio *new, struct folio *old) {} diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 55ed2f245f80..67e205286dbf 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1041,6 +1041,84 @@ static void kernel_init_pages(struct page *page, int numpages) kasan_enable_current(); } +#ifdef CONFIG_MEM_ALLOC_PROFILING + +/* Should be called only if mem_alloc_profiling_enabled() */ +void __clear_page_tag_ref(struct page *page) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + set_codetag_empty(&ref); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline_if_mem_alloc_prof +void __pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_add(&ref, task->alloc_tag, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) +{ + if (mem_alloc_profiling_enabled()) + __pgalloc_tag_add(page, task, nr); +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline_if_mem_alloc_prof +void __pgalloc_tag_sub(struct page *page, unsigned int nr) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(page, &ref, &handle)) { + alloc_tag_sub(&ref, PAGE_SIZE * nr); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } +} + +static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) +{ + if (mem_alloc_profiling_enabled()) + __pgalloc_tag_sub(page, nr); +} + +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) +{ + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + tag = __pgalloc_tag_get(page); + if (tag) + this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) {} +static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} +static inline void pgalloc_tag_sub_pages(struct page *page, unsigned int nr) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + __always_inline bool free_pages_prepare(struct page *page, unsigned int order) { -- 2.48.1.262.g85cc9f2d1e-goog