From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4750D27B34E for ; Thu, 18 Dec 2025 22:16:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.50 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766096205; cv=none; b=V5mMhSAetul34qVTB/JRvYu7ruKecaFkfltnwiphAU8xmKNuJaLYBU2YA/UU8jqYJM86pz8tC9jezipTdtMcZdEpjGiqVfz4XLj7GQQWrW96QNE4AJ8N8ksEHj7AIuFF5Gb7ohxAPsy5sB7nt57PD9f6BOGF5Ai84LZuIx6le4g= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766096205; c=relaxed/simple; bh=0xd5VQJphUdTkSt449psxow9J9WCcxUvSivv8VMNdlw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=hzwgn2QCGMH7XpNCDSrvP2Bw3DA+HNDMh17suEwt0160M0hIeEJ1yOwjSH6E/G70dK6HpBGGj6LHTR/zXQp4AUw8xtaBMQ0gjPLaf81OLsT12wzKV5+KqsxLs2SY2ohCUHrZ6ogNAFHmEj0/sSnf+2c0y+wZ9EbuEgSnmfKmvdY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PEaxzYGb; arc=none smtp.client-ip=209.85.219.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PEaxzYGb" Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-8885b3c06caso16402806d6.1 for ; Thu, 18 Dec 2025 14:16:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766096202; x=1766701002; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:feedback-id:from:to:cc:subject:date :message-id:reply-to; bh=3sXHinIvqqn4NKYj3YLqEcLxxqdNblij/w/+MvNbYRQ=; b=PEaxzYGbB2hvC1O6TovmlaRJjq3RyJPYJS7TFbM1kCZM7cA3Msg/eu4VqVybqfT3rw bHsoh63pAhnnGcFzrBZHVJEG6qZs5CIE3DQReUEr7KWrroP7s2X0qpC1/1ucz2at38vb H7GVd1BSGhYGzM5cScVJt9NhAs/J+XiW/H8jfSSeBSdHpFnN/3qbjVsPHllW2Ai73teA xKKdPaQjpPtd6aAr/19IseFmM3zA2qWVPFS1c962pnG+iUXNZzWPtPc7BEAEb3VjL97r +qCNjlDW3GHDlkCQrrgZC0lWts0CGUkCT1cjZJpCOeXiz20Wdi/X/sbitDGl3xc0lUPs 1VOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766096202; x=1766701002; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:feedback-id:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3sXHinIvqqn4NKYj3YLqEcLxxqdNblij/w/+MvNbYRQ=; b=sK/TRIN6B/kdGwJ5ZpSgtQlge/y2J7kEh3VT4gOm/4olTueE20qHiKmlHLmdYunJz4 ct5ltiy4dVxD16f2DYqw9VpA2hFPu22/0AmnDOz8ADZPPS/FwPUzZdcnLYJ9BjkNzwKk IATrSJECaRFkIJgVvWxK9xsfw18QnKp9j/+IWYeOg8WgNytApEtJj9EsVFvfyF9MC8zA +QeVRC69S0p908RdO3dXo705KRU+eKXbD2c26dthe2hXqtVUCkTjpNb3ftqim3RjZUdT 7tDO0EXZODUwVvmjU1ytITBCH6y+TAwVk1NI+JdLgttqt9Gl3AsFI5/UfAQiUU5Y5Iv8 glyg== X-Forwarded-Encrypted: i=1; AJvYcCXzmspzVEQ+gVRuG+So+VB0Vv2esUpn4QhHiVCeZ5KNiTORkgsJbBzRFxOJZ0H7DejeR8BYk1amw8R1Vok=@vger.kernel.org X-Gm-Message-State: AOJu0Yyge8+bovW11xkUp1ugAxA8TGPDjS3LKzFTSab/IJkyQjkalaB3 PLne2nEqX1CEBlam/yghep7YM3ioFEYmQZVK4S8IEK+Y1e9PtYi3rUtu X-Gm-Gg: AY/fxX6zjGjwQ+k+JtNHTUsTukPSHJPJMOHcrwniyzpwVb8kXrVxHRwlCgB3qMVHnpw N2PK+ibEzY8FGPSJNDquTxrkcHP5wvvbOjzxgHoBznPokXiydDQgNlFeQl85Jjt4z2uPhMTwIZE k2pJXwfkLVc7ob8dJywVsqrFG08+sFN4/qTxDmeKkQP4useln2MeJTTU8aDH0v6s7lLIhd6rV63 S8JCUqqDsnZUbjgM0uL+7OEAim9xfgW3vm5WSPVwi07sm2xjTle3mjKgyBcDnheYCK+Tw3oAv9V v2wWPayN246Pw8yjX/Tjzh/A3QhZ6gc/BjBuNDm4xqpvWJc4CZVCF83BcOMrbWuI6fdkSIqc9ox P+BXZhb7HGbwz5X6P+JwitcQ+U+q3hsAm+JTki30XWaBl5dOuxfNULSRsMZqlinT8/AKdaJX3lO aSopFm7SXs3BsjCJeSEBLfunADXdl9+58oBKB76uMEe3a27g39S2HgWmLjhsve0Mbdm++jlZJmG nSI9SC1wxc7VOM= X-Google-Smtp-Source: AGHT+IH9dlus8CLBnccpPwM7k0azGKuifjOTEiDe/F0ZElbIptxHmkvJntfFb5aH3wD+BBl6lym7+A== X-Received: by 2002:ad4:55e6:0:b0:888:58e9:fda1 with SMTP id 6a1803df08f44-88d851fd95cmr13819516d6.14.1766096202075; Thu, 18 Dec 2025 14:16:42 -0800 (PST) Received: from fauth-a2-smtp.messagingengine.com (fauth-a2-smtp.messagingengine.com. [103.168.172.201]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-88d997aeef5sm5249626d6.27.2025.12.18.14.16.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Dec 2025 14:16:41 -0800 (PST) Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfauth.phl.internal (Postfix) with ESMTP id AA726F4006D; Thu, 18 Dec 2025 17:16:40 -0500 (EST) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Thu, 18 Dec 2025 17:16:40 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefgedrtddtgdegieeitdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhephedugfduffffteeutddvheeuveelvdfhleelieevtdeguefhgeeuveeiudffiedv necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqh hunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddu jeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvg drnhgrmhgvpdhnsggprhgtphhtthhopeeffedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtohepmhgrthhhihgvuhdruggvshhnohihvghrshesvghffhhitghiohhsrdgtohhmpd hrtghpthhtohepjhhovghlsehjohgvlhhfvghrnhgrnhguvghsrdhorhhgpdhrtghpthht ohepphgruhhlmhgtkheskhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhhinhhugidqkh gvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepnhhpihhgghhi nhesghhmrghilhdrtghomhdprhgtphhtthhopehmphgvsegvlhhlvghrmhgrnhdrihgurd gruhdprhgtphhtthhopehgrhgvghhkhheslhhinhhugihfohhunhgurghtihhonhdrohhr ghdprhgtphhtthhopegsihhgvggrshihsehlihhnuhhtrhhonhhigidruggvpdhrtghpth htohepfihilhhlsehkvghrnhgvlhdrohhrgh X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 18 Dec 2025 17:16:39 -0500 (EST) Date: Fri, 19 Dec 2025 07:16:37 +0900 From: Boqun Feng To: Mathieu Desnoyers Cc: Joel Fernandes , "Paul E. McKenney" , linux-kernel@vger.kernel.org, Nicholas Piggin , Michael Ellerman , Greg Kroah-Hartman , Sebastian Andrzej Siewior , Will Deacon , Peter Zijlstra , Alan Stern , John Stultz , Neeraj Upadhyay , Linus Torvalds , Andrew Morton , Frederic Weisbecker , Josh Triplett , Uladzislau Rezki , Steven Rostedt , Lai Jiangshan , Zqiang , Ingo Molnar , Waiman Long , Mark Rutland , Thomas Gleixner , Vlastimil Babka , maged.michael@gmail.com, Mateusz Guzik , Jonas Oberhauser , rcu@vger.kernel.org, linux-mm@kvack.org, lkmm@lists.linux.dev Subject: Re: [RFC PATCH v4 4/4] hazptr: Migrate per-CPU slots to backup slot on context switch Message-ID: References: <20251218014531.3793471-1-mathieu.desnoyers@efficios.com> <20251218014531.3793471-5-mathieu.desnoyers@efficios.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20251218014531.3793471-5-mathieu.desnoyers@efficios.com> On Wed, Dec 17, 2025 at 08:45:31PM -0500, Mathieu Desnoyers wrote: > Integrate with the scheduler to migrate per-CPU slots to the backup slot > on context switch. This ensures that the per-CPU slots won't be used by > blocked or preempted tasks holding on hazard pointers for a long time. > > Signed-off-by: Mathieu Desnoyers > Cc: Nicholas Piggin > Cc: Michael Ellerman > Cc: Greg Kroah-Hartman > Cc: Sebastian Andrzej Siewior > Cc: "Paul E. McKenney" > Cc: Will Deacon > Cc: Peter Zijlstra > Cc: Boqun Feng > Cc: Alan Stern > Cc: John Stultz > Cc: Neeraj Upadhyay > Cc: Linus Torvalds > Cc: Andrew Morton > Cc: Boqun Feng > Cc: Frederic Weisbecker > Cc: Joel Fernandes > Cc: Josh Triplett > Cc: Uladzislau Rezki > Cc: Steven Rostedt > Cc: Lai Jiangshan > Cc: Zqiang > Cc: Ingo Molnar > Cc: Waiman Long > Cc: Mark Rutland > Cc: Thomas Gleixner > Cc: Vlastimil Babka > Cc: maged.michael@gmail.com > Cc: Mateusz Guzik > Cc: Jonas Oberhauser > Cc: rcu@vger.kernel.org > Cc: linux-mm@kvack.org > Cc: lkmm@lists.linux.dev > --- > include/linux/hazptr.h | 63 ++++++++++++++++++++++++++++++++++++++++-- > include/linux/sched.h | 4 +++ > init/init_task.c | 3 ++ > kernel/Kconfig.preempt | 10 +++++++ > kernel/fork.c | 3 ++ > kernel/sched/core.c | 2 ++ > 6 files changed, 83 insertions(+), 2 deletions(-) > > diff --git a/include/linux/hazptr.h b/include/linux/hazptr.h > index 70c066ddb0f5..10ac53a42a7a 100644 > --- a/include/linux/hazptr.h > +++ b/include/linux/hazptr.h > @@ -24,6 +24,7 @@ > #include > #include > #include > +#include > > /* 8 slots (each sizeof(void *)) fit in a single cache line. */ > #define NR_HAZPTR_PERCPU_SLOTS 8 > @@ -46,6 +47,9 @@ struct hazptr_ctx { > struct hazptr_slot *slot; > /* Backup slot in case all per-CPU slots are used. */ > struct hazptr_backup_slot backup_slot; > +#ifdef CONFIG_PREEMPT_HAZPTR I would suggest we make CONFIG_PREEMPT_HAZPTR always enabled hence no need for a config, do we have the measurement of the additional cost? > + struct list_head preempt_node; > +#endif > }; > > struct hazptr_percpu_slots { > @@ -98,6 +102,50 @@ bool hazptr_slot_is_backup(struct hazptr_ctx *ctx, struct hazptr_slot *slot) > return slot == &ctx->backup_slot.slot; > } > > +#ifdef CONFIG_PREEMPT_HAZPTR > +static inline > +void hazptr_chain_task_ctx(struct hazptr_ctx *ctx) > +{ > + list_add(&ctx->preempt_node, ¤t->hazptr_ctx_list); > +} > + > +static inline > +void hazptr_unchain_task_ctx(struct hazptr_ctx *ctx) > +{ > + list_del(&ctx->preempt_node); > +} > + I think you need to add interrupt disabling for chain/unchain because of the potential readers in interrupt and then you can avoid the preempt disabling in hazptr_release() I think. Let's aim for supporting readers in interrupt handler, because at least lockdep needs that. Regards, Boqun > +static inline > +void hazptr_note_context_switch(void) > +{ > + struct hazptr_ctx *ctx; > + > + list_for_each_entry(ctx, ¤t->hazptr_ctx_list, preempt_node) { > + struct hazptr_slot *slot; > + > + if (hazptr_slot_is_backup(ctx, ctx->slot)) > + continue; > + slot = hazptr_chain_backup_slot(ctx); > + /* > + * Move hazard pointer from per-CPU slot to backup slot. > + * This requires hazard pointer synchronize to iterate > + * on per-CPU slots with load-acquire before iterating > + * on the overflow list. > + */ > + WRITE_ONCE(slot->addr, ctx->slot->addr); > + /* > + * store-release orders store to backup slot addr before > + * store to per-CPU slot addr. > + */ > + smp_store_release(&ctx->slot->addr, NULL); > + } > +} > +#else > +static inline void hazptr_chain_task_ctx(struct hazptr_ctx *ctx) { } > +static inline void hazptr_unchain_task_ctx(struct hazptr_ctx *ctx) { } > +static inline void hazptr_note_context_switch(void) { } > +#endif > + > /* > * hazptr_acquire: Load pointer at address and protect with hazard pointer. > * > @@ -114,6 +162,7 @@ void *hazptr_acquire(struct hazptr_ctx *ctx, void * const * addr_p) > struct hazptr_slot *slot = NULL; > void *addr, *addr2; > > + ctx->slot = NULL; > /* > * Load @addr_p to know which address should be protected. > */ > @@ -121,7 +170,9 @@ void *hazptr_acquire(struct hazptr_ctx *ctx, void * const * addr_p) > for (;;) { > if (!addr) > return NULL; > + > guard(preempt)(); > + hazptr_chain_task_ctx(ctx); > if (likely(!hazptr_slot_is_backup(ctx, slot))) { > slot = hazptr_get_free_percpu_slot(); > /* > @@ -140,8 +191,11 @@ void *hazptr_acquire(struct hazptr_ctx *ctx, void * const * addr_p) > * Re-load @addr_p after storing it to the hazard pointer slot. > */ > addr2 = READ_ONCE(*addr_p); /* Load A */ > - if (likely(ptr_eq(addr2, addr))) > + if (likely(ptr_eq(addr2, addr))) { > + ctx->slot = slot; > + /* Success. Break loop, enable preemption and return. */ > break; > + } > /* > * If @addr_p content has changed since the first load, > * release the hazard pointer and try again. > @@ -150,11 +204,14 @@ void *hazptr_acquire(struct hazptr_ctx *ctx, void * const * addr_p) > if (!addr2) { > if (hazptr_slot_is_backup(ctx, slot)) > hazptr_unchain_backup_slot(ctx); > + hazptr_unchain_task_ctx(ctx); > + /* Loaded NULL. Enable preemption and return NULL. */ > return NULL; > } > addr = addr2; > + hazptr_unchain_task_ctx(ctx); > + /* Enable preemption and retry. */ > } > - ctx->slot = slot; > /* > * Use addr2 loaded from the second READ_ONCE() to preserve > * address dependency ordering. > @@ -170,11 +227,13 @@ void hazptr_release(struct hazptr_ctx *ctx, void *addr) > > if (!addr) > return; > + guard(preempt)(); > slot = ctx->slot; > WARN_ON_ONCE(slot->addr != addr); > smp_store_release(&slot->addr, NULL); > if (unlikely(hazptr_slot_is_backup(ctx, slot))) > hazptr_unchain_backup_slot(ctx); > + hazptr_unchain_task_ctx(ctx); > } [...]