From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 205673E2752 for ; Tue, 5 May 2026 08:26:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777969615; cv=none; b=sq81rxhDpd94Kq/2B8hzjliIAeUG+7w/qwg4SII/fUJdPThxq1J0CK/6vblVELJcVIsQ18kx5TFdSoGSCCvtxFJnOtgaJOOxxDajRv58Y/qgjSmZB7eHIo6XSijeMeFtHkcwn1H41FzTmP6L4Muxd9MrTHGaJSrNxpMfR9R7A5M= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777969615; c=relaxed/simple; bh=+ocCNrmOThZDtwyYwAwCZLnSSO0XgSjvLv6B1ebvYv8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JGDXp6f1oaHoDhGJ2MB3zklcx2Cg5gq3bNvnsS4O8xmy9pEzC2BdjlPiYrIw8iM2wgKFP7Lx68zoPOxiYgi5MQbh8KXd6EKnWZ3zZGak0JEl7mx6AyqGdO0MlqqJlgwDF/BKrkpoCqsyzxrdERI7tJUt6sJ0caJpKa8gNx+iBII= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=g4TcVkbm; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="g4TcVkbm" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1777969613; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c3MM+Rrr4gllCDOAXA/Nf1PZQdNA+vK+kEYbUU2YmTg=; b=g4TcVkbmfwPKxRzsKiyChRvnclPjstx1Ezwl7oeZxsW4Q6V0pgCPA4+qfoP0ywpVrAT4hX m5ktNUKItPej8Fzgd5Xj/fu7M5Xhby8/aQndGOeA69fW56kI6L72HIEtWGqA5KA/Hl3MJZ ohF+fYykOJtlsHMFX513wKlxh7qZlY8= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-484-hB-RglEGNquoXBpJTHj0OA-1; Tue, 05 May 2026 04:26:49 -0400 X-MC-Unique: hB-RglEGNquoXBpJTHj0OA-1 X-Mimecast-MFC-AGG-ID: hB-RglEGNquoXBpJTHj0OA_1777969605 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 44EFF19560A6; Tue, 5 May 2026 08:26:45 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.44.48.109]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1519F30001A1; Tue, 5 May 2026 08:26:31 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: "Peter Zijlstra (Intel)" , Nicolas Saenz Julienne , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Arnaldo Carvalho de Melo , Josh Poimboeuf , Paolo Bonzini , Arnd Bergmann , Frederic Weisbecker , "Paul E. McKenney" , Jason Baron , Steven Rostedt , Ard Biesheuvel , Sami Tolvanen , "David S. Miller" , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Mel Gorman , Andrew Morton , Masahiro Yamada , Han Shen , Rik van Riel , Jann Horn , Dan Carpenter , Oleg Nesterov , Juri Lelli , Clark Williams , Tomas Glozar , Yair Podemsky , Marcelo Tosatti , Daniel Wagner , Petr Tesarik , Shrikanth Hegde Subject: [PATCH v9 09/10] context_tracking,x86: Defer kernel text patching IPIs when tracking CR3 switches Date: Tue, 5 May 2026 10:23:54 +0200 Message-ID: <20260505082355.1982003-10-vschneid@redhat.com> In-Reply-To: <20260505082355.1982003-1-vschneid@redhat.com> References: <20260505082355.1982003-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 text_poke_bp_batch() sends IPIs to all online CPUs to synchronize them vs the newly patched instruction. CPUs that are executing in userspace do not need this synchronization to happen immediately, and this is actually harmful interference for NOHZ_FULL CPUs. As the synchronization IPIs are sent using a blocking call, returning from text_poke_bp_batch() implies all CPUs will observe the patched instruction(s), and this should be preserved even if the IPI is deferred. In other words, to safely defer this synchronization, any kernel instruction leading to the execution of the deferred instruction sync must *not* be mutable (patchable) at runtime. This means we must pay attention to mutable instructions in the early entry code: - alternatives - static keys - static calls - all sorts of probes (kprobes/ftrace/bpf/???) The early entry code is noinstr, which gets rid of the probes. Alternatives are safe, because it's boot-time patching (before SMP is even brought up) which is before any IPI deferral can happen. This leaves us with static keys and static calls. Any static key used in early entry code should be only forever-enabled at boot time, IOW __ro_after_init (pretty much like alternatives). Exceptions to that will now be caught by objtool. The deferred instruction sync is the CR3 RMW done as part of kPTI when switching to the kernel page table: SDM vol2 chapter 4.3 - Move to/from control registers: ``` MOV CR* instructions, except for MOV CR8, are serializing instructions. ``` Leverage the new kernel_cr3_loaded signal and the kPTI CR3 RMW to defer sync_core() IPIs targeting NOHZ_FULL CPUs. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Nicolas Saenz Julienne Signed-off-by: Valentin Schneider --- arch/x86/include/asm/text-patching.h | 5 +++ arch/x86/kernel/alternative.c | 57 +++++++++++++++++++++++++--- arch/x86/kernel/kprobes/core.c | 4 +- arch/x86/kernel/kprobes/opt.c | 4 +- arch/x86/kernel/module.c | 2 +- include/asm-generic/sections.h | 14 +++++++ 6 files changed, 75 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index f2d142a0a862e..628e80f8318cd 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -33,6 +33,11 @@ extern void text_poke_apply_relocation(u8 *buf, const u8 * const instr, size_t i */ extern void *text_poke(void *addr, const void *opcode, size_t len); extern void smp_text_poke_sync_each_cpu(void); +#ifdef CONFIG_TRACK_CR3 +extern void smp_text_poke_sync_each_cpu_deferrable(void); +#else +#define smp_text_poke_sync_each_cpu_deferrable smp_text_poke_sync_each_cpu +#endif extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern void *text_poke_copy(void *addr, const void *opcode, size_t len); #define text_poke_copy text_poke_copy diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 62936a3bde19b..e2d185e6cb7ca 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -13,6 +14,7 @@ #include #include #include +#include int __read_mostly alternatives_patched; @@ -2768,10 +2770,43 @@ static void do_sync_core(void *info) sync_core(); } +static void __smp_text_poke_sync_each_cpu(smp_cond_func_t cond_func) +{ + on_each_cpu_cond(cond_func, do_sync_core, NULL, 1); +} + void smp_text_poke_sync_each_cpu(void) { - on_each_cpu(do_sync_core, NULL, 1); + __smp_text_poke_sync_each_cpu(NULL); +} + +#ifdef CONFIG_TRACK_CR3 +static bool do_sync_core_defer_cond(int cpu, void *info) +{ + /* + * Send the IPI if the target CPU is a housekeeping one, or if it is + * already executing in kernelspace. + */ + bool ret = housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE); + + /* + * Pairs with the LOCK in NOTE_KERNEL_CR3 + * + * Ensures any previous operations are visible on a remote CPU + * entering the kernel and setting @kernel_cr3_loaded, if this one + * decides to defer the IPI. + */ + smp_mb(); + ret |= per_cpu(kernel_cr3_loaded, cpu); + + return ret; +} + +void smp_text_poke_sync_each_cpu_deferrable(void) +{ + __smp_text_poke_sync_each_cpu(do_sync_core_defer_cond); } +#endif /* * NOTE: crazy scheme to allow patching Jcc.d32 but not increase the size of @@ -2940,6 +2975,7 @@ noinstr int smp_text_poke_int3_handler(struct pt_regs *regs) */ void smp_text_poke_batch_finish(void) { + void (*sync_fn)(void) = smp_text_poke_sync_each_cpu_deferrable; unsigned char int3 = INT3_INSN_OPCODE; unsigned int i; int do_sync; @@ -2976,11 +3012,20 @@ void smp_text_poke_batch_finish(void) * First step: add a INT3 trap to the address that will be patched. */ for (i = 0; i < text_poke_array.nr_entries; i++) { - text_poke_array.vec[i].old = *(u8 *)text_poke_addr(&text_poke_array.vec[i]); - text_poke(text_poke_addr(&text_poke_array.vec[i]), &int3, INT3_INSN_SIZE); + void *addr = text_poke_addr(&text_poke_array.vec[i]); + + /* + * There's no safe way to defer IPIs for patching text in + * entry, record whether there is at least one such poke. + */ + if (is_kernel_entrytext((unsigned long)addr)) + sync_fn = smp_text_poke_sync_each_cpu; + + text_poke_array.vec[i].old = *((u8 *)addr); + text_poke(addr, &int3, INT3_INSN_SIZE); } - smp_text_poke_sync_each_cpu(); + sync_fn(); /* * Second step: update all but the first byte of the patched range. @@ -3042,7 +3087,7 @@ void smp_text_poke_batch_finish(void) * not necessary and we'd be safe even without it. But * better safe than sorry (plus there's not only Intel). */ - smp_text_poke_sync_each_cpu(); + sync_fn(); } /* @@ -3063,7 +3108,7 @@ void smp_text_poke_batch_finish(void) } if (do_sync) - smp_text_poke_sync_each_cpu(); + sync_fn(); /* * Remove and wait for refs to be zero. diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index c1fac3a9fecc2..61a93ba30f255 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -789,7 +789,7 @@ void arch_arm_kprobe(struct kprobe *p) u8 int3 = INT3_INSN_OPCODE; text_poke(p->addr, &int3, 1); - smp_text_poke_sync_each_cpu(); + smp_text_poke_sync_each_cpu_deferrable(); perf_event_text_poke(p->addr, &p->opcode, 1, &int3, 1); } @@ -799,7 +799,7 @@ void arch_disarm_kprobe(struct kprobe *p) perf_event_text_poke(p->addr, &int3, 1, &p->opcode, 1); text_poke(p->addr, &p->opcode, 1); - smp_text_poke_sync_each_cpu(); + smp_text_poke_sync_each_cpu_deferrable(); } void arch_remove_kprobe(struct kprobe *p) diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 6f826a00eca29..3b3be66da320c 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -509,11 +509,11 @@ void arch_unoptimize_kprobe(struct optimized_kprobe *op) JMP32_INSN_SIZE - INT3_INSN_SIZE); text_poke(addr, new, INT3_INSN_SIZE); - smp_text_poke_sync_each_cpu(); + smp_text_poke_sync_each_cpu_deferrable(); text_poke(addr + INT3_INSN_SIZE, new + INT3_INSN_SIZE, JMP32_INSN_SIZE - INT3_INSN_SIZE); - smp_text_poke_sync_each_cpu(); + smp_text_poke_sync_each_cpu_deferrable(); perf_event_text_poke(op->kp.addr, old, JMP32_INSN_SIZE, new, JMP32_INSN_SIZE); } diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index 11c45ce42694c..0894b1f38de77 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -209,7 +209,7 @@ static int write_relocate_add(Elf64_Shdr *sechdrs, write, apply); if (!early) { - smp_text_poke_sync_each_cpu(); + smp_text_poke_sync_each_cpu_deferrable(); mutex_unlock(&text_mutex); } diff --git a/include/asm-generic/sections.h b/include/asm-generic/sections.h index 0755bc39b0d80..7496d26a85a4c 100644 --- a/include/asm-generic/sections.h +++ b/include/asm-generic/sections.h @@ -199,6 +199,20 @@ static inline bool is_kernel_inittext(unsigned long addr) addr < (unsigned long)_einittext; } +/** + * is_kernel_entrytext - checks if the pointer address is located in the + * .entry.text section + * + * @addr: address to check + * + * Returns: true if the address is located in .entry.text, false otherwise. + */ +static inline bool is_kernel_entrytext(unsigned long addr) +{ + return addr >= (unsigned long)__entry_text_start && + addr < (unsigned long)__entry_text_end; +} + /** * __is_kernel_text - checks if the pointer address is located in the * .text section -- 2.52.0