From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 169C73264FB for ; Tue, 14 Jul 2026 18:32:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784053967; cv=none; b=M6hy6bjZ7tHRVLXW52ltuaOgXqcolPPYHFpNmiJ6P9foR5lDg5l9Rch77nFtqohvd/h3S0Xb+bCzIu7EAYeJxAZ9pNBqblXVLIRE2ElAgYS0eKQGAuflfq6OeDMQ7i6SF78VoMn2CWsgKZY+2HOywlWxz9/iMim8p9kDQsDFQro= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784053967; c=relaxed/simple; bh=U8Z0nvRnaFFrw3lPHvPCl/wITya9sSFlUIbcZxe00bU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j/oEgSah6EdKtFTBHI6BPWiIYgJKoTtAV8sPeLK+NHHfelIJon5LNLlBO3XSx70rSOU9lukN6K2LTNDM6SmIn0/olU/amP3MqLVNxfdo9+zXn5WsegIyaBXfIUcWCND4BKNzowTpxzNR8Hw6yPFpSHCj3p0UqdUtI9e/JB57KpE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=l6FYtvg9; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="l6FYtvg9" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-920f33347f5so317191785a.3 for ; Tue, 14 Jul 2026 11:32:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784053965; x=1784658765; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=JG+YXpLaf6eTeKDHjsfPWQsS8e418Ifa944UVLgQD7U=; b=l6FYtvg9K7qTr9PRUV9BLwvHtUGDx48DgF7q5vOQqGr0YH7Q14H5Kodm8HCyVugJl3 4e+LaK37kdjGNqYiLjxI8ECiu+ZUIFV0y0OP+KCadlKsdUD40QRV1F6EeGK71rxxB5HE BUizXm9wvsLq+Wa50677s7e+fg3UlcEWyk8iJndeZjadc296WmkqBbR67JvWsrZXiGbb gmzwKwE3hFBVJkXwLD1XCai9/+ias7Ge1D/ANkcEDMOqA5uF8Z1gDpqwcFrRU9E8HVnc hRWSXYsBdAxnn0YTl1OxnKe7zoWHg5nc/piaWsjXS0dROWE2pruHRC+VwUdqGRfxA9gs qokA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784053965; x=1784658765; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=JG+YXpLaf6eTeKDHjsfPWQsS8e418Ifa944UVLgQD7U=; b=jXK571OGmPs5T4/ReIGnrFa03ACfFx6BFZ1dwLue4kLfaVhiDUixH0Fnht9lsDEDOS SGjboYpmn6LHWeeDgidFgvYHenkV1q+XfSgccgbQUBcESQMd3d0dez2qAAjU+pHYS1VP z4m0N2EkaP1nuyKnfB6igCK0ysCnpNvoStowy1J6HDcx/KOx9juEn5c0efHXCRsH7zpQ dS0huTKJKKYNK8bmbVdEBYxyD0oicdnJUpJ8FyURcKCHptT/OoDxW9YS0Xj+CyXm9xNZ 6TcoCDd+CDuz+T1b8Re/smapQWU53BlvfFLOZln7jG0QoLp9mAIexD4e49VdyOFF/AS3 WXNg== X-Forwarded-Encrypted: i=1; AHgh+RrSLl5sg1ptZjpLD4jl1OX7s7ApxiSXQ5vgwqxKGjy4tVHJR50KqhVCwm2rhH1TMMx6ks5VXlZq2eE=@vger.kernel.org X-Gm-Message-State: AOJu0YxpqO6lZpHu7XWJhrye1NMEGvHvnull4R2CGgPXmFEfrJbXjXoN buTGMK5Avu/tCTi58u3v3tU40HP2dXFZlBTofMobx1G/86rYJc0/YkKK X-Gm-Gg: AfdE7cl5PAhwy1Dqw8RChUW99KFCYDvJ3NCrEJz9bikZ0N3Gv2WxuLlFIhrIcL91WAa +KTHiTQWgcEZr3fDNSu+NdASIQ6o0Pn8Trm77Oll2VbHv0+DG/5ZsD3Hqf7OJazSrouWIAedVcy qE4b4b1+A+DDViJfOWTvVxj2S/59a7jYH2biheqz3QPI4Y6mz/INBobrWnTcZ4tVAgHGZC2BEqv OhVlpTrCcbcElj25i7wVc1WaJWpAxNH/vWojhRQ5u4ivmdWOYMHnNlAZmOh6kHQ/c+/scnfW45+ /uzfmfsfJSnNlzTF04dunnPLBD94/pHp5uR7LPihfj5wLL95D9NAEcVtYV4vwyebOGPujaVco3m L6UpwxS1J5vodxzA768t4FpCnepSdwFri38B7j5jTKoM9x0rk+Dmtqpvfez7ELvEl6DUxv8QazH MPBzKKsjd4HKxJUpvJUw2xzLgyIGvCLU962BNEv71iL2aZcEDVpDE= X-Received: by 2002:a05:620a:4012:b0:92e:d63b:455b with SMTP id af79cd13be357-92ef2b12452mr1436818585a.6.1784053964792; Tue, 14 Jul 2026 11:32:44 -0700 (PDT) Received: from localhost ([48.45.163.146]) by smtp.gmail.com with ESMTPSA id af79cd13be357-92ee5b88a38sm1484900685a.12.2026.07.14.11.32.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jul 2026 11:32:44 -0700 (PDT) From: Jinchao Wang To: Andrew Morton , Peter Zijlstra , Thomas Gleixner , Steven Rostedt , Masami Hiramatsu Cc: Ingo Molnar , Borislav Petkov , Dave Hansen , "H . Peter Anvin" , x86@kernel.org, Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Mathieu Desnoyers , David Hildenbrand , Jonathan Corbet , Matthew Wilcox , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-doc@vger.kernel.org, Jinchao Wang Subject: [RFC PATCH 09/13] mm/kwatch: add probe lifecycle runtime Date: Wed, 15 Jul 2026 02:32:26 +0800 Message-ID: <20260714183226.12756-1-wangjinchao600@gmail.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260714182243.10687-1-wangjinchao600@gmail.com> References: <20260714182243.10687-1-wangjinchao600@gmail.com> Precedence: bulk X-Mailing-List: linux-doc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Open and close the watch window with a kretprobe on the target function: the entry handler tracks per-task nesting depth and, when the configured depth is reached, resolves the watch expression and arms a watchpoint; the exit handler disarms it. An optional kprobe at func_offset arms mid-function instead of at entry. Functions running in a real NMI(-like) context are rejected once, at function entry, by comparing the NMI nesting count against the one NMI-like layer that int3-based kprobe delivery itself adds; a companion kprobe with a post_handler pins the probe point so jump optimization cannot change the delivery mechanism after it is sampled. Rejections are counted and exposed to the control plane. A global epoch versioning scheme invalidates stale per-task state across reconfigurations, and a per-CPU mute flag keeps window management quiet while a CPU rewrites its own debug registers. Signed-off-by: Jinchao Wang --- mm/kwatch/Makefile | 2 +- mm/kwatch/probe.c | 263 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 264 insertions(+), 1 deletion(-) create mode 100644 mm/kwatch/probe.c diff --git a/mm/kwatch/Makefile b/mm/kwatch/Makefile index b2bc3003c89b..f04673cc5b1c 100644 --- a/mm/kwatch/Makefile +++ b/mm/kwatch/Makefile @@ -1,3 +1,3 @@ obj-$(CONFIG_KWATCH) += kwatch.o -kwatch-y := deref.o task_ctx.o hwbp.o +kwatch-y := deref.o task_ctx.o hwbp.o probe.o diff --git a/mm/kwatch/probe.c b/mm/kwatch/probe.c new file mode 100644 index 000000000000..af6e0af45c10 --- /dev/null +++ b/mm/kwatch/probe.c @@ -0,0 +1,263 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include + +#include "kwatch.h" +#define TRAMPOLINE_CHECK_DEPTH 16 +static DEFINE_PER_CPU(bool, kwatch_probe_cpu_muted); + +struct kwatch_probe_ctx { + struct kprobe kp; + struct kretprobe rp; + struct kprobe pin_kp; + const struct kwatch_config *cfg; + bool rp_via_int3; + + u32 epoch; +}; + +static struct kwatch_probe_ctx kwatch_probe_ctx; +static atomic_long_t kwatch_nmi_rejected; + +unsigned long kwatch_probe_nmi_rejected(void) +{ + return atomic_long_read(&kwatch_nmi_rejected); +} + +/* + * True if the probed function itself runs in an NMI-like context. + * int3-based kprobe delivery adds one NMI-like layer of its own; + * delivery is pinned at registration so the subtraction stays exact. + */ +static bool kwatch_probed_ctx_in_nmi(bool via_int3) +{ + return (preempt_count() & NMI_MASK) > (via_int3 ? NMI_OFFSET : 0); +} + +static void kwatch_pin_post_handler(struct kprobe *p, struct pt_regs *regs, + unsigned long flags) +{ + /* a post_handler pins the probepoint: no jump optimization */ +} + +bool kwatch_probe_validate_hit(struct pt_regs *regs, + struct task_struct *arm_tsk) +{ + struct kwatch_tsk_ctx *ctx = kwatch_tsk_ctx_get(false); + + if (unlikely(!ctx)) + return true; + + if (arm_tsk != current || + ctx->depth != kwatch_probe_ctx.cfg->depth + 1) + return true; + + return false; +} + +void kwatch_probe_mute(bool mute) +{ + __this_cpu_write(kwatch_probe_cpu_muted, mute); +} + +static inline bool kwatch_probe_is_muted(void) +{ + return __this_cpu_read(kwatch_probe_cpu_muted); +} + +enum kwatch_probe_position { + KWATCH_PROBE_POSITION_ENTRY, + KWATCH_PROBE_POSITION_ACTIVE, + KWATCH_PROBE_POSITION_EXIT +}; + +static bool kwatch_tsk_ctx_check(enum kwatch_probe_position pos) +{ + struct kwatch_tsk_ctx *ctx = kwatch_tsk_ctx_get(true); + u32 epoch; + + if (unlikely(!ctx)) + return false; + + /* Pairs with smp_store_release() in kwatch_probe_start/stop() */ + epoch = smp_load_acquire(&kwatch_probe_ctx.epoch); + + if (unlikely(ctx->epoch != epoch)) + kwatch_tsk_ctx_reset(ctx, epoch); + + if (unlikely(!epoch)) + return false; + + switch (pos) { + case KWATCH_PROBE_POSITION_ENTRY: + ctx->depth++; + return true; + case KWATCH_PROBE_POSITION_ACTIVE: + return true; + case KWATCH_PROBE_POSITION_EXIT: + if (unlikely(ctx->depth == 0)) { + kwatch_tsk_ctx_put(); + return false; + } + + ctx->depth--; + if (ctx->depth == 0) { + kwatch_tsk_ctx_put(); + return false; + } + return true; + } + return false; +} + +static int kwatch_activate_handler(struct kprobe *p, struct pt_regs *regs) +{ + struct kwatch_tsk_ctx *ctx = kwatch_tsk_ctx_get(false); + unsigned long watch_addr; + u16 watch_len; + + if (unlikely(!ctx)) + return 0; + + if (unlikely(kwatch_probe_is_muted())) + return 0; + + if (unlikely(!kwatch_tsk_ctx_check(KWATCH_PROBE_POSITION_ACTIVE))) + return 0; + + if (ctx->depth != kwatch_probe_ctx.cfg->depth + 1 || ctx->wp) + return 0; + + if (kwatch_deref_resolve(kwatch_probe_ctx.cfg, regs, &watch_addr, + &watch_len)) + return 0; + + if (kwatch_hwbp_get(&ctx->wp)) + return 0; + + kwatch_hwbp_arm(ctx->wp, watch_addr, watch_len); + return 0; +} + +static int kwatch_lifecycle_entry(struct kretprobe_instance *ri, + struct pt_regs *regs) +{ + /* + * Single policy point: the target function's context is judged once + * here. A rejected invocation never increments depth, so the offset + * kprobe path inherits the verdict through the depth check. + */ + if (unlikely(kwatch_probed_ctx_in_nmi(kwatch_probe_ctx.rp_via_int3))) { + atomic_long_inc(&kwatch_nmi_rejected); + return 1; /* NMI context is unsupported: no window, no return hook */ + } + + if (!kwatch_tsk_ctx_check(KWATCH_PROBE_POSITION_ENTRY)) + return 0; + + if (kwatch_probe_ctx.cfg->func_offset == 0) + kwatch_activate_handler(NULL, regs); + + return 0; +} + +static int kwatch_lifecycle_exit(struct kretprobe_instance *ri, + struct pt_regs *regs) +{ + struct kwatch_tsk_ctx *ctx = kwatch_tsk_ctx_get(false); + + if (unlikely(!ctx)) + return 0; + + if (!kwatch_tsk_ctx_check(KWATCH_PROBE_POSITION_EXIT)) + return 0; + + if (ctx->depth == kwatch_probe_ctx.cfg->depth) { + struct kwatch_watchpoint *wp = xchg(&ctx->wp, NULL); + + if (wp) + kwatch_hwbp_put(wp); + } + + return 0; +} + +int kwatch_probe_start(struct kwatch_config *cfg) +{ + static u32 next_epoch; + u32 current_epoch; + int ret; + + /* + * Lockless check to prevent concurrent starts. Strictly serialized + * by the control plane mutex, but serves as a sanity check. + */ + if (smp_load_acquire(&kwatch_probe_ctx.epoch) != 0) + return -EBUSY; + + memset(&kwatch_probe_ctx, 0, sizeof(kwatch_probe_ctx)); + kwatch_probe_ctx.cfg = cfg; + + /* + * Pin the entry probepoint before the kretprobe registers, so its + * delivery (int3 vs ftrace) can never change under jump optimization. + * register_kretprobe() clears kp.post_handler, hence the companion. + */ + kwatch_probe_ctx.pin_kp.symbol_name = cfg->func_name; + kwatch_probe_ctx.pin_kp.post_handler = kwatch_pin_post_handler; + ret = register_kprobe(&kwatch_probe_ctx.pin_kp); + if (ret < 0) + return ret; + + kwatch_probe_ctx.rp.entry_handler = kwatch_lifecycle_entry; + kwatch_probe_ctx.rp.handler = kwatch_lifecycle_exit; + kwatch_probe_ctx.rp.kp.symbol_name = cfg->func_name; + + ret = register_kretprobe(&kwatch_probe_ctx.rp); + if (ret < 0) { + unregister_kprobe(&kwatch_probe_ctx.pin_kp); + return ret; + } + kwatch_probe_ctx.rp_via_int3 = !kprobe_ftrace(&kwatch_probe_ctx.rp.kp); + + if (cfg->func_offset) { + kwatch_probe_ctx.kp.symbol_name = cfg->func_name; + kwatch_probe_ctx.kp.offset = cfg->func_offset; + kwatch_probe_ctx.kp.pre_handler = kwatch_activate_handler; + + ret = register_kprobe(&kwatch_probe_ctx.kp); + if (ret) { + unregister_kretprobe(&kwatch_probe_ctx.rp); + unregister_kprobe(&kwatch_probe_ctx.pin_kp); + return ret; + } + } + + current_epoch = ++next_epoch; + if (unlikely(!current_epoch)) + current_epoch = ++next_epoch; + + /* Pairs with smp_load_acquire() in kwatch_tsk_ctx_check() */ + smp_store_release(&kwatch_probe_ctx.epoch, current_epoch); + + return 0; +} + +void kwatch_probe_stop(void) +{ + if (!kwatch_probe_ctx.epoch) + return; + + /* Pairs with smp_load_acquire() in kwatch_tsk_ctx_check() */ + smp_store_release(&kwatch_probe_ctx.epoch, 0); + + if (kwatch_probe_ctx.cfg->func_offset > 0) + unregister_kprobe(&kwatch_probe_ctx.kp); + + unregister_kretprobe(&kwatch_probe_ctx.rp); + unregister_kprobe(&kwatch_probe_ctx.pin_kp); +} -- 2.53.0