[PATCH v1] kernel: add a simple timer based software watchpoint

[PATCH v1] kernel: add a simple timer based software watchpoint

[Feng Tang]

During debugging some bios/hardware related nasty memory corruption
issues, we found using periodic timer to monitor specific dram/mmio
physical address is very useful for debugging, which acts like
a basic software watchpoint.

For those bugs,  who (and when) change(corrupt) those dram or mmio
register is hard to trace, and sometimes even hardware jtag debugger
can't help (say the physical address watchpoint doesn't work).

The biggest shortcoming of this method is it can never capture the
exact point like a hardware watchpoint. The idea is trying to
approach the point by adjusting the timer interval, hoping the caught
context have enough debug info (which did help us in solving quite
some bios/hardware bugs)

The working flow is simple: after suspected address is identified,
start periodic timer polling it to catch if its value is changed to
target 'magic' value, then halt the cpu (better limit to have only
one cpu online), or panic, or print out system information, so that
the error environment is frozen for further check , or let
kexec/kdump to record the vmcore, etc.

One real use case was:
"
On an arm64 platform, some BIOS/HW config caused OS boot easy to
stall in systemd init phase, then the reproducing was simplified it
by making it boot to console with a function-reduced rootfs, and
always triggering 'segmentation fault' when running 'less' command.

By using GDB, some static array of 'less' is found corrupted before
being written, and one byte in its memory is always '0x33'.At this
stage the static array is in bss segment first, and backed by kernel
zero page after first read, so it was an obvious memory corruption.

HW engineers tried to capture HW traces after the issue happened, but
could not find valuable hints, as the corruption could happen long
before 'less' is run, and the trace/context of that time was gone.

As physical address of kernel zero page was known, and offset of the
corrupted byte was fixed, the address was A. But HW debugger failed
to breakpoint the point that address A was written with '0x33'. Then
this method was used to monitor 'writing 0x33 to A' with 30ms
interval, and halted the system  by 'while (1);' (the system was made
to a UP by using 'nr_cpus=1' cmdline parameter) once hit, then HW
people collected the HW trace they need and root caused it to be a
bad config.
"

The culprits of memory corruption issues we met are mainly:
* broken devices (like ethernet card)
* BIOS runtime service
* silicon bugs
* kernel itself

As kernel already have many useful debug featues like slub_debug,
kasan, kfence, kmemleak etc.., this method could be more fit for the
upper three types.

All the settings are module parameters:

  watch_interval_ms: SW watchpoint check interval in ms
  paddr_dram_to_watch: Physical dram address to monitor.
  target_dram_val: Expected value at the dram address that triggers the watchpoint.
  paddr_mmio_to_watch: Physical mmio address to monitor. Must be 32-bit aligned.
  target_mmio_val: Expected value at the mmio address that triggers the watchpoint.
  panic_on_hit: Trigger kernel panic when watchpoint condition hits.
  hang_on_hit: halt the CPU (wait for HW debugger)

This provides the basic function, and there are more todoes:
  * add a ftrace mode to do function level monitoring with ftrace hook,
    which is more accurate timing wise, as suggested by Steven Rostedt
  * merge the dram/mmio interface to auto detect it's dram or mmio
  * support runtime changing the address
  * move the starting point earlier in boot phase
  * monitor a whole memory region
  * currently is monitoring 'changing to a value', add support
    for 'changing from a value'

Signed-off-by: Feng Tang <feng.tang@linux.alibaba.com>
---
Changelog:

  since RFC:
      * move the code to separate kernel/watch_mem.c (Petr)
	  * add real world use case in commit log (Petr)
	  * tried several ftrace methods, though not included in V1 (Steven)
	  * address some comments from Sashiko: https://sashiko.dev/#/patchset/20260527034324.51136-1-feng.tang%40linux.alibaba.com
	  * simplify ioremap code
 
   RFC: https://lore.kernel.org/lkml/20260527034324.51136-1-feng.tang@linux.alibaba.com/

 kernel/Makefile    |   1 +
 kernel/watch_mem.c | 129 +++++++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug  |   8 +++
 3 files changed, 138 insertions(+)
 create mode 100644 kernel/watch_mem.c

diff --git a/kernel/Makefile b/kernel/Makefile
index 1e1a31673577..931a31a583a6 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -105,6 +105,7 @@ obj-$(CONFIG_DETECT_HUNG_TASK) += hung_task.o
 obj-$(CONFIG_LOCKUP_DETECTOR) += watchdog.o
 obj-$(CONFIG_HARDLOCKUP_DETECTOR_BUDDY) += watchdog_buddy.o
 obj-$(CONFIG_HARDLOCKUP_DETECTOR_PERF) += watchdog_perf.o
+obj-$(CONFIG_SOFTWARE_WATCHPOINT) += watch_mem.o
 obj-$(CONFIG_SECCOMP) += seccomp.o
 obj-$(CONFIG_RELAY) += relay.o
 obj-$(CONFIG_SYSCTL) += utsname_sysctl.o
diff --git a/kernel/watch_mem.c b/kernel/watch_mem.c
new file mode 100644
index 000000000000..91cc46bbf37e
--- /dev/null
+++ b/kernel/watch_mem.c
@@ -0,0 +1,129 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/kernel.h>
+#include <linux/ftrace.h>
+#include <linux/io.h>
+#include <linux/memremap.h>
+#include <linux/sys_info.h>
+
+/* Time interval for SW watchpoin timer (default 100 ms). Setting to 0 will disable the timer */
+static unsigned long watch_interval_ms = 100;
+module_param(watch_interval_ms, ulong, 0644);
+
+static unsigned long paddr_dram_to_watch;
+module_param(paddr_dram_to_watch, ulong, 0644);
+MODULE_PARM_DESC(paddr_dram_to_watch, "Physical DRAM address to watch");
+static unsigned long *vaddr_dram;
+
+static unsigned long target_dram_val;
+module_param(target_dram_val, ulong, 0644);
+MODULE_PARM_DESC(target_dram_val, "Target DRAM value to trigger watchpoint");
+
+/* The MMIO address should be 32b aligned */
+static unsigned long paddr_mmio_to_watch;
+module_param(paddr_mmio_to_watch, ulong, 0644);
+MODULE_PARM_DESC(paddr_mmio_to_watch, "Physical MMIO address to watch (32bit aligned)");
+static void __iomem *vaddr_mmio;
+
+static unsigned int target_mmio_val;
+module_param(target_mmio_val, uint, 0644);
+MODULE_PARM_DESC(target_mmio_val, "Target MMIO value to trigger watchpoint");
+
+static bool panic_on_hit;
+module_param(panic_on_hit, bool, 0644);
+
+static bool hang_on_hit;
+module_param(hang_on_hit, bool, 0644);
+
+/* Stop the watchpoint after first hit */
+static bool check_once = true;
+module_param(check_once, bool, 0644);
+MODULE_PARM_DESC(check_once, "Stop watching after first hit");
+
+static struct timer_list sw_watchpoint_timer;
+
+static void sw_watchpoint_timer_fn(struct timer_list *unused)
+{
+	bool hit = false;
+
+	if (vaddr_mmio && (readl(vaddr_mmio) == target_mmio_val)) {
+		pr_info("MMIO [@0x%lx] hit the target value [0x%x]!\n",
+			paddr_mmio_to_watch, target_mmio_val);
+		hit = true;
+	}
+
+	/*
+	 * memremap for a RAM address could return a linear mapping address,
+	 * and should be unpoisoned before accessing to avoid KASAN warnings.
+	 */
+	if (vaddr_dram)
+		kasan_unpoison_pages(phys_to_page(paddr_dram_to_watch), 0, false);
+
+	if (vaddr_dram && (READ_ONCE(*vaddr_dram) == target_dram_val)) {
+		pr_info("DRAM [@0x%lx] hit the target value [0x%lx]!\n",
+			paddr_dram_to_watch, target_dram_val);
+		hit = true;
+	}
+
+	if (hit) {
+		sys_info(0);
+
+		/*
+		 * Useful for stopping the OS (together with configuring system
+		 * to UP with cmdline parameter 'nr_cpus=1') and waiting for HW
+		 * debugger being attached.
+		 */
+		if (hang_on_hit) {
+			pr_warn("Will dead loop on this CPU\n");
+			while (1);
+		}
+
+		/* Could be used to trigger kexec/kdump */
+		if (panic_on_hit)
+			panic("SW watchpoint hit!");
+
+		if (check_once)
+			return;
+	}
+
+	mod_timer(&sw_watchpoint_timer, jiffies + msecs_to_jiffies(watch_interval_ms));
+}
+
+static int __init sw_watchpoint_timer_init(void)
+{
+	bool has_watch = false;
+
+	if (!watch_interval_ms)
+		return 0;
+
+	if (paddr_dram_to_watch) {
+		vaddr_dram = memremap(paddr_dram_to_watch, sizeof(*vaddr_dram),
+				      MEMREMAP_WB);
+		if (!vaddr_dram)
+			return -ENOMEM;
+
+		has_watch = true;
+	}
+
+	if (paddr_mmio_to_watch) {
+		if (paddr_mmio_to_watch & 0x3) {
+			pr_info("paddr_mmio_to_watch should be 32 bits aligned!\n");
+			return -EINVAL;
+		}
+
+		vaddr_mmio = ioremap(paddr_mmio_to_watch, 4);
+		if (!vaddr_mmio)
+			return -ENOMEM;
+
+		has_watch = true;
+	}
+
+	if (!has_watch)
+		return 0;
+
+	timer_setup(&sw_watchpoint_timer, sw_watchpoint_timer_fn, 0);
+	sw_watchpoint_timer.expires = jiffies + msecs_to_jiffies(watch_interval_ms);
+	add_timer(&sw_watchpoint_timer);
+	return 0;
+}
+core_initcall(sw_watchpoint_timer_init);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 1244dcac2294..7f8b67d9581e 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1076,6 +1076,14 @@ config MEM_ALLOC_PROFILING_DEBUG
 	  Adds warnings with helpful error messages for memory allocation
 	  profiling.
 
+config SOFTWARE_WATCHPOINT
+	bool "Softwre watchpoint to watch memory corruption"
+	depends on  DEBUG_KERNEL
+	help
+	  Using periodc timer checking specific physical DRAM/MMIO address
+	  to capture memory corruption triggered by kernel, hardware devices,
+	  BIOS runtime code or silicon.
+
 source "lib/Kconfig.kasan"
 source "lib/Kconfig.kfence"
 source "lib/Kconfig.kmsan"

base-commit: ef0c9f75a19532d7675384708fc8621e10850104
-- 
2.39.5 (Apple Git-154)

Re: [PATCH v1] kernel: add a simple timer based software watchpoint

[David Hildenbrand (Arm)]

On 6/22/26 10:14, Feng Tang wrote:
> During debugging some bios/hardware related nasty memory corruption
> issues, we found using periodic timer to monitor specific dram/mmio
> physical address is very useful for debugging, which acts like
> a basic software watchpoint.
> 
> For those bugs,  who (and when) change(corrupt) those dram or mmio
> register is hard to trace, and sometimes even hardware jtag debugger
> can't help (say the physical address watchpoint doesn't work).
> 
> The biggest shortcoming of this method is it can never capture the
> exact point like a hardware watchpoint. The idea is trying to
> approach the point by adjusting the timer interval, hoping the caught
> context have enough debug info (which did help us in solving quite
> some bios/hardware bugs)
> 
> The working flow is simple: after suspected address is identified,
> start periodic timer polling it to catch if its value is changed to
> target 'magic' value, then halt the cpu (better limit to have only
> one cpu online), or panic, or print out system information, so that
> the error environment is frozen for further check , or let
> kexec/kdump to record the vmcore, etc.
> 
> One real use case was:
> "
> On an arm64 platform, some BIOS/HW config caused OS boot easy to
> stall in systemd init phase, then the reproducing was simplified it
> by making it boot to console with a function-reduced rootfs, and
> always triggering 'segmentation fault' when running 'less' command.
> 
> By using GDB, some static array of 'less' is found corrupted before
> being written, and one byte in its memory is always '0x33'.At this
> stage the static array is in bss segment first, and backed by kernel
> zero page after first read, so it was an obvious memory corruption.
> 
> HW engineers tried to capture HW traces after the issue happened, but
> could not find valuable hints, as the corruption could happen long
> before 'less' is run, and the trace/context of that time was gone.
> 
> As physical address of kernel zero page was known, and offset of the
> corrupted byte was fixed, the address was A. But HW debugger failed
> to breakpoint the point that address A was written with '0x33'. Then
> this method was used to monitor 'writing 0x33 to A' with 30ms
> interval, and halted the system  by 'while (1);' (the system was made
> to a UP by using 'nr_cpus=1' cmdline parameter) once hit, then HW
> people collected the HW trace they need and root caused it to be a
> bad config.
> "
> 
> The culprits of memory corruption issues we met are mainly:
> * broken devices (like ethernet card)
> * BIOS runtime service
> * silicon bugs
> * kernel itself
> 
> As kernel already have many useful debug featues like slub_debug,
> kasan, kfence, kmemleak etc.., this method could be more fit for the
> upper three types.
> 
> All the settings are module parameters:
> 
>   watch_interval_ms: SW watchpoint check interval in ms
>   paddr_dram_to_watch: Physical dram address to monitor.
>   target_dram_val: Expected value at the dram address that triggers the watchpoint.
>   paddr_mmio_to_watch: Physical mmio address to monitor. Must be 32-bit aligned.
>   target_mmio_val: Expected value at the mmio address that triggers the watchpoint.
>   panic_on_hit: Trigger kernel panic when watchpoint condition hits.
>   hang_on_hit: halt the CPU (wait for HW debugger)
> 
> This provides the basic function, and there are more todoes:
>   * add a ftrace mode to do function level monitoring with ftrace hook,
>     which is more accurate timing wise, as suggested by Steven Rostedt
>   * merge the dram/mmio interface to auto detect it's dram or mmio
>   * support runtime changing the address
>   * move the starting point earlier in boot phase
>   * monitor a whole memory region
>   * currently is monitoring 'changing to a value', add support
>     for 'changing from a value'

That really looks more like the kind of thing you would want to carry as a OOT
hack for your special debugging needs :)

-- 
Cheers,

David

Re: [PATCH v1] kernel: add a simple timer based software watchpoint

[Thomas Gleixner]

On Mon, Jun 22 2026 at 10:42, David Hildenbrand wrote:
> On 6/22/26 10:14, Feng Tang wrote:
>>   * currently is monitoring 'changing to a value', add support
>>     for 'changing from a value'
>
> That really looks more like the kind of thing you would want to carry as a OOT
> hack for your special debugging needs :)

Indeed.

Re: [PATCH v1] kernel: add a simple timer based software watchpoint

[Feng Tang]

Hi David,

On Mon, Jun 22, 2026 at 10:42:06AM +0200, David Hildenbrand (Arm) wrote:
> On 6/22/26 10:14, Feng Tang wrote:
> > During debugging some bios/hardware related nasty memory corruption
> > issues, we found using periodic timer to monitor specific dram/mmio
> > physical address is very useful for debugging, which acts like
> > a basic software watchpoint.
> > 
> > For those bugs,  who (and when) change(corrupt) those dram or mmio
> > register is hard to trace, and sometimes even hardware jtag debugger
> > can't help (say the physical address watchpoint doesn't work).
> > 
> > The biggest shortcoming of this method is it can never capture the
> > exact point like a hardware watchpoint. The idea is trying to
> > approach the point by adjusting the timer interval, hoping the caught
> > context have enough debug info (which did help us in solving quite
> > some bios/hardware bugs)
> > 
> > The working flow is simple: after suspected address is identified,
> > start periodic timer polling it to catch if its value is changed to
> > target 'magic' value, then halt the cpu (better limit to have only
> > one cpu online), or panic, or print out system information, so that
> > the error environment is frozen for further check , or let
> > kexec/kdump to record the vmcore, etc.
> > 
> > One real use case was:
> > "
> > On an arm64 platform, some BIOS/HW config caused OS boot easy to
> > stall in systemd init phase, then the reproducing was simplified it
> > by making it boot to console with a function-reduced rootfs, and
> > always triggering 'segmentation fault' when running 'less' command.
> > 
> > By using GDB, some static array of 'less' is found corrupted before
> > being written, and one byte in its memory is always '0x33'.At this
> > stage the static array is in bss segment first, and backed by kernel
> > zero page after first read, so it was an obvious memory corruption.
> > 
> > HW engineers tried to capture HW traces after the issue happened, but
> > could not find valuable hints, as the corruption could happen long
> > before 'less' is run, and the trace/context of that time was gone.
> > 
> > As physical address of kernel zero page was known, and offset of the
> > corrupted byte was fixed, the address was A. But HW debugger failed
> > to breakpoint the point that address A was written with '0x33'. Then
> > this method was used to monitor 'writing 0x33 to A' with 30ms
> > interval, and halted the system  by 'while (1);' (the system was made
> > to a UP by using 'nr_cpus=1' cmdline parameter) once hit, then HW
> > people collected the HW trace they need and root caused it to be a
> > bad config.
> > "
> > 
> > The culprits of memory corruption issues we met are mainly:
> > * broken devices (like ethernet card)
> > * BIOS runtime service
> > * silicon bugs
> > * kernel itself
> > 
> > As kernel already have many useful debug featues like slub_debug,
> > kasan, kfence, kmemleak etc.., this method could be more fit for the
> > upper three types.
> > 
> > All the settings are module parameters:
> > 
> >   watch_interval_ms: SW watchpoint check interval in ms
> >   paddr_dram_to_watch: Physical dram address to monitor.
> >   target_dram_val: Expected value at the dram address that triggers the watchpoint.
> >   paddr_mmio_to_watch: Physical mmio address to monitor. Must be 32-bit aligned.
> >   target_mmio_val: Expected value at the mmio address that triggers the watchpoint.
> >   panic_on_hit: Trigger kernel panic when watchpoint condition hits.
> >   hang_on_hit: halt the CPU (wait for HW debugger)
> > 
> > This provides the basic function, and there are more todoes:
> >   * add a ftrace mode to do function level monitoring with ftrace hook,
> >     which is more accurate timing wise, as suggested by Steven Rostedt
> >   * merge the dram/mmio interface to auto detect it's dram or mmio
> >   * support runtime changing the address
> >   * move the starting point earlier in boot phase
> >   * monitor a whole memory region
> >   * currently is monitoring 'changing to a value', add support
> >     for 'changing from a value'
> 
> That really looks more like the kind of thing you would want to carry as a OOT
> hack for your special debugging needs :)

Thanks for the review and bringing up a very good question that whether
the patch is worthy!

Personally, I think it is.  

Firstly, IMHO, memory corruption is a common issue.  Petr asked me to give
some real-world examples, and I talked about two cases in 
[1]. https://lore.kernel.org/lkml/aiaRLiaNs9M9c2q-@U-2FWC9VHC-2323.local/

I have met more memory corruptions, and I would divide them to 2 types:
kernel-triggered and non-kernel (device/BIOS/silicon) triggered.

For non-kernel triggered: mainly are from devices (ethernet cards, USB
controllers etc), many devices may corrupt kernel memory during dma
transfer, suspend/resume cycle or warm reboot, like some version of
Aquantia NIC , mainstream MLXCX5 NIC, or octeon-hcd, which are general
hardware devices:

[2]. https://bugzilla.kernel.org/show_bug.cgi?id=217854
[3]. https://lore.kernel.org/lkml/20221130085451.3390992-2-feng.tang@intel.com/

For kernel triggered: 
* kernel array overflows is common case for memory corruption (which
  should be easy captured by watchpoint backed by ftrace method) 
* kernel zero page case, which I don't know the detail but I guess it comes
  from some corruption to zero page similar like what I met.
[4]. https://lore.kernel.org/all/20260526175846.2694125-31-ardb+git@google.com/

As my own experience is very limited, and I think there should be much
more real-world memory corruptions cases than I know.

And IMHO, these non-kernel trigged ones are more difficult to handle
than kernel triggered once, as the symptom is more random, and hard
to generalize a pattern.

Secondly, it is low cost and could be built into production kernel.
* It is under a "default n" config, so won't bother normal users/OSVs
* The binary is small when built into kernel, and won't cost kernel
  cycles when not enabled
* For production kernel running on big server fleets, the kasan/slub_debug
  option usually won't be enabled as affecting performance, so many
  kernel-triggered memory corruption can't be helped by them. 

Thanks,
Feng

> 
> -- 
> Cheers,
> 
> David

Re: [PATCH v1] kernel: add a simple timer based software watchpoint

[David Hildenbrand (Arm)]

>>> This provides the basic function, and there are more todoes:
>>>   * add a ftrace mode to do function level monitoring with ftrace hook,
>>>     which is more accurate timing wise, as suggested by Steven Rostedt
>>>   * merge the dram/mmio interface to auto detect it's dram or mmio
>>>   * support runtime changing the address
>>>   * move the starting point earlier in boot phase
>>>   * monitor a whole memory region
>>>   * currently is monitoring 'changing to a value', add support
>>>     for 'changing from a value'
>>
>> That really looks more like the kind of thing you would want to carry as a OOT
>> hack for your special debugging needs :)
> 
> Thanks for the review and bringing up a very good question that whether
> the patch is worthy!
> 
> Personally, I think it is.  

The problem is that the interface is odd (kernel parameters) and the
implementation is shaky (what happens if memory does not exist? What happens if
we unplug memory?).

I don't think it's a good use of our time to polish all that up and then
maintain it.

-- 
Cheers,

David