Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

[Dave Hansen]

On 7/17/26 05:50, Jinchao Wang wrote:
>  24 files changed, 2115 insertions(+), 63 deletions(-)
Reading this, I wonder how many kernel debugging features we need. I
don't even think we have a centralized list of them. They all just live
in their own silos.

This one really seems like a super specialized tool. It has to be
enabled at compile time and specifically aimed at a specific function.

Maybe this should live off on the side for a while. If folks end up
actually needing it, they can point their friendly LLM over to its tree.

Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

[Borislav Petkov]

On Fri, Jul 17, 2026 at 06:41:49AM -0700, Dave Hansen wrote:
> On 7/17/26 05:50, Jinchao Wang wrote:
> >  24 files changed, 2115 insertions(+), 63 deletions(-)
> Reading this, I wonder how many kernel debugging features we need. I
> don't even think we have a centralized list of them. They all just live
> in their own silos.
> 
> This one really seems like a super specialized tool. It has to be
> enabled at compile time and specifically aimed at a specific function.
> 
> Maybe this should live off on the side for a while. If folks end up
> actually needing it, they can point their friendly LLM over to its tree.

Right, and in my mbox right under this mail thread there's a

https://lore.kernel.org/all/178429796992.157981.3393977217853767915.stgit@devnote2/

which Masami has been blasting almost every day this week which contains two
of the patches from this set here...

Looks to me like folks need to sit down and agree on strategy first.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

[Masami Hiramatsu]

On Fri, 17 Jul 2026 11:10:04 -0700
Borislav Petkov <bp@alien8.de> wrote:

> On Fri, Jul 17, 2026 at 06:41:49AM -0700, Dave Hansen wrote:
> > On 7/17/26 05:50, Jinchao Wang wrote:
> > >  24 files changed, 2115 insertions(+), 63 deletions(-)
> > Reading this, I wonder how many kernel debugging features we need. I
> > don't even think we have a centralized list of them. They all just live
> > in their own silos.
> > 
> > This one really seems like a super specialized tool. It has to be
> > enabled at compile time and specifically aimed at a specific function.
> > 
> > Maybe this should live off on the side for a while. If folks end up
> > actually needing it, they can point their friendly LLM over to its tree.
> 
> Right, and in my mbox right under this mail thread there's a
> 
> https://lore.kernel.org/all/178429796992.157981.3393977217853767915.stgit@devnote2/
> 
> which Masami has been blasting almost every day this week which contains two
> of the patches from this set here...

Sorry about that. I tried to fix the reviewed comment from Sashiko,
but I should wait for someone's comment.

> 
> Looks to me like folks need to sit down and agree on strategy first.

Yeah, I need to talk with Jinchao.

Like kprobe, wprobe itself integrates watchpoint functionality into
the tracing subsystem. Jinchao, I would like to ask you to redesign
the kwatch tool based on fprobe and wprobe events in user space?
I think that could reduce redundant efforts on the similar feature.

Using fprobe entry and exit event trigger, we can make a Function-
scoped watch window. Also, since fprobe and wprobe are having BTF
typecast support, you can access to the members of data structures
more naturally.

If you are interested, I would be open to having you handle the
development of the necessary features for wprobe.

Thank you,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

[Marco Elver]

On Fri, 17 Jul 2026 at 14:50, Jinchao Wang <wangjinchao600@gmail.com> wrote:
>
> Motivation
> ==========
>
> The hardest memory corruption bugs are the silent ones: a rogue writer
> scribbles over a live object through a stale pointer or a race, and

The stale/dangling pointer (use-after-free) case is the main one you're after?

> the victim crashes in a code path far away from the culprit. Any
> single developer hits such a bug rarely, but across the kernel's code
> base and install base they keep arriving, and each one is
> disproportionately expensive to localize. The question to answer is
> "who wrote to this object, and from where?", and it is hard to get at
> with the existing tools:
>
>  - The kernel's own reports - an oops on a clobbered pointer, a
>    BUG_ON, a list-corruption warning - fire at the victim's access,
>    not at the corrupting write.
>  - KASAN/KFENCE catch memory-safety violations: out-of-bounds
>    accesses and use-after-free. But they have a blind spot: a
>    corrupting write can be fully memory-safe - a *valid* pointer, in

Language-semantically speaking, a use-after-free write to recycled
memory (be it in heap or stack) is NOT memory-safe. The detectors may
not always catch these (KASAN relies on quarantine for that to
increase the chances, but yeah, not guaranteed..).

>    bounds, to a live object, written just at the wrong time or to the
>    wrong place - and then they stay silent by design. And even for
>    the bugs they can catch, KASAN's rebuild, overhead and redzones
>    change timing and layout enough that racy corruption often no
>    longer reproduces.

There's also tag-based KASAN (KASAN_SW_TAGS or KASAN_HW_TAGS), of
which KASAN_SW_TAGS has an x86 version based on LAM (not yet merged
though? see https://lwn.net/ml/all/cover.1773164688.git.m.wieczorretman@pm.me/).

One property of tag-based schemes (and any other lock+key detection
scheme) is that upon recycling some memory, the tag (or key) to that
location changes, and any previous pointer that still has the old tag
will fault. The caveat with current pointer-tagging schemes is that
entropy is relatively low (4 bit tags).

Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

[Jinchao Wang]

On 7/20/2026 10:24 AM, Masami Hiramatsu (Google) wrote:
> On Fri, 17 Jul 2026 11:10:04 -0700
> Borislav Petkov <bp@alien8.de> wrote:

>> Looks to me like folks need to sit down and agree on strategy first.
> 
> Yeah, I need to talk with Jinchao.
> 
> Like kprobe, wprobe itself integrates watchpoint functionality into
> the tracing subsystem. Jinchao, I would like to ask you to redesign
> the kwatch tool based on fprobe and wprobe events in user space?
> I think that could reduce redundant efforts on the similar feature.
> 
> Using fprobe entry and exit event trigger, we can make a Function-
> scoped watch window. Also, since fprobe and wprobe are having BTF
> typecast support, you can access to the members of data structures
> more naturally.
> 
> If you are interested, I would be open to having you handle the
> development of the necessary features for wprobe.
> 
> Thank you,
> 
Thank you, Boris and Masami.

Boris is right that we should agree on the overall strategy before
moving either series forward, especially since they share the low-level
HWBP work. Masami is also right that fprobe and wprobe provide a more
appropriate foundation than introducing KWatch as another standalone
debugging tool.

After reconsidering the design, I will stop the standalone KWatch
series. Instead, I would like to work with Masami on extending wprobe
to support the function-scoped dynamic-watchpoint use cases that
motivated KWatch.

Masami, thank you for being open to having me develop the necessary
wprobe features. I am interested in taking that direction.
---
Thanks,
Jinchao

Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

[Jinchao Wang]

On 7/17/2026 9:41 AM, Dave Hansen wrote:
> On 7/17/26 05:50, Jinchao Wang wrote:
>>  24 files changed, 2115 insertions(+), 63 deletions(-)
> Reading this, I wonder how many kernel debugging features we need. I
> don't even think we have a centralized list of them. They all just live
> in their own silos.
> 
> This one really seems like a super specialized tool. It has to be
> enabled at compile time and specifically aimed at a specific function.
> 
> Maybe this should live off on the side for a while. If folks end up
> actually needing it, they can point their friendly LLM over to its tree.

Thanks, Dave.

I agree that adding another standalone in-kernel debugging facility of
this size is not the right direction.

The motivation was practical. I have encountered several silent memory
corruption bugs for which I wrote similar one-off instrumentation to
identify the writer. KWatch was an attempt to make that technique
reusable instead of rebuilding it for each bug, and to reduce the same
setup cost for other developers facing similar problems.

This approach proved useful for localizing the dummy_hcd corruption:

https://lore.kernel.org/all/20260714064829.172098-1-wangjinchao600@gmail.com/

It identified the copy that overwrote req->complete and recorded the
writer's call stack. So I believe the debugging use case is valid, but
it does not require a separate KWatch implementation.

As discussed in my reply to Masami and Boris, I will stop pursuing the
standalone KWatch series and work with Masami on implementing the
necessary scoped watchpoint features in wprobe/tracing instead.

Thanks,
Jinchao

Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

[Jinchao Wang]

On 7/20/2026 11:26 AM, Marco Elver wrote:
> On Fri, 17 Jul 2026 at 14:50, Jinchao Wang <wangjinchao600@gmail.com> wrote:
>>
>> Motivation
>> ==========
>>
>> The hardest memory corruption bugs are the silent ones: a rogue writer
>> scribbles over a live object through a stale pointer or a race, and
> 
Thanks, Marco. These are good points. My cover letter did not clearly
separate two different classes of bugs.

> The stale/dangling pointer (use-after-free) case is the main one you're after?

Not exclusively. KASAN already does a good job as a general
use-after-free detector. However, a stale-pointer write after the
storage has been recycled, which may no longer be reported by Generic
KASAN, is one of the cases that the scoped watchpoint can help
localize.

The broader target is silent corruption of a known live victim object,
including non-UAF cases such as an in-bounds racing write to an object
whose allocation is still live.

>> the victim crashes in a code path far away from the culprit. Any
>> single developer hits such a bug rarely, but across the kernel's code
>> base and install base they keep arriving, and each one is
>> disproportionately expensive to localize. The question to answer is
>> "who wrote to this object, and from where?", and it is hard to get at
>> with the existing tools:
>>
>>  - The kernel's own reports - an oops on a clobbered pointer, a
>>    BUG_ON, a list-corruption warning - fire at the victim's access,
>>    not at the corrupting write.
>>  - KASAN/KFENCE catch memory-safety violations: out-of-bounds
>>    accesses and use-after-free. But they have a blind spot: a
>>    corrupting write can be fully memory-safe - a *valid* pointer, in
> 
> Language-semantically speaking, a use-after-free write to recycled
> memory (be it in heap or stack) is NOT memory-safe. The detectors may
> not always catch these (KASAN relies on quarantine for that to
> increase the chances, but yeah, not guaranteed..).
> 

You are right that a stale-pointer write to recycled storage remains a
use-after-free, so "memory-safe" was the wrong term. What I meant was
that such a corrupting write may not be detected by Generic KASAN at
the point where it occurs. KWatch was intended to use a hardware
watchpoint to catch the write and report the actual writer and its call
stack.

>>    bounds, to a live object, written just at the wrong time or to the
>>    wrong place - and then they stay silent by design. And even for
>>    the bugs they can catch, KASAN's rebuild, overhead and redzones
>>    change timing and layout enough that racy corruption often no
>>    longer reproduces.
> 
> There's also tag-based KASAN (KASAN_SW_TAGS or KASAN_HW_TAGS), of
> which KASAN_SW_TAGS has an x86 version based on LAM (not yet merged
> though? see https://lwn.net/ml/all/cover.1773164688.git.m.wieczorretman@pm.me/).
> 
> One property of tag-based schemes (and any other lock+key detection
> scheme) is that upon recycling some memory, the tag (or key) to that
> location changes, and any previous pointer that still has the old tag
> will fault. The caveat with current pointer-tagging schemes is that
> entropy is relatively low (4 bit tags).
Thanks for pointing this out. I agree that tag-based KASAN can detect a
recycled-storage UAF when the new allocation receives a different tag,
so it covers part of the intended use case.

There are still cases it cannot detect, such as an in-bounds corrupting
write to the same still-live allocation, or a recycled allocation which
reuses the same tag. Software tag-based KASAN also still instruments
memory accesses and can perturb a timing-sensitive bug enough that it
no longer reproduces; hardware tag checking reduces this overhead but
is not available everywhere.

The original goal of KWatch, which I now plan to pursue through wprobe,
was to provide a simple and targeted interface to hardware watchpoints.
It does not instrument every memory access system-wide, so it can have
lower overall overhead and less timing perturbation while identifying
the actual writer. I see this as complementary to KASAN rather than a
replacement for it.

Thanks,
Jinchao

Re: [RFC PATCH v2 00/13] mm/kwatch: dynamic hardware watchpoints for hunting memory corruption

[Masami Hiramatsu]

On Tue, 21 Jul 2026 08:36:01 -0400
Jinchao Wang <wangjinchao600@gmail.com> wrote:

> On 7/20/2026 10:24 AM, Masami Hiramatsu (Google) wrote:
> > On Fri, 17 Jul 2026 11:10:04 -0700
> > Borislav Petkov <bp@alien8.de> wrote:
> 
> >> Looks to me like folks need to sit down and agree on strategy first.
> > 
> > Yeah, I need to talk with Jinchao.
> > 
> > Like kprobe, wprobe itself integrates watchpoint functionality into
> > the tracing subsystem. Jinchao, I would like to ask you to redesign
> > the kwatch tool based on fprobe and wprobe events in user space?
> > I think that could reduce redundant efforts on the similar feature.
> > 
> > Using fprobe entry and exit event trigger, we can make a Function-
> > scoped watch window. Also, since fprobe and wprobe are having BTF
> > typecast support, you can access to the members of data structures
> > more naturally.
> > 
> > If you are interested, I would be open to having you handle the
> > development of the necessary features for wprobe.
> > 
> > Thank you,
> > 
> Thank you, Boris and Masami.
> 
> Boris is right that we should agree on the overall strategy before
> moving either series forward, especially since they share the low-level
> HWBP work. Masami is also right that fprobe and wprobe provide a more
> appropriate foundation than introducing KWatch as another standalone
> debugging tool.

Agreed. I just sent v10 because v9 lacked base-commit and Sashiko did
not reviewed. (so it just add a trigger-side BTF support which was
dropped in v9).
Think of v10 as the complete set of ideas we wanted to implement
with this wprobe.

> 
> After reconsidering the design, I will stop the standalone KWatch
> series. Instead, I would like to work with Masami on extending wprobe
> to support the function-scoped dynamic-watchpoint use cases that
> motivated KWatch.

Thanks Jinchao, I appreciate to work with you on this project.

> 
> Masami, thank you for being open to having me develop the necessary
> wprobe features. I am interested in taking that direction.

OK, please share your thoughts on watchpoint usage in the kernel.

Basically, I will fix the points raised in the review, but please
let me know if you have any other suggestions.

Thank you,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>