Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[David Hildenbrand (Arm)]

On 6/3/26 15:11, Zhuo, Qiuxu wrote:
> Hi,
> 
>  
> 
> Laiyi reported that the userspace rasdaemon fails to enable memory_failure_event
> on kernels >= v6.19.
> 
>  
> 
> Kernel commit 31807483d395 ("mm/memory-failure: remove the selection of RAS"),
> merged in v6.19-rc1,
> 
> moved the memory_failure_event tracepoint from the "ras" subsystem to
> "memory_failure".
> 
> However, rasdaemon still tries to enable:
> 
>  
> 
>     ras:memory_failure_event
> 
>  
> 
> while on v6.19+ kernels, the tracepoint is:
> 
>  
> 
>     memory_failure:memory_failure_event
> 
>  
> 
> As a result, rasdaemon fails to start:
> 
>  
> 
>     …
> 
>    Can't write to set_event
> 
>    Huh! something got wrong. Aborting.
> 
>    …
> 
>  
> 
> Reproducer:
> 
>  
> 
>     rasdaemon --enable
> 
>  
> 
> Could you please let me know whether the preferred solution is to revert the
> kernel change,
> 
> or to update rasdaemon to support both tracepoint names for backward/forward
> compatibility?


Likely the latter. BPF [1] documents:

Q: Are tracepoints part of the stable ABI?
A: NO. Tracepoints are tied to internal implementation details hence they are
subject to change and can break with newer kernels. BPF programs need to change
accordingly when this happens.

The Kernel ABI document explicitly doesn't list them AFAIKS.

There were previous discussions on the stability of tracepints [2], I don't know
what changed in the meantime. CCing Steve

[1] https://www.kernel.org/doc/html/latest/bpf/bpf_design_QA.html
[2] https://lwn.net/Articles/747256/
[3] https://www.kernel.org/doc/html/latest/admin-guide/abi.html

-- 
Cheers,

David

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Steven Rostedt]

On Wed, 3 Jun 2026 15:44:54 +0200
"David Hildenbrand (Arm)" <david@kernel.org> wrote:

> Likely the latter. BPF [1] documents:
> 
> Q: Are tracepoints part of the stable ABI?
> A: NO. Tracepoints are tied to internal implementation details hence they are
> subject to change and can break with newer kernels. BPF programs need to change
> accordingly when this happens.
> 
> The Kernel ABI document explicitly doesn't list them AFAIKS.
> 
> There were previous discussions on the stability of tracepints [2], I don't know
> what changed in the meantime. CCing Steve
> 
> [1] https://www.kernel.org/doc/html/latest/bpf/bpf_design_QA.html
> [2] https://lwn.net/Articles/747256/
> [3] https://www.kernel.org/doc/html/latest/admin-guide/abi.html

Tracepoints are not stable or BPF programs only. But other applications
they are[1].

Adding Linus as he's the Supreme Judge on the matter.

-- Steve

[1] https://lwn.net/Articles/442113/

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Borislav Petkov]

On Wed, Jun 03, 2026 at 12:17:07PM -0400, Steven Rostedt wrote:
> On Wed, 3 Jun 2026 15:44:54 +0200
> "David Hildenbrand (Arm)" <david@kernel.org> wrote:
> 
> > Likely the latter. BPF [1] documents:
> > 
> > Q: Are tracepoints part of the stable ABI?
> > A: NO. Tracepoints are tied to internal implementation details hence they are
> > subject to change and can break with newer kernels. BPF programs need to change
> > accordingly when this happens.
> > 
> > The Kernel ABI document explicitly doesn't list them AFAIKS.
> > 
> > There were previous discussions on the stability of tracepints [2], I don't know
> > what changed in the meantime. CCing Steve
> > 
> > [1] https://www.kernel.org/doc/html/latest/bpf/bpf_design_QA.html
> > [2] https://lwn.net/Articles/747256/
> > [3] https://www.kernel.org/doc/html/latest/admin-guide/abi.html
> 
> Tracepoints are not stable or BPF programs only. But other applications
> they are[1].
> 
> Adding Linus as he's the Supreme Judge on the matter.

I *think* tools or libtraceevent can't really anticipate the TP namespace
change so we might have to revert, I'm afraid...

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[David Hildenbrand (Arm)]

On 6/3/26 18:19, Borislav Petkov wrote:
> On Wed, Jun 03, 2026 at 12:17:07PM -0400, Steven Rostedt wrote:
>> On Wed, 3 Jun 2026 15:44:54 +0200
>> "David Hildenbrand (Arm)" <david@kernel.org> wrote:
>>
>>> Likely the latter. BPF [1] documents:
>>>
>>> Q: Are tracepoints part of the stable ABI?
>>> A: NO. Tracepoints are tied to internal implementation details hence they are
>>> subject to change and can break with newer kernels. BPF programs need to change
>>> accordingly when this happens.
>>>
>>> The Kernel ABI document explicitly doesn't list them AFAIKS.
>>>
>>> There were previous discussions on the stability of tracepints [2], I don't know
>>> what changed in the meantime. CCing Steve
>>>
>>> [1] https://www.kernel.org/doc/html/latest/bpf/bpf_design_QA.html
>>> [2] https://lwn.net/Articles/747256/
>>> [3] https://www.kernel.org/doc/html/latest/admin-guide/abi.html
>>
>> Tracepoints are not stable or BPF programs only. But other applications
>> they are[1].
>>
>> Adding Linus as he's the Supreme Judge on the matter.
> 
> I *think* tools or libtraceevent can't really anticipate the TP namespace
> change so we might have to revert, I'm afraid...

Yeah, I was fearing that when I read in [2]:

	"It has become clear in the past that this promise extends to
	 tracepoints, most notably in 2011 when a tracepoint change broke
	 powertop and had to be reverted."

Which means that I now also fully understand

	"Some kernel maintainers prohibit or severely restrict the addition of
	 tracepoints to their subsystems out of fear that a similar thing could
	 happen to them. "

Whatever the result of this discussion will be, I'll try to document it.

-- 
Cheers,

David

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Steven Rostedt]

On Wed, 3 Jun 2026 18:26:24 +0200
"David Hildenbrand (Arm)" <david@kernel.org> wrote:

> Yeah, I was fearing that when I read in [2]:
> 
> 	"It has become clear in the past that this promise extends to
> 	 tracepoints, most notably in 2011 when a tracepoint change broke
> 	 powertop and had to be reverted."

Technically the issue is with trace events and not tracepoints. The
difference is that a trace event is created via the TRACE_EVENT() macro
which defines what is to be collected from the tracepoint and exposes that
information to tracefs which applications can easily see.

A tracepoint is simply the hook in the code that you can attach to. Trace
events create a callback from that hook to extract the data from the
tracepoint to fill in the fields.

> 
> Which means that I now also fully understand
> 
> 	"Some kernel maintainers prohibit or severely restrict the addition of
> 	 tracepoints to their subsystems out of fear that a similar thing could
> 	 happen to them. "
> 
> Whatever the result of this discussion will be, I'll try to document it.

You can still create a tracepoint without creating a trace event by using
the DECLARE_TRACE() macro. The scheduler subsystem uses that quite
extensively. That creates a tracepoint without exposing it to tracefs. The
runtime verifier uses these hooks to monitor the scheduler.

But you can still connect to these tracepoints from tracefs via a tprobe. A
tprobe hooks to tracepoints that you need the source code to find (just
like a fprobe hooks to any function). Thus applications *can't* rely on
them because there's nothing there to tell you it exists or not.

For example, for the given tracepoint:

 # cd /sys/kernel/tracing
 # echo 't:rfail memory_failure_event pfn=pfn type=type result=result' > dynamic_events
 # cat events/tracepoints/rfail/format 
name: rfail
ID: 1894
format:
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:unsigned long __probe_ip;	offset:8;	size:8;	signed:0;
	field:u64 pfn;	offset:16;	size:8;	signed:0;
	field:s32 type;	offset:24;	size:4;	signed:1;
	field:s32 result;	offset:28;	size:4;	signed:1;

print fmt: "(%lx) pfn=%Lu type=%d result=%d", REC->__probe_ip, REC->pfn, REC->type, REC->result

It requires that BTF exists and the above doesn't annotate the result as
nicely. But you can get data directly from tracepoints this way.

-- Steve

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[David Hildenbrand (Arm)]

On 6/3/26 19:00, Steven Rostedt wrote:
> On Wed, 3 Jun 2026 18:26:24 +0200
> "David Hildenbrand (Arm)" <david@kernel.org> wrote:
> 
>> Yeah, I was fearing that when I read in [2]:
>>
>> 	"It has become clear in the past that this promise extends to
>> 	 tracepoints, most notably in 2011 when a tracepoint change broke
>> 	 powertop and had to be reverted."
> 
> Technically the issue is with trace events and not tracepoints. The
> difference is that a trace event is created via the TRACE_EVENT() macro
> which defines what is to be collected from the tracepoint and exposes that
> information to tracefs which applications can easily see.
> 
> A tracepoint is simply the hook in the code that you can attach to. Trace
> events create a callback from that hook to extract the data from the
> tracepoint to fill in the fields.
> 
>>
>> Which means that I now also fully understand
>>
>> 	"Some kernel maintainers prohibit or severely restrict the addition of
>> 	 tracepoints to their subsystems out of fear that a similar thing could
>> 	 happen to them. "
>>
>> Whatever the result of this discussion will be, I'll try to document it.
> 
> You can still create a tracepoint without creating a trace event by using
> the DECLARE_TRACE() macro. The scheduler subsystem uses that quite
> extensively. That creates a tracepoint without exposing it to tracefs. The
> runtime verifier uses these hooks to monitor the scheduler.
> 
> But you can still connect to these tracepoints from tracefs via a tprobe. A
> tprobe hooks to tracepoints that you need the source code to find (just
> like a fprobe hooks to any function). Thus applications *can't* rely on
> them because there's nothing there to tell you it exists or not.

Thanks, that makes sense!

So, would it be fair to say that, in general, what's exposed through

	/sys/kernel/tracing/events/

is stable ABI?


Would the following be sufficient to avoid a full revert and the dependency on CONFIG_RAS?

diff --git a/include/trace/events/memory-failure.h b/include/trace/events/memory-failure.h
index aa57cc8f896b..c46b17602578 100644
--- a/include/trace/events/memory-failure.h
+++ b/include/trace/events/memory-failure.h
@@ -1,6 +1,7 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 #undef TRACE_SYSTEM
-#define TRACE_SYSTEM memory_failure
+/* Some user space relies on ras/memory_failure_event */
+#define TRACE_SYSTEM ras
 #define TRACE_INCLUDE_FILE memory-failure
 
 #if !defined(_TRACE_MEMORY_FAILURE_H) || defined(TRACE_HEADER_MULTI_READ)


-- 
Cheers,

David

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Steven Rostedt]

On Wed, 3 Jun 2026 21:13:30 +0200
"David Hildenbrand (Arm)" <david@kernel.org> wrote:

> Would the following be sufficient to avoid a full revert and the dependency on CONFIG_RAS?
> 
> diff --git a/include/trace/events/memory-failure.h b/include/trace/events/memory-failure.h
> index aa57cc8f896b..c46b17602578 100644
> --- a/include/trace/events/memory-failure.h
> +++ b/include/trace/events/memory-failure.h
> @@ -1,6 +1,7 @@
>  /* SPDX-License-Identifier: GPL-2.0 */
>  #undef TRACE_SYSTEM
> -#define TRACE_SYSTEM memory_failure
> +/* Some user space relies on ras/memory_failure_event */
> +#define TRACE_SYSTEM ras

If that puts back the original path then yeah, all would be good.

-- Steve
  

>  #define TRACE_INCLUDE_FILE memory-failure
>  
>  #if !defined(_TRACE_MEMORY_FAILURE_H) || defined(TRACE_HEADER_MULTI_READ)

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Steven Rostedt]

On Wed, 3 Jun 2026 21:13:30 +0200
"David Hildenbrand (Arm)" <david@kernel.org> wrote:

> Thanks, that makes sense!
> 
> So, would it be fair to say that, in general, what's exposed through
> 
> 	/sys/kernel/tracing/events/
> 
> is stable ABI?

It's only stable if something depends on it. It changes all the time.
It's only when someone complains about it that it becomes "stable"!

-- Steve

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[David Hildenbrand (Arm)]

On 6/3/26 21:31, Steven Rostedt wrote:
> On Wed, 3 Jun 2026 21:13:30 +0200
> "David Hildenbrand (Arm)" <david@kernel.org> wrote:
> 
>> Thanks, that makes sense!
>>
>> So, would it be fair to say that, in general, what's exposed through
>>
>> 	/sys/kernel/tracing/events/
>>
>> is stable ABI?
> 
> It's only stable if something depends on it. It changes all the time.
> It's only when someone complains about it that it becomes "stable"!

Heh, so we only know that it's stable when we break it ...

Let me figure out how to document that.

-- 
Cheers,

David

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Steven Rostedt]

On June 5, 2026 4:52:28 AM EDT, "David Hildenbrand (Arm)" <david@kernel.org> wrote:
>On 6/3/26 21:31, Steven Rostedt wrote:
>> On Wed, 3 Jun 2026 21:13:30 +0200
>> "David Hildenbrand (Arm)" <david@kernel.org> wrote:
>> 
>>> Thanks, that makes sense!
>>>
>>> So, would it be fair to say that, in general, what's exposed through
>>>
>>> 	/sys/kernel/tracing/events/
>>>
>>> is stable ABI?
>> 
>> It's only stable if something depends on it. It changes all the time.
>> It's only when someone complains about it that it becomes "stable"!
>
>Heh, so we only know that it's stable when we break it ...
>
>Let me figure out how to document that.
>

Yep. That's basically Linus's rule. He even said we break user space API all the time. What we don't allow is to break actual user space. The problem is that you can break user space by fixing an API without knowing something depended on the bug.

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Xie Yuanbin]

On Wed, 3 Jun 2026 21:13:30 +0200, David Hildenbrand (Arm) wrote:
> Would the following be sufficient to avoid a full revert and the dependency on CONFIG_RAS?
>
> diff --git a/include/trace/events/memory-failure.h b/include/trace/events/memory-failure.h
> index aa57cc8f896b..c46b17602578 100644
> --- a/include/trace/events/memory-failure.h
> +++ b/include/trace/events/memory-failure.h
> @@ -1,6 +1,7 @@
>  /* SPDX-License-Identifier: GPL-2.0 */
>  #undef TRACE_SYSTEM
> -#define TRACE_SYSTEM memory_failure
> +/* Some user space relies on ras/memory_failure_event */
> +#define TRACE_SYSTEM ras
>  #define TRACE_INCLUDE_FILE memory-failure
>
>  #if !defined(_TRACE_MEMORY_FAILURE_H) || defined(TRACE_HEADER_MULTI_READ)

Yes, it should be. In fact, when I sent the V2 patch, I had already
considered this issue, and that's exactly what I did:
Link: https://lore.kernel.org/20251104072306.100738-3-xieyuanbin1@huawei.com

However, David Hildenbrand advised me at that time to completely
remove the dependence on RAS:
Link: https://lore.kernel.org/01b44e0f-ea2e-406f-9f65-b698b5504f42@kernel.org

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[David Hildenbrand (Arm)]

On 6/4/26 03:46, Xie Yuanbin wrote:
> On Wed, 3 Jun 2026 21:13:30 +0200, David Hildenbrand (Arm) wrote:
>> Would the following be sufficient to avoid a full revert and the dependency on CONFIG_RAS?
>>
>> diff --git a/include/trace/events/memory-failure.h b/include/trace/events/memory-failure.h
>> index aa57cc8f896b..c46b17602578 100644
>> --- a/include/trace/events/memory-failure.h
>> +++ b/include/trace/events/memory-failure.h
>> @@ -1,6 +1,7 @@
>>  /* SPDX-License-Identifier: GPL-2.0 */
>>  #undef TRACE_SYSTEM
>> -#define TRACE_SYSTEM memory_failure
>> +/* Some user space relies on ras/memory_failure_event */
>> +#define TRACE_SYSTEM ras
>>  #define TRACE_INCLUDE_FILE memory-failure
>>
>>  #if !defined(_TRACE_MEMORY_FAILURE_H) || defined(TRACE_HEADER_MULTI_READ)
> 
> Yes, it should be. In fact, when I sent the V2 patch, I had already
> considered this issue, and that's exactly what I did:
> Link: https://lore.kernel.org/20251104072306.100738-3-xieyuanbin1@huawei.com
> 
> However, David Hildenbrand advised me at that time to completely
> remove the dependence on RAS:
> Link: https://lore.kernel.org/01b44e0f-ea2e-406f-9f65-b698b5504f42@kernel.org

Yeah, if only I had known that we would break user space by changing trace
events ... now we know :)

Do you have capacity to send a fix?

-- 
Cheers,

David

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Xie Yuanbin]

On Thu, 4 Jun 2026 08:42:37 +0200, David Hildenbrand (Arm) wrote:
> Yeah, if only I had known that we would break user space by changing trace
> events ... now we know :)
>
> Do you have capacity to send a fix?

Sure, with pleasure.

RE: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Zhuo, Qiuxu]

> From: Xie Yuanbin <xieyuanbin1@huawei.com>
> Sent: Thursday, June 4, 2026 9:42 PM
> To: david@kernel.org; Zhuo, Qiuxu <qiuxu.zhuo@intel.com>; bp@alien8.de;
> akpm@linux-foundation.org; rostedt@goodmis.org; linmiaohe@huawei.com
> Cc: linux-edac@vger.kernel.org; linux-kernel@vger.kernel.org; linux-
> mm@kvack.org; linux-trace-kernel@vger.kernel.org;
> mchehab+huawei@kernel.org; Luck, Tony <tony.luck@intel.com>;
> torvalds@linux-foundation.org; xieyuanbin1@huawei.com; Lai, Yi1
> <yi1.lai@intel.com>
> Subject: Re: mm/memory-failure tracepoint change breaks userspace
> rasdaemon
> 
> On Thu, 4 Jun 2026 08:42:37 +0200, David Hildenbrand (Arm) wrote:
> > Yeah, if only I had known that we would break user space by changing
> > trace events ... now we know :)
> >
> > Do you have capacity to send a fix?
> 
> Sure, with pleasure.

Thanks Yuanbin,

When your patch is ready, we can help test it again if needed.

-Qiuxu

RE: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Zhuo, Qiuxu]

> From: David Hildenbrand (Arm) <david@kernel.org>
> [...]
> Would the following be sufficient to avoid a full revert and the dependency
> on CONFIG_RAS?
> 
> diff --git a/include/trace/events/memory-failure.h
> b/include/trace/events/memory-failure.h
> index aa57cc8f896b..c46b17602578 100644
> --- a/include/trace/events/memory-failure.h
> +++ b/include/trace/events/memory-failure.h
> @@ -1,6 +1,7 @@
>  /* SPDX-License-Identifier: GPL-2.0 */
>  #undef TRACE_SYSTEM
> -#define TRACE_SYSTEM memory_failure
> +/* Some user space relies on ras/memory_failure_event */ #define
> +TRACE_SYSTEM ras
>  #define TRACE_INCLUDE_FILE memory-failure
> 

Thanks all for the discussion on this issue.

We applied David's above fix to v7.1-rc3, tested it, and confirmed that rasdaemon 
can again enable and receive the memory_failure event.

   Rasdaemon logs:
     ...
     rasdaemon: ras:memory_failure_event event enabled
     rasdaemon: Enabled event ras:memory_failure_event
    ...
    <...>-2513  [000] .....     0.000021 memory_failure_event [ALERT] 2026-06-04 23:30:43 +0800 pfn=0x144e6f page_type=dirty LRU page action_result=Recovered
    ...

-Qiuxu

Re: mm/memory-failure tracepoint change breaks userspace rasdaemon

[Andrew Morton]

On Wed, 3 Jun 2026 13:00:06 -0400 Steven Rostedt <rostedt@goodmis.org> wrote:

> On Wed, 3 Jun 2026 18:26:24 +0200
> "David Hildenbrand (Arm)" <david@kernel.org> wrote:
> 
> > Yeah, I was fearing that when I read in [2]:
> > 
> > 	"It has become clear in the past that this promise extends to
> > 	 tracepoints, most notably in 2011 when a tracepoint change broke
> > 	 powertop and had to be reverted."
> 
> Technically the issue is with trace events and not tracepoints. The
> difference is that a trace event is created via the TRACE_EVENT() macro
> which defines what is to be collected from the tracepoint and exposes that
> information to tracefs which applications can easily see.
> 
> A tracepoint is simply the hook in the code that you can attach to. Trace
> events create a callback from that hook to extract the data from the
> tracepoint to fill in the fields.

The problem here appears to be that "ras:memory_failure_event" became
"memory_failure:memory_failure_event".

Perhaps we can add infrastructure to permit aliasing "ras" onto
"memory_failure".  So if we make these namespace alterations we can
easily preserve back-compatibility?