Re: [PATCH v2] riscv: mm: Trace TLB flush path selection

[Jisheng Zhang <jszhang@kernel.org>]

On Wed, Oct 14, 2026 at 05:58:42PM +0200, Roman Storozhenko wrote:
> Hi Jisheng,
> 
> Thanks for the clarification.
> 
> Since Radim's tracepoint code-generation results are now part of this
> discussion, I've added him to Cc in case I misinterpret any aspect of
> that issue.
> 
> I agree with you on one point regarding the compiler overhead. Since
> the kernel supports older GCC versions, we have to take the code
> generated by those supported compilers into account today. So
> describing the issue simply as temporary was too strong.
> 
> However, I think there are two separate questions here: whether the
> event exposes useful information, and whether the disabled-path cost
> of its current implementation is acceptable.
> 
> Regarding kprobes, most of the raw fields in this event are indeed
> already arguments of `__flush_tlb_range()`. My point is not that these
> values are impossible to obtain dynamically.
> 
> Kprobes can be used to observe a very large part of the kernel. If the
> ability to reconstruct an event with kprobes were by itself sufficient
> reason not to add a tracepoint, the same reasoning would apply to many
> existing tracepoints.
> 
> The distinction I see here is between observing implementation details
> and exposing the semantic event directly.
> 
> For this case, the interesting information is not only `start`,
> `size`, `stride`, `cmask`, etc., but their correlation with the path
> Linux selected for that particular request:
> 
>         local
>         SBI RFENCE
>         cross-CPU call
> 
> Calculating the local case from `cmask` may be simple with the current
> implementation, but the tracing program still has to reproduce
> knowledge of the selection logic. Likewise, observing downstream
> functions ties the tracing program to the current call graph.
> 
> I also don't think `/proc/interrupts` provides equivalent information here.
> 
> The RISC-V IPI statistics are aggregate counters. In particular, Linux
> accounts `IPI_CALL_FUNC` as "Function call interrupts", while SBI
> RFENCE is handled below Linux by firmware; for example, OpenSBI
> implements the remote TLB operation using its own `IPI_TLB` event.

I think you misunderstood something. A riscv platform uses either
SBI RFENCE or IPI for tlb shootdown, and won't change. So you can
know whether the system is SBI RFENCE powered or IPI powered by
cat /proc/interrupts. It's much simpler.

> 
> So checking `/proc/interrupts` can certainly be useful as a sanity
> check in a controlled workload: an increase in function-call IPIs can
> indicate use of the Linux cross-CPU path.
> 
> But it cannot associate that IPI activity with a particular TLB
> invalidation request, and it cannot correlate it with the range, ASID,
> stride and target cpumask. Other users of the call-function IPI also
> contribute to the same counters. In a concurrent workload this becomes
> an inference rather than an observation of the TLB path decision.
> 
> That per-request correlation is what I am trying to expose with the tracepoint.
> 
> Regarding the performance argument, I agree that the disabled path
> must be cheap and that the current RISC-V tracepoint code-generation
> issue is relevant when evaluating the patch.
> 
> I would not claim that all tracepoints have exactly the same overhead.
> My narrower point is that the issue referenced in Radim's thread is a
> general interaction between RISC-V tracepoints and compiler code
> generation rather than something specific to the semantics of this
> event.
> 
> If the generated disabled path is considered too expensive, that is a
> valid implementation concern and may require improvement or
> mitigation. But I don't think it shows that the TLB path-selection
> event itself is redundant because the information can be reconstructed
> with kprobes.
> 
> There is also no contradiction, in my view, between using the event
> for performance analysis and the event having overhead when enabled.
> Tracing normally perturbs the workload when it is enabled. The
> intended use here is primarily to understand things such as the
> distribution of local/SBI/IPI decisions and the request context
> associated with those decisions, rather than to use the tracepoint
> itself for cycle-accurate measurement of the cost of a TLB flush.
> 
> The important requirement is therefore that the disabled path be
> sufficiently cheap.
> 
> I think the PowerPC TLB tracepoints also remain a useful precedent
> here: they show that permanent architecture-specific TLB observability
> can be worthwhile when the architecture-specific information is
> useful.
> 
> So I agree that convenience, maintenance cost and disabled-path
> overhead all need to be considered. I just don't think that kprobes or
> `/proc/interrupts` provide the same abstraction as recording the
> path-selection decision directly.
> 
> Thanks,
> Roman
> 
> On Wed, Oct 14, 2026 at 7:25 AM Jisheng Zhang <jszhang@kernel.org> wrote:
> >
> > On Sun, Oct 11, 2026 at 02:43:05PM +0200, Roman Storozhenko wrote:
> > > On Sat, Oct 10, 2026 at 6:44 AM Jisheng Zhang <jszhang@kernel.org> wrote:
> > > >
> > > > On Sat, Oct 03, 2026 at 10:54:44AM +0200, Roman Storozhenko wrote:
> > > > > On Fri, Oct 2, 2026 at 6:04 PM Jisheng Zhang <jszhang@kernel.org> wrote:
> > > > > >
> > > > > > On Sun, Aug 30, 2026 at 04:36:37PM +0200, Roman 'Hedin' Storozhenko wrote:
> > > > > > > Make RISC-V TLB flush path selection observable. Record whether Linux
> > > > > > > handles an invalidation locally, delegates it to SBI RFENCE, or executes
> > > > > > > it through a cross-CPU call, so MM activity can be correlated with the
> > > > > > > RISC-V, firmware, or Linux cross-CPU path carrying the request.
> > > > > >
> > > > > > I didn't see too much usefullness of the trace here. why not use kprobe
> > > > > > instead? can you plz show a real usage example of the trace? Which
> > > > > > bug/performance problem can be solved conveniently with this trace only?
> > > > > >
> > > > > > From another side, except x86, other architectures don't have the trace
> > > > > > in the TLB path.
> > > > > >
> > > > >
> > > > > Hi Jisheng,
> > > > >
> > > > > Thanks for the review. I agree that the current commit message describes
> > > > > the intended use rather abstractly.
> > > > >
> > > > > A kprobe can certainly be used to investigate this. My motivation
> > > > > for a tracepoint is not that dynamic tracing cannot obtain the
> > > > > information, but that doing so requires knowledge of the current
> > > > > implementation and reconstruction of a semantic decision made inside
> > > > > __flush_tlb_range() / flush_tlb_all().
> > > >
> > > > I prefer aligning with other architectures(except x86), I.E not
> > > > add this kind of trace. However, I would leave the decision to riscv
> > > > maintainers.
> > > >
> > > > FYI, you may notice that the riscv trace overhead is not trivial per
> > > > https://lore.kernel.org/linux-riscv/20250619190315.2603194-4-rkrcmar@ventanamicro.com/
> > >
> > > Hi Jisheng,
> > >
> > > Thanks for the feedback.
> > >
> > > Regarding the comparison with other architectures, thanks to Michael
> > > Ellerman for pointing me to the PowerPC TLB tracing when we happened
> > > to discuss this at LPC 2026.
> > >
> > > PowerPC defines the powerpc:tlbie and powerpc:tlbia trace events in
> > > arch/powerpc/include/asm/trace.h, and the corresponding trace calls
> > > are used in the PowerPC TLB invalidation paths.
> > >
> > > These events are not identical to the proposed RISC-V event: the
> > > PowerPC events primarily describe architecture invalidation
> > > operations, whereas this patch records the path selected by Linux
> > > before dispatch. However, I think they provide useful precedent for
> > > architecture-specific TLB observability beyond x86.
> > >
> > > Regarding kprobes, I agree that dynamic tracing can be used to investigate this.
> > >
> > > My concern is that obtaining the same information requires knowledge
> > > of the current implementation and reconstruction of the decision made
> > > inside __flush_tlb_range() / flush_tlb_all().
> > >
> > > For example, probing __flush_tlb_range() gives us the incoming
> > > request, but does not directly tell us whether Linux selected the
> > > local flush, SBI RFENCE, or cross-CPU-call path. A tracing tool has to
> > > probe and correlate implementation-specific downstream functions, or
> > > reproduce the selection logic itself.
> >
> > Hmm, let's look at where the tracepoints added:
> > void __flush_tlb_range(struct mm_struct *mm,
> >                               const struct cpumask *cmask,
> >                               unsigned long start, unsigned long size,
> >                               unsigned long stride)
> > void flush_tlb_all(void)
> >
> > most of the information you exported with the tracepoints are just
> > parameters of flush_tlb_range.
> > So kprobe or function trace can also get most of them, while whether
> > local or not needs to calcuate the cmask, but getting this info is
> > also easy via. adding one LoC to the kprobe routine.
> >
> > The tlb invalidation is frequently executed, so we need to carefully
> > evaluate the convience VS maitanince offort and performance overhead when
> > adding permanent tracepoints, given current riscv tracepoint overhead
> > isn't trivial.
> >
> > BTW: whether SBI RFENCE or cross-cpu-call is much easier. Just search
> > IPI in /proc/interrupts. No need to kprobe/function trace/tracepoint
> >
> > >
> > > The proposed event records this semantic decision directly at the
> > > point where Linux makes it. Users therefore do not need to depend on
> > > the current helper names, call graph, or exact implementation of the
> > > selection logic.
> > >
> > > Regarding the tracepoint overhead issue, I agree that disabled-path
> > > overhead is important, especially in the TLB flush path.
> > >
> > > However, I see the referenced SBI tracepoint case primarily as a
> > > RISC-V compiler/code-generation issue rather than as an argument
> > > against providing this kind of observability.
> > >
> > > The referenced discussion itself shows that the generated code depends
> > > significantly on compiler behavior, and that this is an area being
> > > improved. The undesirable overhead is not an inherent semantic
> > > requirement of the tracepoint.
> > >
> > > Even if the same code-generation problem currently affects this
> > > tracepoint, I don't think a temporary compiler limitation should by
> > > itself determine whether the tracepoint is useful or whether this
> > > observability belongs in the kernel.
> > >
> > > If the compiler currently generates unacceptable disabled-path
> > > overhead, that is certainly something that should be improved or
> > > mitigated in the implementation. But I would separate that issue from
> > > the question of whether exposing the RISC-V TLB path-selection
> > > decision is useful.
> > >
> >
> > But we need to live with it. Per the Radim email, gcc14 can still
> > see the issue, given the minimum gcc version of kernel building
> > requirement is gcc8 now, gcc14 has a long time to bailout.
> >
> > And another tricky point here is
> > one of your motivations of adding the tracepoint is to profile then
> > improve the tlb related code path performance, but adding the
> > tracepoint brings non-trivial overhead to tlb code path.
> 
> 
> 
> -- 
> Kind regards,
> Roman 'Hedin' Storozhenko