Re: [PATCH v2] fuse: Wake requests on the same cpu

[Peter Zijlstra <peterz@infradead.org>]

On Wed, Oct 15, 2025 at 03:19:31PM -0700, Joanne Koong wrote:

> > > Won't this lose cache locality for all the other data that is in the
> > > client thread's cache on the previous CPU? It seems to me like on
> > > average this would be a costlier miss overall? What are your thoughts
> > > on this?
> >
> > So as in the introduction, which b4 made a '---' comment below,
> > initially I thought this should be a conditional on queue-per-core.
> > With queue-per-core it should be easy to explain, I think.
> >
> > App submits request on core-X, waits/sleeps, request gets handle on
> > core-X by queue-X.
> > If there are more applications running on this core, they
> > get likely re-scheduled to another core, as the libfuse queue thread is
> > core bound. If other applications don't get re-scheduled either the
> > entire system is overloaded or someone sets manual application core
> > affinity - we can't do much about that in either case. With
> > queue-per-core there is also no debate about "previous CPU".
> > Worse is actually scheduler behavior here, although the ring thread
> > itself goes to sleep soon enough. Application gets still quite often
> > re-scheduled to another core. Without wake-on-same core behavior is
> > even worse and it jumps across all the time. Not good for CPU cache...
> 
> Maybe this is a lack of my understanding of scheduler internals,  but
> I'm having a hard time seeing what the benefit of
> wake_up_on_current_cpu() is over wake_up() for the queue-per-core
> case.
> 
> As I understand it, with wake_up() the scheduler already will try to
> wake up the thread and put it back on the same core to maintain cache
> locality, which in this case is the same core
> "wake_up_on_current_cpu()" is trying to put it on. If there's too much
> load imbalance then regardless of whether you call wake_up() or
> wake_up_on_current_cpu(), the scheduler will migrate the task to
> whatever other core is better for it.
> 
> So I guess the main benefit of calling wake_up_on_current_cpu() over
> wake_up() is that for situations where there is only some but not too
> much load imbalance we force the application to run on the current
> core even despite the scheduler thinking it's better for overall
> system health to distribute the load? I don't see an issue if the
> application thread runs very briefly but it seems more likely that the
> application thread could be work intensive in which case it seems like
> the thread would get migrated anyways or lead to more latency in the
> long term with trying to compete on an overloaded core?

So the scheduler will try and wake on the previous CPU, but if that CPU
is not idle it will look for any non-idle CPU in the same L3 and very
aggressively move tasks around.

Notably if Task-A is waking Task-B, and Task-A is running on CPU-1 and
Task-B was previously running on CPU-1, then the wakeup will see CPU-1
is not idle (it is running Task-A) and it will try and find another CPU
in the same L3.

This is fine if Task-A continues running; however in the case where
Task-A is going to sleep right after doing the wakeup, this is perhaps
sub-optimal, CPU-1 will end up idle.

We have the WF_SYNC (wake-flag) that tries to indicate this latter case;
trouble is, it often gets used where it should not be, it is unreliable.
Therefore it not a strong hint.

Then we 'recently' grew WF_CURRENT_CPU, that forces the wakeup to the
same CPU. If you abuse, you keep pieces :-)

So it all depends a bit on the workload, machine and situation.

Some machines L3 is fine, some machines L3 has exclusive L2 and it hurts
more to move tasks. Some workloads don't fit L2 so it doesn't matter
anyway. TL;DR is we need this damn crystal ball instruction :-)