Re: [PATCH v2] fuse: Wake requests on the same cpu

[Peter Zijlstra <peterz@infradead.org>]

On Thu, Oct 16, 2025 at 10:58:14AM +0200, Peter Zijlstra wrote:
> On Wed, Oct 15, 2025 at 03:19:31PM -0700, Joanne Koong wrote:
> 
> > > > Won't this lose cache locality for all the other data that is in the
> > > > client thread's cache on the previous CPU? It seems to me like on
> > > > average this would be a costlier miss overall? What are your thoughts
> > > > on this?
> > >
> > > So as in the introduction, which b4 made a '---' comment below,
> > > initially I thought this should be a conditional on queue-per-core.
> > > With queue-per-core it should be easy to explain, I think.
> > >
> > > App submits request on core-X, waits/sleeps, request gets handle on
> > > core-X by queue-X.
> > > If there are more applications running on this core, they
> > > get likely re-scheduled to another core, as the libfuse queue thread is
> > > core bound. If other applications don't get re-scheduled either the
> > > entire system is overloaded or someone sets manual application core
> > > affinity - we can't do much about that in either case. With
> > > queue-per-core there is also no debate about "previous CPU".
> > > Worse is actually scheduler behavior here, although the ring thread
> > > itself goes to sleep soon enough. Application gets still quite often
> > > re-scheduled to another core. Without wake-on-same core behavior is
> > > even worse and it jumps across all the time. Not good for CPU cache...
> > 
> > Maybe this is a lack of my understanding of scheduler internals,  but
> > I'm having a hard time seeing what the benefit of
> > wake_up_on_current_cpu() is over wake_up() for the queue-per-core
> > case.
> > 
> > As I understand it, with wake_up() the scheduler already will try to
> > wake up the thread and put it back on the same core to maintain cache
> > locality, which in this case is the same core
> > "wake_up_on_current_cpu()" is trying to put it on. If there's too much
> > load imbalance then regardless of whether you call wake_up() or
> > wake_up_on_current_cpu(), the scheduler will migrate the task to
> > whatever other core is better for it.
> > 
> > So I guess the main benefit of calling wake_up_on_current_cpu() over
> > wake_up() is that for situations where there is only some but not too
> > much load imbalance we force the application to run on the current
> > core even despite the scheduler thinking it's better for overall
> > system health to distribute the load? I don't see an issue if the
> > application thread runs very briefly but it seems more likely that the
> > application thread could be work intensive in which case it seems like
> > the thread would get migrated anyways or lead to more latency in the
> > long term with trying to compete on an overloaded core?
> 
> So the scheduler will try and wake on the previous CPU, but if that CPU
> is not idle it will look for any non-idle CPU in the same L3 and very

Typing hard: s/non-//

> aggressively move tasks around.
> 
> Notably if Task-A is waking Task-B, and Task-A is running on CPU-1 and
> Task-B was previously running on CPU-1, then the wakeup will see CPU-1
> is not idle (it is running Task-A) and it will try and find another CPU
> in the same L3.
> 
> This is fine if Task-A continues running; however in the case where
> Task-A is going to sleep right after doing the wakeup, this is perhaps
> sub-optimal, CPU-1 will end up idle.
> 
> We have the WF_SYNC (wake-flag) that tries to indicate this latter case;
> trouble is, it often gets used where it should not be, it is unreliable.
> Therefore it not a strong hint.
> 
> Then we 'recently' grew WF_CURRENT_CPU, that forces the wakeup to the
> same CPU. If you abuse, you keep pieces :-)
> 
> So it all depends a bit on the workload, machine and situation.
> 
> Some machines L3 is fine, some machines L3 has exclusive L2 and it hurts
> more to move tasks. Some workloads don't fit L2 so it doesn't matter
> anyway. TL;DR is we need this damn crystal ball instruction :-)