Re: [PATCH v2] fuse: Wake requests on the same cpu

[Bernd Schubert <bschubert@ddn.com>]

On 10/16/25 22:13, Joanne Koong wrote:
> On Thu, Oct 16, 2025 at 2:00 AM Peter Zijlstra <peterz@infradead.org> wrote:
>>
>> On Thu, Oct 16, 2025 at 10:58:14AM +0200, Peter Zijlstra wrote:
>>> On Wed, Oct 15, 2025 at 03:19:31PM -0700, Joanne Koong wrote:
>>>
>>>>>> Won't this lose cache locality for all the other data that is in the
>>>>>> client thread's cache on the previous CPU? It seems to me like on
>>>>>> average this would be a costlier miss overall? What are your thoughts
>>>>>> on this?
>>>>>
>>>>> So as in the introduction, which b4 made a '---' comment below,
>>>>> initially I thought this should be a conditional on queue-per-core.
>>>>> With queue-per-core it should be easy to explain, I think.
>>>>>
>>>>> App submits request on core-X, waits/sleeps, request gets handle on
>>>>> core-X by queue-X.
>>>>> If there are more applications running on this core, they
>>>>> get likely re-scheduled to another core, as the libfuse queue thread is
>>>>> core bound. If other applications don't get re-scheduled either the
>>>>> entire system is overloaded or someone sets manual application core
>>>>> affinity - we can't do much about that in either case. With
>>>>> queue-per-core there is also no debate about "previous CPU".
>>>>> Worse is actually scheduler behavior here, although the ring thread
>>>>> itself goes to sleep soon enough. Application gets still quite often
>>>>> re-scheduled to another core. Without wake-on-same core behavior is
>>>>> even worse and it jumps across all the time. Not good for CPU cache...
>>>>
>>>> Maybe this is a lack of my understanding of scheduler internals,  but
>>>> I'm having a hard time seeing what the benefit of
>>>> wake_up_on_current_cpu() is over wake_up() for the queue-per-core
>>>> case.
>>>>
>>>> As I understand it, with wake_up() the scheduler already will try to
>>>> wake up the thread and put it back on the same core to maintain cache
>>>> locality, which in this case is the same core
>>>> "wake_up_on_current_cpu()" is trying to put it on. If there's too much
>>>> load imbalance then regardless of whether you call wake_up() or
>>>> wake_up_on_current_cpu(), the scheduler will migrate the task to
>>>> whatever other core is better for it.
>>>>
>>>> So I guess the main benefit of calling wake_up_on_current_cpu() over
>>>> wake_up() is that for situations where there is only some but not too
>>>> much load imbalance we force the application to run on the current
>>>> core even despite the scheduler thinking it's better for overall
>>>> system health to distribute the load? I don't see an issue if the
>>>> application thread runs very briefly but it seems more likely that the
>>>> application thread could be work intensive in which case it seems like
>>>> the thread would get migrated anyways or lead to more latency in the
>>>> long term with trying to compete on an overloaded core?
>>>
>>> So the scheduler will try and wake on the previous CPU, but if that CPU
>>> is not idle it will look for any non-idle CPU in the same L3 and very
>>
>> Typing hard: s/non-//
>>
>>> aggressively move tasks around.
>>>
>>> Notably if Task-A is waking Task-B, and Task-A is running on CPU-1 and
>>> Task-B was previously running on CPU-1, then the wakeup will see CPU-1
>>> is not idle (it is running Task-A) and it will try and find another CPU
>>> in the same L3.
>>>
>>> This is fine if Task-A continues running; however in the case where
>>> Task-A is going to sleep right after doing the wakeup, this is perhaps
>>> sub-optimal, CPU-1 will end up idle.
>>>
>>> We have the WF_SYNC (wake-flag) that tries to indicate this latter case;
>>> trouble is, it often gets used where it should not be, it is unreliable.
>>> Therefore it not a strong hint.
>>>
>>> Then we 'recently' grew WF_CURRENT_CPU, that forces the wakeup to the
>>> same CPU. If you abuse, you keep pieces :-)
>>>
>>> So it all depends a bit on the workload, machine and situation.
>>>
>>> Some machines L3 is fine, some machines L3 has exclusive L2 and it hurts
>>> more to move tasks. Some workloads don't fit L2 so it doesn't matter
>>> anyway. TL;DR is we need this damn crystal ball instruction :-)
> 
> Thanks for the explanation! I found it very helpful.
> 
> In light of that information, it seems to me that the original
> wake_up() would be more optimal here than wake_up_on_current_cpu()
> then. After fuse_request_end(), the thread still has work to do with
> fetching and servicing the next requests. If it wakes up the
> application on its cpu, then with queue-per-core the thread would be
> forced to sleep since on the libfuse side during setup the thread is
> pinned to the core, which would prevent any migration while the
> application task runs. Or am I misassuming something in this analysis,
> Bernd?


Well, the numbers speak a different language. And I still don't see
why wouldn't want to take on the current CPU at least if we have
queue-per-core.
Thanks a lot @Peter for the explanation. To my understanding WF_SYNC
should do the trick, i.e. wake on the same core, when the current task
is going to sleep anyway and there is nothing else running on that core.
For a blocking IO with queue-per-core this is exactly what we have.

Example

+ echo 'Running: example/passthrough_hp' -o allow_other --foreground --nopassthrough -o io_uring -o io_uring_nr_qs=2 /tmp/source /tmp/dest


And then 

bschubert2@imesrv3 ~>fio --directory=/tmp/dest --name=iops.\$jobnum --rw=randread --bs=4k --size=1G --numjobs=1 --iodepth=1 --time_based --runtime=30s --group_reporting --ioengine=psync --direct=1 


With WF_CURRENT_CPU: READ: bw=269MiB/s
With WF_SYNC:        READ: bw=214MiB/s
With plain wake_up:  READ: bw=217MiB/s

With WF_SYNC and plain wake_up I see a persistent core switching
of the fio process between two cores on one numa node - so much 
about L1/L2 cpu cache.
With more fuse-io-uring queues there also would be additional
switching for that and even lower perf, but with only one queue
per numa that gets a bit restricted.

My guess is that WF_SYNC doesn't detect that the current libfuse
ring thread will go to sleep in the io_uring_enter() system call
rather quickly.
I can try to get some time and figure out why the fio process
bounces between two cores. I had already started to ftrace
things, because even WF_CURRENT_CPU isn't ideal. Although this
discussion here goes the other direction.


Thanks,
Bernd