Re: [RFC PATCH v6 29/35] KVM: arm64: Pin the SPE buffer in the host and map it at stage 2

James Clark
2026-01-09T16:35:49Z

On 09/01/2026 4:29 pm, James Clark wrote:
>
>
> On 14/11/2025 4:07 pm, Alexandru Elisei wrote:
>> If the SPU encounters a translation fault when it attempts to write a
>> profiling record to memory, it stops profiling and asserts the PMBIRQ
>> interrupt.  Interrupts are not delivered instantaneously to the CPU, and
>> this creates a profiling blackout window where the profiled CPU executes
>> instructions, but no samples are collected.
>>
>> This is not desirable, and the SPE driver avoids it by keeping the buffer
>> mapped for the entire the profiling session.
>>
>> KVM maps memory at stage 2 when the guest accesses it, following a
>> fault on
>> a missing stage 2 translation, which means that the problem is present
>> in a
>> SPE enabled virtual machine. Worse yet, the blackout windows are
>> unpredictable: the guest profiling the same process can during one
>> profiling session, not trigger any stage 2 faults (the entire buffer
>> memory
>> is already mapped at stage 2), but worst case scenario, during another
>> profiling session, trigger stage 2 faults for every record it attempts to
>> write (if KVM keeps removing the buffer pages from stage 2), or something
>> in between - some records trigger a stage 2 fault, some don't.
>>
>> The solution is for KVM to follow what the SPE driver does: keep the
>> buffer
>> mapped at stage 2 while ProfilingBufferEnabled() is true. To accomplish
>
> Hi Alex,
>
> The problem is that the driver enables and disables the buffer every
> time the target process is switched out unless you explicitly ask for
> per-CPU mode. Is there some kind of heuristic you can add to prevent
> pinning and unpinning unless something actually changes?
>
> Otherwise it's basically unusable with normal perf commands and larger
> buffer sizes. Take these basic examples were I've added a filter so no
> SPE data is even recorded:
>
>  $ perf record -e arm_spe/min_latency=1000,event_filter=10/ -m,256M --\
>      true
>
> On a kernel with lockep and kmemleak etc this takes 20s to complete. On
> a normal kernel build it still takes 4s.
>
> Much worse is anything more complicated than just 'true' which will have
> more context switching:
>
> $ perf record -e arm_spe/min_latency=1000,event_filter=10/ -m,256M --\
>     perf stat true
>
> This takes 3 minutes or 50 seconds to complete (with and without kernel
> debugging features respectively)
>
> For comparison, running these on the host all take less than half a
> second. I measured each pin/unpin taking about 0.2s and the basic 'true'
> example resulting in 100 context switches which adds up to the 20s.
>
> Another interesting stat is that the second example says 'true' ends up
> running at an average clock speed of 4Mhz:
>
>           12683357      cycles   #    0.004 GHz
>
> You also get warnings like this
>
>  rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
>  rcu:     Tasks blocked on level-0 rcu_node (CPUs 0-0): P53/1:b..l
>  rcu:     (detected by 0, t=6503 jiffies, g=8461, q=43 ncpus=1)
>  task:perf            state:R  running task     stack:0     pid:53
> tgid:53    ppid:52     task_flags:0x400000 flags:0x00000008
>  Call trace:
>   __switch_to+0x1b8/0x2d8 (T)
>   __schedule+0x8b4/0x1050
>   preempt_schedule_common+0x2c/0xb8
>   preempt_schedule+0x30/0x38
>   _raw_spin_unlock+0x60/0x70
>   finish_fault+0x330/0x408
>   do_pte_missing+0x7d4/0x1188
>   handle_mm_fault+0x244/0x568
>   do_page_fault+0x21c/0x548
>   do_translation_fault+0x44/0x68
>   do_mem_abort+0x4c/0x100
>   el0_da+0x58/0x200
>   el0t_64_sync_handler+0xc0/0x130
>   el0t_64_sync+0x198/0x1a0
>
> If we can't add a heuristic to keep the buffer pinned, it almost seems
> like the random blackouts would be preferable to pinning being so slow.
>

One other comment to add to this, is that increasing the buffer size is
the normal reaction to profiling overheads being high. I think that's
how I came across this in the first place. Or if you want to avoid
overhead entirely you set a buffer that's large enough to handle the
whole run.

In this case in a VM it actually has the opposite effect to doing the
same thing on a host. The bigger you make it the worse the problem gets.