Re: [PATCH v4 1/1] powerpc: enable dynamic preemption

[Shrikanth Hegde <sshegde@linux.ibm.com>]

Hi Jirka,
Please avoid top-posting.

On 7/27/26 3:43 PM, Jirka Hladky wrote:
> Hi Shrikanth,
> 
> Thanks for the quick response and for looping in Paul.
> 
>> This is true only if user selected PREEMPT_DYNAMIC option i think.
> 
> Yes, exactly. The issue is that CONFIG_PREEMPT_DYNAMIC=y has been in
> the Fedora/RHEL kernel config since kernel 5.16 (2021). It was
> silently ignored on ppc64le until your 6.16 patch added
> HAVE_PREEMPT_DYNAMIC_KEY, so this is the first time PREEMPT_RCU
> actually takes effect for all Fedora/RHEL ppc64le users.
> 
>> Does your preemption mode remain the same in two cases?
> 
> Verified. On 6.15-rc6 (before PREEMPT_DYNAMIC takes effect):
> 
>    CONFIG_PREEMPT_VOLUNTARY=y
>    (no CONFIG_PREEMPT_DYNAMIC, no /sys/kernel/debug/sched/preempt)
> 
> On both 7.1 and 7.2-rc4 (with PREEMPT_DYNAMIC active):
> 
>    CONFIG_PREEMPT_LAZY=y
>    CONFIG_PREEMPT_DYNAMIC=y
>    CONFIG_PREEMPT_RCU=y
>    cat /sys/kernel/debug/sched/preempt: "full (lazy)"
> 
> So the runtime preemption mode is the same on 7.1 and 7.2. The
> difference between 6.15 and 7.1+ is that the preemption model changed
> from voluntary (static) to full/lazy (dynamic), which is what pulls
> in PREEMPT_RCU.

That means comparison is between preempt=voluntary vs preempt=lazy.

If you make it full preemption in 6.15 you will likely see similar data as 7.1+.
Only on 7.0/7.1 there is force switch to lazy/full. Can you give that a try?

If it shows same data, that implies the regression is mainly due to change
of preemption modes, rather than the static key stuff.

> 
>> Weak memory model would need barriers irrespective of
>> HAVE_PREEMPT_DYNAMIC_CALL or HAVE_PREEMPT_DYNAMIC_KEY.
>> Static key too is expected to minimal cost.
> 
> You're right, I should clarify -- the overhead is not in the static
> key mechanism itself. The cost comes from __rcu_read_lock() and
> __rcu_read_unlock() which are called when CONFIG_PREEMPT_RCU=y.
> These need real memory barriers (lwsync/isync) on ppc64le regardless
> of whether the dynamic mechanism uses keys or calls.
> 
> The real question is: why does enabling PREEMPT_RCU cost ~33% on
> ppc64le but only ~3% on x86_64? The answer is that x86_64's TSO
> memory model makes the barriers in __rcu_read_lock/__rcu_read_unlock
> essentially free, while ppc64le's weak ordering requires explicit
> lwsync/isync instructions, which are expensive when called thousands
> of times per second in the SELinux AVC hot path.

Plus, it may call schedule in lazy/pull preemption.

> 
> I also have new data from SELinux isolation testing that helps
> quantify this. On kernel 7.1 (which already has PREEMPT_RCU=y):
> 
> SELinux mode    kill bogo-ops/sec    vs enforcing
> ------------    -----------------    ------------
> Enforcing       69,107               baseline
> Permissive      70,248               +1.6%
> Disabled        93,566               +35.4%
> 
> Permissive ~ enforcing confirms the overhead is not in SELinux policy
> evaluation. Disabling SELinux removes the rcu_read_lock/unlock call
> sites in the AVC path and recovers most of the performance -- but
> still leaves a ~13% gap vs 6.12 (no PREEMPT_RCU), which is the base
> cost of PREEMPT_RCU in the non-SELinux parts of the kill() path.
> 

This seems strange. How come rcu lock/unlock depends on SELinux policy?
One should call rcu lock/unlock if they are working with rcu updated fields.

Does the policy change itself protected with rcu lock/unlock?
I will check it up.

> So the core issue is: on ppc64le, CONFIG_PREEMPT_RCU makes
> rcu_read_lock/unlock significantly more expensive, and the kill()
> syscall path hits them very heavily through SELinux AVC lookups.
> 
> Thanks,
> Jirka
> 
> On Mon, Jul 27, 2026 at 6:19 AM Shrikanth Hegde <sshegde@linux.ibm.com> wrote:
>>
>> +cc paul for any further/RCU insights.
>>
>> On 7/27/26 12:03 AM, Jirka Hladky wrote:
>>>     Hi Shrikanth, Christophe,
>>>
>>
>> Hi Jirka, thanks for the report.
>>
>>>    I'm seeing a significant performance regression on ppc64le after this
>>>    patch landed in 6.16, caused by CONFIG_PREEMPT_RCU becoming active
>>>    once HAVE_PREEMPT_DYNAMIC_KEY is selected.
>>>
>>
>> This is true only if user selected PREEMPT_DYNAMIC option i think.
>>
>> config PREEMPT_RCU
>>           bool
>>           default y if (PREEMPT || PREEMPT_RT || PREEMPT_DYNAMIC)
>>           select TREE_RCU
>>
>>
>>>    Benchmark: stress-ng kill stressor (tight kill() syscall loop),
>>>    single thread, POWER10 LPAR (8 vCPUs, 1 core SMT-8).
>>>
>>>    Bisected across Fedora ELN kernel builds on ppc64le:
>>>
>>>      kernel               CONFIG_PREEMPT_RCU   kill bogo-ops/sec
>>>    ---  6.15-rc6 (eln148)    no                   103,207
>>>      6.16     (eln150)          yes                   70,281    (-32%)
>>>      6.18     (eln154)          yes                   72,552    (-30%)
>>>
>>
>> Does your preemption mode remain the same in two cases?
>>
>>>    For comparison, x86_64 (AMD EPYC 9355P) with the same config change
>>>    shows only a 2.8% regression:
>>>
>>>      6.12 x86_64          37,436
>>>      7.2  x86_64           36,392    (-2.8%)
>>>
>>>    perf report shows the overhead comes from rcu_read_lock/unlock in the
>>>    SELinux AVC path (check_kill_permission -> security_task_kill ->
>>>    selinux_task_kill -> avc_has_perm -> avc_lookup):
>>>
>>>      Function               6.15 (no PREEMPT_RCU)   6.16 (PREEMPT_RCU)
>>>    ---  avc_lookup                          15.23%              24.79%
>>>      __rcu_read_lock                      ~0%                 4.52%
>>>      __rcu_read_unlock                    ~0%                 4.17%
>>>      selinux_task_kill                    6.23%               7.35%
>>>      audit_signal_info*                   0.94%               3.59%
>>>
>>>    On x86_64, rcu_read_lock/unlock are cheap thanks to static calls
>>>    (HAVE_PREEMPT_DYNAMIC_CALL). On ppc64le with the KEY-based
>>>    implementation, the weak memory model requires real barriers
>>>    (lwsync/isync) making each RCU read-side critical section
>>>    significantly more expensive.
>>
>> Weak memory model would need barriers irrespective of HAVE_PREEMPT_DYNAMIC_CALL
>> or HAVE_PREEMPT_DYNAMIC_KEY. That's my assumption. I will look
>> more into it. Also i don't know much about PREEMPT_RCU. So might take a while.
>>
>>>
>>>    This aligns with Christophe's earlier review comment that
>>>    HAVE_PREEMPT_DYNAMIC_CALL should be more performant. Would
>>>    implementing static calls for ppc64 be feasible to close this gap?
>>>
>>
>> Static key too is expected to minimal cost. There maybe more into this.
>>
>>>    Test details:
>>>    - Machine: IBM POWER10 (pvr 0080 0200), pHyp virtualization
>>>    - stress-ng 0.21.03, gcc 14.3.1, glibc 2.39
>>>    - Tuned profile: virtual-guest
>>>    - SELinux: enforcing (permissive recovers only ~7%)
>>>
>>>    Happy to run additional tests if needed.
>>>
>>>
>>
> 
>