Re: [PATCH 02/27] sched/isolation: Introduce housekeeping per-cpu rwsem

[Waiman Long <llong@redhat.com>]

On 6/25/25 10:18 AM, Frederic Weisbecker wrote:
> Le Mon, Jun 23, 2025 at 01:34:58PM -0400, Waiman Long a écrit :
>> On 6/20/25 11:22 AM, Frederic Weisbecker wrote:
>>> The HK_TYPE_DOMAIN isolation cpumask, and further the
>>> HK_TYPE_KERNEL_NOISE cpumask will be made modifiable at runtime in the
>>> future.
>>>
>>> The affected subsystems will need to synchronize against those cpumask
>>> changes so that:
>>>
>>> * The reader get a coherent snapshot
>>> * The housekeeping subsystem can safely propagate a cpumask update to
>>>     the susbsytems after it has been published.
>>>
>>> Protect against readsides that can sleep with per-cpu rwsem. Updates are
>>> expected to be very rare given that CPU isolation is a niche usecase and
>>> related cpuset setup happen only in preparation work. On the other hand
>>> read sides can occur in more frequent paths.
>>>
>>> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
>> Thanks for the patch series and it certainly has some good ideas. However I
>> am a bit concern about the overhead of using percpu-rwsem for
>> synchronization especially when the readers have to wait for the completion
>> on the writer side. From my point of view, during the transition period when
>> new isolated CPUs are being added or old ones being removed, the reader will
>> either get the old CPU data or the new one depending on the exact timing.
>> The effect the CPU selection may persist for a while after the end of the
>> critical section.
> It depends.
>
> 1) If the read side queues a work and wait for it
>    (case of work_on_cpu()), we can protect the whole under the same
>    sleeping lock and there is no persistance beyond.
>
> 2) But if the read side just queues some work or defines some cpumask
>     for future queue then there is persistance and some action must be
>     taken by housekeeping after the update to propagare the new cpumask
>     (flush pending works, etc...)

I don't mind doing actions to make sure that the cpumask is properly 
propagated after changing housekeeping cpumasks. I just don't want to 
introduce too much latency on the reader which could be a latency 
sensitive task running on an isolated CPU.

I would say it should be OK to have a grace period (reusing the RCU 
term) after changing the housekeeping cpumasks that tasks running on 
those CPUs that are affected by cpumask changes may or may not 
experience the full effect of the cpumask change. However, we should 
minimize the overhead of those tasks that run on CPUs unrelated to the 
cpumask change ASAP.

>> Can we just rely on RCU to make sure that it either get the new one or the
>> old one but nothing in between without the additional overhead?
> This is the case as well and it is covered by 2) above.
> The sleeping parts handled in 1) would require more thoughts.
>
>> My current thinking is to make use CPU hotplug to enable better CPU
>> isolation. IOW, I would shut down the affected CPUs, change the housekeeping
>> masks and then bring them back online again. That means the writer side will
>> take a while to complete.
> You mean that an isolated partition should only be set on offline CPUs ? That's
> the plan for nohz_full but it may be too late for domain isolation.

Actually I was talking mainly about nohz_full, but we should handle 
changes in HK_TYPE_DOMAIN cpumask the same way.

Cheers,
Longman