Re: [PATCH 16/23] sched: Use lightweight hazard pointers to grab lazy mms

[Nadav Amit <nadav.amit@gmail.com>]

> On Jan 9, 2022, at 11:37 AM, Rik van Riel <riel@surriel.com> wrote:
> 
> On Sun, 2022-01-09 at 11:34 -0800, Nadav Amit wrote:
>> 
>> 
>>> On Jan 9, 2022, at 11:22 AM, Rik van Riel <riel@surriel.com> wrote:
>>> 
>>> On Sun, 2022-01-09 at 00:49 -0800, Nadav Amit wrote:
>>>> 
>>>> It is possible for instance to get rid of is_lazy, keep the CPU
>>>> on mm_cpumask when switching to kernel thread, and then if/when
>>>> an IPI is received remove it from cpumask to avoid further
>>>> unnecessary TLB shootdown IPIs.
>>>> 
>>>> I do not know whether it is a pure win, but there is a tradeoff.
>>> 
>>> That's not a win at all. It is what we used to have before
>>> the lazy mm stuff was re-introduced, and it led to quite a
>>> few unnecessary IPIs, which are especially slow on virtual
>>> machines, where the target CPU may not be running.
>> 
>> You make a good point about VMs.
>> 
>> IIUC Lazy-TLB serves several goals:
>> 
>> 1. Avoid arch address-space switch to save switching time and
>>    TLB misses.
>> 2. Prevent unnecessary IPIs while kernel threads run.
>> 3. Avoid cache-contention on mm_cpumask when switching to a kernel
>>    thread.
>> 
>> Your concern is with (2), and this one is easy to keep regardless
>> of the rest.
>> 
>> I am not sure that (3) is actually helpful, since it might lead
>> to more cache activity than without lazy-TLB, but that is somewhat
>> orthogonal to everything else.
> 
> I have seen problems with (3) in practice, too.
> 
> For many workloads, context switching is much, much more
> of a hot path than TLB shootdowns, which are relatively
> infrequent by comparison.
> 
> Basically ASID took away only the first concern from your
> list above, not the other two.

I agree, but the point I was trying to make is that you can keep lazy
TLB for (2) and (3), but still switch the address-space. If you
already accept PTI, then the 600 cycles or so of switching the
address space back and forth, which should occur more infrequently
than those on syscalls/exceptions, are not that painful.

You can also make a case that it is “safer” to switch the address
space, although SMAP/SMEP protection provides similar properties.