Re: [PATCH 1/3] x86: avoid flush IPI when possible

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 10/02/16 15:37, Jan Beulich wrote:
>>>> On 10.02.16 at 16:00, <andrew.cooper3@citrix.com> wrote:
>> On 10/02/16 12:56, Jan Beulich wrote:
>>> Since CLFLUSH, other than WBINVD, is a coherency domain wide flush,
>> I can't parse this sentence.
> Should have been "..., is a cache coherency domain wide flush, ..." -
> does it read any better then?

I believe, given the code in the patch, your intent is "if we WBINVD, we
don't need to IPI other cores cache flushing reasons".

However, given your comment below...

>
>> CLFUSH states "Invalidates from every level of the cache hierarchy in
>> the cache coherence domain"
>>
>> WBINVD however states "The instruction then issues a special-function
>> bus cycle that directs external caches to also write back modified data
>> and another bus cycle to indicate that the external caches should be
>> invalidated."
>>
>> I think we need input from Intel and AMD here as to the behaviour and
>> terminology here, and in particular, where the coherency domain
>> boundaries are.  All CPUs, even across multiple sockets, see coherent
>> caching, but it is unclear whether this qualifies them to be in the same
>> cache coherency domain per the instruction spec.
> Linux already doing what this patch switches us to, I'm not sure
> we need much extra input.
>
>> In particular, given the architecture of 8-socket systems and 45MB of
>> RAM in L3 caches, does wbinvd seriously drain all caches everywhere? 
> Not everywhere, just on the local socket (assuming there's no external
> cache).

If this is true, then it is clearly not safe to omit the IPIs.

>
>> Causing 45MB of data to move to remote memory controllers all at once
>> would cause a massive system stall.
> That's why it takes (as we know) so long. See the figure in SDM Vol 3
> section "Invalidating Caches and TLBs".

I presume you mean Figure 2-10. WBINVD Invalidation of Shared and
Non-Shared Cache Hierarchy?

This quite clearly shows that WBINVD will not invalidate or write back
the L1 caches for other cores in the same processor.

Have I misunderstood the logic for choosing when to omit the IPIs?

~Andrew