From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: [PATCH 1/3] x86: avoid flush IPI when possible Date: Wed, 10 Feb 2016 17:51:25 +0000 Message-ID: <56BB789D.9080701@citrix.com> References: <56BB3DE902000078000D087A@prv-mh.provo.novell.com> <56BB419B02000078000D08A9@prv-mh.provo.novell.com> <56BB5077.8090704@citrix.com> <56BB673A02000078000D0A7D@prv-mh.provo.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aTYvJ-0001Dm-Dq for xen-devel@lists.xenproject.org; Wed, 10 Feb 2016 17:51:29 +0000 In-Reply-To: <56BB673A02000078000D0A7D@prv-mh.provo.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: xen-devel , Keir Fraser List-Id: xen-devel@lists.xenproject.org On 10/02/16 15:37, Jan Beulich wrote: >>>> On 10.02.16 at 16:00, wrote: >> On 10/02/16 12:56, Jan Beulich wrote: >>> Since CLFLUSH, other than WBINVD, is a coherency domain wide flush, >> I can't parse this sentence. > Should have been "..., is a cache coherency domain wide flush, ..." - > does it read any better then? I believe, given the code in the patch, your intent is "if we WBINVD, we don't need to IPI other cores cache flushing reasons". However, given your comment below... > >> CLFUSH states "Invalidates from every level of the cache hierarchy in >> the cache coherence domain" >> >> WBINVD however states "The instruction then issues a special-function >> bus cycle that directs external caches to also write back modified data >> and another bus cycle to indicate that the external caches should be >> invalidated." >> >> I think we need input from Intel and AMD here as to the behaviour and >> terminology here, and in particular, where the coherency domain >> boundaries are. All CPUs, even across multiple sockets, see coherent >> caching, but it is unclear whether this qualifies them to be in the same >> cache coherency domain per the instruction spec. > Linux already doing what this patch switches us to, I'm not sure > we need much extra input. > >> In particular, given the architecture of 8-socket systems and 45MB of >> RAM in L3 caches, does wbinvd seriously drain all caches everywhere? > Not everywhere, just on the local socket (assuming there's no external > cache). If this is true, then it is clearly not safe to omit the IPIs. > >> Causing 45MB of data to move to remote memory controllers all at once >> would cause a massive system stall. > That's why it takes (as we know) so long. See the figure in SDM Vol 3 > section "Invalidating Caches and TLBs". I presume you mean Figure 2-10. WBINVD Invalidation of Shared and Non-Shared Cache Hierarchy? This quite clearly shows that WBINVD will not invalidate or write back the L1 caches for other cores in the same processor. Have I misunderstood the logic for choosing when to omit the IPIs? ~Andrew