From mboxrd@z Thu Jan 1 00:00:00 1970 From: James Bottomley Subject: Re: Rough notes from sys_membarrier() lightning BoF Date: Thu, 21 Sep 2017 10:23:39 -0700 Message-ID: <1506014619.3848.59.camel@HansenPartnership.com> References: <20170917223608.GA14577@linux.vnet.ibm.com> <1264439870.15044.1505931230400.JavaMail.zimbra@efficios.com> <20170921130950.nwfbyiil34psoyua@hirez.programming.kicks-ass.net> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Return-path: In-Reply-To: <20170921130950.nwfbyiil34psoyua@hirez.programming.kicks-ass.net> Sender: linux-kernel-owner@vger.kernel.org To: Peter Zijlstra , Mathieu Desnoyers Cc: Andy Lutomirski , "Paul E. McKenney" , Will Deacon , Alan Stern , Michael Ellerman , linux-kernel , linux-arch , Dave Watson , maged michael List-Id: linux-arch.vger.kernel.org On Thu, 2017-09-21 at 15:09 +0200, Peter Zijlstra wrote: > On Wed, Sep 20, 2017 at 06:13:50PM +0000, Mathieu Desnoyers wrote: > > > > > > > > > Also, can you elaborate on the PPC issue?  PPC appears to track > > > mm_cpumask more or less just like x86.  Is the issue just that > > > this > > > tracking has no implied barriers?  If so, how does TLB flush on > > > ppc > > > work?  It really does seem impressive to me that an architecture > > > can > > > efficiently support munmap() but not an expedited private > > > membarrier. > > > > I'll leave this question to the PPC experts :) > > IIRC PPC does not keep a tight mm_cpumask, it only sets bit, it never > clears bits. The atomic op required to set bits does not imply any > memory barrier on PPC. > > TLB invalidation is a TLBI instruction, it sends TLBI broadcast > packets over the interconnect, it doesn't require IPIs like x86. I believe this to be true for all SMP RISC systems ... it's certainly true for PA-RISC as well.  There are so many RISC coherency issues that the CPUs pretty much have to have a private bus to broadcast and interlock coherency operations.  We have one system that locks up if multiple CPUs have outstanding coherency operations on the private bus, but that's only one annoying CPU (which we manage with a special lock inside the PA-RISC mmu code). James From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bedivere.hansenpartnership.com ([66.63.167.143]:53270 "EHLO bedivere.hansenpartnership.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751737AbdIURXl (ORCPT ); Thu, 21 Sep 2017 13:23:41 -0400 Message-ID: <1506014619.3848.59.camel@HansenPartnership.com> Subject: Re: Rough notes from sys_membarrier() lightning BoF From: James Bottomley Date: Thu, 21 Sep 2017 10:23:39 -0700 In-Reply-To: <20170921130950.nwfbyiil34psoyua@hirez.programming.kicks-ass.net> References: <20170917223608.GA14577@linux.vnet.ibm.com> <1264439870.15044.1505931230400.JavaMail.zimbra@efficios.com> <20170921130950.nwfbyiil34psoyua@hirez.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-arch-owner@vger.kernel.org List-ID: To: Peter Zijlstra , Mathieu Desnoyers Cc: Andy Lutomirski , "Paul E. McKenney" , Will Deacon , Alan Stern , Michael Ellerman , linux-kernel , linux-arch , Dave Watson , maged michael Message-ID: <20170921172339.K3mSKBmpMSFunOqGP1bA4NRxS5lcP7sa3_6GgM4d1L8@z> On Thu, 2017-09-21 at 15:09 +0200, Peter Zijlstra wrote: > On Wed, Sep 20, 2017 at 06:13:50PM +0000, Mathieu Desnoyers wrote: > > > > > > > > > Also, can you elaborate on the PPC issue?  PPC appears to track > > > mm_cpumask more or less just like x86.  Is the issue just that > > > this > > > tracking has no implied barriers?  If so, how does TLB flush on > > > ppc > > > work?  It really does seem impressive to me that an architecture > > > can > > > efficiently support munmap() but not an expedited private > > > membarrier. > > > > I'll leave this question to the PPC experts :) > > IIRC PPC does not keep a tight mm_cpumask, it only sets bit, it never > clears bits. The atomic op required to set bits does not imply any > memory barrier on PPC. > > TLB invalidation is a TLBI instruction, it sends TLBI broadcast > packets over the interconnect, it doesn't require IPIs like x86. I believe this to be true for all SMP RISC systems ... it's certainly true for PA-RISC as well.  There are so many RISC coherency issues that the CPUs pretty much have to have a private bus to broadcast and interlock coherency operations.  We have one system that locks up if multiple CPUs have outstanding coherency operations on the private bus, but that's only one annoying CPU (which we manage with a special lock inside the PA-RISC mmu code). James