From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BF29C48BE5 for ; Thu, 17 Jun 2021 14:16:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 818DF6135C for ; Thu, 17 Jun 2021 14:16:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 818DF6135C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=efficios.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 14E4F6B006E; Thu, 17 Jun 2021 10:16:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 132C76B0071; Thu, 17 Jun 2021 10:16:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBA326B0072; Thu, 17 Jun 2021 10:16:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0008.hostedemail.com [216.40.44.8]) by kanga.kvack.org (Postfix) with ESMTP id BAD2E6B006E for ; Thu, 17 Jun 2021 10:16:20 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3D7AD181AEF0B for ; Thu, 17 Jun 2021 14:16:20 +0000 (UTC) X-FDA: 78263415720.07.68C9A9C Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 1D092C00F790 for ; Thu, 17 Jun 2021 14:16:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 98BF933AEF8; Thu, 17 Jun 2021 10:16:14 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id NQ5nwdb_CSGL; Thu, 17 Jun 2021 10:16:13 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 940CC33AEF7; Thu, 17 Jun 2021 10:16:13 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 940CC33AEF7 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1623939373; bh=PPPq0X6QUD3zkAKDaJYB0a4dhhEK2iuXQUmjCr94ufo=; h=Date:From:To:Message-ID:MIME-Version; b=Tp6w/FWv/15PBsm+sZdeYHS2fGUFFDxva9d8Q43DW/PFsLMkGpEoAhtU5H4ys8LTy wdQds1SYsU0tW0Sv3/Esbm4SptIGscsIXbQfaj2lkEY42W5H2LxA0NI7hSOg/Tk0R8 RuyJatVGwuM+3gSDUWA+2IX7j9/BHtwfexP+veVKe9KhQBLes6fLpcEv38p6y/8c7X sSRx5na+oQl3zvjKAoY7rELzuLwdK4qDfzcvfMwlbVoe8MyoBeQmM+bdg0u5Wje6Xy vd5EhWDcJXMMF0rFANM29DBE7EfdJMQmKodEQJyejT3EU2NRBEecWnLUkXf1jQqHJM iB+i559Gejy1g== X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id eSxuNMgKD58p; Thu, 17 Jun 2021 10:16:13 -0400 (EDT) Received: from localhost (unknown [192.222.236.144]) by mail.efficios.com (Postfix) with ESMTPSA id 3B0FD33B0B7; Thu, 17 Jun 2021 10:16:13 -0400 (EDT) Date: Thu, 17 Jun 2021 10:16:12 -0400 From: Mathieu Desnoyers To: Mark Rutland Cc: Andy Lutomirski , "Russell King (Oracle)" , the arch/x86 maintainers , Dave Hansen , Linux Kernel Mailing List , linux-mm@kvack.org, Andrew Morton , Nicholas Piggin , "Peter Zijlstra (Intel)" , linux-arm-kernel@lists.infradead.org Subject: Re: [PATCH 7/8] membarrier: Remove arm (32) support for SYNC_CORE Message-ID: <20210617141612.GA18005@localhost> References: <2142129092ff9aa00e600c42a26c4015b7f5ceec.1623813516.git.luto@kernel.org> <20210617103524.GA82133@C02TD0UTHF1T.local> <20210617112305.GK22278@shell.armlinux.org.uk> <20210617113349.GB82133@C02TD0UTHF1T.local> <394219d4-36a6-4e7f-a03c-8590551b099a@www.fastmail.com> <20210617135133.GA86101@C02TD0UTHF1T.local> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20210617135133.GA86101@C02TD0UTHF1T.local> User-Agent: Mutt/1.10.1 (2018-07-13) Content-Transfer-Encoding: quoted-printable Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=efficios.com header.s=default header.b="Tp6w/FWv"; dmarc=pass (policy=none) header.from=efficios.com; spf=pass (imf22.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 167.114.26.124 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com X-Rspamd-Server: rspam02 X-Stat-Signature: ua4kce7mki93sfgg9owfd4qk47uayrmu X-Rspamd-Queue-Id: 1D092C00F790 X-HE-Tag: 1623939364-941361 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 17-Jun-2021 02:51:33 PM, Mark Rutland wrote: > On Thu, Jun 17, 2021 at 06:41:41AM -0700, Andy Lutomirski wrote: > >=20 > >=20 > > On Thu, Jun 17, 2021, at 4:33 AM, Mark Rutland wrote: > > > On Thu, Jun 17, 2021 at 12:23:05PM +0100, Russell King (Oracle) wro= te: > > > > On Thu, Jun 17, 2021 at 11:40:46AM +0100, Mark Rutland wrote: > > > > > On Tue, Jun 15, 2021 at 08:21:12PM -0700, Andy Lutomirski wrote= : > > > > > > On arm32, the only way to safely flush icache from usermode i= s to call > > > > > > cacheflush(2). This also handles any required pipeline flush= es, so > > > > > > membarrier's SYNC_CORE feature is useless on arm. Remove it. > > > > >=20 > > > > > Unfortunately, it's a bit more complicated than that, and these= days > > > > > SYNC_CORE is equally necessary on arm as on arm64. This is some= thing > > > > > that changed in the architecture over time, but since ARMv7 we = generally > > > > > need both the cache maintenance *and* a context synchronization= event > > > > > (the latter must occur on the CPU which will execute the instru= ctions). > > > > >=20 > > > > > If you look at the latest ARMv7-AR manual (ARM DDI 406C.d), sec= tion > > > > > A3.5.4 "Concurrent modification and execution of instructions" = covers > > > > > this. That manual can be found at: > > > > >=20 > > > > > https://developer.arm.com/documentation/ddi0406/latest/ > > > >=20 > > > > Looking at that, sys_cacheflush() meets this. The manual details = a > > > > series of cache maintenance calls in "step 1" that the modifying = thread > > > > must issue - this is exactly what sys_cacheflush() does. The same= is > > > > true for ARMv6, except the "ISB" terminology is replaced by a > > > > "PrefetchFlush" terminology. (I checked DDI0100I). > > > >=20 > > > > "step 2" requires an ISB on the "other CPU" prior to executing th= at > > > > code. As I understand it, in ARMv7, userspace can issue an ISB it= self. > > > >=20 > > > > For ARMv6K, it doesn't have ISB, but instead has a CP15 instructi= on > > > > for this that isn't availble to userspace. This is where we come = to > > > > the situation about ARM 11MPCore, and whether we continue to supp= ort > > > > it or not. > > > >=20 > > > > So, I think we're completely fine with ARMv7 under 32-bit ARM ker= nels > > > > as userspace has everything that's required. ARMv6K is a differen= t > > > > matter as we've already identified for several reasons. > > >=20 > > > Sure, and I agree we should not change cacheflush(). > > >=20 > > > The point of membarrier(SYNC_CORE) is that you can move the cost of= that > > > ISB out of the fast-path in the executing thread(s) and into the > > > slow-path on the thread which generated the code. > > >=20 > > > So e.g. rather than an executing thread always having to do: > > >=20 > > > LDR , [] > > > ISB // in case funcptr was just updated > > > BLR > > >=20 > > > ... you have the thread generating the code use membarrier(SYNC_COR= E) > > > prior to plublishing the funcptr, and the fast-path on all the exec= uting > > > threads can be: > > >=20 > > > LDR [] > > > BLR > > >=20 > > > ... and thus I think we still want membarrier(SYNC_CORE) so that pe= ople > > > can do this, even if there are other means to achieve the same > > > functionality. > >=20 > > I had the impression that sys_cacheflush() did that. Am I wrong? >=20 > Currently sys_cacheflush() doesn't do this, and IIUC it has never done > remote context synchronization even for architectures that need that > (e.g. x86 requiring a serializing instruction). >=20 > > In any event, I=E2=80=99m even more convinced that no new SYNC_CORE a= rches > > should be added. We need a new API that just does the right thing.=20 >=20 > My intuition is the other way around, and that this is a gnereally > useful thing for architectures that require context synchronization. >=20 > It's not clear to me what "the right thing" would mean specifically, an= d > on architectures with userspace cache maintenance JITs can usually do > the most optimal maintenance, and only need help for the context > synchronization. If I can attempt to summarize the current situation for ARMv7: - In addition to the cache flushing on the core doing the code update, the architecture requires every core to perform a context synchronizing instruction before executing the updated code. - sys_cacheflush() don't do this core sync on every core. It also takes a single address range as parameter. - ARM, ARM64, powerpc, powerpc64, x86, x86-64 all currently handle the context synchronization requirement for updating user-space code on SMP with sys_membarrier SYNC_CORE. It's not, however, meant to replace explicit cache flushing operations if those are needed. So removing membarrier SYNC_CORE from ARM would be a step backward here. On ARMv7, the SYNC_CORE is needed _in addition_ to sys_cacheflush. Adding a sync-core operation at the end of sys_cacheflush would be inefficient for common GC use-cases where a rather large set of address ranges are invalidated in one go: for this, we either want the GC to: - Invoke sys_cacheflush for each targeted range, and then issue a single sys_membarrier SYNC_CORE, or - Implement a new "sys_cacheflush_iov" which takes an iovec input. There I see that it could indeed invalidate all relevant cache lines *and* issue the SYNC_CORE at the end. But shoehorning the SYNC_CORE in the pre-existing sys_cacheflush after the fact seems like a bad idea. Thanks, Mathieu --=20 Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com