From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C03CCDB47E for ; Fri, 13 Oct 2023 18:50:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tesc0t9QIhO2JRbIiULgFVBIJNG3zzvYNoy0ePoSNr8=; b=nOW9OSXOBsLuWU tK03PCbPRHCADxYmSVylGk9kx5iXU8qaZYxo4WWLkAz3yvs+12B1BSrwB6hhbGJgMc/e+DNrzzZ3t rIswFhdo6tNAPwWEtBMtqRu8LMKvnMocapiJ8fTWJDK0tAix9e9XAQ28TPIQ6MkO32Lt+LkNosBJt WOO8BwshwFMti4/JpoJqSs98pEepQNFPVpgLs0N5BXVlrmtoBZ3HAxuytAJ8Q2dZnsi/SAHW+iLnr COC06MO+qKrvZFdPC47NJXMCRWeldh1MNBDuBTRRbbiaNT6ZXd1LrUV229s+01rmdwI/cGehyF/7y Ifl82FD3JD2Bek8aE1ng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qrNEm-0045O4-1y; Fri, 13 Oct 2023 18:50:16 +0000 Received: from [2607:5300:203:b2ee::31e5] (helo=smtpout.efficios.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qrNEh-0045MN-35 for linux-riscv@lists.infradead.org; Fri, 13 Oct 2023 18:50:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1697222992; bh=5lJYynDVEnZ8b1vOKPXx6IKj1N5D/WYZRCy3pdZBk9g=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=kTyupYGOBFQ+FOo49cXR/LNgg1bHI49BVwQhTQtsMIDJPHpsbq2qONbpsiLQr740Q JSUFYoTT4XbkCEzyc6mR/PlHjx8g8LR2fb09kRn+ImPbRdNiY9lOQJtU//zI+r4mxV M3vWkiD64qZ5yyZ9o7Chp6sB8K0ZJ8UEO98OauQG6NcAg7zo3MW4yKNw2rS7I5dgnT OK9I0VOGoVnmkyxQIUVcfeWryh6R5vNNm0ae8lCQf2DX53uXnolJppOF38pBf+4p7R WqV+Bx/bbXlqK+B3S5f/Gls/d9doxwqLuAVKvO+SBU1Cs7GoMeDK8eI+eBK8TodyuE krPkiDFQdBrag== Received: from [172.16.0.134] (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4S6bCh2l5yz1X8t; Fri, 13 Oct 2023 14:49:52 -0400 (EDT) Message-ID: <65e98129-0617-49ca-9802-8e3a46d58d29@efficios.com> Date: Fri, 13 Oct 2023 14:49:56 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] membarrier: riscv: Provide core serializing command Content-Language: en-US To: Palmer Dabbelt , parri.andrea@gmail.com, charlie@rivosinc.com, rehn@rivosinc.com Cc: paulmck@kernel.org, Paul Walmsley , aou@eecs.berkeley.edu, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, mmaas@google.com, hboehm@google.com, striker@us.ibm.com References: From: Mathieu Desnoyers In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231013_115012_083838_7EC39185 X-CRM114-Status: GOOD ( 48.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gMjAyMy0xMC0xMyAxMzoyOSwgUGFsbWVyIERhYmJlbHQgd3JvdGU6Cj4gT24gTW9uLCAwNyBB dWcgMjAyMyAwNjoxOToxOCBQRFQgKC0wNzAwKSwgcGFycmkuYW5kcmVhQGdtYWlsLmNvbSB3cm90 ZToKPj4+IE9uZSBtb3JlIG5vdGV3b3J0aHkgZGV0YWlsOiBpZiBhIHN5c3RlbSBjYWxsIHNpbWls YXIgdG8gQVJNIAo+Pj4gY2FjaGVmbHVzaCgyKSBpcyBpbXBsZW1lbnRlZCBmb3IKPj4+IFJJU0Mt ViwgcGVyaGFwcyBhbiBpb3ZlYyBBQkkgKHNpbWlsYXIgdG8gcmVhZHYoMikvd3JpdGV2KDIpKSB3 b3VsZCBiZSAKPj4+IHJlbGV2YW50IHRvIGhhbmRsZQo+Pj4gYmF0Y2hpbmcgb2YgY2FjaGUgZmx1 c2hpbmcgd2hlbiBhZGRyZXNzIHJhbmdlcyBhcmUgbm90IGNvbnRpZ3VvdXMuIAo+Pj4gTWF5YmUg d2l0aCBhIG5ldyBuYW1lCj4+PiBsaWtlICJjYWNoZWZsdXNodigyKSIsIHNvIGV2ZW50dWFsbHkg b3RoZXIgYXJjaGl0ZWN0dXJlcyBjb3VsZCAKPj4+IGltcGxlbWVudCBpdCBhcyB3ZWxsID8KPj4K Pj4gSSBiZWxpZXZlIHRoYXQncyBhIHNlbnNpYmxlIGlkZWEuwqAgQnV0IHRoZSBSSVNDLVYgbWFp bnRhaW5lcnMgY2FuIHByb3ZpZGUKPj4gYSBtb3JlIHJlbGlhYmxlIGZlZWRiYWNrLgo+IAo+IFNv cnJ5IEkgbWlzc2VkIHRoaXMsIEknbSBzdGlsbCBhIGJpdCBiYWNrbG9nZ2VkIGZyb20gQ09WSUQu wqAgQSBmZXcgb2YgdXMgCj4gd2VyZSBoYXZpbmcgYSBtZWV0aW5nLCBqdXN0IHRvIHRyeSBhbmQg c3VtbWFyaXplIChtYW55IG9mIHRoZXNlIHBvaW50cyAKPiBjYW1lIHVwIGluIHRoZSB0aHJlYWQs IHNvIHNvcnJ5IGZvciByZWhhc2hpbmcgdGhpbmdzKToKPiAKPiBXZSBkb24ndCBoYXZlIGEgZmVu Y2UuaSBpbiB0aGUgc2NoZWR1bGluZyBwYXRoLCBhcyBmZW5jZS5pIGlzIHZlcnkgc2xvdyAKPiBv biBzeXN0ZW1zIHRoYXQgaW1wbGVtZW50IGl0IGJ5IGZsdXNoaW5nIHRoZSBpY2FjaGUuwqAgSW5z dGVhZCB3ZSBoYXZlIGEgCj4gbWVjaGFuaXNtIGZvciBkZWZlcnJpbmcgdGhlIGZlbmNlcyAoc2Vl IGZsdXNoX2ljYWNoZV9kZWZlcnJlZCwgdGhvdWdoIAo+IEknbSBubyBsb25nZXIgc3VyZSB0aGF0 J3MgY29ycmVjdCB3aGljaCBJJ2xsIG1lbnRpb24gYmVsb3cpLsKgIEFzIGEgCj4gcmVzdWx0IHVz ZXJzcGFjZSBjYW4ndCBkbyBhIGZlbmNlLmkgZGlyZWN0bHksIGJ1dCBpbnN0ZWFkIG5lZWRzIHRv IG1ha2UgCj4gYSBzeXNjYWxsL3Zkc29jYWxsIHNvIHRoZSBrZXJuZWwgY2FuIGRvIHRoaXMgYm9v a2tlZXBpbmcuwqAgVGhlcmUncyBzb21lIAo+IHByb3Bvc2FscyBmb3IgSVNBIGV4dGVuc2lvbnMg dGhhdCByZXBsYWNlIGZlbmNlLmksIGJ1dCB0aGV5J3JlIHN0aWxsIFdJUCAKPiBhbmQgdGhlcmUn cyBhIGxvdCBvZiBmZW5jZS5pLW9ubHkgaGFyZHdhcmUgc28gd2UnbGwgaGF2ZSB0byBkZWFsIHdp dGggaXQuCj4gCj4gV2hlbiB3ZSBkaWQgdGhpcyB3ZSBoYWQgYSBmZWVsaW5nIHRoaXMgbWF5IGJl IHN1Yi1vcHRpbWFsIGZvciBzeXN0ZW1zIAo+IHRoYXQgaGF2ZSBmYXN0ZXIgZmVuY2UuaSBpbXBs ZW1lbnRhdGlvbnMgKGllLCBjb2hlcmVudCBpbnN0cnVjdGlvbiAKPiBjYWNoZXMpLCBidXQgbm9i b2R5J3MgZ290dGVuIGFyb3VuZCB0byBkb2luZyB0aGF0IHlldCAtLSBhbmQgbWF5YmUgCj4gdGhl cmUncyBubyBoYXJkd2FyZSB0aGF0IGJlaGF2ZXMgdGhpcyB3YXkuwqAgVGhlIHJvdWdoIHBsYW4g d2FzIGFsb25nIHRoZSAKPiBsaW5lcyBvZiBhZGRpbmcgYSBwcmN0bCgpIHdoZXJlIHVzZXJzcGFj ZSBjYW4gcmVxdWVzdCB0aGUgYWJpbGl0eSB0byAKPiBkaXJlY3RseSBlbWl0IGZlbmNlLmksIHdo aWNoIHdvdWxkIHRoZW4gcmVzdWx0IGluIHRoZSBrZXJuZWwgZWFnZXJseSAKPiBlbWl0dGluZyBm ZW5jZS5pIHdoZW4gc2NoZWR1bGluZy7CoCBTb21lIG9mIHRoZSBKYXZhIHBlb3BsZSBoYXZlIGJl ZW4gCj4gYXNraW5nIGZvciB0aGlzIHNvcnQgb2YgZmVhdHVyZS4KClRoZXJlIGlzIGEgbWVtYmFy cmllcigyKSByZWdpc3RyYXRpb24gc2NoZW1lIHRvIGVuc3VyZSB0aGF0IGNvcmUgc2VyaWFsaXpp bmcKaW5zdHJ1Y3Rpb25zIGFyZSBpc3N1ZWQgaW4gdGhlIHJlbGV2YW50IHNjZW5hcmlvcy4KClNl ZSBNRU1CQVJSSUVSX0NNRF9SRUdJU1RFUl9QUklWQVRFX0VYUEVESVRFRF9TWU5DX0NPUkUgYW5k CkRvY3VtZW50YXRpb24vZmVhdHVyZXMvc2NoZWQvbWVtYmFycmllci1zeW5jLWNvcmUvYXJjaC1z dXBwb3J0LnR4dAoKVGhlIGNvcmUgc2VyaWFsaXppbmcgaW5zdHJ1Y3Rpb25zIGFyZSB0eXBpY2Fs bHkgaXNzdWVkIG9uIHJldHVybiB0byB1c2Vyc3BhY2UKb24gdmFyaW91cyBhcmNoaXRlY3R1cmVz LCBlbHNlIHdlIHJlbHkgb24gc3dpdGNoX21tKCkgZW1pdHRpbmcgdGhlIGZlbmNlIGJldHdlZW4K dXBkYXRlIHRvIHJxLT5jdXJyLT5tbSBhbmQgcmV0dXJuIHRvIHVzZXJzcGFjZS4gQW5kIG9uIHRo ZSByYXJlIGNhc2Ugd2hlcmUKcnEtPmN1cnItPm1tIGlzIGNoYW5nZWQgd2l0aG91dCBpbnZva2lu ZyBzd2l0Y2hfbW0oKSAodHJhbnNpdGlvbiB0byBhCmtlcm5lbCB0aHJlYWQpLCB0aGVuIHdlJ3Zl IGFkZGVkIHRoZSByZWxldmFudCBjb2RlIGluIHRoZSBzY2hlZHVsZXIgdG8gYWRkCmEgY29yZSBz ZXJpYWxpemluZyBpbnN0cnVjdGlvbiBmb3IgcmVnaXN0ZXJlZCBwcm9jZXNzZXMuCgo+IAo+ICBG cm9tIGxvb2tpbmcgYXQgdGhlIG1lbWJhcnJpZXIgYXJjaC9zY2hlZHVsZXIgaG9va3MsIEkgdGhp bmsgd2UgbWlnaHQgCj4gaGF2ZSBhIGJ1ZyBpbiBvdXIgZGVmZXJyZWQgaWNhY2hlIGZsdXNoaW5n IG1lY2hhbmlzbTogc3BlY2lmaWNhbGx5IHdlIAo+IGhvb2sgaW50byBzd2l0Y2hfbW0oKSwgd2hp Y2ggdGhpcyBjb21tZW50IGhhcyBtZSB3b3JyaWVkIGFib3V0Cj4gCj4gIMKgwqDCoMKgwqDCoMKg ICogV2hlbiBzd2l0Y2hpbmcgdGhyb3VnaCBhIGtlcm5lbCB0aHJlYWQsIHRoZSBsb29wIGluCj4g IMKgwqDCoMKgwqDCoMKgICogbWVtYmFycmllcl97cHJpdmF0ZSxnbG9iYWx9X2V4cGVkaXRlZCgp IG1heSBoYXZlIG9ic2VydmVkIHRoYXQKPiAgwqDCoMKgwqDCoMKgwqAgKiBrZXJuZWwgdGhyZWFk IGFuZCBub3QgaXNzdWVkIGFuIElQSS4gSXQgaXMgdGhlcmVmb3JlIHBvc3NpYmxlIHRvCj4gIMKg wqDCoMKgwqDCoMKgICogc2NoZWR1bGUgYmV0d2VlbiB1c2VyLT5rZXJuZWwtPnVzZXIgdGhyZWFk cyB3aXRob3V0IHBhc3NpbmcgCj4gdGhvdWdoCj4gIMKgwqDCoMKgwqDCoMKgICogc3dpdGNoX21t KCkuIE1lbWJhcnJpZXIgcmVxdWlyZXMgYSBiYXJyaWVyIGFmdGVyIHN0b3JpbmcgdG8KPiAgwqDC oMKgwqDCoMKgwqAgKiBycS0+Y3VyciwgYmVmb3JlIHJldHVybmluZyB0byB1c2Vyc3BhY2UsIHNv IHByb3ZpZGUgdGhlbSBoZXJlOgoKSSBndWVzcyB5b3Ugd29uZGVyIGlmIHRoZSBvbl9lYWNoX2Nw dV9tYXNrIGlwaXMgYmFzZWQgb24gdGhlIG1tX2NwdW1hc2sobW0pCmdpdmVzIGFueSBsZXZlbCBv ZiBndWFyYW50ZWUgd2l0aCByZXNwZWN0IHRvIHN3aXRjaF9tbSgpIG1vZGlmeWluZyB0aGlzCm1h c2suIChpbiBmbHVzaF9pY2FjaGVfbW0oKSkKCkluIG1lbWJhcnJpZXIsIHdlIGRlY2lkZWQgYWdh aW5zdCB1c2luZyB0aGUgbW1fY3B1bWFzayBmb3IgdmFyaW91cyByZWFzb25zOgoKLSBBRkFJUiwg b24gc29tZSBhcmNoaXRlY3R1cmVzLCB0aGUgbW1fY3B1bWFzayBpcyBhIHN1cGVyc2V0IG9mIHRo ZSBDUFVzIGFjdHVhbGx5CiAgIHVzZWQgKGl0J3MgbmV2ZXIgY2xlYXJlZCksCi0gdGhlIHBvaW50 IHdoZXJlIHRoZSBtbV9jcHVtYXNrIGlzIHVwZGF0ZWQgd2l0aCByZXNwZWN0IHRvIG1lbW9yeSBi YXJyaWVycwogICBpbiB0aGUgc2NoZWR1bGVyIGNvZGUgaXMgbm90IGFzIGNvbnZlbmllbnQgYXMg aXQgaXMgZm9yIHVwZGF0ZXMgdG8KICAgInJxLT5jdXJyIiBieSB0aGUgc2NoZWR1bGVyLiBUaGlz IG1hdHRlcnMgZm9yIHRoZSBvdGhlciBwdXJwb3NlcyBvZgogICBtZW1iYXJyaWVyKDIpIHdoaWNo IGlzIHRvIGlzc3VlIG1lbW9yeSBiYXJyaWVycyBvbiBhbGwgdGhyZWFkcyBiZWxvbmdpbmcKICAg dG8gYSBwcm9jZXNzLgoKYW5kIGluc3RlYWQgd2UgaXRlcmF0ZSBvbiBlYWNoIG9ubGluZSBjcHVz LCBhbmQgY29tcGFyZSB0aGUgInJxLT5jdXJyLT5tbSIKcG9pbnRlciB0byB0aGUgY3VycmVudCB0 YXNrLiBUaGVuIHdlIG1hZGUgc3VyZSB0byBkb2N1bWVudCBhbGwgdGhlCnJlbGV2YW50IG1lbW9y eSBiYXJyaWVycyBhbmQgY29yZSBzZXJpYWxpemluZyBpbnN0cnVjdGlvbiBleHBlY3RhdGlvbnMK YXJvdW5kIHJxLT5jdXJyLT5tbSB1cGRhdGUgYnkgdGhlIHNjaGVkdWxlci4KCkJ1dCBiYWNrIHRv IHRoZSBSSVNDLVYgZmx1c2hfaWNhY2hlX21tKCkgc2NoZW1lLCBiZWNhdXNlIGl0IGRvZXMgbm90 IHJlbHkKb24gInJxLT5jdXJyIiBhdCBhbGwsIHRoZW4gaXQgYWxsIGRlcGVuZHMgb24gd2hldGhl ciB0aGUgY3B1IGlzIHN0aWxsIGluCnRoZSBtbV9jcHVtYXNrIG9mIHRoZSBtbSB3aGVuIHRoYXQg Y3B1IHRlbXBvcmFyaWx5IHNjaGVkdWxlcyBhIGtlcm5lbCB0aHJlYWQuCkFGQUlSLCBzY2hlZHVs aW5nIGEga2VybmVsIHRocmVhZCBkb2VzIG5vdCB0cmlnZ2VyIGFueSBjYWxsIHRvIHN3aXRjaF9t bQooaXQgYW4gb3B0aW1pemF0aW9uIHdoaWNoIGxlYXZlcyB0aGUgbW0gaW4gcGxhY2Ugd2hpbGUg cnVubmluZyB0aGUga2VybmVsCnRocmVhZCksIHNvIHRoZSBvbl9lYWNoX2NwdV9tYXNrIHVzaW5n IHRoZSBtbV9jcHVtYXNrIHdvdWxkIGJlIE9LLgoKPiAKPiBFdmVuIGlmIHRoZXJlJ3Mgbm90IGEg YnVnIGluIHRoZSBSSVNDLVYgc3R1ZmYsIGl0IHNlZW1zIHRoYXQgd2UndmUgZW5kZWQgCj4gdXAg d2l0aCBwcmV0dHkgc2ltaWxhciBzY2hlbWVzIGhlcmUgYW5kIHdlIGNvdWxkIHJlbW92ZSBzb21l IAo+IGFyY2gtc3BlY2lmaWMgY29kZSBieSBkZS1kdXBsaWNhdGluZyB0aGluZ3MgLS0gSUlSQyB0 aGVyZSB3YXMgbm8gCj4gbWVtYmFycmllciB3aGVuIHdlIGRpZCB0aGUgb3JpZ2luYWwgcG9ydCwg c28gSSB0aGluayB3ZSd2ZSBqdXN0IG1pc3NlZCBhIAo+IGNsZWFudXAgb3Bwb3J0dW5pdHkuCgpB Y3R1YWxseSwgbWVtYmFycmllcigyKSBNRU1CQVJSSUVSX0NNRF9QUklWQVRFX0VYUEVESVRFRF9T WU5DX0NPUkUgYXBwZWFyZWQKaW4gTGludXggNC4xNiwgd2hlcmVhcyB0aGUgaW5pdGlhbCBwb3J0 IG9mIFJJU0MtViBhcHBlYXJlZCBpbiBMaW51eCA0LjE1LgpTbyB0aGlzIGRlLWR1cGxpY2F0aW9u IGhhcyBiZWVuIG1pc3NlZCBieSBhIG5hcnJvdyB3aW5kb3cgOikKClllcywgaXQgd291bGQgbWFr ZSBzZW5zZSB0byBkbyB0aGlzIGRlLWR1cGxpY2F0aW9uLgoKPiAKPiBTbyBJJ2QgcHJvcG9zZSBk b2luZyB0aGUgZm9sbG93aW5nOgo+IAo+ICogUGljayB1cCBhIHBhdGNoIGxpa2UgdGhpcy7CoCBN bWF5YmUgZXhhY3RseSB0aGlzLCBJJ20gZ29pbmcgdG8gZ2l2ZSBpdCAKPiAgwqBhIHByb3BlciBy ZXZpZXcgdG8gbWFrZSBzdXJlLgoKQUZBSVIgdGhpcyBwYXRjaCBpbXBsZW1lbnRzIHN5bmNfY29y ZV9iZWZvcmVfdXNlcm1vZGUgd2hpY2ggZ2V0cyB1c2VkIGJ5Cm1lbWJhcnJpZXJfbW1fc3luY19j b3JlX2JlZm9yZV91c2VybW9kZSgpIHRvIGhhbmRsZSB0aGUgdXRocmVhZC0+a3RocmVhZC0+dXRo cmVhZApjYXNlLiBJdCByZWxpZXMgb24gc3dpdGNoX21tIGlzc3VpbmcgYSBjb3JlIHNlcmlhbGl6 aW5nIGluc3RydWN0aW9uIGFzIHdlbGwuCgpMb29raW5nIGF0IFJJU0MtViBzd2l0Y2hfbW0oKSwg SSBzZWUgdGhhdCBzd2l0Y2hfbW0oKSBjYWxsczoKCiAgIGZsdXNoX2ljYWNoZV9kZWZlcnJlZChu ZXh0LCBjcHUpOwoKd2hpY2ggb25seSBpc3N1ZXMgYSBmZW5jZS5pIGlmIGEgZGVmZXJyZWQgaWNh Y2hlIGZsdXNoIHdhcyByZXF1aXJlZC4gV2UncmUKbWlzc2luZyB0aGUgcGFydCB0aGF0IHNldHMg dGhlIGljYWNoZV9zdGFsZV9tYXNrIGNwdW1hc2sgYml0cyB3aGVuIGEKTUVNQkFSUklFUl9DTURf UFJJVkFURV9FWFBFRElURURfU1lOQ19DT1JFIGlzIGludm9rZWQuCgoKPiAqIFJlbW92ZSB0aGUg UklTQy1WIGltcGxlbWVuYXRpb24gb2YgZGVmZXJyZWQgaWNhY2hlIGZsdXNoZXMgYW5kIGluc3Rl YWQgCj4gIMKganVzdCBjYWxsIGludG8gbWVtYmFycmllci7CoCBXZSBtaWdodCBuZWVkIHRvIGFk ZCBzb21lIG1vcmUgYm9va2tlZXBpbmcgCj4gIMKgaGVyZSwgYnV0IGZyb20gYSBxdWljayBsb29r IGl0IHNlZW1zIG1lbWJhcnJpZXIgaXMgZG9pbmcgcHJldHR5IG11Y2ggCj4gIMKgdGhlIHNhbWUg dGhpbmcuCgpUaGUgb25seSBwYXJ0IHdoZXJlIEkgdGhpbmsgeW91IG1heSB3YW50IHRvIGtlZXAg c29tZSBsZXZlbCBvZiBkZWZlcnJlZAppY2FjaGUgZmx1c2hpbmcgYXMgeW91IGRvIG5vdyBpcyBh cyBmb2xsb3dzOgoKLSB3aGVuIG1lbWJhcnJpZXIgTUVNQkFSUklFUl9DTURfUFJJVkFURV9FWFBF RElURURfU1lOQ19DT1JFIGlzIGludm9rZWQsCiAgIGNhbGwgYSBuZXcgYXJjaGl0ZWN0dXJlIGhv b2sgd2hpY2ggc2V0cyBjcHVtYXNrIGJpdHMgaW4gdGhlIG1tIGNvbnRleHQKICAgdGhhdCB0ZWxs cyB0aGUgbmV4dCBzd2l0Y2hfbW0gb24gZWFjaCBjcHUgdG8gaXNzdWUgZmVuY2UuaSBmb3IgdGhh dCBtbS4KLSBrZWVwIHNvbWV0aGluZyBsaWtlIGZsdXNoX2ljYWNoZV9kZWZlcnJlZCBhcyB5b3Ug aGF2ZSBub3cuCgpPdGhlcndpc2UsIEkgZmVhciB0aGUgb3ZlcmhlYWQgb2YgYSB2ZXJ5IGV4cGVu c2l2ZSBmZW5jZS5pIHdvdWxkIGJlIHRvbwptdWNoIHdoZW4gcHJvY2Vzc2VzIHJlZ2lzdGVyaW5n IHdpdGggTUVNQkFSUklFUl9DTURfUkVHSVNURVJfUFJJVkFURV9FWFBFRElURURfU1lOQ19DT1JF CmFuZCBzdGFydCBkb2luZyBmZW5jZS5pIG9uIGVhY2ggYW5kIGV2ZXJ5IHN3aXRjaF9tbSgpLgoK U28geW91J2QgYmFzaWNhbGx5IHJlbHkgb24gbWVtYmFycmllciB0byBvbmx5IGlzc3VlIElQSXMg dG8gdGhlIENQVXMgd2hpY2ggYXJlCmN1cnJlbnRseSBydW5uaW5nIHRocmVhZHMgYmVsb25naW5n IHRvIHRoZSBtbSwgYW5kIGhhbmRsZSB0aGUgc3dpdGNoX21tIHdpdGgKdGhlIHN5bmNfY29yZV9i ZWZvcmVfdXNlcm1vZGUoKSBmb3IgdXRocmVhZC0+a3RocmVhZC0+dXRocmVhZCBjYXNlLCBhbmQg aW1wbGVtZW50CmEgZGVmZXJyZWQgaWNhY2hlIGZsdXNoIGZvciB0aGUgdHlwaWNhbCBzd2l0Y2hf bW0oKSBjYXNlLgoKCj4gKiBJbXBsZW1lbnQgdGhhdCBwcmN0bCB0aGF0IGFsbG93cyB1c2Vyc3Bh Y2UgdG8gYXNrIGZvciBwZXJtaXNzaW9uIHRvIGRvIAo+ICDCoGRpcmVjdCBmZW5jZS5pIGluc3Ry dWN0aW9ucyAtLSBzb3J0IG9mIGEgZGlmZmVyZW50IHByb2plY3QsIGJ1dCBpZiAKPiAgwqB3ZSdy ZSBnb2luZyB0byBiZSB0ZWFyaW5nIGludG8gYWxsIHRoaXMgY29kZSB3ZSBtaWdodCBhcyB3ZWxs IGRvIGl0IMKgbm93LgoKQnV0IGZlbmNlLmkgd291bGQgb25seSBoYXZlIGVmZmVjdHMgb24gdGhl IGhhcnQgaXQgaXMgYmVpbmcgY2FsbGVkIGZyb20sIHJpZ2h0ID8KV2hhdCBpcyB0aGUgdXNlLWNh c2UgZm9yIGFsbG93aW5nIHVzZXItc3BhY2UgdG8gaXNzdWUgdGhpcyBpbnN0cnVjdGlvbiA/CgpP bmUgbW9yZSB0aGluZzogbWVtYmFycmllcigyKSBzeW5jX2NvcmUgb25seSBpc3N1ZXMgdGhpbmdz IGxpa2UgImZlbmNlLmkiIG9uCnRoZSB2YXJpb3VzIGNvcmVzIGluIHRoZSBzeXN0ZW0gcnVubmlu ZyB0aHJlYWRzIGJlbG9uZ2luZyB0byB0aGUgcHJvY2VzcywgYnV0CmRvZXMgbm90IGludGVuZCB0 byB0YWtlIGNhcmUgb2YgZG9pbmcgYW55IGtpbmQgb2YgY2FjaGUgaW52YWxpZGF0aW9uIHBlciBz ZQooZS5nLiBpbnZhbGlkYXRpbmcgYW4gYWRkcmVzcyByYW5nZSB3b3J0aCBvZiBjYWNoZSkuIE9u IEFSTSwgdGhpcyBpcyBkb25lIGJ5IGEKc2VwYXJhdGUgc3lzdGVtIGNhbGwgKGUuZy4gY2FjaGVm bHVzaCgyKSksIG9yIGNhbiBiZSBkb25lIGJ5IGluc3RydWN0aW9ucwphdmFpbGFibGUgZnJvbSB1 c2Vyc3BhY2UgaW4gc29tZSBjYXNlcy4KCkRvIHlvdSBleHBlY3QgdG8gaGF2ZSBhIG5lZWQgZm9y IGZsdXNoaW5nIG9ubHkgc3BlY2lmaWMgaWNhY2hlIGxpbmVzLCBvciBpcwp0aGUgaW50ZW50IHRv IGFsd2F5cyBmbHVzaCB0aGUgZW50aXJlIGljYWNoZSA/Cgo+IAo+IENoYXJsaWUgaXMgdm9sdW50 ZWVyaW5nIHRvIGRvIHRoZSB3b3JrIGhlcmUsIHNvIGhvcGVmdWxseSB3ZSdsbCBoYXZlIAo+IHNv bWV0aGluZyBtb3ZpbmcgZm9yd2FyZC4KClRoYXQncyBncmVhdCEgSSBob3BlIG15IGZlZWRiYWNr IHdpbGwgaGVscC4KClRoYW5rcywKCk1hdGhpZXUKCj4gCj4+Cj4+IMKgIEFuZHJlYQoKLS0gCk1h dGhpZXUgRGVzbm95ZXJzCkVmZmljaU9TIEluYy4KaHR0cHM6Ly93d3cuZWZmaWNpb3MuY29tCgoK X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtcmlz Y3YgbWFpbGluZyBsaXN0CmxpbnV4LXJpc2N2QGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xp c3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1yaXNjdgo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 248DBCDB47E for ; Fri, 13 Oct 2023 18:50:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231447AbjJMSt7 (ORCPT ); Fri, 13 Oct 2023 14:49:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229679AbjJMSt6 (ORCPT ); Fri, 13 Oct 2023 14:49:58 -0400 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E136283 for ; Fri, 13 Oct 2023 11:49:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1697222992; bh=5lJYynDVEnZ8b1vOKPXx6IKj1N5D/WYZRCy3pdZBk9g=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=kTyupYGOBFQ+FOo49cXR/LNgg1bHI49BVwQhTQtsMIDJPHpsbq2qONbpsiLQr740Q JSUFYoTT4XbkCEzyc6mR/PlHjx8g8LR2fb09kRn+ImPbRdNiY9lOQJtU//zI+r4mxV M3vWkiD64qZ5yyZ9o7Chp6sB8K0ZJ8UEO98OauQG6NcAg7zo3MW4yKNw2rS7I5dgnT OK9I0VOGoVnmkyxQIUVcfeWryh6R5vNNm0ae8lCQf2DX53uXnolJppOF38pBf+4p7R WqV+Bx/bbXlqK+B3S5f/Gls/d9doxwqLuAVKvO+SBU1Cs7GoMeDK8eI+eBK8TodyuE krPkiDFQdBrag== Received: from [172.16.0.134] (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4S6bCh2l5yz1X8t; Fri, 13 Oct 2023 14:49:52 -0400 (EDT) Message-ID: <65e98129-0617-49ca-9802-8e3a46d58d29@efficios.com> Date: Fri, 13 Oct 2023 14:49:56 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] membarrier: riscv: Provide core serializing command Content-Language: en-US To: Palmer Dabbelt , parri.andrea@gmail.com, charlie@rivosinc.com, rehn@rivosinc.com Cc: paulmck@kernel.org, Paul Walmsley , aou@eecs.berkeley.edu, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, mmaas@google.com, hboehm@google.com, striker@us.ibm.com References: From: Mathieu Desnoyers In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2023-10-13 13:29, Palmer Dabbelt wrote: > On Mon, 07 Aug 2023 06:19:18 PDT (-0700), parri.andrea@gmail.com wrote: >>> One more noteworthy detail: if a system call similar to ARM >>> cacheflush(2) is implemented for >>> RISC-V, perhaps an iovec ABI (similar to readv(2)/writev(2)) would be >>> relevant to handle >>> batching of cache flushing when address ranges are not contiguous. >>> Maybe with a new name >>> like "cacheflushv(2)", so eventually other architectures could >>> implement it as well ? >> >> I believe that's a sensible idea.  But the RISC-V maintainers can provide >> a more reliable feedback. > > Sorry I missed this, I'm still a bit backlogged from COVID.  A few of us > were having a meeting, just to try and summarize (many of these points > came up in the thread, so sorry for rehashing things): > > We don't have a fence.i in the scheduling path, as fence.i is very slow > on systems that implement it by flushing the icache.  Instead we have a > mechanism for deferring the fences (see flush_icache_deferred, though > I'm no longer sure that's correct which I'll mention below).  As a > result userspace can't do a fence.i directly, but instead needs to make > a syscall/vdsocall so the kernel can do this bookkeeping.  There's some > proposals for ISA extensions that replace fence.i, but they're still WIP > and there's a lot of fence.i-only hardware so we'll have to deal with it. > > When we did this we had a feeling this may be sub-optimal for systems > that have faster fence.i implementations (ie, coherent instruction > caches), but nobody's gotten around to doing that yet -- and maybe > there's no hardware that behaves this way.  The rough plan was along the > lines of adding a prctl() where userspace can request the ability to > directly emit fence.i, which would then result in the kernel eagerly > emitting fence.i when scheduling.  Some of the Java people have been > asking for this sort of feature. There is a membarrier(2) registration scheme to ensure that core serializing instructions are issued in the relevant scenarios. See MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE and Documentation/features/sched/membarrier-sync-core/arch-support.txt The core serializing instructions are typically issued on return to userspace on various architectures, else we rely on switch_mm() emitting the fence between update to rq->curr->mm and return to userspace. And on the rare case where rq->curr->mm is changed without invoking switch_mm() (transition to a kernel thread), then we've added the relevant code in the scheduler to add a core serializing instruction for registered processes. > > From looking at the membarrier arch/scheduler hooks, I think we might > have a bug in our deferred icache flushing mechanism: specifically we > hook into switch_mm(), which this comment has me worried about > >         * When switching through a kernel thread, the loop in >         * membarrier_{private,global}_expedited() may have observed that >         * kernel thread and not issued an IPI. It is therefore possible to >         * schedule between user->kernel->user threads without passing > though >         * switch_mm(). Membarrier requires a barrier after storing to >         * rq->curr, before returning to userspace, so provide them here: I guess you wonder if the on_each_cpu_mask ipis based on the mm_cpumask(mm) gives any level of guarantee with respect to switch_mm() modifying this mask. (in flush_icache_mm()) In membarrier, we decided against using the mm_cpumask for various reasons: - AFAIR, on some architectures, the mm_cpumask is a superset of the CPUs actually used (it's never cleared), - the point where the mm_cpumask is updated with respect to memory barriers in the scheduler code is not as convenient as it is for updates to "rq->curr" by the scheduler. This matters for the other purposes of membarrier(2) which is to issue memory barriers on all threads belonging to a process. and instead we iterate on each online cpus, and compare the "rq->curr->mm" pointer to the current task. Then we made sure to document all the relevant memory barriers and core serializing instruction expectations around rq->curr->mm update by the scheduler. But back to the RISC-V flush_icache_mm() scheme, because it does not rely on "rq->curr" at all, then it all depends on whether the cpu is still in the mm_cpumask of the mm when that cpu temporarily schedules a kernel thread. AFAIR, scheduling a kernel thread does not trigger any call to switch_mm (it an optimization which leaves the mm in place while running the kernel thread), so the on_each_cpu_mask using the mm_cpumask would be OK. > > Even if there's not a bug in the RISC-V stuff, it seems that we've ended > up with pretty similar schemes here and we could remove some > arch-specific code by de-duplicating things -- IIRC there was no > membarrier when we did the original port, so I think we've just missed a > cleanup opportunity. Actually, membarrier(2) MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE appeared in Linux 4.16, whereas the initial port of RISC-V appeared in Linux 4.15. So this de-duplication has been missed by a narrow window :) Yes, it would make sense to do this de-duplication. > > So I'd propose doing the following: > > * Pick up a patch like this.  Mmaybe exactly this, I'm going to give it >  a proper review to make sure. AFAIR this patch implements sync_core_before_usermode which gets used by membarrier_mm_sync_core_before_usermode() to handle the uthread->kthread->uthread case. It relies on switch_mm issuing a core serializing instruction as well. Looking at RISC-V switch_mm(), I see that switch_mm() calls: flush_icache_deferred(next, cpu); which only issues a fence.i if a deferred icache flush was required. We're missing the part that sets the icache_stale_mask cpumask bits when a MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE is invoked. > * Remove the RISC-V implemenation of deferred icache flushes and instead >  just call into membarrier.  We might need to add some more bookkeeping >  here, but from a quick look it seems membarrier is doing pretty much >  the same thing. The only part where I think you may want to keep some level of deferred icache flushing as you do now is as follows: - when membarrier MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE is invoked, call a new architecture hook which sets cpumask bits in the mm context that tells the next switch_mm on each cpu to issue fence.i for that mm. - keep something like flush_icache_deferred as you have now. Otherwise, I fear the overhead of a very expensive fence.i would be too much when processes registering with MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE and start doing fence.i on each and every switch_mm(). So you'd basically rely on membarrier to only issue IPIs to the CPUs which are currently running threads belonging to the mm, and handle the switch_mm with the sync_core_before_usermode() for uthread->kthread->uthread case, and implement a deferred icache flush for the typical switch_mm() case. > * Implement that prctl that allows userspace to ask for permission to do >  direct fence.i instructions -- sort of a different project, but if >  we're going to be tearing into all this code we might as well do it  now. But fence.i would only have effects on the hart it is being called from, right ? What is the use-case for allowing user-space to issue this instruction ? One more thing: membarrier(2) sync_core only issues things like "fence.i" on the various cores in the system running threads belonging to the process, but does not intend to take care of doing any kind of cache invalidation per se (e.g. invalidating an address range worth of cache). On ARM, this is done by a separate system call (e.g. cacheflush(2)), or can be done by instructions available from userspace in some cases. Do you expect to have a need for flushing only specific icache lines, or is the intent to always flush the entire icache ? > > Charlie is volunteering to do the work here, so hopefully we'll have > something moving forward. That's great! I hope my feedback will help. Thanks, Mathieu > >> >>   Andrea -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com