From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Jones Date: Fri, 19 Apr 2024 15:53:22 +0200 Subject: [PATCH v2 0/6] riscv: Apply Zawrs when available Message-ID: <20240419135321.70781-8-ajones@ventanamicro.com> List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Zawrs provides two instructions (wrs.nto and wrs.sto), where both are meant to allow the hart to enter a low-power state while waiting on a store to a memory location. The instructions also both wait an implementation-defined "short" duration (unless the implementation terminates the stall for another reason). The difference is that while wrs.sto will terminate when the duration elapses, wrs.nto, depending on configuration, will either just keep waiting or an ILL exception will be raised. Linux will use wrs.nto, so if platforms have an implementation which falls in the "just keep waiting" category, then it should _not_ advertise Zawrs in the hardware description. Text to that effect has been added to the Zawrs DT definition. Like wfi (and with the same {m,h}status bits to configure it), when wrs.nto is configured to raise exceptions it's expected that the higher privilege level will see the instruction was a wait instruction, do something, and then resume execution following the instruction. For example, KVM does configure exceptions for wfi (hstatus.VTW=1) and therefore also for wrs.nto. KVM does this for wfi since it's better to allow other tasks to be scheduled while a VCPU waits for an interrupt. For waits such as those where wrs.nto/sto would be used, which are typically locks, it is also a good idea for KVM to be involved, as it can attempt to schedule the lock holding VCPU. This series starts with Christoph's addition of riscv smp_cond_load* functions which apply wrs.sto when available. That patch has been reworked to use wrs.nto and to use the same approach as Arm for the wait loop, since we can't have arbitrary C code between the load- reserved and the wrs. Then, hwprobe support is added (since the instructions are also usable from usermode), and finally KVM is taught about wrs.nto, allowing guests to see and use the Zawrs extension. We still don't have test results from hardware, and it's not possible to prove that using Zawrs is a win when testing on QEMU, not even when oversubscribing VCPUs to guests. However, it is possible to use KVM selftests to force a scenario where we can prove Zawrs does its job and does it well. [4] is a test which does this and, on my machine, without Zawrs it takes 16 seconds to complete and with Zawrs it takes 0.25 seconds. This series is also available here [1]. In order to use QEMU for testing a build with [2] is needed. In order to enable guests to use Zawrs with KVM using kvmtool, the branch at [3] may be used. [1] https://github.com/jones-drew/linux/commits/riscv/zawrs-v2/ [2] https://lore.kernel.org/all/20240312152901.512001-2-ajones at ventanamicro.com/ [3] https://github.com/jones-drew/kvmtool/commits/riscv/zawrs/ [4] https://github.com/jones-drew/linux/commit/9311702bcd118bdbfa8b9be4a8ec355c40559499 Thanks, drew v2: - Added DT bindings patch with additional Linux specifications due to wrs.nto potentially never terminating, as suggested by Palmer - Added patch to share pause insn definition - Rework main Zawrs support patch to use Arm approach (which is also the approach that Andrea Parri suggested) - Dropped the riscv implementation of smp_cond_load_acquire(). afaict, the generic implementation, which will use the riscv implementation of smp_cond_load_relaxed() is sufficient for riscv. - The rework was large enough (IMO) to drop Heiko's s-o-b and to add myself as a co-developer Andrew Jones (5): riscv: Provide a definition for 'pause' dt-bindings: riscv: Add Zawrs ISA extension description riscv: hwprobe: export Zawrs ISA extension KVM: riscv: Support guest wrs.nto KVM: riscv: selftests: Add Zawrs extension to get-reg-list test Christoph M?llner (1): riscv: Add Zawrs support for spinlocks Documentation/arch/riscv/hwprobe.rst | 4 ++ .../devicetree/bindings/riscv/extensions.yaml | 12 +++++ arch/riscv/Kconfig | 20 +++++--- arch/riscv/Makefile | 3 -- arch/riscv/include/asm/barrier.h | 45 ++++++++++------ arch/riscv/include/asm/cmpxchg.h | 51 +++++++++++++++++++ arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/insn-def.h | 4 ++ arch/riscv/include/asm/kvm_host.h | 1 + arch/riscv/include/asm/vdso/processor.h | 8 +-- arch/riscv/include/uapi/asm/hwprobe.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + arch/riscv/kernel/sys_hwprobe.c | 1 + arch/riscv/kvm/vcpu.c | 1 + arch/riscv/kvm/vcpu_insn.c | 15 ++++++ arch/riscv/kvm/vcpu_onereg.c | 2 + .../selftests/kvm/riscv/get-reg-list.c | 4 ++ 18 files changed, 144 insertions(+), 31 deletions(-) -- 2.44.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57365C04FFE for ; Fri, 19 Apr 2024 13:53:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=TU20SWNImduPs60Kv6JKU/BCPWw87XMrfkuxCpFujqo=; b=kZPMP57VYGLOor H9VGb6Hx5edjE2U7N26HnPQEcDpJWfy8AFY8e7w+rLhilzW/J/7M1Oo/5V8CqFFHPMUVowopsvZ5H IAvuv4L6515N1VPcK4jbVbxDO7AsInrfWnkRXAjWFTIVGs4X352Lk8kYZIBqJItzzSRqAdALnGDb+ SAS5VN14s3HMuzLE/buujk9sligoi+x7m3MLcFFrRwSiHyHz7GEsTIAiJUHISSHq3LJd1cXoGKU4B qKgXflKEGXJY/tybhezBUhxdRUMYUIgYS+riT8TiQ3WMqD94gPyRIfAmU7lqCZ7vZmAW/KfmnZNkF tT89aIXuho89x2CUd7wg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxogD-00000005pD3-3LYe; Fri, 19 Apr 2024 13:53:29 +0000 Received: from mail-lf1-x133.google.com ([2a00:1450:4864:20::133]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxogA-00000005pB8-2mO2 for linux-riscv@lists.infradead.org; Fri, 19 Apr 2024 13:53:28 +0000 Received: by mail-lf1-x133.google.com with SMTP id 2adb3069b0e04-51967f75729so2401719e87.0 for ; Fri, 19 Apr 2024 06:53:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; t=1713534803; x=1714139603; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Vygojg6kK2t93LE6Jccui0sWTwt0Dla9UnPOwn3MABA=; b=EqpVD4341UgPm8HoXDFa6bk+/nohjOIe8tGzMQRXH7zL47NHFv4w+ZrzlHWfRZST48 g01Cl7Gq24zv4/u4H+S5bENTQZJ4Pgl9SMwZ0Yyq0gxdFHuBTXrBS8JNbst3Ule/E7ls VA2cv8CvECwnAOLlEXB6JWNgGkja59+T88UCcfgcSwnmK1cUZUCm9T5LAQbUIsySWrPm pyE2WZD8kD8QH+Tnvc5sHnvroIt9CY8iS+Mg/fV8miOipt82yzVQICuPwtgqfunNggPh VvmuQrkGU7OFuduRUZBw4BfUa3j1vyBX4u380yGrJYUEUPkEGt0h7v+yxY2Y/ymFrhaD Dujg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713534803; x=1714139603; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Vygojg6kK2t93LE6Jccui0sWTwt0Dla9UnPOwn3MABA=; b=jCFdYjq8epa0vbm/7fuR1XAU9Jb4d1dTUhbxY33A89NHkAIP3Ki/6g8C5CuSp/lUnP Qv+k35KPoPn3/xSHaN/HILPCXIgqGnlxjvxXFhncwBPTbFa02cwBNTN2g40ZdewCRNug 60S0RXENICXQnVINgZm7rMtH8KYJx853z5r2ZgUY1rr/g8BHX/dUBBRGH9v41VH+k6DF kjqLOP2vrsASKjYpxtIIvjCy/xI3pbBWAeVFo9cplDdsrD5tmj1yluDh+BJVLcr8Xy+1 PV0sCtyLJ+rVY1FkPuO2NyMbhHL0MMvHONuH0P7Kf2wr9a0rQ5RECJwMbhFQT6ir6n65 JJ3w== X-Gm-Message-State: AOJu0Ywo8b9uZQi7blNnygBPDJoJ7cqfNctLho5+nLb+7E5Qp2Foin1B h11HH3OKl6hiT0U5AjyBFwDDzF32y9xIIq/1ZAE8i7jJ0CsgABZuaXYCnmD4FiI0tdiolpHBRrK FTDk= X-Google-Smtp-Source: AGHT+IFFzEV9Mdl8G4cryaHfYH+SN86dJZ7B9fY5aMYsjcaqBCQ0YnmeqoSttiNY9Cm5AgowfnUBMg== X-Received: by 2002:ac2:5388:0:b0:519:14a2:72da with SMTP id g8-20020ac25388000000b0051914a272damr1453236lfh.67.1713534803347; Fri, 19 Apr 2024 06:53:23 -0700 (PDT) Received: from localhost (2001-1ae9-1c2-4c00-20f-c6b4-1e57-7965.ip6.tmcz.cz. [2001:1ae9:1c2:4c00:20f:c6b4:1e57:7965]) by smtp.gmail.com with ESMTPSA id lj26-20020a170906f9da00b00a5254ec731esm2242648ejb.176.2024.04.19.06.53.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Apr 2024 06:53:22 -0700 (PDT) From: Andrew Jones To: linux-riscv@lists.infradead.org, kvm-riscv@lists.infradead.org, devicetree@vger.kernel.org Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, conor.dooley@microchip.com, anup@brainfault.org, atishp@atishpatra.org, robh@kernel.org, krzysztof.kozlowski+dt@linaro.org, conor+dt@kernel.org, christoph.muellner@vrull.eu, heiko@sntech.de, charlie@rivosinc.com, David.Laight@ACULAB.COM, parri.andrea@gmail.com, luxu.kernel@bytedance.com Subject: [PATCH v2 0/6] riscv: Apply Zawrs when available Date: Fri, 19 Apr 2024 15:53:22 +0200 Message-ID: <20240419135321.70781-8-ajones@ventanamicro.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240419_065326_899309_697A6313 X-CRM114-Status: GOOD ( 22.39 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org WmF3cnMgcHJvdmlkZXMgdHdvIGluc3RydWN0aW9ucyAod3JzLm50byBhbmQgd3JzLnN0byksIHdo ZXJlIGJvdGggYXJlCm1lYW50IHRvIGFsbG93IHRoZSBoYXJ0IHRvIGVudGVyIGEgbG93LXBvd2Vy IHN0YXRlIHdoaWxlIHdhaXRpbmcgb24gYQpzdG9yZSB0byBhIG1lbW9yeSBsb2NhdGlvbi4gVGhl IGluc3RydWN0aW9ucyBhbHNvIGJvdGggd2FpdCBhbgppbXBsZW1lbnRhdGlvbi1kZWZpbmVkICJz aG9ydCIgZHVyYXRpb24gKHVubGVzcyB0aGUgaW1wbGVtZW50YXRpb24KdGVybWluYXRlcyB0aGUg c3RhbGwgZm9yIGFub3RoZXIgcmVhc29uKS4gVGhlIGRpZmZlcmVuY2UgaXMgdGhhdCB3aGlsZQp3 cnMuc3RvIHdpbGwgdGVybWluYXRlIHdoZW4gdGhlIGR1cmF0aW9uIGVsYXBzZXMsIHdycy5udG8s IGRlcGVuZGluZyBvbgpjb25maWd1cmF0aW9uLCB3aWxsIGVpdGhlciBqdXN0IGtlZXAgd2FpdGlu ZyBvciBhbiBJTEwgZXhjZXB0aW9uIHdpbGwgYmUKcmFpc2VkLiBMaW51eCB3aWxsIHVzZSB3cnMu bnRvLCBzbyBpZiBwbGF0Zm9ybXMgaGF2ZSBhbiBpbXBsZW1lbnRhdGlvbgp3aGljaCBmYWxscyBp biB0aGUgImp1c3Qga2VlcCB3YWl0aW5nIiBjYXRlZ29yeSwgdGhlbiBpdCBzaG91bGQgX25vdF8K YWR2ZXJ0aXNlIFphd3JzIGluIHRoZSBoYXJkd2FyZSBkZXNjcmlwdGlvbi4gVGV4dCB0byB0aGF0 IGVmZmVjdCBoYXMKYmVlbiBhZGRlZCB0byB0aGUgWmF3cnMgRFQgZGVmaW5pdGlvbi4KCkxpa2Ug d2ZpIChhbmQgd2l0aCB0aGUgc2FtZSB7bSxofXN0YXR1cyBiaXRzIHRvIGNvbmZpZ3VyZSBpdCks IHdoZW4Kd3JzLm50byBpcyBjb25maWd1cmVkIHRvIHJhaXNlIGV4Y2VwdGlvbnMgaXQncyBleHBl Y3RlZCB0aGF0IHRoZSBoaWdoZXIKcHJpdmlsZWdlIGxldmVsIHdpbGwgc2VlIHRoZSBpbnN0cnVj dGlvbiB3YXMgYSB3YWl0IGluc3RydWN0aW9uLCBkbwpzb21ldGhpbmcsIGFuZCB0aGVuIHJlc3Vt ZSBleGVjdXRpb24gZm9sbG93aW5nIHRoZSBpbnN0cnVjdGlvbi4gRm9yCmV4YW1wbGUsIEtWTSBk b2VzIGNvbmZpZ3VyZSBleGNlcHRpb25zIGZvciB3ZmkgKGhzdGF0dXMuVlRXPTEpIGFuZAp0aGVy ZWZvcmUgYWxzbyBmb3Igd3JzLm50by4gS1ZNIGRvZXMgdGhpcyBmb3Igd2ZpIHNpbmNlIGl0J3Mg YmV0dGVyIHRvCmFsbG93IG90aGVyIHRhc2tzIHRvIGJlIHNjaGVkdWxlZCB3aGlsZSBhIFZDUFUg d2FpdHMgZm9yIGFuIGludGVycnVwdC4KRm9yIHdhaXRzIHN1Y2ggYXMgdGhvc2Ugd2hlcmUgd3Jz Lm50by9zdG8gd291bGQgYmUgdXNlZCwgd2hpY2ggYXJlCnR5cGljYWxseSBsb2NrcywgaXQgaXMg YWxzbyBhIGdvb2QgaWRlYSBmb3IgS1ZNIHRvIGJlIGludm9sdmVkLCBhcyBpdApjYW4gYXR0ZW1w dCB0byBzY2hlZHVsZSB0aGUgbG9jayBob2xkaW5nIFZDUFUuCgpUaGlzIHNlcmllcyBzdGFydHMg d2l0aCBDaHJpc3RvcGgncyBhZGRpdGlvbiBvZiByaXNjdiBzbXBfY29uZF9sb2FkKgpmdW5jdGlv bnMgd2hpY2ggYXBwbHkgd3JzLnN0byB3aGVuIGF2YWlsYWJsZS4gVGhhdCBwYXRjaCBoYXMgYmVl bgpyZXdvcmtlZCB0byB1c2Ugd3JzLm50byBhbmQgdG8gdXNlIHRoZSBzYW1lIGFwcHJvYWNoIGFz IEFybSBmb3IgdGhlCndhaXQgbG9vcCwgc2luY2Ugd2UgY2FuJ3QgaGF2ZSBhcmJpdHJhcnkgQyBj b2RlIGJldHdlZW4gdGhlIGxvYWQtCnJlc2VydmVkIGFuZCB0aGUgd3JzLiBUaGVuLCBod3Byb2Jl IHN1cHBvcnQgaXMgYWRkZWQgKHNpbmNlIHRoZQppbnN0cnVjdGlvbnMgYXJlIGFsc28gdXNhYmxl IGZyb20gdXNlcm1vZGUpLCBhbmQgZmluYWxseSBLVk0gaXMKdGF1Z2h0IGFib3V0IHdycy5udG8s IGFsbG93aW5nIGd1ZXN0cyB0byBzZWUgYW5kIHVzZSB0aGUgWmF3cnMKZXh0ZW5zaW9uLgoKV2Ug c3RpbGwgZG9uJ3QgaGF2ZSB0ZXN0IHJlc3VsdHMgZnJvbSBoYXJkd2FyZSwgYW5kIGl0J3Mgbm90 IHBvc3NpYmxlIHRvCnByb3ZlIHRoYXQgdXNpbmcgWmF3cnMgaXMgYSB3aW4gd2hlbiB0ZXN0aW5n IG9uIFFFTVUsIG5vdCBldmVuIHdoZW4Kb3ZlcnN1YnNjcmliaW5nIFZDUFVzIHRvIGd1ZXN0cy4g SG93ZXZlciwgaXQgaXMgcG9zc2libGUgdG8gdXNlIEtWTQpzZWxmdGVzdHMgdG8gZm9yY2UgYSBz Y2VuYXJpbyB3aGVyZSB3ZSBjYW4gcHJvdmUgWmF3cnMgZG9lcyBpdHMgam9iIGFuZApkb2VzIGl0 IHdlbGwuIFs0XSBpcyBhIHRlc3Qgd2hpY2ggZG9lcyB0aGlzIGFuZCwgb24gbXkgbWFjaGluZSwg d2l0aG91dApaYXdycyBpdCB0YWtlcyAxNiBzZWNvbmRzIHRvIGNvbXBsZXRlIGFuZCB3aXRoIFph d3JzIGl0IHRha2VzIDAuMjUKc2Vjb25kcy4KClRoaXMgc2VyaWVzIGlzIGFsc28gYXZhaWxhYmxl IGhlcmUgWzFdLiBJbiBvcmRlciB0byB1c2UgUUVNVSBmb3IgdGVzdGluZwphIGJ1aWxkIHdpdGgg WzJdIGlzIG5lZWRlZC4gSW4gb3JkZXIgdG8gZW5hYmxlIGd1ZXN0cyB0byB1c2UgWmF3cnMgd2l0 aApLVk0gdXNpbmcga3ZtdG9vbCwgdGhlIGJyYW5jaCBhdCBbM10gbWF5IGJlIHVzZWQuCgpbMV0g aHR0cHM6Ly9naXRodWIuY29tL2pvbmVzLWRyZXcvbGludXgvY29tbWl0cy9yaXNjdi96YXdycy12 Mi8KWzJdIGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2FsbC8yMDI0MDMxMjE1MjkwMS41MTIwMDEt Mi1ham9uZXNAdmVudGFuYW1pY3JvLmNvbS8KWzNdIGh0dHBzOi8vZ2l0aHViLmNvbS9qb25lcy1k cmV3L2t2bXRvb2wvY29tbWl0cy9yaXNjdi96YXdycy8KWzRdIGh0dHBzOi8vZ2l0aHViLmNvbS9q b25lcy1kcmV3L2xpbnV4L2NvbW1pdC85MzExNzAyYmNkMTE4YmRiZmE4YjliZTRhOGVjMzU1YzQw NTU5NDk5CgpUaGFua3MsCmRyZXcKCnYyOgogLSBBZGRlZCBEVCBiaW5kaW5ncyBwYXRjaCB3aXRo IGFkZGl0aW9uYWwgTGludXggc3BlY2lmaWNhdGlvbnMgZHVlCiAgIHRvIHdycy5udG8gcG90ZW50 aWFsbHkgbmV2ZXIgdGVybWluYXRpbmcsIGFzIHN1Z2dlc3RlZCBieSBQYWxtZXIKIC0gQWRkZWQg cGF0Y2ggdG8gc2hhcmUgcGF1c2UgaW5zbiBkZWZpbml0aW9uCiAtIFJld29yayBtYWluIFphd3Jz IHN1cHBvcnQgcGF0Y2ggdG8gdXNlIEFybSBhcHByb2FjaCAod2hpY2ggaXMKICAgYWxzbyB0aGUg YXBwcm9hY2ggdGhhdCBBbmRyZWEgUGFycmkgc3VnZ2VzdGVkKQogLSBEcm9wcGVkIHRoZSByaXNj diBpbXBsZW1lbnRhdGlvbiBvZiBzbXBfY29uZF9sb2FkX2FjcXVpcmUoKS4KICAgYWZhaWN0LCB0 aGUgZ2VuZXJpYyBpbXBsZW1lbnRhdGlvbiwgd2hpY2ggd2lsbCB1c2UgdGhlIHJpc2N2CiAgIGlt cGxlbWVudGF0aW9uIG9mIHNtcF9jb25kX2xvYWRfcmVsYXhlZCgpIGlzIHN1ZmZpY2llbnQgZm9y IHJpc2N2LgogLSBUaGUgcmV3b3JrIHdhcyBsYXJnZSBlbm91Z2ggKElNTykgdG8gZHJvcCBIZWlr bydzIHMtby1iIGFuZCB0bwogICBhZGQgbXlzZWxmIGFzIGEgY28tZGV2ZWxvcGVyCgoKQW5kcmV3 IEpvbmVzICg1KToKICByaXNjdjogUHJvdmlkZSBhIGRlZmluaXRpb24gZm9yICdwYXVzZScKICBk dC1iaW5kaW5nczogcmlzY3Y6IEFkZCBaYXdycyBJU0EgZXh0ZW5zaW9uIGRlc2NyaXB0aW9uCiAg cmlzY3Y6IGh3cHJvYmU6IGV4cG9ydCBaYXdycyBJU0EgZXh0ZW5zaW9uCiAgS1ZNOiByaXNjdjog U3VwcG9ydCBndWVzdCB3cnMubnRvCiAgS1ZNOiByaXNjdjogc2VsZnRlc3RzOiBBZGQgWmF3cnMg ZXh0ZW5zaW9uIHRvIGdldC1yZWctbGlzdCB0ZXN0CgpDaHJpc3RvcGggTcO8bGxuZXIgKDEpOgog IHJpc2N2OiBBZGQgWmF3cnMgc3VwcG9ydCBmb3Igc3BpbmxvY2tzCgogRG9jdW1lbnRhdGlvbi9h cmNoL3Jpc2N2L2h3cHJvYmUucnN0ICAgICAgICAgIHwgIDQgKysKIC4uLi9kZXZpY2V0cmVlL2Jp bmRpbmdzL3Jpc2N2L2V4dGVuc2lvbnMueWFtbCB8IDEyICsrKysrCiBhcmNoL3Jpc2N2L0tjb25m aWcgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAyMCArKysrKy0tLQogYXJjaC9yaXNjdi9N YWtlZmlsZSAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgIDMgLS0KIGFyY2gvcmlzY3YvaW5j bHVkZS9hc20vYmFycmllci5oICAgICAgICAgICAgICB8IDQ1ICsrKysrKysrKystLS0tLS0KIGFy Y2gvcmlzY3YvaW5jbHVkZS9hc20vY21weGNoZy5oICAgICAgICAgICAgICB8IDUxICsrKysrKysr KysrKysrKysrKysKIGFyY2gvcmlzY3YvaW5jbHVkZS9hc20vaHdjYXAuaCAgICAgICAgICAgICAg ICB8ICAxICsKIGFyY2gvcmlzY3YvaW5jbHVkZS9hc20vaW5zbi1kZWYuaCAgICAgICAgICAgICB8 ICA0ICsrCiBhcmNoL3Jpc2N2L2luY2x1ZGUvYXNtL2t2bV9ob3N0LmggICAgICAgICAgICAgfCAg MSArCiBhcmNoL3Jpc2N2L2luY2x1ZGUvYXNtL3Zkc28vcHJvY2Vzc29yLmggICAgICAgfCAgOCAr LS0KIGFyY2gvcmlzY3YvaW5jbHVkZS91YXBpL2FzbS9od3Byb2JlLmggICAgICAgICB8ICAxICsK IGFyY2gvcmlzY3YvaW5jbHVkZS91YXBpL2FzbS9rdm0uaCAgICAgICAgICAgICB8ICAxICsKIGFy Y2gvcmlzY3Yva2VybmVsL2NwdWZlYXR1cmUuYyAgICAgICAgICAgICAgICB8ICAxICsKIGFyY2gv cmlzY3Yva2VybmVsL3N5c19od3Byb2JlLmMgICAgICAgICAgICAgICB8ICAxICsKIGFyY2gvcmlz Y3Yva3ZtL3ZjcHUuYyAgICAgICAgICAgICAgICAgICAgICAgICB8ICAxICsKIGFyY2gvcmlzY3Yv a3ZtL3ZjcHVfaW5zbi5jICAgICAgICAgICAgICAgICAgICB8IDE1ICsrKysrKwogYXJjaC9yaXNj di9rdm0vdmNwdV9vbmVyZWcuYyAgICAgICAgICAgICAgICAgIHwgIDIgKwogLi4uL3NlbGZ0ZXN0 cy9rdm0vcmlzY3YvZ2V0LXJlZy1saXN0LmMgICAgICAgIHwgIDQgKysKIDE4IGZpbGVzIGNoYW5n ZWQsIDE0NCBpbnNlcnRpb25zKCspLCAzMSBkZWxldGlvbnMoLSkKCi0tIAoyLjQ0LjAKCgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1yaXNjdiBt YWlsaW5nIGxpc3QKbGludXgtcmlzY3ZAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMu aW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LXJpc2N2Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86771200D3 for ; Fri, 19 Apr 2024 13:53:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713534807; cv=none; b=gvdN4gH0hZiSeAdEB+WGQmyE3+2LO0c7WmvS8YdwGYKcvq5FZrTTsLXXyg2JQXSZgYoTMuLHM+57pkOPlQpl68mxA7NsYQ1dbIYaItkEbjMQq5oG45uMPH//RVV3lS/fzTRR8XxOu+syjrBPwcX3ICkUzkWV1f2tpdPwDquqhyc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713534807; c=relaxed/simple; bh=xfzH6BZrc06a1HLflSF6QEIDYR7DUr3ByDtP2wGK+7s=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type: Content-type; b=euTPsPXvPEPEs6XXvPrFmaFcFkftzcjlnzBBIUyC1zGZr4iReI5c3JOiwjnAITWpeyH2Uc+6hH60JeFCMGuzxjothSKpileFd+PUlgbozauDMR978NaQctuJHk1md5JbuvrVt9qr4dyU+ODQ5tzU91qHU6Jrj19p3VmWp5ur2qk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ventanamicro.com; spf=pass smtp.mailfrom=ventanamicro.com; dkim=pass (2048-bit key) header.d=ventanamicro.com header.i=@ventanamicro.com header.b=C7WM6U+8; arc=none smtp.client-ip=209.85.167.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ventanamicro.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ventanamicro.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ventanamicro.com header.i=@ventanamicro.com header.b="C7WM6U+8" Received: by mail-lf1-f45.google.com with SMTP id 2adb3069b0e04-51967f75729so2401718e87.0 for ; Fri, 19 Apr 2024 06:53:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; t=1713534803; x=1714139603; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Vygojg6kK2t93LE6Jccui0sWTwt0Dla9UnPOwn3MABA=; b=C7WM6U+822osLeE7GZj2fZXKOvtKQ9QQ2wNttnc+Sh7CcKVn4GOOMA8RX+YPmhpUux 3I4m/HvLh8WnZ6cATSjuQfL44q7HpJQkUNbAf6KXn7sfawpVexE7oWMUpx3cSh7gCrvd Ku5dRiNC7DwFKuKmjqX1KjeDvsmqI7b349ywIPccgZrNRYAJXWpSW6eKUGPPIIuYRB6s slsXjyMB54loulzQAchlsYbot0u8gh6MwxJ311BEXqyixLBObfICXGUFtNyfswCxFio+ CFJBeBH8TUd4qjdmp3K8Bf+I98OfF0u7K8Ryvx7dPj2W0Ppx1Hbj1Xuo4N2S3uwI8cN5 NUpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713534803; x=1714139603; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Vygojg6kK2t93LE6Jccui0sWTwt0Dla9UnPOwn3MABA=; b=NOGd/lDj498dg3f5TczuyfYsSAQCWxL/p53XYFpL3sNHP3S2+Vzi4OCf1zbMgWy6/e JBcGU++sVgCZH8Y/OpKjNCaMKUa2RXtcM1JjUmiL8WEjJZu9pJjv6sIbx8c+s/KUn/mV 2U1P6Wra4gZ0Rvf9xEcfG3sxCYVTuojb79QXcFJSPHRXeXVKKbwlm2Pfk3hFOjOTVZgt SHDoycd1h9ceK9/oaIAv+8PD9Qyq0XK1ByJbFbxW9RHtJf5fgBOxn3RK5N07yXXpzsgD 8jguIEEzHnu7kjRelqxIOBnUi7lr2lHmQl/OcIEHpZ+qnKSOkrEcAe3DkRQ/aNg2OP3d 7qEQ== X-Forwarded-Encrypted: i=1; AJvYcCVPYkorRhcid4pEQe65u7irg+PytWrtdrxv1p86y/86VkSDJJUhVVAu/6MEr9H9ukZ1RYpdkbDiAQBgk9eza5R+hMC4Eha+ZVqCtA== X-Gm-Message-State: AOJu0YyZWD/PxmjLDDgE8GA/T6VGIB5uVtaHanfmeJERYBrTW4TdEesi 60qGr5Mk4HYW8M4R2ZDMPpKaO9rCrQCNM4xcXiN2+Y0ICGA1ZpXRhil48CFQZjU= X-Google-Smtp-Source: AGHT+IFFzEV9Mdl8G4cryaHfYH+SN86dJZ7B9fY5aMYsjcaqBCQ0YnmeqoSttiNY9Cm5AgowfnUBMg== X-Received: by 2002:ac2:5388:0:b0:519:14a2:72da with SMTP id g8-20020ac25388000000b0051914a272damr1453236lfh.67.1713534803347; Fri, 19 Apr 2024 06:53:23 -0700 (PDT) Received: from localhost (2001-1ae9-1c2-4c00-20f-c6b4-1e57-7965.ip6.tmcz.cz. [2001:1ae9:1c2:4c00:20f:c6b4:1e57:7965]) by smtp.gmail.com with ESMTPSA id lj26-20020a170906f9da00b00a5254ec731esm2242648ejb.176.2024.04.19.06.53.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Apr 2024 06:53:22 -0700 (PDT) From: Andrew Jones To: linux-riscv@lists.infradead.org, kvm-riscv@lists.infradead.org, devicetree@vger.kernel.org Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, conor.dooley@microchip.com, anup@brainfault.org, atishp@atishpatra.org, robh@kernel.org, krzysztof.kozlowski+dt@linaro.org, conor+dt@kernel.org, christoph.muellner@vrull.eu, heiko@sntech.de, charlie@rivosinc.com, David.Laight@ACULAB.COM, parri.andrea@gmail.com, luxu.kernel@bytedance.com Subject: [PATCH v2 0/6] riscv: Apply Zawrs when available Date: Fri, 19 Apr 2024 15:53:22 +0200 Message-ID: <20240419135321.70781-8-ajones@ventanamicro.com> X-Mailer: git-send-email 2.44.0 Precedence: bulk X-Mailing-List: devicetree@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-type: text/plain Content-Transfer-Encoding: 8bit Zawrs provides two instructions (wrs.nto and wrs.sto), where both are meant to allow the hart to enter a low-power state while waiting on a store to a memory location. The instructions also both wait an implementation-defined "short" duration (unless the implementation terminates the stall for another reason). The difference is that while wrs.sto will terminate when the duration elapses, wrs.nto, depending on configuration, will either just keep waiting or an ILL exception will be raised. Linux will use wrs.nto, so if platforms have an implementation which falls in the "just keep waiting" category, then it should _not_ advertise Zawrs in the hardware description. Text to that effect has been added to the Zawrs DT definition. Like wfi (and with the same {m,h}status bits to configure it), when wrs.nto is configured to raise exceptions it's expected that the higher privilege level will see the instruction was a wait instruction, do something, and then resume execution following the instruction. For example, KVM does configure exceptions for wfi (hstatus.VTW=1) and therefore also for wrs.nto. KVM does this for wfi since it's better to allow other tasks to be scheduled while a VCPU waits for an interrupt. For waits such as those where wrs.nto/sto would be used, which are typically locks, it is also a good idea for KVM to be involved, as it can attempt to schedule the lock holding VCPU. This series starts with Christoph's addition of riscv smp_cond_load* functions which apply wrs.sto when available. That patch has been reworked to use wrs.nto and to use the same approach as Arm for the wait loop, since we can't have arbitrary C code between the load- reserved and the wrs. Then, hwprobe support is added (since the instructions are also usable from usermode), and finally KVM is taught about wrs.nto, allowing guests to see and use the Zawrs extension. We still don't have test results from hardware, and it's not possible to prove that using Zawrs is a win when testing on QEMU, not even when oversubscribing VCPUs to guests. However, it is possible to use KVM selftests to force a scenario where we can prove Zawrs does its job and does it well. [4] is a test which does this and, on my machine, without Zawrs it takes 16 seconds to complete and with Zawrs it takes 0.25 seconds. This series is also available here [1]. In order to use QEMU for testing a build with [2] is needed. In order to enable guests to use Zawrs with KVM using kvmtool, the branch at [3] may be used. [1] https://github.com/jones-drew/linux/commits/riscv/zawrs-v2/ [2] https://lore.kernel.org/all/20240312152901.512001-2-ajones@ventanamicro.com/ [3] https://github.com/jones-drew/kvmtool/commits/riscv/zawrs/ [4] https://github.com/jones-drew/linux/commit/9311702bcd118bdbfa8b9be4a8ec355c40559499 Thanks, drew v2: - Added DT bindings patch with additional Linux specifications due to wrs.nto potentially never terminating, as suggested by Palmer - Added patch to share pause insn definition - Rework main Zawrs support patch to use Arm approach (which is also the approach that Andrea Parri suggested) - Dropped the riscv implementation of smp_cond_load_acquire(). afaict, the generic implementation, which will use the riscv implementation of smp_cond_load_relaxed() is sufficient for riscv. - The rework was large enough (IMO) to drop Heiko's s-o-b and to add myself as a co-developer Andrew Jones (5): riscv: Provide a definition for 'pause' dt-bindings: riscv: Add Zawrs ISA extension description riscv: hwprobe: export Zawrs ISA extension KVM: riscv: Support guest wrs.nto KVM: riscv: selftests: Add Zawrs extension to get-reg-list test Christoph Müllner (1): riscv: Add Zawrs support for spinlocks Documentation/arch/riscv/hwprobe.rst | 4 ++ .../devicetree/bindings/riscv/extensions.yaml | 12 +++++ arch/riscv/Kconfig | 20 +++++--- arch/riscv/Makefile | 3 -- arch/riscv/include/asm/barrier.h | 45 ++++++++++------ arch/riscv/include/asm/cmpxchg.h | 51 +++++++++++++++++++ arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/insn-def.h | 4 ++ arch/riscv/include/asm/kvm_host.h | 1 + arch/riscv/include/asm/vdso/processor.h | 8 +-- arch/riscv/include/uapi/asm/hwprobe.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + arch/riscv/kernel/sys_hwprobe.c | 1 + arch/riscv/kvm/vcpu.c | 1 + arch/riscv/kvm/vcpu_insn.c | 15 ++++++ arch/riscv/kvm/vcpu_onereg.c | 2 + .../selftests/kvm/riscv/get-reg-list.c | 4 ++ 18 files changed, 144 insertions(+), 31 deletions(-) -- 2.44.0