From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-oo1-f53.google.com (mail-oo1-f53.google.com [209.85.161.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A8A943146 for ; Tue, 16 Apr 2024 04:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.53 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713240736; cv=none; b=Vhn5XXlCbNrRNQZoZAitt94EEuNzgO/GpzHmYrezZLrlUCUeiIuRXx468D0r2AZlNExecePtZbARo7LzofOwwXTFRU+ajvktdFM+6m85j1CPrWSl7eW3bi8o6jXc5PXrJunu18htWjIasKVretiLXQqX01W+9QkQ/m7XZ+97bTU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713240736; c=relaxed/simple; bh=EUo/8P9TxYm5Z8gBeogX8nOGhGz5D7qWLLRReTJSe00=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=fo9mMJdomUB7LucqNyR4T+sFWCZLDOYFjEzaa3oWnbCwT9otAVIc5BfvKT44oYd/i2ltmMs5Y8BhU6awMHbr25EBdzwwc7qKcoi2L0O1BDLo8HPWjxd1z5pzV1DM3iAHvu1O6ws31MvQ85cK9vOvhgRZ4RKeLmSQ4odZsSvYL8E= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=BOH4mkpB; arc=none smtp.client-ip=209.85.161.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="BOH4mkpB" Received: by mail-oo1-f53.google.com with SMTP id 006d021491bc7-5aa2dcd4454so2755412eaf.2 for ; Mon, 15 Apr 2024 21:12:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1713240733; x=1713845533; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=pfgEouA2o3txDoLsnSFQ1dO9dPIDHDcBgqnFwikm9Z4=; b=BOH4mkpB9RbrV74CfvdtZEUzs4DfbP/OYhPg7lAepAxFaDsxOy3YYfzfvFM7UyzvGd 5iBiPFgmUrouXbmzSxIeXODSXOVd6IlIT8qFDrinxn5mAl3DTfHiVblJ1eg6LvKF5Ro7 rLgjf1BKhRd+rZOnmUG9GijSfNN8tSfVOsGYpyuMDq2jsgx9f2IA2v8c5sBr15sTY5lr TAxKk1zEFsg6sreCaKMBlr6e8eyCavPAP8HWDIumsoGx3/R69mTvU9BPfW8aWEHF74tL X8wr17YF12l2k1RR10OUZtlqz5bzuE4fw8mswBqgSUD4ozp2ByPYPHKExY1sgQfQlFLZ QxRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713240733; x=1713845533; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pfgEouA2o3txDoLsnSFQ1dO9dPIDHDcBgqnFwikm9Z4=; b=E3Evr4m8EINeGNfYmIctOSZb8npd78hHiBirixp2AbVLL6Ik5oYgs8d4RzhnBaX8Ce 8Zy9GndBNgID/Cmum2ODLE8WAmoRPf0yf5aGfSuNTCu55ykJ5hlP9UULgtwPmtKgtZ1D mMID2HJJahX5wzke9UjxnFBhqHjVjU66pAgcrJQUfJri4hjrWlAL9V95BY8IiDXObcfs QrKfmRhU5QClL3Wbf6PYp7Pb7UPDX4UxkUXhrha96viKbzas8SckhKrT46NSpbp4Rp7V kVEp1ql8c44jY+0l1q6tPULEidkc8e/wyBzR5zUbv5WaF5Axd5ut7UgpO/uOZIYV+qen pY7A== X-Forwarded-Encrypted: i=1; AJvYcCUhnaw3hERUWlJ8Fsp5U+SeRtmsCa1ng8Z05yTnLCQO9RCT4hphNtA4/VqMmzm4gO0nG2DFwJ8H65Lf2RGbeAygmGjdxrAo7vHv3A== X-Gm-Message-State: AOJu0Yy0dWFo6oajr0NqD4G3fXGh7bF8vV/JaEFJplnaf6/Ar01xpxz8 Nzf95k6Kf2kkAfUuQ5X8t+JcquLEMhy7n2NtgmP4GMJLfIbXTbDfidfKjBQa4KA= X-Google-Smtp-Source: AGHT+IEkWKCmfS18iCDatrUQ8+gUH7AmhQlIUh3jbL94Sbp8FTPKm2WfVOtKvazKrIc/nVfvWnVHCQ== X-Received: by 2002:a05:6870:7010:b0:22e:cc63:9b26 with SMTP id u16-20020a056870701000b0022ecc639b26mr13242263oae.15.1713240732726; Mon, 15 Apr 2024 21:12:12 -0700 (PDT) Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id z184-20020a6265c1000000b006ed4c430acesm8383149pfb.40.2024.04.15.21.12.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 21:12:11 -0700 (PDT) From: Charlie Jenkins Date: Mon, 15 Apr 2024 21:12:04 -0700 Subject: [PATCH v2 07/17] riscv: Introduce vendor variants of extension helpers Precedence: bulk X-Mailing-List: devicetree@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20240415-dev-charlie-support_thead_vector_6_9-v2-7-c7d68c603268@rivosinc.com> References: <20240415-dev-charlie-support_thead_vector_6_9-v2-0-c7d68c603268@rivosinc.com> In-Reply-To: <20240415-dev-charlie-support_thead_vector_6_9-v2-0-c7d68c603268@rivosinc.com> To: Conor Dooley , Rob Herring , Krzysztof Kozlowski , Paul Walmsley , Palmer Dabbelt , Albert Ou , Guo Ren , Conor Dooley , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , Conor Dooley , Evan Green , =?utf-8?q?Cl=C3=A9ment_L=C3=A9ger?= , Jonathan Corbet , Shuah Khan Cc: linux-riscv@lists.infradead.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Palmer Dabbelt , linux-arm-kernel@lists.infradead.org, linux-sunxi@lists.linux.dev, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Charlie Jenkins X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713240719; l=11791; i=charlie@rivosinc.com; s=20231120; h=from:subject:message-id; bh=EUo/8P9TxYm5Z8gBeogX8nOGhGz5D7qWLLRReTJSe00=; b=Q0x4Mz7P7q6WFelFlp1y5rSIZoXPi8vlvCpVUPr+N3sy5vC2A1kMHvi3EW9gb6Ay3s/PW+Mxk ZD4o95nKjphDcab+24J6lgRl1n0bJFwXHfQUGSW0TwWbIKaQKKIWdA2 X-Developer-Key: i=charlie@rivosinc.com; a=ed25519; pk=t4RSWpMV1q5lf/NWIeR9z58bcje60/dbtxxmoSfBEcs= Vendor extensions are maintained in riscv_isa_vendor (separate from standard extensions which live in riscv_isa). Create vendor variants for the existing extension helpers to interface with the riscv_isa_vendor bitmap. There is a good amount of overlap between these functions, so the alternative checking code can be factored out. Signed-off-by: Charlie Jenkins --- arch/riscv/errata/sifive/errata.c | 2 + arch/riscv/errata/thead/errata.c | 2 + arch/riscv/include/asm/cpufeature.h | 142 +++++++++++++++++++++++++++--------- arch/riscv/include/asm/hwprobe.h | 3 + arch/riscv/kernel/cpufeature.c | 53 ++++++++++++-- arch/riscv/kernel/sys_hwprobe.c | 4 +- 6 files changed, 161 insertions(+), 45 deletions(-) diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/errata.c index 3d9a32d791f7..847ff85cc911 100644 --- a/arch/riscv/errata/sifive/errata.c +++ b/arch/riscv/errata/sifive/errata.c @@ -99,6 +99,8 @@ void sifive_errata_patch_func(struct alt_entry *begin, struct alt_entry *end, for (alt = begin; alt < end; alt++) { if (alt->vendor_id != SIFIVE_VENDOR_ID) continue; + if (alt->patch_id >= RISCV_ISA_VENDOR_EXT_BASE) + continue; if (alt->patch_id >= ERRATA_SIFIVE_NUMBER) { WARN(1, "This errata id:%d is not in kernel errata list", alt->patch_id); continue; diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c index b1c410bbc1ae..6e3eabfe92af 100644 --- a/arch/riscv/errata/thead/errata.c +++ b/arch/riscv/errata/thead/errata.c @@ -163,6 +163,8 @@ void thead_errata_patch_func(struct alt_entry *begin, struct alt_entry *end, for (alt = begin; alt < end; alt++) { if (alt->vendor_id != THEAD_VENDOR_ID) continue; + if (alt->patch_id >= RISCV_ISA_VENDOR_EXT_BASE) + continue; if (alt->patch_id >= ERRATA_THEAD_NUMBER) continue; diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 50fa174cccb9..12dd36bafa2a 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -110,23 +110,19 @@ bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned i #define riscv_isa_extension_available(isa_bitmap, ext) \ __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) +bool __riscv_isa_vendor_extension_available(const unsigned long *vendor_isa_bitmap, + unsigned int bit); +#define riscv_isa_vendor_extension_available(isa_bitmap, ext) \ + __riscv_isa_vendor_extension_available(isa_bitmap, RISCV_ISA_VENDOR_EXT_##ext) + static __always_inline bool -riscv_has_extension_likely(const unsigned long ext) +__riscv_has_extension_likely_alternatives(const unsigned long vendor, const unsigned long ext) { - compiletime_assert(ext < RISCV_ISA_EXT_MAX, - "ext must be < RISCV_ISA_EXT_MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) { - asm goto( - ALTERNATIVE("j %l[l_no]", "nop", 0, %[ext], 1) - : - : [ext] "i" (ext) - : - : l_no); - } else { - if (!__riscv_isa_extension_available(NULL, ext)) - goto l_no; - } + asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_no); return true; l_no: @@ -134,42 +130,118 @@ riscv_has_extension_likely(const unsigned long ext) } static __always_inline bool -riscv_has_extension_unlikely(const unsigned long ext) +__riscv_has_extension_unlikely_alternatives(const unsigned long vendor, const unsigned long ext) { - compiletime_assert(ext < RISCV_ISA_EXT_MAX, - "ext must be < RISCV_ISA_EXT_MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) { - asm goto( - ALTERNATIVE("nop", "j %l[l_yes]", 0, %[ext], 1) - : - : [ext] "i" (ext) - : - : l_yes); - } else { - if (__riscv_isa_extension_available(NULL, ext)) - goto l_yes; - } + asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_yes); return false; l_yes: return true; } +/* Standard extension helpers */ + +static __always_inline bool +riscv_has_extension_likely(const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, + "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_likely_alternatives(0, ext); + else + return __riscv_isa_extension_available(NULL, ext); +} + +static __always_inline bool +riscv_has_extension_unlikely(const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, + "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_unlikely_alternatives(0, ext); + else + return __riscv_isa_extension_available(NULL, ext); +} + static __always_inline bool riscv_cpu_has_extension_likely(int cpu, const unsigned long ext) { - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE) && riscv_has_extension_likely(ext)) - return true; + compiletime_assert(ext < RISCV_ISA_EXT_MAX, + "ext must be < RISCV_ISA_EXT_MAX"); - return __riscv_isa_extension_available(hart_isa[cpu].isa, ext); + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE) && + __riscv_has_extension_likely_alternatives(0, ext)) + return true; + else + return __riscv_isa_extension_available(hart_isa[cpu].isa, ext); } static __always_inline bool riscv_cpu_has_extension_unlikely(int cpu, const unsigned long ext) { - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE) && riscv_has_extension_unlikely(ext)) + compiletime_assert(ext < RISCV_ISA_EXT_MAX, + "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE) && + __riscv_has_extension_unlikely_alternatives(0, ext)) return true; + else + return __riscv_isa_extension_available(hart_isa[cpu].isa, ext); +} + +/* Vendor extension helpers */ + +static __always_inline bool +riscv_has_vendor_extension_likely(const unsigned long vendor, const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_VENDOR_EXT_MAX, + "ext must be < RISCV_ISA_VENDOR_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_likely_alternatives(vendor, ext); + else + return __riscv_isa_vendor_extension_available(NULL, ext); +} - return __riscv_isa_extension_available(hart_isa[cpu].isa, ext); +static __always_inline bool +riscv_has_vendor_extension_unlikely(const unsigned long vendor, const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_VENDOR_EXT_MAX, + "ext must be < RISCV_ISA_VENDOR_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_unlikely_alternatives(vendor, ext); + else + return __riscv_isa_vendor_extension_available(NULL, ext); +} + +static __always_inline bool riscv_cpu_has_vendor_extension_likely(const unsigned long vendor, + int cpu, const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_VENDOR_EXT_MAX, + "ext must be < RISCV_ISA_VENDOR_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_likely_alternatives(vendor, ext); + else + return __riscv_isa_vendor_extension_available(hart_isa_vendor[cpu].isa, ext); +} + +static __always_inline bool riscv_cpu_has_vendor_extension_unlikely(const unsigned long vendor, + int cpu, + const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_VENDOR_EXT_MAX, + "ext must be < RISCV_ISA_VENDOR_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_unlikely_alternatives(vendor, ext); + else + return __riscv_isa_vendor_extension_available(hart_isa_vendor[cpu].isa, ext); } #endif diff --git a/arch/riscv/include/asm/hwprobe.h b/arch/riscv/include/asm/hwprobe.h index 630507dff5ea..1378c3c9401a 100644 --- a/arch/riscv/include/asm/hwprobe.h +++ b/arch/riscv/include/asm/hwprobe.h @@ -6,6 +6,7 @@ #ifndef _ASM_HWPROBE_H #define _ASM_HWPROBE_H +#include #include #define RISCV_HWPROBE_MAX_KEY 6 @@ -39,4 +40,6 @@ static inline bool riscv_hwprobe_pair_cmp(struct riscv_hwprobe *pair, return pair->value == other_pair->value; } +void hwprobe_arch_id(struct riscv_hwprobe *pair, const struct cpumask *cpus); + #endif diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index d7a33e017a15..799ec2d2e9e0 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -82,6 +83,30 @@ bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned i } EXPORT_SYMBOL_GPL(__riscv_isa_extension_available); +/** + * __riscv_isa_vendor_extension_available() - Check whether given vendor + * extension is available or not. The vendor extension must be associated + * with the same vendor that was used to populate isa_bitmap. + * + * @isa_bitmap: ISA bitmap to use + * @bit: bit position of the desired extension + * Return: true or false + * + * NOTE: If isa_bitmap is NULL then Host ISA bitmap will be used. + */ +bool __riscv_isa_vendor_extension_available(const unsigned long *isa_bitmap, unsigned int bit) +{ + const unsigned long *bmap = (isa_bitmap) ? isa_bitmap : riscv_isa_vendor; + + bit -= RISCV_ISA_VENDOR_EXT_BASE; + + if (bit < 0 || bit >= RISCV_ISA_VENDOR_EXT_MAX) + return false; + + return test_bit(bit, bmap) ? true : false; +} +EXPORT_SYMBOL_GPL(__riscv_isa_vendor_extension_available); + static bool riscv_isa_extension_check(int id) { switch (id) { @@ -832,25 +857,37 @@ void __init_or_module riscv_cpufeature_patch_func(struct alt_entry *begin, { struct alt_entry *alt; void *oldptr, *altptr; - u16 id, value; + u16 id, value, vendor; if (stage == RISCV_ALTERNATIVES_EARLY_BOOT) return; - for (alt = begin; alt < end; alt++) { - if (alt->vendor_id != 0) - continue; + struct riscv_hwprobe mvendorid = { + .key = RISCV_HWPROBE_KEY_MVENDORID, + .value = 0 + }; + hwprobe_arch_id(&mvendorid, cpu_possible_mask); + + for (alt = begin; alt < end; alt++) { id = PATCH_ID_CPUFEATURE_ID(alt->patch_id); + vendor = PATCH_ID_CPUFEATURE_ID(alt->vendor_id); - if (id >= RISCV_ISA_EXT_MAX) { + if (id >= RISCV_ISA_VENDOR_EXT_BASE) { + if (vendor != mvendorid.value || + !__riscv_isa_vendor_extension_available(NULL, id)) + continue; + } else if (id < RISCV_ISA_EXT_MAX) { + if (alt->vendor_id != 0) + continue; + + if (!__riscv_isa_extension_available(NULL, id)) + continue; + } else { WARN(1, "This extension id:%d is not in ISA extension list", id); continue; } - if (!__riscv_isa_extension_available(NULL, id)) - continue; - value = PATCH_ID_CPUFEATURE_VALUE(alt->patch_id); if (!riscv_cpufeature_patch_check(id, value)) continue; diff --git a/arch/riscv/kernel/sys_hwprobe.c b/arch/riscv/kernel/sys_hwprobe.c index 8cae41a502dd..394f1343490c 100644 --- a/arch/riscv/kernel/sys_hwprobe.c +++ b/arch/riscv/kernel/sys_hwprobe.c @@ -13,11 +13,11 @@ #include #include #include +#include #include -static void hwprobe_arch_id(struct riscv_hwprobe *pair, - const struct cpumask *cpus) +void hwprobe_arch_id(struct riscv_hwprobe *pair, const struct cpumask *cpus) { u64 id = -1ULL; bool first = true; -- 2.44.0