[PATCH v13 0/6] iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs

[PATCH v13 0/6] iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs

[Bibek Kumar Patro]

This patch series consist of six parts and covers the following:

1. Re-enable context caching for Qualcomm SoCs to retain prefetcher
   settings during reset and runtime suspend.

2. Remove cfg inside qcom_smmu structure and replace it with single
   pointer to qcom_smmu_match_data avoiding replication of multiple
   members from same.

3. Introduce intital set of driver changes to implement ACTLR register
   for custom prefetcher settings in Qualcomm SoCs.

4. Add ACTLR data and implementation operations for SM8550.

5. Add ACTLR data and implementation operations for SC7280.

6. Add support for ACTLR PRR bit setup via adreno-smmu-priv interface.

Changes in v13 from v12:
 - Fix the compilation issues reported by kernel test robot [1].
 [1] https://lore.kernel.org/all/202406281241.xEX0TWjt-lkp@intel.com/#t
 Link to v12:
 https://lore.kernel.org/all/20240626143020.3682243-1-quic_bibekkum@quicinc.com/

Changes in v12 from v11:
 Changes to incorporate suggestion from Rob:
 - Fix the set and reset logic for prr bit as pointed out in v11-6/6.
 - Rename set_actlr_bit function name to set_prr.
 - Add extension for PRR name as Partially-Resident-Region in comments
   for set_prr function.
 - Add few missing sids for sc7280 in patch-5/6.
 Link to v11:
 https://lore.kernel.org/all/20240605121713.3596499-1-quic_bibekkum@quicinc.com/

Changes in v11 from v10:
 - Include a new patch 6/6 to add support for ACTLR PRR bit
   through adreno-smmu-priv interface as suggested by Rob and Dmitry.
 Link to v10:
 https://lore.kernel.org/all/20240524131800.2288259-1-quic_bibekkum@quicinc.com/

Changes in v10 from v9:
 - Added reviewed-by tags 1/5,2/5,3/5.
 Changes incorporated:
 - Remove redundant PRR bit setting from gfx actlr table(patch 4/5,5/5)
   as this bit needs special handling in the gfx driver along with
   the associated register settings.
 Link to discussion on PRR bit:
 https://lore.kernel.org/all/f2222714-1e00-424e-946d-c314d55541b8@quicinc.com/
 Link to v9:
 https://lore.kernel.org/all/20240123144543.9405-1-quic_bibekkum@quicinc.com/

Changes in v9 from v8:
 Changes to incorporate suggestions from Konrad as follows:
 - Re-wrap struct members of actlr_variant in patch 4/5,5/5
   in a cleaner way.
 - Move actlr_config members to the header.
 Link to v8:
 https://lore.kernel.org/all/20240116150411.23876-1-quic_bibekkum@quicinc.com/

Changes in v8 from v7:
 - Added reviewed-by tags on patch 1/5, 2/5.
 Changes to incorporate suggestions from Pavan and Konrad:
 - Remove non necessary extra lines.
 - Use num_smmu and num_actlrcfg to store the array size and use the
   same to traverse the table and save on sentinel space along with
   indentation levels.
 - Refactor blocks containing qcom_smmu_set_actlr to remove block
   repetition in patch 3/5.
 - Change copyright year from 2023 to 2022-2023 in patch 3/5.
 - Modify qcom_smmu_match_data.actlrvar and actlr_variant.actlrcfg to
   const pointer to a const resource.
 - use C99 designated initializers and put the address first.
 Link to v7:
 https://lore.kernel.org/all/20240109114220.30243-1-quic_bibekkum@quicinc.com/

Changes in v7 from v6:
 Changes to incorporate suggestions from Dmitry as follows:
 - Use io_start address instead of compatible string to identify the
   correct instance by comparing with smmu start address and check for
   which smmu the corresponding actlr table is to be picked.
Link to v6:
https://lore.kernel.org/all/20231220133808.5654-1-quic_bibekkum@quicinc.com/

Changes in v6 from v5:
 - Remove extra Suggested-by tags.
 - Add return check for arm_mmu500_reset in 1/5 as discussed.
Link to v5:
https://lore.kernel.org/all/20231219135947.1623-1-quic_bibekkum@quicinc.com/

Changes in v5 from v4:
 New addition:
 - Modify copyright year in arm-smmu-qcom.h to 2023 from 2022.
 Changes to incorporate suggestions from Dmitry as follows:
 - Modify the defines for prefetch in (foo << bar) format
   as suggested.(FIELD_PREP could not be used in defines
   is not inside any block/function)
 Changes to incorporate suggestions from Konrad as follows:
 - Shift context caching enablement patch as 1/5 instead of 5/5 to
   be picked up as independent patch.
 - Fix the codestyle to orient variables in reverse xmas tree format
   for patch 1/5.
 - Fix variable name in patch 1/5 as suggested.
 Link to v4:
https://lore.kernel.org/all/20231215101827.30549-1-quic_bibekkum@quicinc.com/

Changes in v4 from v3:
 New addition:
 - Remove actlrcfg_size and use NULL end element instead to traverse
   the actlr table, as this would be a cleaner approach by removing
   redundancy of actlrcfg_size.
 - Renaming of actlr set function to arm_smmu_qcom based proprietary
   convention.
 - break from loop once sid is found and ACTLR value is initialized
   in qcom_smmu_set_actlr.
 - Modify the GFX prefetch value separating into 2 sensible defines.
 - Modify comments for prefetch defines as per SMMU-500 TRM.
 Changes to incorporate suggestions from Konrad as follows:
 - Use Reverse-Christmas-tree sorting wherever applicable.
 - Pass arguments directly to arm_smmu_set_actlr instead of creating
   duplicate variables.
 - Use array indexing instead of direct pointer addressed by new
   addition of eliminating actlrcfg_size.
 - Switch the HEX value's case from upper to lower case in SC7280
   actlrcfg table.
 Changes to incorporate suggestions from Dmitry as follows:
 - Separate changes not related to ACTLR support to different commit
   with patch 5/5.
 - Using pointer to struct for arguments in smr_is_subset().
 Changes to incorporate suggestions from Bjorn as follows:
 - fix the commit message for patch 2/5 to properly document the
   value space to avoid confusion.
 Fixed build issues reported by kernel test robot [1] for
 arm64-allyesconfig [2].
 [1]: https://lore.kernel.org/all/202312011750.Pwca3TWE-lkp@intel.com/
 [2]:
https://download.01.org/0day-ci/archive/20231201/202312011750.Pwca3TWE-lkp@intel.com/config
 Link to v3:
https://lore.kernel.org/all/20231127145412.3981-1-quic_bibekkum@quicinc.com/

Changes in v3 from v2:
 New addition:
 - Include patch 3/4 for adding ACTLR support and data for SC7280.
 - Add driver changes for actlr support in gpu smmu.
 - Add target wise actlr data and implementation ops for gpu smmu.
 Changes to incorporate suggestions from Robin as follows:
 - Match the ACTLR values with individual corresponding SID instead
   of assuming that any SMR will be programmed to match a superset of
   the data.
 - Instead of replicating each elements from qcom_smmu_match_data to
   qcom_smmu structre during smmu device creation, replace the
   replicated members with qcom_smmu_match_data structure inside
   qcom_smmu structre and handle the dereference in places that
   requires them.
 Changes to incorporate suggestions from Dmitry and Konrad as follows:
 - Maintain actlr table inside a single structure instead of
   nested structure.
 - Rename prefetch defines to more appropriately describe their
   behavior.
 - Remove SM8550 specific implementation ops and roll back to default
   qcom_smmu_500_impl implementation ops.
 - Add back the removed comments which are NAK.
 - Fix commit description for patch 4/4.
 Link to v2:
https://lore.kernel.org/all/20231114135654.30475-1-quic_bibekkum@quicinc.com/

Changes in v2 from v1:
 - Incorporated suggestions on v1 from Dmitry,Konrad,Pratyush.
 - Added defines for ACTLR values.
 - Linked sm8550 implementation structure to corresponding
   compatible string.
 - Repackaged actlr value set implementation to separate function.
 - Fixed indentation errors.
 - Link to v1:
https://lore.kernel.org/all/20231103215124.1095-1-quic_bibekkum@quicinc.com/

Changes in v1 from RFC:
 - Incorporated suggestion form Robin on RFC
 - Moved the actlr data table into driver, instead of maintaining
   it inside soc specific DT and piggybacking on exisiting iommus
   property (iommu = <SID, MASK, ACTLR>) to set this value during
   smmu probe.
 - Link to RFC:
https://lore.kernel.org/all/a01e7e60-6ead-4a9e-ba90-22a8a6bbd03f@quicinc.com/

Bibek Kumar Patro (6):
  iommu/arm-smmu: re-enable context caching in smmu reset operation
  iommu/arm-smmu: refactor qcom_smmu structure to include single pointer
  iommu/arm-smmu: introduction of ACTLR for custom prefetcher settings
  iommu/arm-smmu: add ACTLR data and support for SM8550
  iommu/arm-smmu: add ACTLR data and support for SC7280
  iommu/arm-smmu: add support for PRR bit setup

 .../iommu/arm/arm-smmu/arm-smmu-qcom-debug.c  |   2 +-
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c    | 269 +++++++++++++++++-
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h    |  18 +-
 drivers/iommu/arm/arm-smmu/arm-smmu.c         |   5 +-
 drivers/iommu/arm/arm-smmu/arm-smmu.h         |   7 +
 include/linux/adreno-smmu-priv.h              |   6 +-
 6 files changed, 296 insertions(+), 11 deletions(-)

--
2.34.1

[PATCH v13 1/6] iommu/arm-smmu: re-enable context caching in smmu reset operation

[Bibek Kumar Patro]

Default MMU-500 reset operation disables context caching in
prefetch buffer. It is however expected for context banks using
the ACTLR register to retain their prefetch value during reset
and runtime suspend.

Replace default MMU-500 reset operation with Qualcomm specific reset
operation which envelope the default reset operation and re-enables
context caching in prefetch buffer for Qualcomm SoCs.

Reviewed-by: Konrad Dybcio <konrad.dybcio@linaro.org>
Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 36 ++++++++++++++++++++--
 1 file changed, 33 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index 25f034677f56..76db4c8d1a9b 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -14,6 +14,16 @@

 #define QCOM_DUMMY_VAL	-1

+/*
+ * SMMU-500 TRM defines BIT(0) as CMTLB (Enable context caching in the
+ * macro TLB) and BIT(1) as CPRE (Enable context caching in the prefetch
+ * buffer). The remaining bits are implementation defined and vary across
+ * SoCs.
+ */
+
+#define CPRE			(1 << 1)
+#define CMTLB			(1 << 0)
+
 static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
 {
 	return container_of(smmu, struct qcom_smmu, smmu);
@@ -379,11 +389,31 @@ static int qcom_smmu_def_domain_type(struct device *dev)
 	return match ? IOMMU_DOMAIN_IDENTITY : 0;
 }

+static int qcom_smmu500_reset(struct arm_smmu_device *smmu)
+{
+	int ret;
+	u32 val;
+	int i;
+
+	ret = arm_mmu500_reset(smmu);
+	if (ret)
+		return ret;
+
+	/* arm_mmu500_reset() disables CPRE which is re-enabled here */
+	for (i = 0; i < smmu->num_context_banks; ++i) {
+		val = arm_smmu_cb_read(smmu, i, ARM_SMMU_CB_ACTLR);
+		val |= CPRE;
+		arm_smmu_cb_write(smmu, i, ARM_SMMU_CB_ACTLR, val);
+	}
+
+	return 0;
+}
+
 static int qcom_sdm845_smmu500_reset(struct arm_smmu_device *smmu)
 {
 	int ret;

-	arm_mmu500_reset(smmu);
+	qcom_smmu500_reset(smmu);

 	/*
 	 * To address performance degradation in non-real time clients,
@@ -410,7 +440,7 @@ static const struct arm_smmu_impl qcom_smmu_500_impl = {
 	.init_context = qcom_smmu_init_context,
 	.cfg_probe = qcom_smmu_cfg_probe,
 	.def_domain_type = qcom_smmu_def_domain_type,
-	.reset = arm_mmu500_reset,
+	.reset = qcom_smmu500_reset,
 	.write_s2cr = qcom_smmu_write_s2cr,
 	.tlb_sync = qcom_smmu_tlb_sync,
 #ifdef CONFIG_ARM_SMMU_QCOM_DEBUG
@@ -443,7 +473,7 @@ static const struct arm_smmu_impl qcom_adreno_smmu_v2_impl = {
 static const struct arm_smmu_impl qcom_adreno_smmu_500_impl = {
 	.init_context = qcom_adreno_smmu_init_context,
 	.def_domain_type = qcom_smmu_def_domain_type,
-	.reset = arm_mmu500_reset,
+	.reset = qcom_smmu500_reset,
 	.alloc_context_bank = qcom_adreno_smmu_alloc_context_bank,
 	.write_sctlr = qcom_adreno_smmu_write_sctlr,
 	.tlb_sync = qcom_smmu_tlb_sync,
--
2.34.1

[PATCH v13 2/6] iommu/arm-smmu: refactor qcom_smmu structure to include single pointer

[Bibek Kumar Patro]

qcom_smmu_match_data is static and constant so refactor qcom_smmu
to store single pointer to qcom_smmu_match_data instead of
replicating multiple child members of the same and handle the further
dereferences in the places that want them.

Suggested-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c | 2 +-
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c       | 2 +-
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h       | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
index 552199cbd9e2..885af324916b 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
@@ -73,7 +73,7 @@ void qcom_smmu_tlb_sync_debug(struct arm_smmu_device *smmu)
 	if (__ratelimit(&rs)) {
 		dev_err(smmu->dev, "TLB sync timed out -- SMMU may be deadlocked\n");

-		cfg = qsmmu->cfg;
+		cfg = qsmmu->data->cfg;
 		if (!cfg)
 			return;

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index 76db4c8d1a9b..573c4c9886f1 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -506,7 +506,7 @@ static struct arm_smmu_device *qcom_smmu_create(struct arm_smmu_device *smmu,
 		return ERR_PTR(-ENOMEM);

 	qsmmu->smmu.impl = impl;
-	qsmmu->cfg = data->cfg;
+	qsmmu->data = data;

 	return &qsmmu->smmu;
 }
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
index 9bb3ae7d62da..addc07623c0b 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
@@ -8,7 +8,7 @@

 struct qcom_smmu {
 	struct arm_smmu_device smmu;
-	const struct qcom_smmu_config *cfg;
+	const struct qcom_smmu_match_data *data;
 	bool bypass_quirk;
 	u8 bypass_cbndx;
 	u32 stall_enabled;
--
2.34.1

[PATCH v13 3/6] iommu/arm-smmu: introduction of ACTLR for custom prefetcher settings

[Bibek Kumar Patro]

Currently in Qualcomm  SoCs the default prefetch is set to 1 which allows
the TLB to fetch just the next page table. MMU-500 features ACTLR
register which is implementation defined and is used for Qualcomm SoCs
to have a custom prefetch setting enabling TLB to prefetch the next set
of page tables accordingly allowing for faster translations.

ACTLR value is unique for each SMR (Stream matching register) and stored
in a pre-populated table. This value is set to the register during
context bank initialisation.

Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 61 ++++++++++++++++++++++
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h | 16 +++++-
 drivers/iommu/arm/arm-smmu/arm-smmu.c      |  5 +-
 drivers/iommu/arm/arm-smmu/arm-smmu.h      |  5 ++
 4 files changed, 84 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index 573c4c9886f1..77c9abffe07d 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -215,10 +215,42 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
 	return true;
 }

+static void qcom_smmu_set_actlr(struct device *dev, struct arm_smmu_device *smmu, int cbndx,
+		const struct actlr_config *actlrcfg, const size_t num_actlrcfg)
+{
+	struct arm_smmu_master_cfg *cfg = dev_iommu_priv_get(dev);
+	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+	struct arm_smmu_smr *smr;
+	u16 mask;
+	int idx;
+	u16 id;
+	int i;
+	int j;
+
+	for (i = 0; i < num_actlrcfg; i++) {
+		id = actlrcfg[i].sid;
+		mask = actlrcfg[i].mask;
+
+		for_each_cfg_sme(cfg, fwspec, j, idx) {
+			smr = &smmu->smrs[idx];
+			if (smr_is_subset(smr, id, mask)) {
+				arm_smmu_cb_write(smmu, cbndx, ARM_SMMU_CB_ACTLR,
+						actlrcfg[i].actlr);
+				break;
+			}
+		}
+	}
+}
+
 static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 		struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
 {
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+	struct qcom_smmu *qsmmu = to_qcom_smmu(smmu);
+	const struct actlr_variant *actlrvar;
+	int cbndx = smmu_domain->cfg.cbndx;
 	struct adreno_smmu_priv *priv;
+	int i;

 	smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;

@@ -248,6 +280,18 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 	priv->set_stall = qcom_adreno_smmu_set_stall;
 	priv->resume_translation = qcom_adreno_smmu_resume_translation;

+	actlrvar = qsmmu->data->actlrvar;
+	if (!actlrvar)
+		return 0;
+
+	for (i = 0; i < qsmmu->data->num_smmu ; i++) {
+		if (actlrvar[i].io_start == smmu->ioaddr) {
+			qcom_smmu_set_actlr(dev, smmu, cbndx, actlrvar[i].actlrcfg,
+				       actlrvar[i].num_actlrcfg);
+			break;
+		}
+	}
+
 	return 0;
 }

@@ -277,7 +321,24 @@ static const struct of_device_id qcom_smmu_client_of_match[] __maybe_unused = {
 static int qcom_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 		struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
 {
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+	struct qcom_smmu *qsmmu = to_qcom_smmu(smmu);
+	const struct actlr_variant *actlrvar;
+	int cbndx = smmu_domain->cfg.cbndx;
+	int i;
+
 	smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
+	actlrvar = qsmmu->data->actlrvar;
+	if (!actlrvar)
+		return 0;
+
+	for (i = 0; i < qsmmu->data->num_smmu ; i++) {
+		if (actlrvar[i].io_start == smmu->ioaddr) {
+			qcom_smmu_set_actlr(dev, smmu, cbndx, actlrvar[i].actlrcfg,
+				       actlrvar[i].num_actlrcfg);
+			break;
+		}
+	}

 	return 0;
 }
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
index addc07623c0b..c51817ff4674 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 /*
- * Copyright (c) 2022, Qualcomm Innovation Center, Inc. All rights reserved.
+ * Copyright (c) 2022-2023, Qualcomm Innovation Center, Inc. All rights reserved.
  */

 #ifndef _ARM_SMMU_QCOM_H
@@ -24,8 +24,22 @@ struct qcom_smmu_config {
 	const u32 *reg_offset;
 };

+struct actlr_config {
+	u16 sid;
+	u16 mask;
+	u32 actlr;
+};
+
+struct actlr_variant {
+	const resource_size_t io_start;
+	const struct actlr_config * const actlrcfg;
+	const size_t num_actlrcfg;
+};
+
 struct qcom_smmu_match_data {
+	const struct actlr_variant * const actlrvar;
 	const struct qcom_smmu_config *cfg;
+	const size_t num_smmu;
 	const struct arm_smmu_impl *impl;
 	const struct arm_smmu_impl *adreno_impl;
 };
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
index 87c81f75cf84..f43d417bf7f6 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
@@ -1003,9 +1003,10 @@ static int arm_smmu_find_sme(struct arm_smmu_device *smmu, u16 id, u16 mask)
 		 * expect simply identical entries for this case, but there's
 		 * no harm in accommodating the generalisation.
 		 */
-		if ((mask & smrs[i].mask) == mask &&
-		    !((id ^ smrs[i].id) & ~smrs[i].mask))
+
+		if (smr_is_subset(&smrs[i], id, mask))
 			return i;
+
 		/*
 		 * If the new entry has any other overlap with an existing one,
 		 * though, then there always exists at least one stream ID
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
index 4765c6945c34..d9c2ef8c1653 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
@@ -503,6 +503,11 @@ static inline void arm_smmu_writeq(struct arm_smmu_device *smmu, int page,
 		writeq_relaxed(val, arm_smmu_page(smmu, page) + offset);
 }

+static inline bool smr_is_subset(struct arm_smmu_smr *smrs, u16 id, u16 mask)
+{
+	return (mask & smrs->mask) == mask && !((id ^ smrs->id) & ~smrs->mask);
+}
+
 #define ARM_SMMU_GR0		0
 #define ARM_SMMU_GR1		1
 #define ARM_SMMU_CB(s, n)	((s)->numpage + (n))
--
2.34.1

[PATCH v13 4/6] iommu/arm-smmu: add ACTLR data and support for SM8550

[Bibek Kumar Patro]

Add ACTLR data table for SM8550 along with support for
same including SM8550 specific implementation operations.

Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 89 ++++++++++++++++++++++
 1 file changed, 89 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index 77c9abffe07d..b4521471ffe9 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -23,6 +23,85 @@

 #define CPRE			(1 << 1)
 #define CMTLB			(1 << 0)
+#define PREFETCH_SHIFT		8
+#define PREFETCH_DEFAULT	0
+#define PREFETCH_SHALLOW	(1 << PREFETCH_SHIFT)
+#define PREFETCH_MODERATE	(2 << PREFETCH_SHIFT)
+#define PREFETCH_DEEP		(3 << PREFETCH_SHIFT)
+
+static const struct actlr_config sm8550_apps_actlr_cfg[] = {
+	{ 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x0800, 0x0020, PREFETCH_DEFAULT | CMTLB },
+	{ 0x1800, 0x00c0, PREFETCH_DEFAULT | CMTLB },
+	{ 0x1820, 0x0000, PREFETCH_DEFAULT | CMTLB },
+	{ 0x1860, 0x0000, PREFETCH_DEFAULT | CMTLB },
+	{ 0x0c01, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c02, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c03, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c04, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c05, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c06, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c07, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c08, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c09, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c0c, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c0d, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c0e, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x0c0f, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1961, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1962, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1963, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1964, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1965, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1966, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1967, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1968, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1969, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x196c, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x196d, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x196e, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x196f, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c1, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c2, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c3, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c4, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c5, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c6, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c7, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c8, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19c9, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19cc, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19cd, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19ce, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x19cf, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1c00, 0x0002, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1c01, 0x0000, PREFETCH_DEFAULT | CMTLB },
+	{ 0x1920, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1923, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1924, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1940, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1941, 0x0004, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1943, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1944, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1947, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+};
+
+static const struct actlr_config sm8550_gfx_actlr_cfg[] = {
+	{ 0x0000, 0x03ff, PREFETCH_DEEP | CPRE | CMTLB },
+};
+
+static const struct actlr_variant sm8550_actlr[] = {
+	{
+		.io_start = 0x15000000,
+		.actlrcfg = sm8550_apps_actlr_cfg,
+		.num_actlrcfg = ARRAY_SIZE(sm8550_apps_actlr_cfg)
+	}, {
+		.io_start = 0x03da0000,
+		.actlrcfg = sm8550_gfx_actlr_cfg,
+		.num_actlrcfg = ARRAY_SIZE(sm8550_gfx_actlr_cfg)
+	},
+};

 static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
 {
@@ -606,6 +685,15 @@ static const struct qcom_smmu_match_data sdm845_smmu_500_data = {
 	/* Also no debug configuration. */
 };

+
+static const struct qcom_smmu_match_data sm8550_smmu_500_impl0_data = {
+	.impl = &qcom_smmu_500_impl,
+	.adreno_impl = &qcom_adreno_smmu_500_impl,
+	.cfg = &qcom_smmu_impl0_cfg,
+	.actlrvar = sm8550_actlr,
+	.num_smmu = ARRAY_SIZE(sm8550_actlr),
+};
+
 static const struct qcom_smmu_match_data qcom_smmu_500_impl0_data = {
 	.impl = &qcom_smmu_500_impl,
 	.adreno_impl = &qcom_adreno_smmu_500_impl,
@@ -640,6 +728,7 @@ static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
 	{ .compatible = "qcom,sm8250-smmu-500", .data = &qcom_smmu_500_impl0_data },
 	{ .compatible = "qcom,sm8350-smmu-500", .data = &qcom_smmu_500_impl0_data },
 	{ .compatible = "qcom,sm8450-smmu-500", .data = &qcom_smmu_500_impl0_data },
+	{ .compatible = "qcom,sm8550-smmu-500", .data = &sm8550_smmu_500_impl0_data },
 	{ .compatible = "qcom,smmu-500", .data = &qcom_smmu_500_impl0_data },
 	{ }
 };
--
2.34.1

[PATCH v13 5/6] iommu/arm-smmu: add ACTLR data and support for SC7280

[Bibek Kumar Patro]

Add ACTLR data table for SC7280 along with support for
same including SC7280 specific implementation operations.

Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 58 +++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index b4521471ffe9..bd101a161d04 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -29,6 +29,55 @@
 #define PREFETCH_MODERATE	(2 << PREFETCH_SHIFT)
 #define PREFETCH_DEEP		(3 << PREFETCH_SHIFT)

+static const struct actlr_config sc7280_apps_actlr_cfg[] = {
+	{ 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
+	{ 0x2040, 0x0000, PREFETCH_DEFAULT | CMTLB },
+	{ 0x2000, 0x0020, PREFETCH_DEFAULT | CMTLB },
+	{ 0x2062, 0x0000, PREFETCH_DEFAULT | CMTLB },
+	{ 0x2080, 0x0020, PREFETCH_DEFAULT | CMTLB },
+	{ 0x20c0, 0x0020, PREFETCH_DEFAULT | CMTLB },
+	{ 0x2100, 0x0020, PREFETCH_DEFAULT | CMTLB },
+	{ 0x2140, 0x0000, PREFETCH_DEFAULT | CMTLB },
+	{ 0x0900, 0x0402, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x0901, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x0d01, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x1181, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1182, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1183, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1184, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1185, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1186, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1187, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1188, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x1189, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x118b, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x118c, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x118d, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x118e, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x118f, 0x0420, PREFETCH_DEEP | CPRE | CMTLB },
+	{ 0x2180, 0x0020, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x2181, 0x0004, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x2183, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x2184, 0x0020, PREFETCH_SHALLOW | CPRE | CMTLB },
+	{ 0x2187, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
+};
+
+static const struct actlr_config sc7280_gfx_actlr_cfg[] = {
+	{ 0x0000, 0x07ff, PREFETCH_DEEP | CPRE | CMTLB },
+};
+
+static const struct actlr_variant sc7280_actlr[] = {
+	{
+		.io_start = 0x15000000,
+		.actlrcfg = sc7280_apps_actlr_cfg,
+		.num_actlrcfg = ARRAY_SIZE(sc7280_apps_actlr_cfg)
+	}, {
+		.io_start = 0x03da0000,
+		.actlrcfg = sc7280_gfx_actlr_cfg,
+		.num_actlrcfg = ARRAY_SIZE(sc7280_gfx_actlr_cfg)
+	},
+};
+
 static const struct actlr_config sm8550_apps_actlr_cfg[] = {
 	{ 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
 	{ 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
@@ -685,6 +734,13 @@ static const struct qcom_smmu_match_data sdm845_smmu_500_data = {
 	/* Also no debug configuration. */
 };

+static const struct qcom_smmu_match_data sc7280_smmu_500_impl0_data = {
+	.impl = &qcom_smmu_500_impl,
+	.adreno_impl = &qcom_adreno_smmu_500_impl,
+	.cfg = &qcom_smmu_impl0_cfg,
+	.actlrvar = sc7280_actlr,
+	.num_smmu = ARRAY_SIZE(sc7280_actlr),
+};

 static const struct qcom_smmu_match_data sm8550_smmu_500_impl0_data = {
 	.impl = &qcom_smmu_500_impl,
@@ -711,7 +767,7 @@ static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
 	{ .compatible = "qcom,qdu1000-smmu-500", .data = &qcom_smmu_500_impl0_data  },
 	{ .compatible = "qcom,sc7180-smmu-500", .data = &qcom_smmu_500_impl0_data },
 	{ .compatible = "qcom,sc7180-smmu-v2", .data = &qcom_smmu_v2_data },
-	{ .compatible = "qcom,sc7280-smmu-500", .data = &qcom_smmu_500_impl0_data },
+	{ .compatible = "qcom,sc7280-smmu-500", .data = &sc7280_smmu_500_impl0_data },
 	{ .compatible = "qcom,sc8180x-smmu-500", .data = &qcom_smmu_500_impl0_data },
 	{ .compatible = "qcom,sc8280xp-smmu-500", .data = &qcom_smmu_500_impl0_data },
 	{ .compatible = "qcom,sdm630-smmu-v2", .data = &qcom_smmu_v2_data },
--
2.34.1

[PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

Add an adreno-smmu-priv interface for drm/msm to call
into arm-smmu-qcom and initiate the PRR bit setup or reset
sequence as per request.

This will be used by GPU to setup the PRR bit and related
configuration registers through adreno-smmu private
interface instead of directly poking the smmu hardware.

Suggested-by: Rob Clark <robdclark@gmail.com>
Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
 drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
 include/linux/adreno-smmu-priv.h           |  6 +++++-
 3 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index bd101a161d04..64571a1c47b8 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -28,6 +28,7 @@
 #define PREFETCH_SHALLOW	(1 << PREFETCH_SHIFT)
 #define PREFETCH_MODERATE	(2 << PREFETCH_SHIFT)
 #define PREFETCH_DEEP		(3 << PREFETCH_SHIFT)
+#define GFX_ACTLR_PRR		(1 << 5)

 static const struct actlr_config sc7280_apps_actlr_cfg[] = {
 	{ 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
@@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
 	arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
 }

+static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
+{
+	struct arm_smmu_domain *smmu_domain = (void *)cookie;
+	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+	u32 reg = 0;
+
+	writel_relaxed(lower_32_bits(page_addr),
+				smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
+
+	writel_relaxed(upper_32_bits(page_addr),
+				smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
+
+	reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
+	reg &= ~GFX_ACTLR_PRR;
+	if (set)
+		reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
+	arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
+
+}
+
 #define QCOM_ADRENO_SMMU_GPU_SID 0

 static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
@@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 	priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
 	priv->set_stall = qcom_adreno_smmu_set_stall;
 	priv->resume_translation = qcom_adreno_smmu_resume_translation;
+	priv->set_prr = qcom_adreno_smmu_set_prr;

 	actlrvar = qsmmu->data->actlrvar;
 	if (!actlrvar)
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
index d9c2ef8c1653..3076bef49e20 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
@@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
 #define ARM_SMMU_SCTLR_M		BIT(0)

 #define ARM_SMMU_CB_ACTLR		0x4
+#define ARM_SMMU_GFX_PRR_CFG_LADDR	0x6008
+#define ARM_SMMU_GFX_PRR_CFG_UADDR	0x600C

 #define ARM_SMMU_CB_RESUME		0x8
 #define ARM_SMMU_RESUME_TERMINATE	BIT(0)
diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
index c637e0997f6d..d6e2ca9f8d8c 100644
--- a/include/linux/adreno-smmu-priv.h
+++ b/include/linux/adreno-smmu-priv.h
@@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
  *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
  *                 the GPU driver must call resume_translation()
  * @resume_translation: Resume translation after a fault
- *
+ * @set_prr:	   Extendible interface to be used by GPU to modify the
+ *		    ACTLR register bits, currently used to configure
+ *		    Partially-Resident-Region (PRR) feature's
+ *		    setup and reset sequence as requested.
  *
  * The GPU driver (drm/msm) and adreno-smmu work together for controlling
  * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
@@ -67,6 +70,7 @@ struct adreno_smmu_priv {
     void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
     void (*set_stall)(const void *cookie, bool enabled);
     void (*resume_translation)(const void *cookie, bool terminate);
+    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
 };

 #endif /* __ADRENO_SMMU_PRIV_H */
--
2.34.1

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
> Add an adreno-smmu-priv interface for drm/msm to call
> into arm-smmu-qcom and initiate the PRR bit setup or reset
> sequence as per request.
>
> This will be used by GPU to setup the PRR bit and related
> configuration registers through adreno-smmu private
> interface instead of directly poking the smmu hardware.
>
> Suggested-by: Rob Clark <robdclark@gmail.com>
> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> ---
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
>  drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>  include/linux/adreno-smmu-priv.h           |  6 +++++-
>  3 files changed, 30 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> index bd101a161d04..64571a1c47b8 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> @@ -28,6 +28,7 @@
>  #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
>  #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
>  #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
> +#define GFX_ACTLR_PRR          (1 << 5)
>
>  static const struct actlr_config sc7280_apps_actlr_cfg[] = {
>         { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>         arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>  }
>
> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
> +{
> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> +       u32 reg = 0;
> +
> +       writel_relaxed(lower_32_bits(page_addr),
> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> +
> +       writel_relaxed(upper_32_bits(page_addr),
> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> +
> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> +       reg &= ~GFX_ACTLR_PRR;
> +       if (set)
> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> +

nit, extra line

Also, if you passed a `struct page *` instead, then you could drop the
bool param, ie. passing NULL for the page would disable PRR.  But I
can go either way if others have a strong preference for phys_addr_t.

Otherwise, lgtm

BR,
-R

> +}
> +
>  #define QCOM_ADRENO_SMMU_GPU_SID 0
>
>  static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>         priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>         priv->set_stall = qcom_adreno_smmu_set_stall;
>         priv->resume_translation = qcom_adreno_smmu_resume_translation;
> +       priv->set_prr = qcom_adreno_smmu_set_prr;
>
>         actlrvar = qsmmu->data->actlrvar;
>         if (!actlrvar)
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> index d9c2ef8c1653..3076bef49e20 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>  #define ARM_SMMU_SCTLR_M               BIT(0)
>
>  #define ARM_SMMU_CB_ACTLR              0x4
> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>
>  #define ARM_SMMU_CB_RESUME             0x8
>  #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> index c637e0997f6d..d6e2ca9f8d8c 100644
> --- a/include/linux/adreno-smmu-priv.h
> +++ b/include/linux/adreno-smmu-priv.h
> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
>   *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
>   *                 the GPU driver must call resume_translation()
>   * @resume_translation: Resume translation after a fault
> - *
> + * @set_prr:      Extendible interface to be used by GPU to modify the
> + *                 ACTLR register bits, currently used to configure
> + *                 Partially-Resident-Region (PRR) feature's
> + *                 setup and reset sequence as requested.
>   *
>   * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>   * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
>      void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>      void (*set_stall)(const void *cookie, bool enabled);
>      void (*resume_translation)(const void *cookie, bool terminate);
> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
>  };
>
>  #endif /* __ADRENO_SMMU_PRIV_H */
> --
> 2.34.1
>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 6/28/2024 7:47 PM, Rob Clark wrote:
> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>> Add an adreno-smmu-priv interface for drm/msm to call
>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>> sequence as per request.
>>
>> This will be used by GPU to setup the PRR bit and related
>> configuration registers through adreno-smmu private
>> interface instead of directly poking the smmu hardware.
>>
>> Suggested-by: Rob Clark <robdclark@gmail.com>
>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>> ---
>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
>>   drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>   include/linux/adreno-smmu-priv.h           |  6 +++++-
>>   3 files changed, 30 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> index bd101a161d04..64571a1c47b8 100644
>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> @@ -28,6 +28,7 @@
>>   #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
>>   #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
>>   #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
>> +#define GFX_ACTLR_PRR          (1 << 5)
>>
>>   static const struct actlr_config sc7280_apps_actlr_cfg[] = {
>>          { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>          arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>   }
>>
>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
>> +{
>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>> +       u32 reg = 0;
>> +
>> +       writel_relaxed(lower_32_bits(page_addr),
>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>> +
>> +       writel_relaxed(upper_32_bits(page_addr),
>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>> +
>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>> +       reg &= ~GFX_ACTLR_PRR;
>> +       if (set)
>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>> +
> 
> nit, extra line
> 

Ack, will remove this. Thanks for pointing out.

> Also, if you passed a `struct page *` instead, then you could drop the
> bool param, ie. passing NULL for the page would disable PRR.  But I
> can go either way if others have a strong preference for phys_addr_t.
> 

Oh okay, this looks simple to reset the prr bit.
But since this page is allocated and is used inside gfx driver
before being utilized for prr bit operation, would it be safe for
drm/gfx driver to keep a reference to this page in smmu driver?

Since we only need the page address for configuring the
CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.

> Otherwise, lgtm
> 
> BR,
> -R
>

Thanks & regards,
Bibek

>> +}
>> +
>>   #define QCOM_ADRENO_SMMU_GPU_SID 0
>>
>>   static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>          priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>          priv->set_stall = qcom_adreno_smmu_set_stall;
>>          priv->resume_translation = qcom_adreno_smmu_resume_translation;
>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
>>
>>          actlrvar = qsmmu->data->actlrvar;
>>          if (!actlrvar)
>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>> index d9c2ef8c1653..3076bef49e20 100644
>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>   #define ARM_SMMU_SCTLR_M               BIT(0)
>>
>>   #define ARM_SMMU_CB_ACTLR              0x4
>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>
>>   #define ARM_SMMU_CB_RESUME             0x8
>>   #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>> index c637e0997f6d..d6e2ca9f8d8c 100644
>> --- a/include/linux/adreno-smmu-priv.h
>> +++ b/include/linux/adreno-smmu-priv.h
>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
>>    *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
>>    *                 the GPU driver must call resume_translation()
>>    * @resume_translation: Resume translation after a fault
>> - *
>> + * @set_prr:      Extendible interface to be used by GPU to modify the
>> + *                 ACTLR register bits, currently used to configure
>> + *                 Partially-Resident-Region (PRR) feature's
>> + *                 setup and reset sequence as requested.
>>    *
>>    * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>    * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
>>       void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>       void (*set_stall)(const void *cookie, bool enabled);
>>       void (*resume_translation)(const void *cookie, bool terminate);
>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
>>   };
>>
>>   #endif /* __ADRENO_SMMU_PRIV_H */
>> --
>> 2.34.1
>>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 6/28/2024 7:47 PM, Rob Clark wrote:
> > On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >> Add an adreno-smmu-priv interface for drm/msm to call
> >> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >> sequence as per request.
> >>
> >> This will be used by GPU to setup the PRR bit and related
> >> configuration registers through adreno-smmu private
> >> interface instead of directly poking the smmu hardware.
> >>
> >> Suggested-by: Rob Clark <robdclark@gmail.com>
> >> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >> ---
> >>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
> >>   drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>   include/linux/adreno-smmu-priv.h           |  6 +++++-
> >>   3 files changed, 30 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> index bd101a161d04..64571a1c47b8 100644
> >> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> @@ -28,6 +28,7 @@
> >>   #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
> >>   #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
> >>   #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
> >> +#define GFX_ACTLR_PRR          (1 << 5)
> >>
> >>   static const struct actlr_config sc7280_apps_actlr_cfg[] = {
> >>          { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
> >> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>          arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>   }
> >>
> >> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
> >> +{
> >> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >> +       u32 reg = 0;
> >> +
> >> +       writel_relaxed(lower_32_bits(page_addr),
> >> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >> +
> >> +       writel_relaxed(upper_32_bits(page_addr),
> >> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >> +
> >> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >> +       reg &= ~GFX_ACTLR_PRR;
> >> +       if (set)
> >> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >> +
> >
> > nit, extra line
> >
>
> Ack, will remove this. Thanks for pointing out.
>
> > Also, if you passed a `struct page *` instead, then you could drop the
> > bool param, ie. passing NULL for the page would disable PRR.  But I
> > can go either way if others have a strong preference for phys_addr_t.
> >
>
> Oh okay, this looks simple to reset the prr bit.
> But since this page is allocated and is used inside gfx driver
> before being utilized for prr bit operation, would it be safe for
> drm/gfx driver to keep a reference to this page in smmu driver?
>
> Since we only need the page address for configuring the
> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.

I don't think the smmu driver needs to keep a reference to the page..
we can just say it is the responsibility of the drm driver to call
set_prr(NULL) before freeing the page

BR,
-R

> > Otherwise, lgtm
> >
> > BR,
> > -R
> >
>
> Thanks & regards,
> Bibek
>
> >> +}
> >> +
> >>   #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>
> >>   static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>          priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>          priv->set_stall = qcom_adreno_smmu_set_stall;
> >>          priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >> +       priv->set_prr = qcom_adreno_smmu_set_prr;
> >>
> >>          actlrvar = qsmmu->data->actlrvar;
> >>          if (!actlrvar)
> >> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >> index d9c2ef8c1653..3076bef49e20 100644
> >> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>   #define ARM_SMMU_SCTLR_M               BIT(0)
> >>
> >>   #define ARM_SMMU_CB_ACTLR              0x4
> >> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>
> >>   #define ARM_SMMU_CB_RESUME             0x8
> >>   #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >> index c637e0997f6d..d6e2ca9f8d8c 100644
> >> --- a/include/linux/adreno-smmu-priv.h
> >> +++ b/include/linux/adreno-smmu-priv.h
> >> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
> >>    *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
> >>    *                 the GPU driver must call resume_translation()
> >>    * @resume_translation: Resume translation after a fault
> >> - *
> >> + * @set_prr:      Extendible interface to be used by GPU to modify the
> >> + *                 ACTLR register bits, currently used to configure
> >> + *                 Partially-Resident-Region (PRR) feature's
> >> + *                 setup and reset sequence as requested.
> >>    *
> >>    * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>    * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
> >>       void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>       void (*set_stall)(const void *cookie, bool enabled);
> >>       void (*resume_translation)(const void *cookie, bool terminate);
> >> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
> >>   };
> >>
> >>   #endif /* __ADRENO_SMMU_PRIV_H */
> >> --
> >> 2.34.1
> >>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 6/28/2024 9:14 PM, Rob Clark wrote:
> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 6/28/2024 7:47 PM, Rob Clark wrote:
>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
>>> <quic_bibekkum@quicinc.com> wrote:
>>>>
>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>> sequence as per request.
>>>>
>>>> This will be used by GPU to setup the PRR bit and related
>>>> configuration registers through adreno-smmu private
>>>> interface instead of directly poking the smmu hardware.
>>>>
>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>> ---
>>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
>>>>    drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>    include/linux/adreno-smmu-priv.h           |  6 +++++-
>>>>    3 files changed, 30 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> index bd101a161d04..64571a1c47b8 100644
>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> @@ -28,6 +28,7 @@
>>>>    #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
>>>>    #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
>>>>    #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>
>>>>    static const struct actlr_config sc7280_apps_actlr_cfg[] = {
>>>>           { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>           arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>    }
>>>>
>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
>>>> +{
>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>> +       u32 reg = 0;
>>>> +
>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>> +
>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>> +
>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>> +       if (set)
>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>> +
>>>
>>> nit, extra line
>>>
>>
>> Ack, will remove this. Thanks for pointing out.
>>
>>> Also, if you passed a `struct page *` instead, then you could drop the
>>> bool param, ie. passing NULL for the page would disable PRR.  But I
>>> can go either way if others have a strong preference for phys_addr_t.
>>>
>>
>> Oh okay, this looks simple to reset the prr bit.
>> But since this page is allocated and is used inside gfx driver
>> before being utilized for prr bit operation, would it be safe for
>> drm/gfx driver to keep a reference to this page in smmu driver?
>>
>> Since we only need the page address for configuring the
>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
> 
> I don't think the smmu driver needs to keep a reference to the page..
> we can just say it is the responsibility of the drm driver to call
> set_prr(NULL) before freeing the page
> 

That makes sense. If we go by this NULL page method to disable the PRR, 
we would have to set the address registers to reset value as well.

The sequence would be like the following as per my understaning:
- Check if it's NULL page
- Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
   these registers
- Reset the PRR bit in actlr register

Similar to this snippet:

#PRR_RESET_ADDR 0x0

--------------
reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
reg &= ~GFX_ACTLR_PRR;
arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);

if (!prr_page) {
	writel_relaxed(PRR_RESET_ADDR,
			smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
	writel_relaxed(PRR_RESET_ADDR),
                         smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
	return;
}


writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
  		smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);

writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
  		smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);

reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
-----------------

If looks good, will implement the same in next version.

Thanks & regards,
Bibek

> BR,
> -R
> 
>>> Otherwise, lgtm
>>>
>>> BR,
>>> -R
>>>
>>
>> Thanks & regards,
>> Bibek
>>
>>>> +}
>>>> +
>>>>    #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>
>>>>    static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>           priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>           priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>           priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
>>>>
>>>>           actlrvar = qsmmu->data->actlrvar;
>>>>           if (!actlrvar)
>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>> index d9c2ef8c1653..3076bef49e20 100644
>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>    #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>
>>>>    #define ARM_SMMU_CB_ACTLR              0x4
>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>
>>>>    #define ARM_SMMU_CB_RESUME             0x8
>>>>    #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
>>>> --- a/include/linux/adreno-smmu-priv.h
>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
>>>>     *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
>>>>     *                 the GPU driver must call resume_translation()
>>>>     * @resume_translation: Resume translation after a fault
>>>> - *
>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
>>>> + *                 ACTLR register bits, currently used to configure
>>>> + *                 Partially-Resident-Region (PRR) feature's
>>>> + *                 setup and reset sequence as requested.
>>>>     *
>>>>     * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>     * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
>>>>        void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>        void (*set_stall)(const void *cookie, bool enabled);
>>>>        void (*resume_translation)(const void *cookie, bool terminate);
>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
>>>>    };
>>>>
>>>>    #endif /* __ADRENO_SMMU_PRIV_H */
>>>> --
>>>> 2.34.1
>>>>

Re: [PATCH v13 4/6] iommu/arm-smmu: add ACTLR data and support for SM8550

[Dmitry Baryshkov]

On Fri, Jun 28, 2024 at 07:34:33PM GMT, Bibek Kumar Patro wrote:
> Add ACTLR data table for SM8550 along with support for
> same including SM8550 specific implementation operations.
> 
> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> ---
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 89 ++++++++++++++++++++++
>  1 file changed, 89 insertions(+)
> 
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> index 77c9abffe07d..b4521471ffe9 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> @@ -23,6 +23,85 @@
> 
>  #define CPRE			(1 << 1)
>  #define CMTLB			(1 << 0)
> +#define PREFETCH_SHIFT		8
> +#define PREFETCH_DEFAULT	0
> +#define PREFETCH_SHALLOW	(1 << PREFETCH_SHIFT)
> +#define PREFETCH_MODERATE	(2 << PREFETCH_SHIFT)
> +#define PREFETCH_DEEP		(3 << PREFETCH_SHIFT)
> +
> +static const struct actlr_config sm8550_apps_actlr_cfg[] = {
> +	{ 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x0800, 0x0020, PREFETCH_DEFAULT | CMTLB },
> +	{ 0x1800, 0x00c0, PREFETCH_DEFAULT | CMTLB },
> +	{ 0x1820, 0x0000, PREFETCH_DEFAULT | CMTLB },
> +	{ 0x1860, 0x0000, PREFETCH_DEFAULT | CMTLB },
> +	{ 0x0c01, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },

- Please keep the list sorted
- Please comment, which devices use these settings.

> +	{ 0x0c02, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c03, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c04, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c05, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c06, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c07, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c08, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c09, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c0c, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c0d, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c0e, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x0c0f, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1961, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1962, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1963, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1964, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1965, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1966, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1967, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1968, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1969, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x196c, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x196d, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x196e, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x196f, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c1, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c2, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c3, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c4, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c5, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c6, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c7, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c8, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19c9, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19cc, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19cd, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19ce, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x19cf, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> +	{ 0x1c00, 0x0002, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x1c01, 0x0000, PREFETCH_DEFAULT | CMTLB },
> +	{ 0x1920, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x1923, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x1924, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x1940, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x1941, 0x0004, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x1943, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x1944, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +	{ 0x1947, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> +};
> +
> +static const struct actlr_config sm8550_gfx_actlr_cfg[] = {
> +	{ 0x0000, 0x03ff, PREFETCH_DEEP | CPRE | CMTLB },
> +};
> +
> +static const struct actlr_variant sm8550_actlr[] = {
> +	{
> +		.io_start = 0x15000000,
> +		.actlrcfg = sm8550_apps_actlr_cfg,
> +		.num_actlrcfg = ARRAY_SIZE(sm8550_apps_actlr_cfg)
> +	}, {
> +		.io_start = 0x03da0000,
> +		.actlrcfg = sm8550_gfx_actlr_cfg,
> +		.num_actlrcfg = ARRAY_SIZE(sm8550_gfx_actlr_cfg)
> +	},
> +};
> 
>  static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>  {
> @@ -606,6 +685,15 @@ static const struct qcom_smmu_match_data sdm845_smmu_500_data = {
>  	/* Also no debug configuration. */
>  };
> 
> +
> +static const struct qcom_smmu_match_data sm8550_smmu_500_impl0_data = {
> +	.impl = &qcom_smmu_500_impl,
> +	.adreno_impl = &qcom_adreno_smmu_500_impl,
> +	.cfg = &qcom_smmu_impl0_cfg,
> +	.actlrvar = sm8550_actlr,
> +	.num_smmu = ARRAY_SIZE(sm8550_actlr),
> +};
> +
>  static const struct qcom_smmu_match_data qcom_smmu_500_impl0_data = {
>  	.impl = &qcom_smmu_500_impl,
>  	.adreno_impl = &qcom_adreno_smmu_500_impl,
> @@ -640,6 +728,7 @@ static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
>  	{ .compatible = "qcom,sm8250-smmu-500", .data = &qcom_smmu_500_impl0_data },
>  	{ .compatible = "qcom,sm8350-smmu-500", .data = &qcom_smmu_500_impl0_data },
>  	{ .compatible = "qcom,sm8450-smmu-500", .data = &qcom_smmu_500_impl0_data },
> +	{ .compatible = "qcom,sm8550-smmu-500", .data = &sm8550_smmu_500_impl0_data },
>  	{ .compatible = "qcom,smmu-500", .data = &qcom_smmu_500_impl0_data },
>  	{ }
>  };
> --
> 2.34.1
> 

-- 
With best wishes
Dmitry

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 6/28/2024 9:14 PM, Rob Clark wrote:
> > On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >>
> >>
> >> On 6/28/2024 7:47 PM, Rob Clark wrote:
> >>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
> >>> <quic_bibekkum@quicinc.com> wrote:
> >>>>
> >>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>> sequence as per request.
> >>>>
> >>>> This will be used by GPU to setup the PRR bit and related
> >>>> configuration registers through adreno-smmu private
> >>>> interface instead of directly poking the smmu hardware.
> >>>>
> >>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>> ---
> >>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
> >>>>    drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>    include/linux/adreno-smmu-priv.h           |  6 +++++-
> >>>>    3 files changed, 30 insertions(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> index bd101a161d04..64571a1c47b8 100644
> >>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> @@ -28,6 +28,7 @@
> >>>>    #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
> >>>>    #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
> >>>>    #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
> >>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>>
> >>>>    static const struct actlr_config sc7280_apps_actlr_cfg[] = {
> >>>>           { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
> >>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>           arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>    }
> >>>>
> >>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
> >>>> +{
> >>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>> +       u32 reg = 0;
> >>>> +
> >>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>> +
> >>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>> +
> >>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>> +       if (set)
> >>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>> +
> >>>
> >>> nit, extra line
> >>>
> >>
> >> Ack, will remove this. Thanks for pointing out.
> >>
> >>> Also, if you passed a `struct page *` instead, then you could drop the
> >>> bool param, ie. passing NULL for the page would disable PRR.  But I
> >>> can go either way if others have a strong preference for phys_addr_t.
> >>>
> >>
> >> Oh okay, this looks simple to reset the prr bit.
> >> But since this page is allocated and is used inside gfx driver
> >> before being utilized for prr bit operation, would it be safe for
> >> drm/gfx driver to keep a reference to this page in smmu driver?
> >>
> >> Since we only need the page address for configuring the
> >> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
> >
> > I don't think the smmu driver needs to keep a reference to the page..
> > we can just say it is the responsibility of the drm driver to call
> > set_prr(NULL) before freeing the page
> >
>
> That makes sense. If we go by this NULL page method to disable the PRR,
> we would have to set the address registers to reset value as well.
>
> The sequence would be like the following as per my understaning:
> - Check if it's NULL page
> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
>    these registers
> - Reset the PRR bit in actlr register
>
> Similar to this snippet:
>
> #PRR_RESET_ADDR 0x0
>
> --------------
> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> reg &= ~GFX_ACTLR_PRR;
> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>
> if (!prr_page) {
>         writel_relaxed(PRR_RESET_ADDR,
>                         smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>         writel_relaxed(PRR_RESET_ADDR),
>                          smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>         return;
> }
>
>
> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
>                 smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>
> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
>                 smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>
> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> -----------------
>
> If looks good, will implement the same in next version.

yeah, that looks like it could work..

you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
and probably could avoid double writing ACTLR, but that is getting
into bikeshedding

BR,
-R

>
> Thanks & regards,
> Bibek
>
> > BR,
> > -R
> >
> >>> Otherwise, lgtm
> >>>
> >>> BR,
> >>> -R
> >>>
> >>
> >> Thanks & regards,
> >> Bibek
> >>
> >>>> +}
> >>>> +
> >>>>    #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>>
> >>>>    static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>           priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>>           priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>>           priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
> >>>>
> >>>>           actlrvar = qsmmu->data->actlrvar;
> >>>>           if (!actlrvar)
> >>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>> index d9c2ef8c1653..3076bef49e20 100644
> >>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>>>    #define ARM_SMMU_SCTLR_M               BIT(0)
> >>>>
> >>>>    #define ARM_SMMU_CB_ACTLR              0x4
> >>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>>>
> >>>>    #define ARM_SMMU_CB_RESUME             0x8
> >>>>    #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >>>> index c637e0997f6d..d6e2ca9f8d8c 100644
> >>>> --- a/include/linux/adreno-smmu-priv.h
> >>>> +++ b/include/linux/adreno-smmu-priv.h
> >>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
> >>>>     *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
> >>>>     *                 the GPU driver must call resume_translation()
> >>>>     * @resume_translation: Resume translation after a fault
> >>>> - *
> >>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
> >>>> + *                 ACTLR register bits, currently used to configure
> >>>> + *                 Partially-Resident-Region (PRR) feature's
> >>>> + *                 setup and reset sequence as requested.
> >>>>     *
> >>>>     * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>>>     * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
> >>>>        void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>>>        void (*set_stall)(const void *cookie, bool enabled);
> >>>>        void (*resume_translation)(const void *cookie, bool terminate);
> >>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
> >>>>    };
> >>>>
> >>>>    #endif /* __ADRENO_SMMU_PRIV_H */
> >>>> --
> >>>> 2.34.1
> >>>>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 7/2/2024 2:01 AM, Rob Clark wrote:
> On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 6/28/2024 9:14 PM, Rob Clark wrote:
>>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
>>> <quic_bibekkum@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
>>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>
>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>>> sequence as per request.
>>>>>>
>>>>>> This will be used by GPU to setup the PRR bit and related
>>>>>> configuration registers through adreno-smmu private
>>>>>> interface instead of directly poking the smmu hardware.
>>>>>>
>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>>> ---
>>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
>>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>>     include/linux/adreno-smmu-priv.h           |  6 +++++-
>>>>>>     3 files changed, 30 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>> index bd101a161d04..64571a1c47b8 100644
>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>> @@ -28,6 +28,7 @@
>>>>>>     #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
>>>>>>     #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
>>>>>>     #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>>>
>>>>>>     static const struct actlr_config sc7280_apps_actlr_cfg[] = {
>>>>>>            { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
>>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>>            arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>>     }
>>>>>>
>>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
>>>>>> +{
>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>> +       u32 reg = 0;
>>>>>> +
>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>> +
>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>> +
>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>>> +       if (set)
>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>> +
>>>>>
>>>>> nit, extra line
>>>>>
>>>>
>>>> Ack, will remove this. Thanks for pointing out.
>>>>
>>>>> Also, if you passed a `struct page *` instead, then you could drop the
>>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
>>>>> can go either way if others have a strong preference for phys_addr_t.
>>>>>
>>>>
>>>> Oh okay, this looks simple to reset the prr bit.
>>>> But since this page is allocated and is used inside gfx driver
>>>> before being utilized for prr bit operation, would it be safe for
>>>> drm/gfx driver to keep a reference to this page in smmu driver?
>>>>
>>>> Since we only need the page address for configuring the
>>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
>>>
>>> I don't think the smmu driver needs to keep a reference to the page..
>>> we can just say it is the responsibility of the drm driver to call
>>> set_prr(NULL) before freeing the page
>>>
>>
>> That makes sense. If we go by this NULL page method to disable the PRR,
>> we would have to set the address registers to reset value as well.
>>
>> The sequence would be like the following as per my understaning:
>> - Check if it's NULL page
>> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
>>     these registers
>> - Reset the PRR bit in actlr register
>>
>> Similar to this snippet:
>>
>> #PRR_RESET_ADDR 0x0
>>
>> --------------
>> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>> reg &= ~GFX_ACTLR_PRR;
>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>
>> if (!prr_page) {
>>          writel_relaxed(PRR_RESET_ADDR,
>>                          smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>          writel_relaxed(PRR_RESET_ADDR),
>>                           smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>          return;
>> }
>>
>>
>> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
>>                  smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>
>> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
>>                  smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>
>> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>> -----------------
>>
>> If looks good, will implement the same in next version.
> 
> yeah, that looks like it could work..
> 
> you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
> and probably could avoid double writing ACTLR, but that is getting
> into bikeshedding
> 

Actually Rob, since you rightly pointed this out.
I crosschecked again on these registers.
PRR_CFG_*ADDR is a global register in SMMU space but
ACTLR register including PRR bit is a per-domain register.
There might also be a situation where PRR feature need to be
disabled or enabled separately for each domain.
So I think it would be cleaner to have two apis, set_prr_addr(),
set_prr_bit().
set_prr_addr() will be used only to set this PRR_CFG_*ADDR
register by passing a 'struct page *'
set_prr_bit() will be used as a switch for PRR feature,
where required smmu_domain will be passed along with
the bool value to set/reset the PRR bit depending on which this
feature will be enabled/disabled for the selected domain.

Thanks & regards,
Bibek

> BR,
> -R
> 
>>
>> Thanks & regards,
>> Bibek
>>
>>> BR,
>>> -R
>>>
>>>>> Otherwise, lgtm
>>>>>
>>>>> BR,
>>>>> -R
>>>>>
>>>>
>>>> Thanks & regards,
>>>> Bibek
>>>>
>>>>>> +}
>>>>>> +
>>>>>>     #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>>>
>>>>>>     static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>            priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>>>            priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>>>            priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
>>>>>>
>>>>>>            actlrvar = qsmmu->data->actlrvar;
>>>>>>            if (!actlrvar)
>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>> index d9c2ef8c1653..3076bef49e20 100644
>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>>>     #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>>>
>>>>>>     #define ARM_SMMU_CB_ACTLR              0x4
>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>>>
>>>>>>     #define ARM_SMMU_CB_RESUME             0x8
>>>>>>     #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
>>>>>> --- a/include/linux/adreno-smmu-priv.h
>>>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
>>>>>>      *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
>>>>>>      *                 the GPU driver must call resume_translation()
>>>>>>      * @resume_translation: Resume translation after a fault
>>>>>> - *
>>>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
>>>>>> + *                 ACTLR register bits, currently used to configure
>>>>>> + *                 Partially-Resident-Region (PRR) feature's
>>>>>> + *                 setup and reset sequence as requested.
>>>>>>      *
>>>>>>      * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>>>      * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
>>>>>>         void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>>>         void (*set_stall)(const void *cookie, bool enabled);
>>>>>>         void (*resume_translation)(const void *cookie, bool terminate);
>>>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
>>>>>>     };
>>>>>>
>>>>>>     #endif /* __ADRENO_SMMU_PRIV_H */
>>>>>> --
>>>>>> 2.34.1
>>>>>>

Re: [PATCH v13 4/6] iommu/arm-smmu: add ACTLR data and support for SM8550

[Bibek Kumar Patro]

On 7/2/2024 12:04 AM, Dmitry Baryshkov wrote:
> On Fri, Jun 28, 2024 at 07:34:33PM GMT, Bibek Kumar Patro wrote:
>> Add ACTLR data table for SM8550 along with support for
>> same including SM8550 specific implementation operations.
>>
>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>> ---
>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 89 ++++++++++++++++++++++
>>   1 file changed, 89 insertions(+)
>>
>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> index 77c9abffe07d..b4521471ffe9 100644
>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> @@ -23,6 +23,85 @@
>>
>>   #define CPRE			(1 << 1)
>>   #define CMTLB			(1 << 0)
>> +#define PREFETCH_SHIFT		8
>> +#define PREFETCH_DEFAULT	0
>> +#define PREFETCH_SHALLOW	(1 << PREFETCH_SHIFT)
>> +#define PREFETCH_MODERATE	(2 << PREFETCH_SHIFT)
>> +#define PREFETCH_DEEP		(3 << PREFETCH_SHIFT)
>> +
>> +static const struct actlr_config sm8550_apps_actlr_cfg[] = {
>> +	{ 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x0800, 0x0020, PREFETCH_DEFAULT | CMTLB },
>> +	{ 0x1800, 0x00c0, PREFETCH_DEFAULT | CMTLB },
>> +	{ 0x1820, 0x0000, PREFETCH_DEFAULT | CMTLB },
>> +	{ 0x1860, 0x0000, PREFETCH_DEFAULT | CMTLB },
>> +	{ 0x0c01, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> 
> - Please keep the list sorted

Sure Dmitry, will sort this list in reverse-christmas-tree order
in next iteration. Thanks for this input.

> - Please comment, which devices use these settings.

As discussed in earlier versions of this patch, these table entries
are kind of just blind values for SMMU device, where SMMU do not have
idea on which SID belong to which client. During probe time when the
clients' Stream-ID has corresponding ACTLR entry then the driver would
set value in register.
Also some might have their prefetch settings as proprietary.
Hence did not add the comments for device using these settings.


Thanks & regards,
Bibek

> 
>> +	{ 0x0c02, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c03, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c04, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c05, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c06, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c07, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c08, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c09, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c0c, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c0d, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c0e, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x0c0f, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1961, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1962, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1963, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1964, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1965, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1966, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1967, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1968, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1969, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x196c, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x196d, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x196e, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x196f, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c1, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c2, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c3, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c4, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c5, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c6, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c7, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c8, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19c9, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19cc, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19cd, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19ce, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x19cf, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>> +	{ 0x1c00, 0x0002, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x1c01, 0x0000, PREFETCH_DEFAULT | CMTLB },
>> +	{ 0x1920, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x1923, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x1924, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x1940, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x1941, 0x0004, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x1943, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x1944, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +	{ 0x1947, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>> +};
>> +
>> +static const struct actlr_config sm8550_gfx_actlr_cfg[] = {
>> +	{ 0x0000, 0x03ff, PREFETCH_DEEP | CPRE | CMTLB },
>> +};
>> +
>> +static const struct actlr_variant sm8550_actlr[] = {
>> +	{
>> +		.io_start = 0x15000000,
>> +		.actlrcfg = sm8550_apps_actlr_cfg,
>> +		.num_actlrcfg = ARRAY_SIZE(sm8550_apps_actlr_cfg)
>> +	}, {
>> +		.io_start = 0x03da0000,
>> +		.actlrcfg = sm8550_gfx_actlr_cfg,
>> +		.num_actlrcfg = ARRAY_SIZE(sm8550_gfx_actlr_cfg)
>> +	},
>> +};
>>
>>   static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>   {
>> @@ -606,6 +685,15 @@ static const struct qcom_smmu_match_data sdm845_smmu_500_data = {
>>   	/* Also no debug configuration. */
>>   };
>>
>> +
>> +static const struct qcom_smmu_match_data sm8550_smmu_500_impl0_data = {
>> +	.impl = &qcom_smmu_500_impl,
>> +	.adreno_impl = &qcom_adreno_smmu_500_impl,
>> +	.cfg = &qcom_smmu_impl0_cfg,
>> +	.actlrvar = sm8550_actlr,
>> +	.num_smmu = ARRAY_SIZE(sm8550_actlr),
>> +};
>> +
>>   static const struct qcom_smmu_match_data qcom_smmu_500_impl0_data = {
>>   	.impl = &qcom_smmu_500_impl,
>>   	.adreno_impl = &qcom_adreno_smmu_500_impl,
>> @@ -640,6 +728,7 @@ static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
>>   	{ .compatible = "qcom,sm8250-smmu-500", .data = &qcom_smmu_500_impl0_data },
>>   	{ .compatible = "qcom,sm8350-smmu-500", .data = &qcom_smmu_500_impl0_data },
>>   	{ .compatible = "qcom,sm8450-smmu-500", .data = &qcom_smmu_500_impl0_data },
>> +	{ .compatible = "qcom,sm8550-smmu-500", .data = &sm8550_smmu_500_impl0_data },
>>   	{ .compatible = "qcom,smmu-500", .data = &qcom_smmu_500_impl0_data },
>>   	{ }
>>   };
>> --
>> 2.34.1
>>
> 

Re: [PATCH v13 4/6] iommu/arm-smmu: add ACTLR data and support for SM8550

[Will Deacon]

On Wed, Jul 03, 2024 at 05:45:23PM +0530, Bibek Kumar Patro wrote:
> 
> 
> On 7/2/2024 12:04 AM, Dmitry Baryshkov wrote:
> > On Fri, Jun 28, 2024 at 07:34:33PM GMT, Bibek Kumar Patro wrote:
> > > Add ACTLR data table for SM8550 along with support for
> > > same including SM8550 specific implementation operations.
> > > 
> > > Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> > > ---
> > >   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 89 ++++++++++++++++++++++
> > >   1 file changed, 89 insertions(+)
> > > 
> > > diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > > index 77c9abffe07d..b4521471ffe9 100644
> > > --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > > @@ -23,6 +23,85 @@
> > > 
> > >   #define CPRE			(1 << 1)
> > >   #define CMTLB			(1 << 0)
> > > +#define PREFETCH_SHIFT		8
> > > +#define PREFETCH_DEFAULT	0
> > > +#define PREFETCH_SHALLOW	(1 << PREFETCH_SHIFT)
> > > +#define PREFETCH_MODERATE	(2 << PREFETCH_SHIFT)
> > > +#define PREFETCH_DEEP		(3 << PREFETCH_SHIFT)
> > > +
> > > +static const struct actlr_config sm8550_apps_actlr_cfg[] = {
> > > +	{ 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> > > +	{ 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> > > +	{ 0x0800, 0x0020, PREFETCH_DEFAULT | CMTLB },
> > > +	{ 0x1800, 0x00c0, PREFETCH_DEFAULT | CMTLB },
> > > +	{ 0x1820, 0x0000, PREFETCH_DEFAULT | CMTLB },
> > > +	{ 0x1860, 0x0000, PREFETCH_DEFAULT | CMTLB },
> > > +	{ 0x0c01, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> > 
> > - Please keep the list sorted
> 
> Sure Dmitry, will sort this list in reverse-christmas-tree order
> in next iteration. Thanks for this input.
> 
> > - Please comment, which devices use these settings.
> 
> As discussed in earlier versions of this patch, these table entries
> are kind of just blind values for SMMU device, where SMMU do not have
> idea on which SID belong to which client. During probe time when the
> clients' Stream-ID has corresponding ACTLR entry then the driver would
> set value in register.

I'm still firmly of the opinion that this stuff needs a higher-level
description in the device-tree and should not be hard-coded in the driver
like this. It's not just a list of opaque values; it describes
SoC-specific topological information that should not be this rigid.

Will

Re: [PATCH v13 4/6] iommu/arm-smmu: add ACTLR data and support for SM8550

[Bibek Kumar Patro]

On 7/3/2024 6:32 PM, Will Deacon wrote:
> On Wed, Jul 03, 2024 at 05:45:23PM +0530, Bibek Kumar Patro wrote:
>>
>>
>> On 7/2/2024 12:04 AM, Dmitry Baryshkov wrote:
>>> On Fri, Jun 28, 2024 at 07:34:33PM GMT, Bibek Kumar Patro wrote:
>>>> Add ACTLR data table for SM8550 along with support for
>>>> same including SM8550 specific implementation operations.
>>>>
>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>> ---
>>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 89 ++++++++++++++++++++++
>>>>    1 file changed, 89 insertions(+)
>>>>
>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> index 77c9abffe07d..b4521471ffe9 100644
>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> @@ -23,6 +23,85 @@
>>>>
>>>>    #define CPRE			(1 << 1)
>>>>    #define CMTLB			(1 << 0)
>>>> +#define PREFETCH_SHIFT		8
>>>> +#define PREFETCH_DEFAULT	0
>>>> +#define PREFETCH_SHALLOW	(1 << PREFETCH_SHIFT)
>>>> +#define PREFETCH_MODERATE	(2 << PREFETCH_SHIFT)
>>>> +#define PREFETCH_DEEP		(3 << PREFETCH_SHIFT)
>>>> +
>>>> +static const struct actlr_config sm8550_apps_actlr_cfg[] = {
>>>> +	{ 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +	{ 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +	{ 0x0800, 0x0020, PREFETCH_DEFAULT | CMTLB },
>>>> +	{ 0x1800, 0x00c0, PREFETCH_DEFAULT | CMTLB },
>>>> +	{ 0x1820, 0x0000, PREFETCH_DEFAULT | CMTLB },
>>>> +	{ 0x1860, 0x0000, PREFETCH_DEFAULT | CMTLB },
>>>> +	{ 0x0c01, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>
>>> - Please keep the list sorted
>>
>> Sure Dmitry, will sort this list in reverse-christmas-tree order
>> in next iteration. Thanks for this input.
>>
>>> - Please comment, which devices use these settings.
>>
>> As discussed in earlier versions of this patch, these table entries
>> are kind of just blind values for SMMU device, where SMMU do not have
>> idea on which SID belong to which client. During probe time when the
>> clients' Stream-ID has corresponding ACTLR entry then the driver would
>> set value in register.
> 
> I'm still firmly of the opinion that this stuff needs a higher-level
> description in the device-tree and should not be hard-coded in the driver
> like this. It's not just a list of opaque values; it describes
> SoC-specific topological information that should not be this rigid.
> 

As per my understanding since ACTLR register is an implementation
defined register,
so I think the placement can also depend on factor of how these
registers are used?

For Qualcomm SoCs, it stores prefetch values for each client, improving
performance without defining hardware design.
Even without setting this value, clients on these Stream-IDs would still
function, albeit with reduced performance.

The SteamID/Mask pair in first two columns <which is a SoC topology> is
only used as reference to find preferred prefetch setting for the
corresponding client on this StreamID

To refer initial discussion and Robin's thoughts on device-tree approach
for this property which we proposed as a part of RFC:
https://lore.kernel.org/all/a01e7e60-6ead-4a9e-ba90-22a8a6bbd03f@quicinc.com/

" On 9/18/2023 4:49 PM, Robin Murphy wrote: "
 >
 > At the very least this would need to be in a implementation-specific
 > backend, since everything about ACTLR is implementation-defined; there
 > could be bits in there that the driver needs to manage itself and
 > clients have absolutely no business overriding (e.g. the MMU-500 errata
 > workarounds). The generic driver can't know what's valid, nor what the
 > consequences are of not being able to satisfy a particular setting. Then
 > there's still the question of what if two clients ask for different
 > settings but want to attach to the same context?
 >
 > It's also questionable whether this would belong in DT at all, since it
 > has a bit of a smell of software policv about it.
 >
 > If it could be
 > sufficiently justified then it would need a proper binding proposal, and
 > "write this opaque value into this register" type properties are
 > generally not approved of.
 >
 > Thanks,
 > Robin.
 >

So as per the initial discussions it felt right to have this data stored
inside driver.
One potential downside is that the driver file could become cluttered
with this data, but this can be mitigated by storing the table in a
separate file if necessary.

For use cases or vendor that implement the ACTLR register differently,
deeply involving SoC topology values or defining hardware design
(something similar to Stream Matching Register),then it might be more
appropriate to place it in the devicetree?

This is just my understanding. I’d appreciate your further thoughts on 
this - Will, Robin, Dmitry, Rob.

Thanks & regards,
Bibek

> Will

Re: [PATCH v13 4/6] iommu/arm-smmu: add ACTLR data and support for SM8550

[Dmitry Baryshkov]

On Wed, 3 Jul 2024 at 15:15, Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 7/2/2024 12:04 AM, Dmitry Baryshkov wrote:
> > On Fri, Jun 28, 2024 at 07:34:33PM GMT, Bibek Kumar Patro wrote:
> >> Add ACTLR data table for SM8550 along with support for
> >> same including SM8550 specific implementation operations.
> >>
> >> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >> ---
> >>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 89 ++++++++++++++++++++++
> >>   1 file changed, 89 insertions(+)
> >>
> >> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> index 77c9abffe07d..b4521471ffe9 100644
> >> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> @@ -23,6 +23,85 @@
> >>
> >>   #define CPRE                       (1 << 1)
> >>   #define CMTLB                      (1 << 0)
> >> +#define PREFETCH_SHIFT              8
> >> +#define PREFETCH_DEFAULT    0
> >> +#define PREFETCH_SHALLOW    (1 << PREFETCH_SHIFT)
> >> +#define PREFETCH_MODERATE   (2 << PREFETCH_SHIFT)
> >> +#define PREFETCH_DEEP               (3 << PREFETCH_SHIFT)
> >> +
> >> +static const struct actlr_config sm8550_apps_actlr_cfg[] = {
> >> +    { 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x0800, 0x0020, PREFETCH_DEFAULT | CMTLB },
> >> +    { 0x1800, 0x00c0, PREFETCH_DEFAULT | CMTLB },
> >> +    { 0x1820, 0x0000, PREFETCH_DEFAULT | CMTLB },
> >> +    { 0x1860, 0x0000, PREFETCH_DEFAULT | CMTLB },
> >> +    { 0x0c01, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >
> > - Please keep the list sorted
>
> Sure Dmitry, will sort this list in reverse-christmas-tree order
> in next iteration. Thanks for this input.

Why? Just sort basing on SID.

>
> > - Please comment, which devices use these settings.
>
> As discussed in earlier versions of this patch, these table entries
> are kind of just blind values for SMMU device, where SMMU do not have
> idea on which SID belong to which client. During probe time when the
> clients' Stream-ID has corresponding ACTLR entry then the driver would
> set value in register.
> Also some might have their prefetch settings as proprietary.
> Hence did not add the comments for device using these settings.

Please mention devices that are going to use the SIDs.


>
>
> Thanks & regards,
> Bibek
>
> >
> >> +    { 0x0c02, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c03, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c04, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c05, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c06, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c07, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c08, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c09, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c0c, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c0d, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c0e, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x0c0f, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1961, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1962, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1963, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1964, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1965, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1966, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1967, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1968, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1969, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x196c, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x196d, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x196e, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x196f, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c1, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c2, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c3, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c4, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c5, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c6, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c7, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c8, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19c9, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19cc, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19cd, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19ce, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x19cf, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
> >> +    { 0x1c00, 0x0002, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x1c01, 0x0000, PREFETCH_DEFAULT | CMTLB },
> >> +    { 0x1920, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x1923, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x1924, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x1940, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x1941, 0x0004, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x1943, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x1944, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +    { 0x1947, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >> +};
> >> +
> >> +static const struct actlr_config sm8550_gfx_actlr_cfg[] = {
> >> +    { 0x0000, 0x03ff, PREFETCH_DEEP | CPRE | CMTLB },
> >> +};
> >> +
> >> +static const struct actlr_variant sm8550_actlr[] = {
> >> +    {
> >> +            .io_start = 0x15000000,
> >> +            .actlrcfg = sm8550_apps_actlr_cfg,
> >> +            .num_actlrcfg = ARRAY_SIZE(sm8550_apps_actlr_cfg)
> >> +    }, {
> >> +            .io_start = 0x03da0000,
> >> +            .actlrcfg = sm8550_gfx_actlr_cfg,
> >> +            .num_actlrcfg = ARRAY_SIZE(sm8550_gfx_actlr_cfg)
> >> +    },
> >> +};
> >>
> >>   static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
> >>   {
> >> @@ -606,6 +685,15 @@ static const struct qcom_smmu_match_data sdm845_smmu_500_data = {
> >>      /* Also no debug configuration. */
> >>   };
> >>
> >> +
> >> +static const struct qcom_smmu_match_data sm8550_smmu_500_impl0_data = {
> >> +    .impl = &qcom_smmu_500_impl,
> >> +    .adreno_impl = &qcom_adreno_smmu_500_impl,
> >> +    .cfg = &qcom_smmu_impl0_cfg,
> >> +    .actlrvar = sm8550_actlr,
> >> +    .num_smmu = ARRAY_SIZE(sm8550_actlr),
> >> +};
> >> +
> >>   static const struct qcom_smmu_match_data qcom_smmu_500_impl0_data = {
> >>      .impl = &qcom_smmu_500_impl,
> >>      .adreno_impl = &qcom_adreno_smmu_500_impl,
> >> @@ -640,6 +728,7 @@ static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
> >>      { .compatible = "qcom,sm8250-smmu-500", .data = &qcom_smmu_500_impl0_data },
> >>      { .compatible = "qcom,sm8350-smmu-500", .data = &qcom_smmu_500_impl0_data },
> >>      { .compatible = "qcom,sm8450-smmu-500", .data = &qcom_smmu_500_impl0_data },
> >> +    { .compatible = "qcom,sm8550-smmu-500", .data = &sm8550_smmu_500_impl0_data },
> >>      { .compatible = "qcom,smmu-500", .data = &qcom_smmu_500_impl0_data },
> >>      { }
> >>   };
> >> --
> >> 2.34.1
> >>
> >



--
With best wishes
Dmitry

Re: [PATCH v13 4/6] iommu/arm-smmu: add ACTLR data and support for SM8550

[Dmitry Baryshkov]

On Thu, 4 Jul 2024 at 12:12, Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 7/3/2024 6:32 PM, Will Deacon wrote:
> > On Wed, Jul 03, 2024 at 05:45:23PM +0530, Bibek Kumar Patro wrote:
> >>
> >>
> >> On 7/2/2024 12:04 AM, Dmitry Baryshkov wrote:
> >>> On Fri, Jun 28, 2024 at 07:34:33PM GMT, Bibek Kumar Patro wrote:
> >>>> Add ACTLR data table for SM8550 along with support for
> >>>> same including SM8550 specific implementation operations.
> >>>>
> >>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>> ---
> >>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 89 ++++++++++++++++++++++
> >>>>    1 file changed, 89 insertions(+)
> >>>>
> >>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> index 77c9abffe07d..b4521471ffe9 100644
> >>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> @@ -23,6 +23,85 @@
> >>>>
> >>>>    #define CPRE                    (1 << 1)
> >>>>    #define CMTLB                   (1 << 0)
> >>>> +#define PREFETCH_SHIFT            8
> >>>> +#define PREFETCH_DEFAULT  0
> >>>> +#define PREFETCH_SHALLOW  (1 << PREFETCH_SHIFT)
> >>>> +#define PREFETCH_MODERATE (2 << PREFETCH_SHIFT)
> >>>> +#define PREFETCH_DEEP             (3 << PREFETCH_SHIFT)
> >>>> +
> >>>> +static const struct actlr_config sm8550_apps_actlr_cfg[] = {
> >>>> +  { 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >>>> +  { 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
> >>>> +  { 0x0800, 0x0020, PREFETCH_DEFAULT | CMTLB },
> >>>> +  { 0x1800, 0x00c0, PREFETCH_DEFAULT | CMTLB },
> >>>> +  { 0x1820, 0x0000, PREFETCH_DEFAULT | CMTLB },
> >>>> +  { 0x1860, 0x0000, PREFETCH_DEFAULT | CMTLB },
> >>>> +  { 0x0c01, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
> >>>
> >>> - Please keep the list sorted
> >>
> >> Sure Dmitry, will sort this list in reverse-christmas-tree order
> >> in next iteration. Thanks for this input.
> >>
> >>> - Please comment, which devices use these settings.
> >>
> >> As discussed in earlier versions of this patch, these table entries
> >> are kind of just blind values for SMMU device, where SMMU do not have
> >> idea on which SID belong to which client. During probe time when the
> >> clients' Stream-ID has corresponding ACTLR entry then the driver would
> >> set value in register.
> >
> > I'm still firmly of the opinion that this stuff needs a higher-level
> > description in the device-tree and should not be hard-coded in the driver
> > like this. It's not just a list of opaque values; it describes
> > SoC-specific topological information that should not be this rigid.
> >
>
> As per my understanding since ACTLR register is an implementation
> defined register,
> so I think the placement can also depend on factor of how these
> registers are used?
>
> For Qualcomm SoCs, it stores prefetch values for each client, improving
> performance without defining hardware design.
> Even without setting this value, clients on these Stream-IDs would still
> function, albeit with reduced performance.
>
> The SteamID/Mask pair in first two columns <which is a SoC topology> is
> only used as reference to find preferred prefetch setting for the
> corresponding client on this StreamID
>
> To refer initial discussion and Robin's thoughts on device-tree approach
> for this property which we proposed as a part of RFC:
> https://lore.kernel.org/all/a01e7e60-6ead-4a9e-ba90-22a8a6bbd03f@quicinc.com/
>
> " On 9/18/2023 4:49 PM, Robin Murphy wrote: "
>  >
>  > At the very least this would need to be in a implementation-specific
>  > backend, since everything about ACTLR is implementation-defined; there
>  > could be bits in there that the driver needs to manage itself and
>  > clients have absolutely no business overriding (e.g. the MMU-500 errata
>  > workarounds). The generic driver can't know what's valid, nor what the
>  > consequences are of not being able to satisfy a particular setting. Then
>  > there's still the question of what if two clients ask for different
>  > settings but want to attach to the same context?
>  >
>  > It's also questionable whether this would belong in DT at all, since it
>  > has a bit of a smell of software policv about it.
>  >
>  > If it could be
>  > sufficiently justified then it would need a proper binding proposal, and
>  > "write this opaque value into this register" type properties are
>  > generally not approved of.
>  >
>  > Thanks,
>  > Robin.
>  >
>
> So as per the initial discussions it felt right to have this data stored
> inside driver.
> One potential downside is that the driver file could become cluttered
> with this data, but this can be mitigated by storing the table in a
> separate file if necessary.
>
> For use cases or vendor that implement the ACTLR register differently,
> deeply involving SoC topology values or defining hardware design
> (something similar to Stream Matching Register),then it might be more
> appropriate to place it in the devicetree?
>
> This is just my understanding. I’d appreciate your further thoughts on
> this - Will, Robin, Dmitry, Rob.

My understanding was that DT should be a place for variable
information. In this case the mapping between Stream-IDs and the
corresponding register programming is more or less fixed for a
particular Soc.
Probably the only way this can be handled outside of the driver is by
increasing #iommu-cells and encoding these values in this extra IOMMU
cell.

-- 
With best wishes
Dmitry

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 7/2/2024 2:01 AM, Rob Clark wrote:
> > On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >>
> >>
> >> On 6/28/2024 9:14 PM, Rob Clark wrote:
> >>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
> >>> <quic_bibekkum@quicinc.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
> >>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
> >>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>
> >>>>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>>>> sequence as per request.
> >>>>>>
> >>>>>> This will be used by GPU to setup the PRR bit and related
> >>>>>> configuration registers through adreno-smmu private
> >>>>>> interface instead of directly poking the smmu hardware.
> >>>>>>
> >>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>>>> ---
> >>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
> >>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>>>     include/linux/adreno-smmu-priv.h           |  6 +++++-
> >>>>>>     3 files changed, 30 insertions(+), 1 deletion(-)
> >>>>>>
> >>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>> index bd101a161d04..64571a1c47b8 100644
> >>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>> @@ -28,6 +28,7 @@
> >>>>>>     #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
> >>>>>>     #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
> >>>>>>     #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
> >>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>>>>
> >>>>>>     static const struct actlr_config sc7280_apps_actlr_cfg[] = {
> >>>>>>            { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
> >>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>>>            arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>>>     }
> >>>>>>
> >>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
> >>>>>> +{
> >>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>>> +       u32 reg = 0;
> >>>>>> +
> >>>>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>> +
> >>>>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>> +
> >>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>>>> +       if (set)
> >>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>> +
> >>>>>
> >>>>> nit, extra line
> >>>>>
> >>>>
> >>>> Ack, will remove this. Thanks for pointing out.
> >>>>
> >>>>> Also, if you passed a `struct page *` instead, then you could drop the
> >>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
> >>>>> can go either way if others have a strong preference for phys_addr_t.
> >>>>>
> >>>>
> >>>> Oh okay, this looks simple to reset the prr bit.
> >>>> But since this page is allocated and is used inside gfx driver
> >>>> before being utilized for prr bit operation, would it be safe for
> >>>> drm/gfx driver to keep a reference to this page in smmu driver?
> >>>>
> >>>> Since we only need the page address for configuring the
> >>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
> >>>
> >>> I don't think the smmu driver needs to keep a reference to the page..
> >>> we can just say it is the responsibility of the drm driver to call
> >>> set_prr(NULL) before freeing the page
> >>>
> >>
> >> That makes sense. If we go by this NULL page method to disable the PRR,
> >> we would have to set the address registers to reset value as well.
> >>
> >> The sequence would be like the following as per my understaning:
> >> - Check if it's NULL page
> >> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
> >>     these registers
> >> - Reset the PRR bit in actlr register
> >>
> >> Similar to this snippet:
> >>
> >> #PRR_RESET_ADDR 0x0
> >>
> >> --------------
> >> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >> reg &= ~GFX_ACTLR_PRR;
> >> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>
> >> if (!prr_page) {
> >>          writel_relaxed(PRR_RESET_ADDR,
> >>                          smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>          writel_relaxed(PRR_RESET_ADDR),
> >>                           smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>          return;
> >> }
> >>
> >>
> >> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
> >>                  smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>
> >> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
> >>                  smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>
> >> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >> -----------------
> >>
> >> If looks good, will implement the same in next version.
> >
> > yeah, that looks like it could work..
> >
> > you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
> > and probably could avoid double writing ACTLR, but that is getting
> > into bikeshedding
> >
>
> Actually Rob, since you rightly pointed this out.
> I crosschecked again on these registers.
> PRR_CFG_*ADDR is a global register in SMMU space but
> ACTLR register including PRR bit is a per-domain register.
> There might also be a situation where PRR feature need to be
> disabled or enabled separately for each domain.
> So I think it would be cleaner to have two apis, set_prr_addr(),
> set_prr_bit().
> set_prr_addr() will be used only to set this PRR_CFG_*ADDR
> register by passing a 'struct page *'
> set_prr_bit() will be used as a switch for PRR feature,
> where required smmu_domain will be passed along with
> the bool value to set/reset the PRR bit depending on which this
> feature will be enabled/disabled for the selected domain.

on a related note, adreno has been using arm-smmu for a number of
generations, I guess not all support PRR?  The drm driver will need to
know whether PRR is supported (and expose that to userspace to let the
UMD know whether to expose certain extensions).  How should this work?

BR,
-R

> Thanks & regards,
> Bibek
>
> > BR,
> > -R
> >
> >>
> >> Thanks & regards,
> >> Bibek
> >>
> >>> BR,
> >>> -R
> >>>
> >>>>> Otherwise, lgtm
> >>>>>
> >>>>> BR,
> >>>>> -R
> >>>>>
> >>>>
> >>>> Thanks & regards,
> >>>> Bibek
> >>>>
> >>>>>> +}
> >>>>>> +
> >>>>>>     #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>>>>
> >>>>>>     static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>>            priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>>>>            priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>>>>            priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
> >>>>>>
> >>>>>>            actlrvar = qsmmu->data->actlrvar;
> >>>>>>            if (!actlrvar)
> >>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>> index d9c2ef8c1653..3076bef49e20 100644
> >>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>>>>>     #define ARM_SMMU_SCTLR_M               BIT(0)
> >>>>>>
> >>>>>>     #define ARM_SMMU_CB_ACTLR              0x4
> >>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>>>>>
> >>>>>>     #define ARM_SMMU_CB_RESUME             0x8
> >>>>>>     #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >>>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
> >>>>>> --- a/include/linux/adreno-smmu-priv.h
> >>>>>> +++ b/include/linux/adreno-smmu-priv.h
> >>>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
> >>>>>>      *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
> >>>>>>      *                 the GPU driver must call resume_translation()
> >>>>>>      * @resume_translation: Resume translation after a fault
> >>>>>> - *
> >>>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
> >>>>>> + *                 ACTLR register bits, currently used to configure
> >>>>>> + *                 Partially-Resident-Region (PRR) feature's
> >>>>>> + *                 setup and reset sequence as requested.
> >>>>>>      *
> >>>>>>      * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>>>>>      * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >>>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
> >>>>>>         void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>>>>>         void (*set_stall)(const void *cookie, bool enabled);
> >>>>>>         void (*resume_translation)(const void *cookie, bool terminate);
> >>>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
> >>>>>>     };
> >>>>>>
> >>>>>>     #endif /* __ADRENO_SMMU_PRIV_H */
> >>>>>> --
> >>>>>> 2.34.1
> >>>>>>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Thu, Jul 4, 2024 at 7:46 AM Rob Clark <robdclark@gmail.com> wrote:
>
> On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
> >
> >
> >
> > On 7/2/2024 2:01 AM, Rob Clark wrote:
> > > On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
> > > <quic_bibekkum@quicinc.com> wrote:
> > >>
> > >>
> > >>
> > >> On 6/28/2024 9:14 PM, Rob Clark wrote:
> > >>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
> > >>> <quic_bibekkum@quicinc.com> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
> > >>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
> > >>>>> <quic_bibekkum@quicinc.com> wrote:
> > >>>>>>
> > >>>>>> Add an adreno-smmu-priv interface for drm/msm to call
> > >>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> > >>>>>> sequence as per request.
> > >>>>>>
> > >>>>>> This will be used by GPU to setup the PRR bit and related
> > >>>>>> configuration registers through adreno-smmu private
> > >>>>>> interface instead of directly poking the smmu hardware.
> > >>>>>>
> > >>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> > >>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> > >>>>>> ---
> > >>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
> > >>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> > >>>>>>     include/linux/adreno-smmu-priv.h           |  6 +++++-
> > >>>>>>     3 files changed, 30 insertions(+), 1 deletion(-)
> > >>>>>>
> > >>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > >>>>>> index bd101a161d04..64571a1c47b8 100644
> > >>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > >>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > >>>>>> @@ -28,6 +28,7 @@
> > >>>>>>     #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
> > >>>>>>     #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
> > >>>>>>     #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
> > >>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
> > >>>>>>
> > >>>>>>     static const struct actlr_config sc7280_apps_actlr_cfg[] = {
> > >>>>>>            { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
> > >>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> > >>>>>>            arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> > >>>>>>     }
> > >>>>>>
> > >>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
> > >>>>>> +{
> > >>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> > >>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> > >>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> > >>>>>> +       u32 reg = 0;
> > >>>>>> +
> > >>>>>> +       writel_relaxed(lower_32_bits(page_addr),
> > >>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> > >>>>>> +
> > >>>>>> +       writel_relaxed(upper_32_bits(page_addr),
> > >>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> > >>>>>> +
> > >>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> > >>>>>> +       reg &= ~GFX_ACTLR_PRR;
> > >>>>>> +       if (set)
> > >>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> > >>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> > >>>>>> +
> > >>>>>
> > >>>>> nit, extra line
> > >>>>>
> > >>>>
> > >>>> Ack, will remove this. Thanks for pointing out.
> > >>>>
> > >>>>> Also, if you passed a `struct page *` instead, then you could drop the
> > >>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
> > >>>>> can go either way if others have a strong preference for phys_addr_t.
> > >>>>>
> > >>>>
> > >>>> Oh okay, this looks simple to reset the prr bit.
> > >>>> But since this page is allocated and is used inside gfx driver
> > >>>> before being utilized for prr bit operation, would it be safe for
> > >>>> drm/gfx driver to keep a reference to this page in smmu driver?
> > >>>>
> > >>>> Since we only need the page address for configuring the
> > >>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
> > >>>
> > >>> I don't think the smmu driver needs to keep a reference to the page..
> > >>> we can just say it is the responsibility of the drm driver to call
> > >>> set_prr(NULL) before freeing the page
> > >>>
> > >>
> > >> That makes sense. If we go by this NULL page method to disable the PRR,
> > >> we would have to set the address registers to reset value as well.
> > >>
> > >> The sequence would be like the following as per my understaning:
> > >> - Check if it's NULL page
> > >> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
> > >>     these registers
> > >> - Reset the PRR bit in actlr register
> > >>
> > >> Similar to this snippet:
> > >>
> > >> #PRR_RESET_ADDR 0x0
> > >>
> > >> --------------
> > >> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> > >> reg &= ~GFX_ACTLR_PRR;
> > >> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> > >>
> > >> if (!prr_page) {
> > >>          writel_relaxed(PRR_RESET_ADDR,
> > >>                          smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> > >>          writel_relaxed(PRR_RESET_ADDR),
> > >>                           smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> > >>          return;
> > >> }
> > >>
> > >>
> > >> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
> > >>                  smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> > >>
> > >> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
> > >>                  smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> > >>
> > >> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> > >> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> > >> -----------------
> > >>
> > >> If looks good, will implement the same in next version.
> > >
> > > yeah, that looks like it could work..
> > >
> > > you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
> > > and probably could avoid double writing ACTLR, but that is getting
> > > into bikeshedding
> > >
> >
> > Actually Rob, since you rightly pointed this out.
> > I crosschecked again on these registers.
> > PRR_CFG_*ADDR is a global register in SMMU space but
> > ACTLR register including PRR bit is a per-domain register.
> > There might also be a situation where PRR feature need to be
> > disabled or enabled separately for each domain.
> > So I think it would be cleaner to have two apis, set_prr_addr(),
> > set_prr_bit().
> > set_prr_addr() will be used only to set this PRR_CFG_*ADDR
> > register by passing a 'struct page *'
> > set_prr_bit() will be used as a switch for PRR feature,
> > where required smmu_domain will be passed along with
> > the bool value to set/reset the PRR bit depending on which this
> > feature will be enabled/disabled for the selected domain.
>
> on a related note, adreno has been using arm-smmu for a number of
> generations, I guess not all support PRR?  The drm driver will need to
> know whether PRR is supported (and expose that to userspace to let the
> UMD know whether to expose certain extensions).  How should this work?

So, I noticed in the x1e80100.dtsi that there is a gpu_prr_mem
reserved section..  maybe we should be connecting this to the smmu
driver in dt, and using that to detect presence of PRR?  Ie. the smmu
driver would configure PRR_CFG_*ADDR based on the reserved mem, and
the interface to drm would just be to enable/disable PRR, returning an
error code if the reserved mem section isn't there.

This simplifies the interface, and handles the question of how to
detect if PRR is supported.

BR,
-R

> BR,
> -R
>
> > Thanks & regards,
> > Bibek
> >
> > > BR,
> > > -R
> > >
> > >>
> > >> Thanks & regards,
> > >> Bibek
> > >>
> > >>> BR,
> > >>> -R
> > >>>
> > >>>>> Otherwise, lgtm
> > >>>>>
> > >>>>> BR,
> > >>>>> -R
> > >>>>>
> > >>>>
> > >>>> Thanks & regards,
> > >>>> Bibek
> > >>>>
> > >>>>>> +}
> > >>>>>> +
> > >>>>>>     #define QCOM_ADRENO_SMMU_GPU_SID 0
> > >>>>>>
> > >>>>>>     static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> > >>>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> > >>>>>>            priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> > >>>>>>            priv->set_stall = qcom_adreno_smmu_set_stall;
> > >>>>>>            priv->resume_translation = qcom_adreno_smmu_resume_translation;
> > >>>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
> > >>>>>>
> > >>>>>>            actlrvar = qsmmu->data->actlrvar;
> > >>>>>>            if (!actlrvar)
> > >>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> > >>>>>> index d9c2ef8c1653..3076bef49e20 100644
> > >>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> > >>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> > >>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> > >>>>>>     #define ARM_SMMU_SCTLR_M               BIT(0)
> > >>>>>>
> > >>>>>>     #define ARM_SMMU_CB_ACTLR              0x4
> > >>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> > >>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> > >>>>>>
> > >>>>>>     #define ARM_SMMU_CB_RESUME             0x8
> > >>>>>>     #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> > >>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> > >>>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
> > >>>>>> --- a/include/linux/adreno-smmu-priv.h
> > >>>>>> +++ b/include/linux/adreno-smmu-priv.h
> > >>>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
> > >>>>>>      *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
> > >>>>>>      *                 the GPU driver must call resume_translation()
> > >>>>>>      * @resume_translation: Resume translation after a fault
> > >>>>>> - *
> > >>>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
> > >>>>>> + *                 ACTLR register bits, currently used to configure
> > >>>>>> + *                 Partially-Resident-Region (PRR) feature's
> > >>>>>> + *                 setup and reset sequence as requested.
> > >>>>>>      *
> > >>>>>>      * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> > >>>>>>      * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> > >>>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
> > >>>>>>         void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> > >>>>>>         void (*set_stall)(const void *cookie, bool enabled);
> > >>>>>>         void (*resume_translation)(const void *cookie, bool terminate);
> > >>>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
> > >>>>>>     };
> > >>>>>>
> > >>>>>>     #endif /* __ADRENO_SMMU_PRIV_H */
> > >>>>>> --
> > >>>>>> 2.34.1
> > >>>>>>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 7/4/2024 9:28 PM, Rob Clark wrote:
> On Thu, Jul 4, 2024 at 7:46 AM Rob Clark <robdclark@gmail.com> wrote:
>>
>> On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
>> <quic_bibekkum@quicinc.com> wrote:
>>>
>>>
>>>
>>> On 7/2/2024 2:01 AM, Rob Clark wrote:
>>>> On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 6/28/2024 9:14 PM, Rob Clark wrote:
>>>>>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
>>>>>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>
>>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>>>>>> sequence as per request.
>>>>>>>>>
>>>>>>>>> This will be used by GPU to setup the PRR bit and related
>>>>>>>>> configuration registers through adreno-smmu private
>>>>>>>>> interface instead of directly poking the smmu hardware.
>>>>>>>>>
>>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>>>>>> ---
>>>>>>>>>      drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
>>>>>>>>>      drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>>>>>      include/linux/adreno-smmu-priv.h           |  6 +++++-
>>>>>>>>>      3 files changed, 30 insertions(+), 1 deletion(-)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>> index bd101a161d04..64571a1c47b8 100644
>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>> @@ -28,6 +28,7 @@
>>>>>>>>>      #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
>>>>>>>>>      #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
>>>>>>>>>      #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
>>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>>>>>>
>>>>>>>>>      static const struct actlr_config sc7280_apps_actlr_cfg[] = {
>>>>>>>>>             { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
>>>>>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>>>>>             arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>>>>>      }
>>>>>>>>>
>>>>>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
>>>>>>>>> +{
>>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>>>>> +       u32 reg = 0;
>>>>>>>>> +
>>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>> +
>>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>> +
>>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>>>>>> +       if (set)
>>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>> +
>>>>>>>>
>>>>>>>> nit, extra line
>>>>>>>>
>>>>>>>
>>>>>>> Ack, will remove this. Thanks for pointing out.
>>>>>>>
>>>>>>>> Also, if you passed a `struct page *` instead, then you could drop the
>>>>>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
>>>>>>>> can go either way if others have a strong preference for phys_addr_t.
>>>>>>>>
>>>>>>>
>>>>>>> Oh okay, this looks simple to reset the prr bit.
>>>>>>> But since this page is allocated and is used inside gfx driver
>>>>>>> before being utilized for prr bit operation, would it be safe for
>>>>>>> drm/gfx driver to keep a reference to this page in smmu driver?
>>>>>>>
>>>>>>> Since we only need the page address for configuring the
>>>>>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
>>>>>>
>>>>>> I don't think the smmu driver needs to keep a reference to the page..
>>>>>> we can just say it is the responsibility of the drm driver to call
>>>>>> set_prr(NULL) before freeing the page
>>>>>>
>>>>>
>>>>> That makes sense. If we go by this NULL page method to disable the PRR,
>>>>> we would have to set the address registers to reset value as well.
>>>>>
>>>>> The sequence would be like the following as per my understaning:
>>>>> - Check if it's NULL page
>>>>> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
>>>>>      these registers
>>>>> - Reset the PRR bit in actlr register
>>>>>
>>>>> Similar to this snippet:
>>>>>
>>>>> #PRR_RESET_ADDR 0x0
>>>>>
>>>>> --------------
>>>>> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>> reg &= ~GFX_ACTLR_PRR;
>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>
>>>>> if (!prr_page) {
>>>>>           writel_relaxed(PRR_RESET_ADDR,
>>>>>                           smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>           writel_relaxed(PRR_RESET_ADDR),
>>>>>                            smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>           return;
>>>>> }
>>>>>
>>>>>
>>>>> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
>>>>>                   smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>
>>>>> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
>>>>>                   smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>
>>>>> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>> -----------------
>>>>>
>>>>> If looks good, will implement the same in next version.
>>>>
>>>> yeah, that looks like it could work..
>>>>
>>>> you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
>>>> and probably could avoid double writing ACTLR, but that is getting
>>>> into bikeshedding
>>>>
>>>
>>> Actually Rob, since you rightly pointed this out.
>>> I crosschecked again on these registers.
>>> PRR_CFG_*ADDR is a global register in SMMU space but
>>> ACTLR register including PRR bit is a per-domain register.
>>> There might also be a situation where PRR feature need to be
>>> disabled or enabled separately for each domain.
>>> So I think it would be cleaner to have two apis, set_prr_addr(),
>>> set_prr_bit().
>>> set_prr_addr() will be used only to set this PRR_CFG_*ADDR
>>> register by passing a 'struct page *'
>>> set_prr_bit() will be used as a switch for PRR feature,
>>> where required smmu_domain will be passed along with
>>> the bool value to set/reset the PRR bit depending on which this
>>> feature will be enabled/disabled for the selected domain.
>>
>> on a related note, adreno has been using arm-smmu for a number of
>> generations, I guess not all support PRR?  The drm driver will need to
>> know whether PRR is supported (and expose that to userspace to let the
>> UMD know whether to expose certain extensions).  How should this work?
> 
> So, I noticed in the x1e80100.dtsi that there is a gpu_prr_mem
> reserved section..  maybe we should be connecting this to the smmu
> driver in dt, and using that to detect presence of PRR?  Ie. the smmu
> driver would configure PRR_CFG_*ADDR based on the reserved mem, and
> the interface to drm would just be to enable/disable PRR, returning an
> error code if the reserved mem section isn't there.
> 
> This simplifies the interface, and handles the question of how to
> detect if PRR is supported.
> 
> BR,
> -R
> 

As I checked gpu_prr_mem reserved mem section is not used for mobile
targets hence not present for other DT only compute targets like
x1e80100.dtsi has the same. PRR looks to be smmu version specific
property.
This feature is present in all the targets using SMMU-500,
I am still checking for targets using versions prior to smmu-500.
Then maybe we can use the smmu compatible string itself (e.g.
qcom,smmu-500 && qcom,adreno-smmu) to connect to smmu driver for
checking the presence of PRR ?
And if the compatible string is different then we can return the
error code signifying PRR feature is not supported on the particular
smmu version.

Thanks & regards,
Bibek

>> BR,
>> -R
>>
>>> Thanks & regards,
>>> Bibek
>>>
>>>> BR,
>>>> -R
>>>>
>>>>>
>>>>> Thanks & regards,
>>>>> Bibek
>>>>>
>>>>>> BR,
>>>>>> -R
>>>>>>
>>>>>>>> Otherwise, lgtm
>>>>>>>>
>>>>>>>> BR,
>>>>>>>> -R
>>>>>>>>
>>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Bibek
>>>>>>>
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>>      #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>>>>>>
>>>>>>>>>      static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>>>>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>>>>             priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>>>>>>             priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>>>>>>             priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>>>>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
>>>>>>>>>
>>>>>>>>>             actlrvar = qsmmu->data->actlrvar;
>>>>>>>>>             if (!actlrvar)
>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>> index d9c2ef8c1653..3076bef49e20 100644
>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>>>>>>      #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>>>>>>
>>>>>>>>>      #define ARM_SMMU_CB_ACTLR              0x4
>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>>>>>>
>>>>>>>>>      #define ARM_SMMU_CB_RESUME             0x8
>>>>>>>>>      #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>>>>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
>>>>>>>>> --- a/include/linux/adreno-smmu-priv.h
>>>>>>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>>>>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
>>>>>>>>>       *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
>>>>>>>>>       *                 the GPU driver must call resume_translation()
>>>>>>>>>       * @resume_translation: Resume translation after a fault
>>>>>>>>> - *
>>>>>>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
>>>>>>>>> + *                 ACTLR register bits, currently used to configure
>>>>>>>>> + *                 Partially-Resident-Region (PRR) feature's
>>>>>>>>> + *                 setup and reset sequence as requested.
>>>>>>>>>       *
>>>>>>>>>       * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>>>>>>       * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>>>>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
>>>>>>>>>          void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>>>>>>          void (*set_stall)(const void *cookie, bool enabled);
>>>>>>>>>          void (*resume_translation)(const void *cookie, bool terminate);
>>>>>>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
>>>>>>>>>      };
>>>>>>>>>
>>>>>>>>>      #endif /* __ADRENO_SMMU_PRIV_H */
>>>>>>>>> --
>>>>>>>>> 2.34.1
>>>>>>>>>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Tue, Jul 9, 2024 at 12:43 PM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 7/4/2024 9:28 PM, Rob Clark wrote:
> > On Thu, Jul 4, 2024 at 7:46 AM Rob Clark <robdclark@gmail.com> wrote:
> >>
> >> On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
> >> <quic_bibekkum@quicinc.com> wrote:
> >>>
> >>>
> >>>
> >>> On 7/2/2024 2:01 AM, Rob Clark wrote:
> >>>> On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
> >>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 6/28/2024 9:14 PM, Rob Clark wrote:
> >>>>>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
> >>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
> >>>>>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
> >>>>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>>>>>>> sequence as per request.
> >>>>>>>>>
> >>>>>>>>> This will be used by GPU to setup the PRR bit and related
> >>>>>>>>> configuration registers through adreno-smmu private
> >>>>>>>>> interface instead of directly poking the smmu hardware.
> >>>>>>>>>
> >>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>>>>>>> ---
> >>>>>>>>>      drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
> >>>>>>>>>      drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>>>>>>      include/linux/adreno-smmu-priv.h           |  6 +++++-
> >>>>>>>>>      3 files changed, 30 insertions(+), 1 deletion(-)
> >>>>>>>>>
> >>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>> index bd101a161d04..64571a1c47b8 100644
> >>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>> @@ -28,6 +28,7 @@
> >>>>>>>>>      #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
> >>>>>>>>>      #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
> >>>>>>>>>      #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
> >>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>>>>>>>
> >>>>>>>>>      static const struct actlr_config sc7280_apps_actlr_cfg[] = {
> >>>>>>>>>             { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
> >>>>>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>>>>>>             arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>>>>>>      }
> >>>>>>>>>
> >>>>>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
> >>>>>>>>> +{
> >>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>>>>>> +       u32 reg = 0;
> >>>>>>>>> +
> >>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>>>> +
> >>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>>>> +
> >>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>>>>>>> +       if (set)
> >>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>>>> +
> >>>>>>>>
> >>>>>>>> nit, extra line
> >>>>>>>>
> >>>>>>>
> >>>>>>> Ack, will remove this. Thanks for pointing out.
> >>>>>>>
> >>>>>>>> Also, if you passed a `struct page *` instead, then you could drop the
> >>>>>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
> >>>>>>>> can go either way if others have a strong preference for phys_addr_t.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Oh okay, this looks simple to reset the prr bit.
> >>>>>>> But since this page is allocated and is used inside gfx driver
> >>>>>>> before being utilized for prr bit operation, would it be safe for
> >>>>>>> drm/gfx driver to keep a reference to this page in smmu driver?
> >>>>>>>
> >>>>>>> Since we only need the page address for configuring the
> >>>>>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
> >>>>>>
> >>>>>> I don't think the smmu driver needs to keep a reference to the page..
> >>>>>> we can just say it is the responsibility of the drm driver to call
> >>>>>> set_prr(NULL) before freeing the page
> >>>>>>
> >>>>>
> >>>>> That makes sense. If we go by this NULL page method to disable the PRR,
> >>>>> we would have to set the address registers to reset value as well.
> >>>>>
> >>>>> The sequence would be like the following as per my understaning:
> >>>>> - Check if it's NULL page
> >>>>> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
> >>>>>      these registers
> >>>>> - Reset the PRR bit in actlr register
> >>>>>
> >>>>> Similar to this snippet:
> >>>>>
> >>>>> #PRR_RESET_ADDR 0x0
> >>>>>
> >>>>> --------------
> >>>>> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>> reg &= ~GFX_ACTLR_PRR;
> >>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>
> >>>>> if (!prr_page) {
> >>>>>           writel_relaxed(PRR_RESET_ADDR,
> >>>>>                           smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>           writel_relaxed(PRR_RESET_ADDR),
> >>>>>                            smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>           return;
> >>>>> }
> >>>>>
> >>>>>
> >>>>> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
> >>>>>                   smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>
> >>>>> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
> >>>>>                   smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>
> >>>>> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>> -----------------
> >>>>>
> >>>>> If looks good, will implement the same in next version.
> >>>>
> >>>> yeah, that looks like it could work..
> >>>>
> >>>> you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
> >>>> and probably could avoid double writing ACTLR, but that is getting
> >>>> into bikeshedding
> >>>>
> >>>
> >>> Actually Rob, since you rightly pointed this out.
> >>> I crosschecked again on these registers.
> >>> PRR_CFG_*ADDR is a global register in SMMU space but
> >>> ACTLR register including PRR bit is a per-domain register.
> >>> There might also be a situation where PRR feature need to be
> >>> disabled or enabled separately for each domain.
> >>> So I think it would be cleaner to have two apis, set_prr_addr(),
> >>> set_prr_bit().
> >>> set_prr_addr() will be used only to set this PRR_CFG_*ADDR
> >>> register by passing a 'struct page *'
> >>> set_prr_bit() will be used as a switch for PRR feature,
> >>> where required smmu_domain will be passed along with
> >>> the bool value to set/reset the PRR bit depending on which this
> >>> feature will be enabled/disabled for the selected domain.
> >>
> >> on a related note, adreno has been using arm-smmu for a number of
> >> generations, I guess not all support PRR?  The drm driver will need to
> >> know whether PRR is supported (and expose that to userspace to let the
> >> UMD know whether to expose certain extensions).  How should this work?
> >
> > So, I noticed in the x1e80100.dtsi that there is a gpu_prr_mem
> > reserved section..  maybe we should be connecting this to the smmu
> > driver in dt, and using that to detect presence of PRR?  Ie. the smmu
> > driver would configure PRR_CFG_*ADDR based on the reserved mem, and
> > the interface to drm would just be to enable/disable PRR, returning an
> > error code if the reserved mem section isn't there.
> >
> > This simplifies the interface, and handles the question of how to
> > detect if PRR is supported.
> >
> > BR,
> > -R
> >
>
> As I checked gpu_prr_mem reserved mem section is not used for mobile
> targets hence not present for other DT only compute targets like
> x1e80100.dtsi has the same. PRR looks to be smmu version specific
> property.

I only see it in gpu_prr_mem in x1e80100.dtsi, but not documented
anywhere.  I'm only assuming based on the name that it is intended to
be for PRR (but not sure why it is larger than 0x1000).  Are the
PRR_CFG_*ADDR regs programmed by the fw (and access blocked in EL1) on
this device?

As far as other DT, we haven't enabled PRR anywhere yet.  I think it
would be perfectly valid to require a new reserved-memory node to
enable PRR.

> This feature is present in all the targets using SMMU-500,
> I am still checking for targets using versions prior to smmu-500.
> Then maybe we can use the smmu compatible string itself (e.g.
> qcom,smmu-500 && qcom,adreno-smmu) to connect to smmu driver for
> checking the presence of PRR ?

If there is a clean break, such as all smmu-500 have PRR and all
smmu-v2 do not, then it would be reasonable to use the compat strings.
If not, I think we need a different way.

BR,
-R

> And if the compatible string is different then we can return the
> error code signifying PRR feature is not supported on the particular
> smmu version.
>
> Thanks & regards,
> Bibek
>
> >> BR,
> >> -R
> >>
> >>> Thanks & regards,
> >>> Bibek
> >>>
> >>>> BR,
> >>>> -R
> >>>>
> >>>>>
> >>>>> Thanks & regards,
> >>>>> Bibek
> >>>>>
> >>>>>> BR,
> >>>>>> -R
> >>>>>>
> >>>>>>>> Otherwise, lgtm
> >>>>>>>>
> >>>>>>>> BR,
> >>>>>>>> -R
> >>>>>>>>
> >>>>>>>
> >>>>>>> Thanks & regards,
> >>>>>>> Bibek
> >>>>>>>
> >>>>>>>>> +}
> >>>>>>>>> +
> >>>>>>>>>      #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>>>>>>>
> >>>>>>>>>      static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>>>>>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>>>>>             priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>>>>>>>             priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>>>>>>>             priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>>>>>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
> >>>>>>>>>
> >>>>>>>>>             actlrvar = qsmmu->data->actlrvar;
> >>>>>>>>>             if (!actlrvar)
> >>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>>>> index d9c2ef8c1653..3076bef49e20 100644
> >>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>>>>>>>>      #define ARM_SMMU_SCTLR_M               BIT(0)
> >>>>>>>>>
> >>>>>>>>>      #define ARM_SMMU_CB_ACTLR              0x4
> >>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>>>>>>>>
> >>>>>>>>>      #define ARM_SMMU_CB_RESUME             0x8
> >>>>>>>>>      #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >>>>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >>>>>>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
> >>>>>>>>> --- a/include/linux/adreno-smmu-priv.h
> >>>>>>>>> +++ b/include/linux/adreno-smmu-priv.h
> >>>>>>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
> >>>>>>>>>       *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
> >>>>>>>>>       *                 the GPU driver must call resume_translation()
> >>>>>>>>>       * @resume_translation: Resume translation after a fault
> >>>>>>>>> - *
> >>>>>>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
> >>>>>>>>> + *                 ACTLR register bits, currently used to configure
> >>>>>>>>> + *                 Partially-Resident-Region (PRR) feature's
> >>>>>>>>> + *                 setup and reset sequence as requested.
> >>>>>>>>>       *
> >>>>>>>>>       * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>>>>>>>>       * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >>>>>>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
> >>>>>>>>>          void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>>>>>>>>          void (*set_stall)(const void *cookie, bool enabled);
> >>>>>>>>>          void (*resume_translation)(const void *cookie, bool terminate);
> >>>>>>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
> >>>>>>>>>      };
> >>>>>>>>>
> >>>>>>>>>      #endif /* __ADRENO_SMMU_PRIV_H */
> >>>>>>>>> --
> >>>>>>>>> 2.34.1
> >>>>>>>>>

Re: [PATCH v13 4/6] iommu/arm-smmu: add ACTLR data and support for SM8550

[Bibek Kumar Patro]

On 7/4/2024 4:53 PM, Dmitry Baryshkov wrote:
> On Wed, 3 Jul 2024 at 15:15, Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 7/2/2024 12:04 AM, Dmitry Baryshkov wrote:
>>> On Fri, Jun 28, 2024 at 07:34:33PM GMT, Bibek Kumar Patro wrote:
>>>> Add ACTLR data table for SM8550 along with support for
>>>> same including SM8550 specific implementation operations.
>>>>
>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>> ---
>>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 89 ++++++++++++++++++++++
>>>>    1 file changed, 89 insertions(+)
>>>>
>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> index 77c9abffe07d..b4521471ffe9 100644
>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> @@ -23,6 +23,85 @@
>>>>
>>>>    #define CPRE                       (1 << 1)
>>>>    #define CMTLB                      (1 << 0)
>>>> +#define PREFETCH_SHIFT              8
>>>> +#define PREFETCH_DEFAULT    0
>>>> +#define PREFETCH_SHALLOW    (1 << PREFETCH_SHIFT)
>>>> +#define PREFETCH_MODERATE   (2 << PREFETCH_SHIFT)
>>>> +#define PREFETCH_DEEP               (3 << PREFETCH_SHIFT)
>>>> +
>>>> +static const struct actlr_config sm8550_apps_actlr_cfg[] = {
>>>> +    { 0x18a0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x18e0, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x0800, 0x0020, PREFETCH_DEFAULT | CMTLB },
>>>> +    { 0x1800, 0x00c0, PREFETCH_DEFAULT | CMTLB },
>>>> +    { 0x1820, 0x0000, PREFETCH_DEFAULT | CMTLB },
>>>> +    { 0x1860, 0x0000, PREFETCH_DEFAULT | CMTLB },
>>>> +    { 0x0c01, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>
>>> - Please keep the list sorted
>>
>> Sure Dmitry, will sort this list in reverse-christmas-tree order
>> in next iteration. Thanks for this input.
> 
> Why? Just sort basing on SID.
> 

Apologies for delayed response.
It's initially sorted based on SID only, but as I see few of them are
not which you're pointing to.
I will check the table again and sort the SIDs which are left unsorted.

>>
>>> - Please comment, which devices use these settings.
>>
>> As discussed in earlier versions of this patch, these table entries
>> are kind of just blind values for SMMU device, where SMMU do not have
>> idea on which SID belong to which client. During probe time when the
>> clients' Stream-ID has corresponding ACTLR entry then the driver would
>> set value in register.
>> Also some might have their prefetch settings as proprietary.
>> Hence did not add the comments for device using these settings.
> 
> Please mention devices that are going to use the SIDs.
> 

Interested to know if there's a compelling reason to mention the device
name.
 From SMMU driver perspective there's not much value proposition by
mentioning the names of devices using the SIDs.
Also sometimes it would be possible that the the SID value might change
a bit for each client from hardware perspective which would need to be
updated back in the driver <kind of an additional overhead for SMMU
driver>.
Hence I was in favour to avoid mentioning the name of devices, not
making it look like a placeholder for SIDs and corresponding ACTLR
values.
Please let me know if you still feel otherwise.

Thanks & regards,
Bibek
>>
>>
>> Thanks & regards,
>> Bibek
>>
>>>
>>>> +    { 0x0c02, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c03, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c04, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c05, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c06, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c07, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c08, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c09, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c0c, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c0d, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c0e, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x0c0f, 0x0020, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1961, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1962, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1963, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1964, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1965, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1966, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1967, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1968, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1969, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x196c, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x196d, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x196e, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x196f, 0x0000, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c1, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c2, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c3, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c4, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c5, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c6, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c7, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c8, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19c9, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19cc, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19cd, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19ce, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x19cf, 0x0010, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +    { 0x1c00, 0x0002, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x1c01, 0x0000, PREFETCH_DEFAULT | CMTLB },
>>>> +    { 0x1920, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x1923, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x1924, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x1940, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x1941, 0x0004, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x1943, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x1944, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +    { 0x1947, 0x0000, PREFETCH_SHALLOW | CPRE | CMTLB },
>>>> +};
>>>> +
>>>> +static const struct actlr_config sm8550_gfx_actlr_cfg[] = {
>>>> +    { 0x0000, 0x03ff, PREFETCH_DEEP | CPRE | CMTLB },
>>>> +};
>>>> +
>>>> +static const struct actlr_variant sm8550_actlr[] = {
>>>> +    {
>>>> +            .io_start = 0x15000000,
>>>> +            .actlrcfg = sm8550_apps_actlr_cfg,
>>>> +            .num_actlrcfg = ARRAY_SIZE(sm8550_apps_actlr_cfg)
>>>> +    }, {
>>>> +            .io_start = 0x03da0000,
>>>> +            .actlrcfg = sm8550_gfx_actlr_cfg,
>>>> +            .num_actlrcfg = ARRAY_SIZE(sm8550_gfx_actlr_cfg)
>>>> +    },
>>>> +};
>>>>
>>>>    static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>>>    {
>>>> @@ -606,6 +685,15 @@ static const struct qcom_smmu_match_data sdm845_smmu_500_data = {
>>>>       /* Also no debug configuration. */
>>>>    };
>>>>
>>>> +
>>>> +static const struct qcom_smmu_match_data sm8550_smmu_500_impl0_data = {
>>>> +    .impl = &qcom_smmu_500_impl,
>>>> +    .adreno_impl = &qcom_adreno_smmu_500_impl,
>>>> +    .cfg = &qcom_smmu_impl0_cfg,
>>>> +    .actlrvar = sm8550_actlr,
>>>> +    .num_smmu = ARRAY_SIZE(sm8550_actlr),
>>>> +};
>>>> +
>>>>    static const struct qcom_smmu_match_data qcom_smmu_500_impl0_data = {
>>>>       .impl = &qcom_smmu_500_impl,
>>>>       .adreno_impl = &qcom_adreno_smmu_500_impl,
>>>> @@ -640,6 +728,7 @@ static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
>>>>       { .compatible = "qcom,sm8250-smmu-500", .data = &qcom_smmu_500_impl0_data },
>>>>       { .compatible = "qcom,sm8350-smmu-500", .data = &qcom_smmu_500_impl0_data },
>>>>       { .compatible = "qcom,sm8450-smmu-500", .data = &qcom_smmu_500_impl0_data },
>>>> +    { .compatible = "qcom,sm8550-smmu-500", .data = &sm8550_smmu_500_impl0_data },
>>>>       { .compatible = "qcom,smmu-500", .data = &qcom_smmu_500_impl0_data },
>>>>       { }
>>>>    };
>>>> --
>>>> 2.34.1
>>>>
>>>
> 
> 
> 
> --
> With best wishes
> Dmitry

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Konrad Dybcio]

On 10.07.2024 7:01 PM, Rob Clark wrote:
> On Tue, Jul 9, 2024 at 12:43 PM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:

[...]

> If there is a clean break, such as all smmu-500 have PRR and all
> smmu-v2 do not, then it would be reasonable to use the compat strings.
> If not, I think we need a different way.

if (a reserved-memory region is passed in dt) {
	init_prr()
}

Konrad

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 7/10/2024 10:31 PM, Rob Clark wrote:
> On Tue, Jul 9, 2024 at 12:43 PM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 7/4/2024 9:28 PM, Rob Clark wrote:
>>> On Thu, Jul 4, 2024 at 7:46 AM Rob Clark <robdclark@gmail.com> wrote:
>>>>
>>>> On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 7/2/2024 2:01 AM, Rob Clark wrote:
>>>>>> On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 6/28/2024 9:14 PM, Rob Clark wrote:
>>>>>>>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
>>>>>>>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
>>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>>>>>>>> sequence as per request.
>>>>>>>>>>>
>>>>>>>>>>> This will be used by GPU to setup the PRR bit and related
>>>>>>>>>>> configuration registers through adreno-smmu private
>>>>>>>>>>> interface instead of directly poking the smmu hardware.
>>>>>>>>>>>
>>>>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>>>>>>>> ---
>>>>>>>>>>>       drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
>>>>>>>>>>>       drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>>>>>>>       include/linux/adreno-smmu-priv.h           |  6 +++++-
>>>>>>>>>>>       3 files changed, 30 insertions(+), 1 deletion(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>> index bd101a161d04..64571a1c47b8 100644
>>>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>> @@ -28,6 +28,7 @@
>>>>>>>>>>>       #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
>>>>>>>>>>>       #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
>>>>>>>>>>>       #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
>>>>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>>>>>>>>
>>>>>>>>>>>       static const struct actlr_config sc7280_apps_actlr_cfg[] = {
>>>>>>>>>>>              { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
>>>>>>>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>>>>>>>              arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>>>>>>>       }
>>>>>>>>>>>
>>>>>>>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
>>>>>>>>>>> +{
>>>>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>>>>>>> +       u32 reg = 0;
>>>>>>>>>>> +
>>>>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>>>> +
>>>>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>>>> +
>>>>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>>>>>>>> +       if (set)
>>>>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>>>> +
>>>>>>>>>>
>>>>>>>>>> nit, extra line
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ack, will remove this. Thanks for pointing out.
>>>>>>>>>
>>>>>>>>>> Also, if you passed a `struct page *` instead, then you could drop the
>>>>>>>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
>>>>>>>>>> can go either way if others have a strong preference for phys_addr_t.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Oh okay, this looks simple to reset the prr bit.
>>>>>>>>> But since this page is allocated and is used inside gfx driver
>>>>>>>>> before being utilized for prr bit operation, would it be safe for
>>>>>>>>> drm/gfx driver to keep a reference to this page in smmu driver?
>>>>>>>>>
>>>>>>>>> Since we only need the page address for configuring the
>>>>>>>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
>>>>>>>>
>>>>>>>> I don't think the smmu driver needs to keep a reference to the page..
>>>>>>>> we can just say it is the responsibility of the drm driver to call
>>>>>>>> set_prr(NULL) before freeing the page
>>>>>>>>
>>>>>>>
>>>>>>> That makes sense. If we go by this NULL page method to disable the PRR,
>>>>>>> we would have to set the address registers to reset value as well.
>>>>>>>
>>>>>>> The sequence would be like the following as per my understaning:
>>>>>>> - Check if it's NULL page
>>>>>>> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
>>>>>>>       these registers
>>>>>>> - Reset the PRR bit in actlr register
>>>>>>>
>>>>>>> Similar to this snippet:
>>>>>>>
>>>>>>> #PRR_RESET_ADDR 0x0
>>>>>>>
>>>>>>> --------------
>>>>>>> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>> reg &= ~GFX_ACTLR_PRR;
>>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>
>>>>>>> if (!prr_page) {
>>>>>>>            writel_relaxed(PRR_RESET_ADDR,
>>>>>>>                            smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>            writel_relaxed(PRR_RESET_ADDR),
>>>>>>>                             smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>            return;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
>>>>>>>                    smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>
>>>>>>> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
>>>>>>>                    smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>
>>>>>>> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>> -----------------
>>>>>>>
>>>>>>> If looks good, will implement the same in next version.
>>>>>>
>>>>>> yeah, that looks like it could work..
>>>>>>
>>>>>> you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
>>>>>> and probably could avoid double writing ACTLR, but that is getting
>>>>>> into bikeshedding
>>>>>>
>>>>>
>>>>> Actually Rob, since you rightly pointed this out.
>>>>> I crosschecked again on these registers.
>>>>> PRR_CFG_*ADDR is a global register in SMMU space but
>>>>> ACTLR register including PRR bit is a per-domain register.
>>>>> There might also be a situation where PRR feature need to be
>>>>> disabled or enabled separately for each domain.
>>>>> So I think it would be cleaner to have two apis, set_prr_addr(),
>>>>> set_prr_bit().
>>>>> set_prr_addr() will be used only to set this PRR_CFG_*ADDR
>>>>> register by passing a 'struct page *'
>>>>> set_prr_bit() will be used as a switch for PRR feature,
>>>>> where required smmu_domain will be passed along with
>>>>> the bool value to set/reset the PRR bit depending on which this
>>>>> feature will be enabled/disabled for the selected domain.
>>>>
>>>> on a related note, adreno has been using arm-smmu for a number of
>>>> generations, I guess not all support PRR?  The drm driver will need to
>>>> know whether PRR is supported (and expose that to userspace to let the
>>>> UMD know whether to expose certain extensions).  How should this work?
>>>
>>> So, I noticed in the x1e80100.dtsi that there is a gpu_prr_mem
>>> reserved section..  maybe we should be connecting this to the smmu
>>> driver in dt, and using that to detect presence of PRR?  Ie. the smmu
>>> driver would configure PRR_CFG_*ADDR based on the reserved mem, and
>>> the interface to drm would just be to enable/disable PRR, returning an
>>> error code if the reserved mem section isn't there.
>>>
>>> This simplifies the interface, and handles the question of how to
>>> detect if PRR is supported.
>>>
>>> BR,
>>> -R
>>>
>>
>> As I checked gpu_prr_mem reserved mem section is not used for mobile
>> targets hence not present for other DT only compute targets like
>> x1e80100.dtsi has the same. PRR looks to be smmu version specific
>> property.
> 
> I only see it in gpu_prr_mem in x1e80100.dtsi, but not documented
> anywhere.  I'm only assuming based on the name that it is intended to
> be for PRR (but not sure why it is larger than 0x1000).  Are the
> PRR_CFG_*ADDR regs programmed by the fw (and access blocked in EL1) on
> this device?
> 

As I checked, if the drm/gfx driver allocates the page for drm, then 
this reserved-memory region is not required.

PRR_CFG_*ADDR regs have read and write access in EL1 only for this 
device, behavior is same as other devices as well. These are not
programmed by fw.

> As far as other DT, we haven't enabled PRR anywhere yet.  I think it
> would be perfectly valid to require a new reserved-memory node to
> enable PRR.
> 

As mentioned above reserved-memory region is not required if gfx/drm
allocates the page.

>> This feature is present in all the targets using SMMU-500,
>> I am still checking for targets using versions prior to smmu-500.
>> Then maybe we can use the smmu compatible string itself (e.g.
>> qcom,smmu-500 && qcom,adreno-smmu) to connect to smmu driver for
>> checking the presence of PRR ?
> 
> If there is a clean break, such as all smmu-500 have PRR and all
> smmu-v2 do not, then it would be reasonable to use the compat strings.
> If not, I think we need a different way.
> 

Yes, from SMMU block perspective PRR bit is present for ACTLR register
on targets with MMU-500, so feature support is available. So I think we 
can just use the mmu-500 compatible string.

Thanks & regards,
Bibek

> BR,
> -R
> 
>> And if the compatible string is different then we can return the
>> error code signifying PRR feature is not supported on the particular
>> smmu version.
>>
>> Thanks & regards,
>> Bibek
>>
>>>> BR,
>>>> -R
>>>>
>>>>> Thanks & regards,
>>>>> Bibek
>>>>>
>>>>>> BR,
>>>>>> -R
>>>>>>
>>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Bibek
>>>>>>>
>>>>>>>> BR,
>>>>>>>> -R
>>>>>>>>
>>>>>>>>>> Otherwise, lgtm
>>>>>>>>>>
>>>>>>>>>> BR,
>>>>>>>>>> -R
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks & regards,
>>>>>>>>> Bibek
>>>>>>>>>
>>>>>>>>>>> +}
>>>>>>>>>>> +
>>>>>>>>>>>       #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>>>>>>>>
>>>>>>>>>>>       static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>>>>>>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>>>>>>              priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>>>>>>>>              priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>>>>>>>>              priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>>>>>>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
>>>>>>>>>>>
>>>>>>>>>>>              actlrvar = qsmmu->data->actlrvar;
>>>>>>>>>>>              if (!actlrvar)
>>>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>>>> index d9c2ef8c1653..3076bef49e20 100644
>>>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>>>>>>>>       #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>>>>>>>>
>>>>>>>>>>>       #define ARM_SMMU_CB_ACTLR              0x4
>>>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>>>>>>>>
>>>>>>>>>>>       #define ARM_SMMU_CB_RESUME             0x8
>>>>>>>>>>>       #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>>>>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>>>>>>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
>>>>>>>>>>> --- a/include/linux/adreno-smmu-priv.h
>>>>>>>>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>>>>>>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
>>>>>>>>>>>        *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
>>>>>>>>>>>        *                 the GPU driver must call resume_translation()
>>>>>>>>>>>        * @resume_translation: Resume translation after a fault
>>>>>>>>>>> - *
>>>>>>>>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
>>>>>>>>>>> + *                 ACTLR register bits, currently used to configure
>>>>>>>>>>> + *                 Partially-Resident-Region (PRR) feature's
>>>>>>>>>>> + *                 setup and reset sequence as requested.
>>>>>>>>>>>        *
>>>>>>>>>>>        * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>>>>>>>>        * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>>>>>>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
>>>>>>>>>>>           void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>>>>>>>>           void (*set_stall)(const void *cookie, bool enabled);
>>>>>>>>>>>           void (*resume_translation)(const void *cookie, bool terminate);
>>>>>>>>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
>>>>>>>>>>>       };
>>>>>>>>>>>
>>>>>>>>>>>       #endif /* __ADRENO_SMMU_PRIV_H */
>>>>>>>>>>> --
>>>>>>>>>>> 2.34.1
>>>>>>>>>>>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 7/11/2024 3:31 AM, Konrad Dybcio wrote:
> On 10.07.2024 7:01 PM, Rob Clark wrote:
>> On Tue, Jul 9, 2024 at 12:43 PM Bibek Kumar Patro
>> <quic_bibekkum@quicinc.com> wrote:
> 
> [...]
> 
>> If there is a clean break, such as all smmu-500 have PRR and all
>> smmu-v2 do not, then it would be reasonable to use the compat strings.
>> If not, I think we need a different way.
> 
> if (a reserved-memory region is passed in dt) {
> 	init_prr()
> }
> 

This would have been a straight forward way to pass as a check for PRR
support, but as I mentioned in the above thread reserved-memory region 
is not required if gfx/drm allocates the page.
So I think compatible string would be more approachable way?

Thanks & regards,
Bibek

> Konrad

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Mon, Jul 15, 2024 at 4:00 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 7/10/2024 10:31 PM, Rob Clark wrote:
> > On Tue, Jul 9, 2024 at 12:43 PM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >>
> >>
> >> On 7/4/2024 9:28 PM, Rob Clark wrote:
> >>> On Thu, Jul 4, 2024 at 7:46 AM Rob Clark <robdclark@gmail.com> wrote:
> >>>>
> >>>> On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
> >>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 7/2/2024 2:01 AM, Rob Clark wrote:
> >>>>>> On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
> >>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 6/28/2024 9:14 PM, Rob Clark wrote:
> >>>>>>>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
> >>>>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
> >>>>>>>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
> >>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>>>>>>>>> sequence as per request.
> >>>>>>>>>>>
> >>>>>>>>>>> This will be used by GPU to setup the PRR bit and related
> >>>>>>>>>>> configuration registers through adreno-smmu private
> >>>>>>>>>>> interface instead of directly poking the smmu hardware.
> >>>>>>>>>>>
> >>>>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>>>>>>>>> ---
> >>>>>>>>>>>       drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
> >>>>>>>>>>>       drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>>>>>>>>       include/linux/adreno-smmu-priv.h           |  6 +++++-
> >>>>>>>>>>>       3 files changed, 30 insertions(+), 1 deletion(-)
> >>>>>>>>>>>
> >>>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>>>> index bd101a161d04..64571a1c47b8 100644
> >>>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>>>> @@ -28,6 +28,7 @@
> >>>>>>>>>>>       #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
> >>>>>>>>>>>       #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
> >>>>>>>>>>>       #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
> >>>>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>>>>>>>>>
> >>>>>>>>>>>       static const struct actlr_config sc7280_apps_actlr_cfg[] = {
> >>>>>>>>>>>              { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
> >>>>>>>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>>>>>>>>              arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>>>>>>>>       }
> >>>>>>>>>>>
> >>>>>>>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
> >>>>>>>>>>> +{
> >>>>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>>>>>>>> +       u32 reg = 0;
> >>>>>>>>>>> +
> >>>>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>>>>>> +
> >>>>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>>>>>> +
> >>>>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>>>>>>>>> +       if (set)
> >>>>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>>>>>> +
> >>>>>>>>>>
> >>>>>>>>>> nit, extra line
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Ack, will remove this. Thanks for pointing out.
> >>>>>>>>>
> >>>>>>>>>> Also, if you passed a `struct page *` instead, then you could drop the
> >>>>>>>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
> >>>>>>>>>> can go either way if others have a strong preference for phys_addr_t.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Oh okay, this looks simple to reset the prr bit.
> >>>>>>>>> But since this page is allocated and is used inside gfx driver
> >>>>>>>>> before being utilized for prr bit operation, would it be safe for
> >>>>>>>>> drm/gfx driver to keep a reference to this page in smmu driver?
> >>>>>>>>>
> >>>>>>>>> Since we only need the page address for configuring the
> >>>>>>>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
> >>>>>>>>
> >>>>>>>> I don't think the smmu driver needs to keep a reference to the page..
> >>>>>>>> we can just say it is the responsibility of the drm driver to call
> >>>>>>>> set_prr(NULL) before freeing the page
> >>>>>>>>
> >>>>>>>
> >>>>>>> That makes sense. If we go by this NULL page method to disable the PRR,
> >>>>>>> we would have to set the address registers to reset value as well.
> >>>>>>>
> >>>>>>> The sequence would be like the following as per my understaning:
> >>>>>>> - Check if it's NULL page
> >>>>>>> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
> >>>>>>>       these registers
> >>>>>>> - Reset the PRR bit in actlr register
> >>>>>>>
> >>>>>>> Similar to this snippet:
> >>>>>>>
> >>>>>>> #PRR_RESET_ADDR 0x0
> >>>>>>>
> >>>>>>> --------------
> >>>>>>> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>>>> reg &= ~GFX_ACTLR_PRR;
> >>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>>
> >>>>>>> if (!prr_page) {
> >>>>>>>            writel_relaxed(PRR_RESET_ADDR,
> >>>>>>>                            smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>>            writel_relaxed(PRR_RESET_ADDR),
> >>>>>>>                             smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>>            return;
> >>>>>>> }
> >>>>>>>
> >>>>>>>
> >>>>>>> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
> >>>>>>>                    smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>>
> >>>>>>> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
> >>>>>>>                    smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>>
> >>>>>>> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>> -----------------
> >>>>>>>
> >>>>>>> If looks good, will implement the same in next version.
> >>>>>>
> >>>>>> yeah, that looks like it could work..
> >>>>>>
> >>>>>> you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
> >>>>>> and probably could avoid double writing ACTLR, but that is getting
> >>>>>> into bikeshedding
> >>>>>>
> >>>>>
> >>>>> Actually Rob, since you rightly pointed this out.
> >>>>> I crosschecked again on these registers.
> >>>>> PRR_CFG_*ADDR is a global register in SMMU space but
> >>>>> ACTLR register including PRR bit is a per-domain register.
> >>>>> There might also be a situation where PRR feature need to be
> >>>>> disabled or enabled separately for each domain.
> >>>>> So I think it would be cleaner to have two apis, set_prr_addr(),
> >>>>> set_prr_bit().
> >>>>> set_prr_addr() will be used only to set this PRR_CFG_*ADDR
> >>>>> register by passing a 'struct page *'
> >>>>> set_prr_bit() will be used as a switch for PRR feature,
> >>>>> where required smmu_domain will be passed along with
> >>>>> the bool value to set/reset the PRR bit depending on which this
> >>>>> feature will be enabled/disabled for the selected domain.
> >>>>
> >>>> on a related note, adreno has been using arm-smmu for a number of
> >>>> generations, I guess not all support PRR?  The drm driver will need to
> >>>> know whether PRR is supported (and expose that to userspace to let the
> >>>> UMD know whether to expose certain extensions).  How should this work?
> >>>
> >>> So, I noticed in the x1e80100.dtsi that there is a gpu_prr_mem
> >>> reserved section..  maybe we should be connecting this to the smmu
> >>> driver in dt, and using that to detect presence of PRR?  Ie. the smmu
> >>> driver would configure PRR_CFG_*ADDR based on the reserved mem, and
> >>> the interface to drm would just be to enable/disable PRR, returning an
> >>> error code if the reserved mem section isn't there.
> >>>
> >>> This simplifies the interface, and handles the question of how to
> >>> detect if PRR is supported.
> >>>
> >>> BR,
> >>> -R
> >>>
> >>
> >> As I checked gpu_prr_mem reserved mem section is not used for mobile
> >> targets hence not present for other DT only compute targets like
> >> x1e80100.dtsi has the same. PRR looks to be smmu version specific
> >> property.
> >
> > I only see it in gpu_prr_mem in x1e80100.dtsi, but not documented
> > anywhere.  I'm only assuming based on the name that it is intended to
> > be for PRR (but not sure why it is larger than 0x1000).  Are the
> > PRR_CFG_*ADDR regs programmed by the fw (and access blocked in EL1) on
> > this device?
> >
>
> As I checked, if the drm/gfx driver allocates the page for drm, then
> this reserved-memory region is not required.
>
> PRR_CFG_*ADDR regs have read and write access in EL1 only for this
> device, behavior is same as other devices as well. These are not
> programmed by fw.

If there is any device which _doesn't_ have EL1 access to these regs,
I think going the reserved memory route seems more future proof?
Otherwise we later on have to deal with two different ways to do
things.  But I'm not sure if there is any such device or risk.

BR,
-R

> > As far as other DT, we haven't enabled PRR anywhere yet.  I think it
> > would be perfectly valid to require a new reserved-memory node to
> > enable PRR.
> >
>
> As mentioned above reserved-memory region is not required if gfx/drm
> allocates the page.
>
> >> This feature is present in all the targets using SMMU-500,
> >> I am still checking for targets using versions prior to smmu-500.
> >> Then maybe we can use the smmu compatible string itself (e.g.
> >> qcom,smmu-500 && qcom,adreno-smmu) to connect to smmu driver for
> >> checking the presence of PRR ?
> >
> > If there is a clean break, such as all smmu-500 have PRR and all
> > smmu-v2 do not, then it would be reasonable to use the compat strings.
> > If not, I think we need a different way.
> >
>
> Yes, from SMMU block perspective PRR bit is present for ACTLR register
> on targets with MMU-500, so feature support is available. So I think we
> can just use the mmu-500 compatible string.
>
> Thanks & regards,
> Bibek
>
> > BR,
> > -R
> >
> >> And if the compatible string is different then we can return the
> >> error code signifying PRR feature is not supported on the particular
> >> smmu version.
> >>
> >> Thanks & regards,
> >> Bibek
> >>
> >>>> BR,
> >>>> -R
> >>>>
> >>>>> Thanks & regards,
> >>>>> Bibek
> >>>>>
> >>>>>> BR,
> >>>>>> -R
> >>>>>>
> >>>>>>>
> >>>>>>> Thanks & regards,
> >>>>>>> Bibek
> >>>>>>>
> >>>>>>>> BR,
> >>>>>>>> -R
> >>>>>>>>
> >>>>>>>>>> Otherwise, lgtm
> >>>>>>>>>>
> >>>>>>>>>> BR,
> >>>>>>>>>> -R
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks & regards,
> >>>>>>>>> Bibek
> >>>>>>>>>
> >>>>>>>>>>> +}
> >>>>>>>>>>> +
> >>>>>>>>>>>       #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>>>>>>>>>
> >>>>>>>>>>>       static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>>>>>>>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>>>>>>>              priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>>>>>>>>>              priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>>>>>>>>>              priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>>>>>>>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
> >>>>>>>>>>>
> >>>>>>>>>>>              actlrvar = qsmmu->data->actlrvar;
> >>>>>>>>>>>              if (!actlrvar)
> >>>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>>>>>> index d9c2ef8c1653..3076bef49e20 100644
> >>>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>>>>>>>>>>       #define ARM_SMMU_SCTLR_M               BIT(0)
> >>>>>>>>>>>
> >>>>>>>>>>>       #define ARM_SMMU_CB_ACTLR              0x4
> >>>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >>>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>>>>>>>>>>
> >>>>>>>>>>>       #define ARM_SMMU_CB_RESUME             0x8
> >>>>>>>>>>>       #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >>>>>>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >>>>>>>>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
> >>>>>>>>>>> --- a/include/linux/adreno-smmu-priv.h
> >>>>>>>>>>> +++ b/include/linux/adreno-smmu-priv.h
> >>>>>>>>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
> >>>>>>>>>>>        *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
> >>>>>>>>>>>        *                 the GPU driver must call resume_translation()
> >>>>>>>>>>>        * @resume_translation: Resume translation after a fault
> >>>>>>>>>>> - *
> >>>>>>>>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
> >>>>>>>>>>> + *                 ACTLR register bits, currently used to configure
> >>>>>>>>>>> + *                 Partially-Resident-Region (PRR) feature's
> >>>>>>>>>>> + *                 setup and reset sequence as requested.
> >>>>>>>>>>>        *
> >>>>>>>>>>>        * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>>>>>>>>>>        * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >>>>>>>>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
> >>>>>>>>>>>           void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>>>>>>>>>>           void (*set_stall)(const void *cookie, bool enabled);
> >>>>>>>>>>>           void (*resume_translation)(const void *cookie, bool terminate);
> >>>>>>>>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
> >>>>>>>>>>>       };
> >>>>>>>>>>>
> >>>>>>>>>>>       #endif /* __ADRENO_SMMU_PRIV_H */
> >>>>>>>>>>> --
> >>>>>>>>>>> 2.34.1
> >>>>>>>>>>>

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Konrad Dybcio]

On 15.07.2024 10:07 PM, Rob Clark wrote:
> On Mon, Jul 15, 2024 at 4:00 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:

[...]

>>>> As I checked gpu_prr_mem reserved mem section is not used for mobile
>>>> targets hence not present for other DT only compute targets like
>>>> x1e80100.dtsi has the same. PRR looks to be smmu version specific
>>>> property.
>>>
>>> I only see it in gpu_prr_mem in x1e80100.dtsi, but not documented
>>> anywhere.  I'm only assuming based on the name that it is intended to
>>> be for PRR (but not sure why it is larger than 0x1000).  Are the
>>> PRR_CFG_*ADDR regs programmed by the fw (and access blocked in EL1) on
>>> this device?
>>>
>>
>> As I checked, if the drm/gfx driver allocates the page for drm, then
>> this reserved-memory region is not required.
>>
>> PRR_CFG_*ADDR regs have read and write access in EL1 only for this
>> device, behavior is same as other devices as well. These are not
>> programmed by fw.
> 
> If there is any device which _doesn't_ have EL1 access to these regs,
> I think going the reserved memory route seems more future proof?
> Otherwise we later on have to deal with two different ways to do
> things.  But I'm not sure if there is any such device or risk.

We can have our cake and eat it too, if we keep the check for a
reserved memory node handle, but make it a dynamic allocation (see
[1] for example), this way there's a way to opt into using this from
the DT and there's no need for adding more properties

Konrad

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/qcom/msm8916.dtsi?h=v6.10#n109

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 7/16/2024 1:37 AM, Rob Clark wrote:
> On Mon, Jul 15, 2024 at 4:00 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 7/10/2024 10:31 PM, Rob Clark wrote:
>>> On Tue, Jul 9, 2024 at 12:43 PM Bibek Kumar Patro
>>> <quic_bibekkum@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 7/4/2024 9:28 PM, Rob Clark wrote:
>>>>> On Thu, Jul 4, 2024 at 7:46 AM Rob Clark <robdclark@gmail.com> wrote:
>>>>>>
>>>>>> On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 7/2/2024 2:01 AM, Rob Clark wrote:
>>>>>>>> On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 6/28/2024 9:14 PM, Rob Clark wrote:
>>>>>>>>>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
>>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
>>>>>>>>>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
>>>>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>>>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>>>>>>>>>> sequence as per request.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This will be used by GPU to setup the PRR bit and related
>>>>>>>>>>>>> configuration registers through adreno-smmu private
>>>>>>>>>>>>> interface instead of directly poking the smmu hardware.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>>>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>        drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
>>>>>>>>>>>>>        drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>>>>>>>>>        include/linux/adreno-smmu-priv.h           |  6 +++++-
>>>>>>>>>>>>>        3 files changed, 30 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>>>> index bd101a161d04..64571a1c47b8 100644
>>>>>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>>>> @@ -28,6 +28,7 @@
>>>>>>>>>>>>>        #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
>>>>>>>>>>>>>        #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
>>>>>>>>>>>>>        #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
>>>>>>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>>>>>>>>>>
>>>>>>>>>>>>>        static const struct actlr_config sc7280_apps_actlr_cfg[] = {
>>>>>>>>>>>>>               { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
>>>>>>>>>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>>>>>>>>>               arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>>>>>>>>>        }
>>>>>>>>>>>>>
>>>>>>>>>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
>>>>>>>>>>>>> +{
>>>>>>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>>>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>>>>>>>>> +       u32 reg = 0;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>>>>>>>>>> +       if (set)
>>>>>>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>>>>>> +
>>>>>>>>>>>>
>>>>>>>>>>>> nit, extra line
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ack, will remove this. Thanks for pointing out.
>>>>>>>>>>>
>>>>>>>>>>>> Also, if you passed a `struct page *` instead, then you could drop the
>>>>>>>>>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
>>>>>>>>>>>> can go either way if others have a strong preference for phys_addr_t.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Oh okay, this looks simple to reset the prr bit.
>>>>>>>>>>> But since this page is allocated and is used inside gfx driver
>>>>>>>>>>> before being utilized for prr bit operation, would it be safe for
>>>>>>>>>>> drm/gfx driver to keep a reference to this page in smmu driver?
>>>>>>>>>>>
>>>>>>>>>>> Since we only need the page address for configuring the
>>>>>>>>>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
>>>>>>>>>>
>>>>>>>>>> I don't think the smmu driver needs to keep a reference to the page..
>>>>>>>>>> we can just say it is the responsibility of the drm driver to call
>>>>>>>>>> set_prr(NULL) before freeing the page
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> That makes sense. If we go by this NULL page method to disable the PRR,
>>>>>>>>> we would have to set the address registers to reset value as well.
>>>>>>>>>
>>>>>>>>> The sequence would be like the following as per my understaning:
>>>>>>>>> - Check if it's NULL page
>>>>>>>>> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
>>>>>>>>>        these registers
>>>>>>>>> - Reset the PRR bit in actlr register
>>>>>>>>>
>>>>>>>>> Similar to this snippet:
>>>>>>>>>
>>>>>>>>> #PRR_RESET_ADDR 0x0
>>>>>>>>>
>>>>>>>>> --------------
>>>>>>>>> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>>>> reg &= ~GFX_ACTLR_PRR;
>>>>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>>
>>>>>>>>> if (!prr_page) {
>>>>>>>>>             writel_relaxed(PRR_RESET_ADDR,
>>>>>>>>>                             smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>>             writel_relaxed(PRR_RESET_ADDR),
>>>>>>>>>                              smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>>             return;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
>>>>>>>>>                     smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>>
>>>>>>>>> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
>>>>>>>>>                     smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>>
>>>>>>>>> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>> -----------------
>>>>>>>>>
>>>>>>>>> If looks good, will implement the same in next version.
>>>>>>>>
>>>>>>>> yeah, that looks like it could work..
>>>>>>>>
>>>>>>>> you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
>>>>>>>> and probably could avoid double writing ACTLR, but that is getting
>>>>>>>> into bikeshedding
>>>>>>>>
>>>>>>>
>>>>>>> Actually Rob, since you rightly pointed this out.
>>>>>>> I crosschecked again on these registers.
>>>>>>> PRR_CFG_*ADDR is a global register in SMMU space but
>>>>>>> ACTLR register including PRR bit is a per-domain register.
>>>>>>> There might also be a situation where PRR feature need to be
>>>>>>> disabled or enabled separately for each domain.
>>>>>>> So I think it would be cleaner to have two apis, set_prr_addr(),
>>>>>>> set_prr_bit().
>>>>>>> set_prr_addr() will be used only to set this PRR_CFG_*ADDR
>>>>>>> register by passing a 'struct page *'
>>>>>>> set_prr_bit() will be used as a switch for PRR feature,
>>>>>>> where required smmu_domain will be passed along with
>>>>>>> the bool value to set/reset the PRR bit depending on which this
>>>>>>> feature will be enabled/disabled for the selected domain.
>>>>>>
>>>>>> on a related note, adreno has been using arm-smmu for a number of
>>>>>> generations, I guess not all support PRR?  The drm driver will need to
>>>>>> know whether PRR is supported (and expose that to userspace to let the
>>>>>> UMD know whether to expose certain extensions).  How should this work?
>>>>>
>>>>> So, I noticed in the x1e80100.dtsi that there is a gpu_prr_mem
>>>>> reserved section..  maybe we should be connecting this to the smmu
>>>>> driver in dt, and using that to detect presence of PRR?  Ie. the smmu
>>>>> driver would configure PRR_CFG_*ADDR based on the reserved mem, and
>>>>> the interface to drm would just be to enable/disable PRR, returning an
>>>>> error code if the reserved mem section isn't there.
>>>>>
>>>>> This simplifies the interface, and handles the question of how to
>>>>> detect if PRR is supported.
>>>>>
>>>>> BR,
>>>>> -R
>>>>>
>>>>
>>>> As I checked gpu_prr_mem reserved mem section is not used for mobile
>>>> targets hence not present for other DT only compute targets like
>>>> x1e80100.dtsi has the same. PRR looks to be smmu version specific
>>>> property.
>>>
>>> I only see it in gpu_prr_mem in x1e80100.dtsi, but not documented
>>> anywhere.  I'm only assuming based on the name that it is intended to
>>> be for PRR (but not sure why it is larger than 0x1000).  Are the
>>> PRR_CFG_*ADDR regs programmed by the fw (and access blocked in EL1) on
>>> this device?
>>>
>>
>> As I checked, if the drm/gfx driver allocates the page for drm, then
>> this reserved-memory region is not required.
>>
>> PRR_CFG_*ADDR regs have read and write access in EL1 only for this
>> device, behavior is same as other devices as well. These are not
>> programmed by fw.
> 
> If there is any device which _doesn't_ have EL1 access to these regs,
> I think going the reserved memory route seems more future proof > Otherwise we later on have to deal with two different ways to do
> things.  But I'm not sure if there is any such device or risk.
> 

PRR is a bit in ACTLR register which is in SMMU space,
so is the PRR_CFG_*ADDR registers - with EL1 having access
to both the registers in all targets released till now with MMU-500.
It's unlikely that this design would change in future
for MMU-500 based targets, so I feel this risk is somewhat negligible.

Also would the reserved memory route look a bit hackish?
Because, since this reserved-memory node is not used when page is
allocated through drm - so it might turn out to be redundant.
If we are aiming for a device-tree handle/node for reference then I
think a better way would be to create a bool parameter inside smmu-node 
indicating presence of PRR ?

Personally,I feel since the PRR enablement mechanism is same for all
MMU-500 targets - compat string would be a robust approach.

Thanks & regards,
Bibek

> BR,
> -R
> 
>>> As far as other DT, we haven't enabled PRR anywhere yet.  I think it
>>> would be perfectly valid to require a new reserved-memory node to
>>> enable PRR.
>>>
>>
>> As mentioned above reserved-memory region is not required if gfx/drm
>> allocates the page.
>>
>>>> This feature is present in all the targets using SMMU-500,
>>>> I am still checking for targets using versions prior to smmu-500.
>>>> Then maybe we can use the smmu compatible string itself (e.g.
>>>> qcom,smmu-500 && qcom,adreno-smmu) to connect to smmu driver for
>>>> checking the presence of PRR ?
>>>
>>> If there is a clean break, such as all smmu-500 have PRR and all
>>> smmu-v2 do not, then it would be reasonable to use the compat strings.
>>> If not, I think we need a different way.
>>>
>>
>> Yes, from SMMU block perspective PRR bit is present for ACTLR register
>> on targets with MMU-500, so feature support is available. So I think we
>> can just use the mmu-500 compatible string.
>>
>> Thanks & regards,
>> Bibek
>>
>>> BR,
>>> -R
>>>
>>>> And if the compatible string is different then we can return the
>>>> error code signifying PRR feature is not supported on the particular
>>>> smmu version.
>>>>
>>>> Thanks & regards,
>>>> Bibek
>>>>
>>>>>> BR,
>>>>>> -R
>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Bibek
>>>>>>>
>>>>>>>> BR,
>>>>>>>> -R
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks & regards,
>>>>>>>>> Bibek
>>>>>>>>>
>>>>>>>>>> BR,
>>>>>>>>>> -R
>>>>>>>>>>
>>>>>>>>>>>> Otherwise, lgtm
>>>>>>>>>>>>
>>>>>>>>>>>> BR,
>>>>>>>>>>>> -R
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks & regards,
>>>>>>>>>>> Bibek
>>>>>>>>>>>
>>>>>>>>>>>>> +}
>>>>>>>>>>>>> +
>>>>>>>>>>>>>        #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>>>>>>>>>>
>>>>>>>>>>>>>        static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>>>>>>>>>>> @@ -407,6 +429,7 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>>>>>>>>               priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>>>>>>>>>>               priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>>>>>>>>>>               priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>>>>>>>>>>> +       priv->set_prr = qcom_adreno_smmu_set_prr;
>>>>>>>>>>>>>
>>>>>>>>>>>>>               actlrvar = qsmmu->data->actlrvar;
>>>>>>>>>>>>>               if (!actlrvar)
>>>>>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>>>>>> index d9c2ef8c1653..3076bef49e20 100644
>>>>>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>>>>>>>>>>        #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>>>>>>>>>>
>>>>>>>>>>>>>        #define ARM_SMMU_CB_ACTLR              0x4
>>>>>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>>>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>>>>>>>>>>
>>>>>>>>>>>>>        #define ARM_SMMU_CB_RESUME             0x8
>>>>>>>>>>>>>        #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>>>>>>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>>>>>>>>>>> index c637e0997f6d..d6e2ca9f8d8c 100644
>>>>>>>>>>>>> --- a/include/linux/adreno-smmu-priv.h
>>>>>>>>>>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>>>>>>>>>>> @@ -49,7 +49,10 @@ struct adreno_smmu_fault_info {
>>>>>>>>>>>>>         *                 before set_ttbr0_cfg().  If stalling on fault is enabled,
>>>>>>>>>>>>>         *                 the GPU driver must call resume_translation()
>>>>>>>>>>>>>         * @resume_translation: Resume translation after a fault
>>>>>>>>>>>>> - *
>>>>>>>>>>>>> + * @set_prr:      Extendible interface to be used by GPU to modify the
>>>>>>>>>>>>> + *                 ACTLR register bits, currently used to configure
>>>>>>>>>>>>> + *                 Partially-Resident-Region (PRR) feature's
>>>>>>>>>>>>> + *                 setup and reset sequence as requested.
>>>>>>>>>>>>>         *
>>>>>>>>>>>>>         * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>>>>>>>>>>         * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>>>>>>>>>>> @@ -67,6 +70,7 @@ struct adreno_smmu_priv {
>>>>>>>>>>>>>            void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>>>>>>>>>>            void (*set_stall)(const void *cookie, bool enabled);
>>>>>>>>>>>>>            void (*resume_translation)(const void *cookie, bool terminate);
>>>>>>>>>>>>> +    void (*set_prr)(const void *cookie, phys_addr_t page_addr, bool set);
>>>>>>>>>>>>>        };
>>>>>>>>>>>>>
>>>>>>>>>>>>>        #endif /* __ADRENO_SMMU_PRIV_H */
>>>>>>>>>>>>> --
>>>>>>>>>>>>> 2.34.1
>>>>>>>>>>>>>
> 

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Wed, Jul 17, 2024 at 3:27 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 7/16/2024 1:37 AM, Rob Clark wrote:
> > On Mon, Jul 15, 2024 at 4:00 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >>
> >>
> >> On 7/10/2024 10:31 PM, Rob Clark wrote:
> >>> On Tue, Jul 9, 2024 at 12:43 PM Bibek Kumar Patro
> >>> <quic_bibekkum@quicinc.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 7/4/2024 9:28 PM, Rob Clark wrote:
> >>>>> On Thu, Jul 4, 2024 at 7:46 AM Rob Clark <robdclark@gmail.com> wrote:
> >>>>>>
> >>>>>> On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
> >>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 7/2/2024 2:01 AM, Rob Clark wrote:
> >>>>>>>> On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
> >>>>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 6/28/2024 9:14 PM, Rob Clark wrote:
> >>>>>>>>>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
> >>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
> >>>>>>>>>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
> >>>>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>>>>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>>>>>>>>>>> sequence as per request.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This will be used by GPU to setup the PRR bit and related
> >>>>>>>>>>>>> configuration registers through adreno-smmu private
> >>>>>>>>>>>>> interface instead of directly poking the smmu hardware.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>>>>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>>>>>>>>>>> ---
> >>>>>>>>>>>>>        drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
> >>>>>>>>>>>>>        drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>>>>>>>>>>        include/linux/adreno-smmu-priv.h           |  6 +++++-
> >>>>>>>>>>>>>        3 files changed, 30 insertions(+), 1 deletion(-)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>>>>>> index bd101a161d04..64571a1c47b8 100644
> >>>>>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>>>>>> @@ -28,6 +28,7 @@
> >>>>>>>>>>>>>        #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
> >>>>>>>>>>>>>        #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
> >>>>>>>>>>>>>        #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
> >>>>>>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>        static const struct actlr_config sc7280_apps_actlr_cfg[] = {
> >>>>>>>>>>>>>               { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
> >>>>>>>>>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>>>>>>>>>>               arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>>>>>>>>>>        }
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
> >>>>>>>>>>>>> +{
> >>>>>>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>>>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>>>>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>>>>>>>>>> +       u32 reg = 0;
> >>>>>>>>>>>>> +
> >>>>>>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>>>>>>>> +
> >>>>>>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>>>>>>>> +
> >>>>>>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>>>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>>>>>>>>>>> +       if (set)
> >>>>>>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>>>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>>>>>>>> +
> >>>>>>>>>>>>
> >>>>>>>>>>>> nit, extra line
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Ack, will remove this. Thanks for pointing out.
> >>>>>>>>>>>
> >>>>>>>>>>>> Also, if you passed a `struct page *` instead, then you could drop the
> >>>>>>>>>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
> >>>>>>>>>>>> can go either way if others have a strong preference for phys_addr_t.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Oh okay, this looks simple to reset the prr bit.
> >>>>>>>>>>> But since this page is allocated and is used inside gfx driver
> >>>>>>>>>>> before being utilized for prr bit operation, would it be safe for
> >>>>>>>>>>> drm/gfx driver to keep a reference to this page in smmu driver?
> >>>>>>>>>>>
> >>>>>>>>>>> Since we only need the page address for configuring the
> >>>>>>>>>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
> >>>>>>>>>>
> >>>>>>>>>> I don't think the smmu driver needs to keep a reference to the page..
> >>>>>>>>>> we can just say it is the responsibility of the drm driver to call
> >>>>>>>>>> set_prr(NULL) before freeing the page
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> That makes sense. If we go by this NULL page method to disable the PRR,
> >>>>>>>>> we would have to set the address registers to reset value as well.
> >>>>>>>>>
> >>>>>>>>> The sequence would be like the following as per my understaning:
> >>>>>>>>> - Check if it's NULL page
> >>>>>>>>> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
> >>>>>>>>>        these registers
> >>>>>>>>> - Reset the PRR bit in actlr register
> >>>>>>>>>
> >>>>>>>>> Similar to this snippet:
> >>>>>>>>>
> >>>>>>>>> #PRR_RESET_ADDR 0x0
> >>>>>>>>>
> >>>>>>>>> --------------
> >>>>>>>>> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>>>>>> reg &= ~GFX_ACTLR_PRR;
> >>>>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>>>>
> >>>>>>>>> if (!prr_page) {
> >>>>>>>>>             writel_relaxed(PRR_RESET_ADDR,
> >>>>>>>>>                             smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>>>>             writel_relaxed(PRR_RESET_ADDR),
> >>>>>>>>>                              smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>>>>             return;
> >>>>>>>>> }
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
> >>>>>>>>>                     smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>>>>
> >>>>>>>>> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
> >>>>>>>>>                     smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>>>>
> >>>>>>>>> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>>>> -----------------
> >>>>>>>>>
> >>>>>>>>> If looks good, will implement the same in next version.
> >>>>>>>>
> >>>>>>>> yeah, that looks like it could work..
> >>>>>>>>
> >>>>>>>> you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
> >>>>>>>> and probably could avoid double writing ACTLR, but that is getting
> >>>>>>>> into bikeshedding
> >>>>>>>>
> >>>>>>>
> >>>>>>> Actually Rob, since you rightly pointed this out.
> >>>>>>> I crosschecked again on these registers.
> >>>>>>> PRR_CFG_*ADDR is a global register in SMMU space but
> >>>>>>> ACTLR register including PRR bit is a per-domain register.
> >>>>>>> There might also be a situation where PRR feature need to be
> >>>>>>> disabled or enabled separately for each domain.
> >>>>>>> So I think it would be cleaner to have two apis, set_prr_addr(),
> >>>>>>> set_prr_bit().
> >>>>>>> set_prr_addr() will be used only to set this PRR_CFG_*ADDR
> >>>>>>> register by passing a 'struct page *'
> >>>>>>> set_prr_bit() will be used as a switch for PRR feature,
> >>>>>>> where required smmu_domain will be passed along with
> >>>>>>> the bool value to set/reset the PRR bit depending on which this
> >>>>>>> feature will be enabled/disabled for the selected domain.
> >>>>>>
> >>>>>> on a related note, adreno has been using arm-smmu for a number of
> >>>>>> generations, I guess not all support PRR?  The drm driver will need to
> >>>>>> know whether PRR is supported (and expose that to userspace to let the
> >>>>>> UMD know whether to expose certain extensions).  How should this work?
> >>>>>
> >>>>> So, I noticed in the x1e80100.dtsi that there is a gpu_prr_mem
> >>>>> reserved section..  maybe we should be connecting this to the smmu
> >>>>> driver in dt, and using that to detect presence of PRR?  Ie. the smmu
> >>>>> driver would configure PRR_CFG_*ADDR based on the reserved mem, and
> >>>>> the interface to drm would just be to enable/disable PRR, returning an
> >>>>> error code if the reserved mem section isn't there.
> >>>>>
> >>>>> This simplifies the interface, and handles the question of how to
> >>>>> detect if PRR is supported.
> >>>>>
> >>>>> BR,
> >>>>> -R
> >>>>>
> >>>>
> >>>> As I checked gpu_prr_mem reserved mem section is not used for mobile
> >>>> targets hence not present for other DT only compute targets like
> >>>> x1e80100.dtsi has the same. PRR looks to be smmu version specific
> >>>> property.
> >>>
> >>> I only see it in gpu_prr_mem in x1e80100.dtsi, but not documented
> >>> anywhere.  I'm only assuming based on the name that it is intended to
> >>> be for PRR (but not sure why it is larger than 0x1000).  Are the
> >>> PRR_CFG_*ADDR regs programmed by the fw (and access blocked in EL1) on
> >>> this device?
> >>>
> >>
> >> As I checked, if the drm/gfx driver allocates the page for drm, then
> >> this reserved-memory region is not required.
> >>
> >> PRR_CFG_*ADDR regs have read and write access in EL1 only for this
> >> device, behavior is same as other devices as well. These are not
> >> programmed by fw.
> >
> > If there is any device which _doesn't_ have EL1 access to these regs,
> > I think going the reserved memory route seems more future proof > Otherwise we later on have to deal with two different ways to do
> > things.  But I'm not sure if there is any such device or risk.
> >
>
> PRR is a bit in ACTLR register which is in SMMU space,
> so is the PRR_CFG_*ADDR registers - with EL1 having access
> to both the registers in all targets released till now with MMU-500.
> It's unlikely that this design would change in future
> for MMU-500 based targets, so I feel this risk is somewhat negligible.

I wasn't worried about the ACTLR register, but the PRR_CFG_*ADDR regs ;-)

IIRC those were in the SMMU global space, why hyp tends to like to own.

> Also would the reserved memory route look a bit hackish?
> Because, since this reserved-memory node is not used when page is
> allocated through drm - so it might turn out to be redundant.
> If we are aiming for a device-tree handle/node for reference then I
> think a better way would be to create a bool parameter inside smmu-node
> indicating presence of PRR ?

tbh, I don't think there is anything better or worse about having the
reserved-memory node vs dynamically allocating it.  (If we dynamically
allocate, we should remove the reserved memory node from
x1e80100.dtsi)

The thing I was more concerned about was whether there was any chance
that some existing or future SoC+fw combo _relied_ on a reserved
memory node and the fw programming PRR_CFG_*ADDR.  If there was any
chance of that, and we went the dynamic allocation route, then we'd
have some devices with a reserved memory node, and some without.  That
seems a bit ugly to me.

If there is no chance of this, then we can go either route.

> Personally,I feel since the PRR enablement mechanism is same for all
> MMU-500 targets - compat string would be a robust approach.

I guess if it is all mmu-500, then we can just pick based on compat
string.  If it turns out some subset of smmu-v2 have PRR, we can just
have a list of compat strings in arm-smmu-qcom.c.. there would only be
a finite # of them ;-)

BR,
-R

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 7/17/2024 8:30 PM, Rob Clark wrote:
> On Wed, Jul 17, 2024 at 3:27 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 7/16/2024 1:37 AM, Rob Clark wrote:
>>> On Mon, Jul 15, 2024 at 4:00 AM Bibek Kumar Patro
>>> <quic_bibekkum@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 7/10/2024 10:31 PM, Rob Clark wrote:
>>>>> On Tue, Jul 9, 2024 at 12:43 PM Bibek Kumar Patro
>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 7/4/2024 9:28 PM, Rob Clark wrote:
>>>>>>> On Thu, Jul 4, 2024 at 7:46 AM Rob Clark <robdclark@gmail.com> wrote:
>>>>>>>>
>>>>>>>> On Wed, Jul 3, 2024 at 4:38 AM Bibek Kumar Patro
>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 7/2/2024 2:01 AM, Rob Clark wrote:
>>>>>>>>>> On Mon, Jul 1, 2024 at 4:01 AM Bibek Kumar Patro
>>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 6/28/2024 9:14 PM, Rob Clark wrote:
>>>>>>>>>>>> On Fri, Jun 28, 2024 at 8:10 AM Bibek Kumar Patro
>>>>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 6/28/2024 7:47 PM, Rob Clark wrote:
>>>>>>>>>>>>>> On Fri, Jun 28, 2024 at 7:05 AM Bibek Kumar Patro
>>>>>>>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>>>>>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>>>>>>>>>>>> sequence as per request.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This will be used by GPU to setup the PRR bit and related
>>>>>>>>>>>>>>> configuration registers through adreno-smmu private
>>>>>>>>>>>>>>> interface instead of directly poking the smmu hardware.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>>>>>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>         drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 23 ++++++++++++++++++++++
>>>>>>>>>>>>>>>         drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>>>>>>>>>>>         include/linux/adreno-smmu-priv.h           |  6 +++++-
>>>>>>>>>>>>>>>         3 files changed, 30 insertions(+), 1 deletion(-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>>>>>> index bd101a161d04..64571a1c47b8 100644
>>>>>>>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>>>>>>>> @@ -28,6 +28,7 @@
>>>>>>>>>>>>>>>         #define PREFETCH_SHALLOW       (1 << PREFETCH_SHIFT)
>>>>>>>>>>>>>>>         #define PREFETCH_MODERATE      (2 << PREFETCH_SHIFT)
>>>>>>>>>>>>>>>         #define PREFETCH_DEEP          (3 << PREFETCH_SHIFT)
>>>>>>>>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>         static const struct actlr_config sc7280_apps_actlr_cfg[] = {
>>>>>>>>>>>>>>>                { 0x0800, 0x04e0, PREFETCH_DEFAULT | CMTLB },
>>>>>>>>>>>>>>> @@ -235,6 +236,27 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>>>>>>>>>>>                arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>>>>>>>>>>>         }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +static void qcom_adreno_smmu_set_prr(const void *cookie, phys_addr_t page_addr, bool set)
>>>>>>>>>>>>>>> +{
>>>>>>>>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>>>>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>>>>>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>>>>>>>>>>> +       u32 reg = 0;
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>>>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>>>>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>>>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>>>>>>>>>>>> +       if (set)
>>>>>>>>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>>>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> nit, extra line
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ack, will remove this. Thanks for pointing out.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, if you passed a `struct page *` instead, then you could drop the
>>>>>>>>>>>>>> bool param, ie. passing NULL for the page would disable PRR.  But I
>>>>>>>>>>>>>> can go either way if others have a strong preference for phys_addr_t.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Oh okay, this looks simple to reset the prr bit.
>>>>>>>>>>>>> But since this page is allocated and is used inside gfx driver
>>>>>>>>>>>>> before being utilized for prr bit operation, would it be safe for
>>>>>>>>>>>>> drm/gfx driver to keep a reference to this page in smmu driver?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Since we only need the page address for configuring the
>>>>>>>>>>>>> CFG_UADDR/CFG_LADDR registers so passed the phys_addr_t.
>>>>>>>>>>>>
>>>>>>>>>>>> I don't think the smmu driver needs to keep a reference to the page..
>>>>>>>>>>>> we can just say it is the responsibility of the drm driver to call
>>>>>>>>>>>> set_prr(NULL) before freeing the page
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That makes sense. If we go by this NULL page method to disable the PRR,
>>>>>>>>>>> we would have to set the address registers to reset value as well.
>>>>>>>>>>>
>>>>>>>>>>> The sequence would be like the following as per my understaning:
>>>>>>>>>>> - Check if it's NULL page
>>>>>>>>>>> - Set the PRR_CFG_UADDR/PRR_CFG_LADDR to reset values i.e - 0x0 for
>>>>>>>>>>>         these registers
>>>>>>>>>>> - Reset the PRR bit in actlr register
>>>>>>>>>>>
>>>>>>>>>>> Similar to this snippet:
>>>>>>>>>>>
>>>>>>>>>>> #PRR_RESET_ADDR 0x0
>>>>>>>>>>>
>>>>>>>>>>> --------------
>>>>>>>>>>> reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>>>>>> reg &= ~GFX_ACTLR_PRR;
>>>>>>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>>>>
>>>>>>>>>>> if (!prr_page) {
>>>>>>>>>>>              writel_relaxed(PRR_RESET_ADDR,
>>>>>>>>>>>                              smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>>>>              writel_relaxed(PRR_RESET_ADDR),
>>>>>>>>>>>                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>>>>              return;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> writel_relaxed(lower_32_bits(page_to_phys(prr_page)),
>>>>>>>>>>>                      smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>>>>
>>>>>>>>>>> writel_relaxed(upper_32_bits(page_to_phys(prr_page)),
>>>>>>>>>>>                      smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>>>>
>>>>>>>>>>> reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>>>>>> arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>>>> -----------------
>>>>>>>>>>>
>>>>>>>>>>> If looks good, will implement the same in next version.
>>>>>>>>>>
>>>>>>>>>> yeah, that looks like it could work..
>>>>>>>>>>
>>>>>>>>>> you probably don't need to zero out the PRR_CFG_*ADDR when disabling,
>>>>>>>>>> and probably could avoid double writing ACTLR, but that is getting
>>>>>>>>>> into bikeshedding
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Actually Rob, since you rightly pointed this out.
>>>>>>>>> I crosschecked again on these registers.
>>>>>>>>> PRR_CFG_*ADDR is a global register in SMMU space but
>>>>>>>>> ACTLR register including PRR bit is a per-domain register.
>>>>>>>>> There might also be a situation where PRR feature need to be
>>>>>>>>> disabled or enabled separately for each domain.
>>>>>>>>> So I think it would be cleaner to have two apis, set_prr_addr(),
>>>>>>>>> set_prr_bit().
>>>>>>>>> set_prr_addr() will be used only to set this PRR_CFG_*ADDR
>>>>>>>>> register by passing a 'struct page *'
>>>>>>>>> set_prr_bit() will be used as a switch for PRR feature,
>>>>>>>>> where required smmu_domain will be passed along with
>>>>>>>>> the bool value to set/reset the PRR bit depending on which this
>>>>>>>>> feature will be enabled/disabled for the selected domain.
>>>>>>>>
>>>>>>>> on a related note, adreno has been using arm-smmu for a number of
>>>>>>>> generations, I guess not all support PRR?  The drm driver will need to
>>>>>>>> know whether PRR is supported (and expose that to userspace to let the
>>>>>>>> UMD know whether to expose certain extensions).  How should this work?
>>>>>>>
>>>>>>> So, I noticed in the x1e80100.dtsi that there is a gpu_prr_mem
>>>>>>> reserved section..  maybe we should be connecting this to the smmu
>>>>>>> driver in dt, and using that to detect presence of PRR?  Ie. the smmu
>>>>>>> driver would configure PRR_CFG_*ADDR based on the reserved mem, and
>>>>>>> the interface to drm would just be to enable/disable PRR, returning an
>>>>>>> error code if the reserved mem section isn't there.
>>>>>>>
>>>>>>> This simplifies the interface, and handles the question of how to
>>>>>>> detect if PRR is supported.
>>>>>>>
>>>>>>> BR,
>>>>>>> -R
>>>>>>>
>>>>>>
>>>>>> As I checked gpu_prr_mem reserved mem section is not used for mobile
>>>>>> targets hence not present for other DT only compute targets like
>>>>>> x1e80100.dtsi has the same. PRR looks to be smmu version specific
>>>>>> property.
>>>>>
>>>>> I only see it in gpu_prr_mem in x1e80100.dtsi, but not documented
>>>>> anywhere.  I'm only assuming based on the name that it is intended to
>>>>> be for PRR (but not sure why it is larger than 0x1000).  Are the
>>>>> PRR_CFG_*ADDR regs programmed by the fw (and access blocked in EL1) on
>>>>> this device?
>>>>>
>>>>
>>>> As I checked, if the drm/gfx driver allocates the page for drm, then
>>>> this reserved-memory region is not required.
>>>>
>>>> PRR_CFG_*ADDR regs have read and write access in EL1 only for this
>>>> device, behavior is same as other devices as well. These are not
>>>> programmed by fw.
>>>
>>> If there is any device which _doesn't_ have EL1 access to these regs,
>>> I think going the reserved memory route seems more future proof > Otherwise we later on have to deal with two different ways to do
>>> things.  But I'm not sure if there is any such device or risk.
>>>
>>
>> PRR is a bit in ACTLR register which is in SMMU space,
>> so is the PRR_CFG_*ADDR registers - with EL1 having access
>> to both the registers in all targets released till now with MMU-500.
>> It's unlikely that this design would change in future
>> for MMU-500 based targets, so I feel this risk is somewhat negligible.
> 
> I wasn't worried about the ACTLR register, but the PRR_CFG_*ADDR regs ;-)
> 
> IIRC those were in the SMMU global space, why hyp tends to like to own.
> 

Yes correct, those are in SMMU global space which hyp likes to own, and
read/write access are avaialble by design for EL1.

>> Also would the reserved memory route look a bit hackish?
>> Because, since this reserved-memory node is not used when page is
>> allocated through drm - so it might turn out to be redundant.
>> If we are aiming for a device-tree handle/node for reference then I
>> think a better way would be to create a bool parameter inside smmu-node
>> indicating presence of PRR ?
> 
> tbh, I don't think there is anything better or worse about having the
> reserved-memory node vs dynamically allocating it.  (If we dynamically
> allocate, we should remove the reserved memory node from
> x1e80100.dtsi)
> 

Right, if the page is dynamically allocated then this reserved memory 
node should be removed as it will stay as an unused region in DDR acting 
as a deadweight. I checked the history on this region for x1e80100 and 
it went as a part of platform upstreaming - we can remove the same.

> The thing I was more concerned about was whether there was any chance
> that some existing or future SoC+fw combo _relied_ on a reserved
> memory node and the fw programming PRR_CFG_*ADDR. If there was any > chance of that, and we went the dynamic allocation route, then we'd
> have some devices with a reserved memory node, and some without.  That
> seems a bit ugly to me.

Agree on the concern, that would surely look a bit ugly - with different 
targets having these allocation differently and having this 
reserved-memory node

But as far as I checked the targets present with MMU-500, none of them 
is relying on reserved-memory node yet.
And the chances are negligible in future targets as well.
Since FW is not setting up these registers - so this reserved region
should not be considered as per me.
Also since in our api we are only taking the address of the prr page to 
setup these registers, so even if client chooses any page allocation 
method <not surely from firmware as per design> then the same api 
smmu-driver is exposing would still work indentical.


> If there is no chance of this, then we can go either route.
> 
>> Personally,I feel since the PRR enablement mechanism is same for all
>> MMU-500 targets - compat string would be a robust approach.
> 
> I guess if it is all mmu-500, then we can just pick based on compat
> string.  If it turns out some subset of smmu-v2 have PRR, we can just
> have a list of compat strings in arm-smmu-qcom.c.. there would only be
> a finite # of them ;-)
> 

Yes right, atleast one thing which we can conclude is - in
MMU-500 version the PRR enablement steps are same from SMMU block and
driver perspective <i.e configure the PRR_CFG_*ADDR regs and set PRR bit
in ACTLR>. So all targets on MMU-500 - we would do the same steps.
Hence under MMU-500 compat string we can keep this.
Agree on the smmu-v2 part as well.


Thanks & regards,
Bibek

> BR,
> -R

Re: [PATCH v13 6/6] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 7/16/2024 5:44 PM, Konrad Dybcio wrote:
> On 15.07.2024 10:07 PM, Rob Clark wrote:
>> On Mon, Jul 15, 2024 at 4:00 AM Bibek Kumar Patro
>> <quic_bibekkum@quicinc.com> wrote:
> 
> [...]
> 
>>>>> As I checked gpu_prr_mem reserved mem section is not used for mobile
>>>>> targets hence not present for other DT only compute targets like
>>>>> x1e80100.dtsi has the same. PRR looks to be smmu version specific
>>>>> property.
>>>>
>>>> I only see it in gpu_prr_mem in x1e80100.dtsi, but not documented
>>>> anywhere.  I'm only assuming based on the name that it is intended to
>>>> be for PRR (but not sure why it is larger than 0x1000).  Are the
>>>> PRR_CFG_*ADDR regs programmed by the fw (and access blocked in EL1) on
>>>> this device?
>>>>
>>>
>>> As I checked, if the drm/gfx driver allocates the page for drm, then
>>> this reserved-memory region is not required.
>>>
>>> PRR_CFG_*ADDR regs have read and write access in EL1 only for this
>>> device, behavior is same as other devices as well. These are not
>>> programmed by fw.
>>
>> If there is any device which _doesn't_ have EL1 access to these regs,
>> I think going the reserved memory route seems more future proof?
>> Otherwise we later on have to deal with two different ways to do
>> things.  But I'm not sure if there is any such device or risk.
> 
> We can have our cake and eat it too, if we keep the check for a
> reserved memory node handle, but make it a dynamic allocation (see
> [1] for example), this way there's a way to opt into using this from
> the DT and there's no need for adding more properties
> 

Coming to allocation methodology, here the client/user(drm/gpu
driver in this case) would be allocating the page and pass the address 
to our SMMU interface which would just configure it in the PRR_CFG_*ADDR 
register.
So I believe the choice of allocation method should be better left to 
the client, rather than restricting them to a specific strategy. (?)
If dynamically allocated without a reserved pool, this region might only 
be present in some targets and not others, which could make it look a 
bit ugly [1].

Thanks & regards,
Bibek

[1]:https://lore.kernel.org/all/CAF6AEGvZWdN+CC9O3tq7kjYPq424U6__KgAnFNCV0bCqE8wPuQ@mail.gmail.com/#:~:text=That%0Aseems%20a%20bit%20ugly%20to%20me.

> Konrad
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/qcom/msm8916.dtsi?h=v6.10#n109