[PATCH RESEND v17 0/5] iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs

[PATCH RESEND v17 0/5] iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs

[Bibek Kumar Patro]

This patch series consist of six parts and covers the following:

1. Provide option to re-enable context caching to retain prefetcher
   settings during reset and runtime suspend.

2. Remove cfg inside qcom_smmu structure and replace it with single
   pointer to qcom_smmu_match_data avoiding replication of multiple
   members from same.

3. Add support for ACTLR PRR bit setup via adreno-smmu-priv interface.

4. Introduce intital set of driver changes to implement ACTLR register
   for custom prefetcher settings in Qualcomm SoCs.

5. Add ACTLR data and support for qcom_smmu_500.

Resend of v17:
 Addition of minor fix of the build warning reported by kernel test robot [1] by powerpc_random config [2].
 [1]:https://lore.kernel.org/all/202411140748.6mcFdJdO-lkp@intel.com/#t
 [2]:https://download.01.org/0day-ci/archive/20241114/202411140748.6mcFdJdO-lkp@intel.com/config
 
Changes in v17 from v16:
 Tags provided earlier not collected yet on patch 1/5, 3/5, 4/5, 5/5
 due to the following revisions:
 - 1/5 : Move the CPRE workaround out of qualcomm specific logic and gate with config
         , update the silicon-errata.rst file
 - 2/5 : No changes - reviewed-by tags collected
 - 3/5 : Move the compatible check before assignment of callback as suggested.
 - 4/5 : Add the actlr setting for *adreno variant* of MMU-500 as well.
 - 5/5 : Due to changes in 1/5, minor refactoring had to be done before adding table.
 Link to v16:
 https://lore.kernel.org/all/20241008125410.3422512-1-quic_bibekkum@quicinc.com/

Changes in v16 from v15:
 - Incorporate Dimitry's suggestion on patch 4/5 to use dev_dbg instead of dev_notice.
 - Fix kernel test robot warning [1] coming for 32bit architecture configuration.
 - Updatingthe tags
 [1]: https://lore.kernel.org/all/202409230343.Q8KnYl2w-lkp@intel.com/
 Link to v15:
 https://lore.kernel.org/all/20240920155813.3434021-1-quic_bibekkum@quicinc.com/

Changes in v15 from v14:
 - As discussed with Robin and Dmitry modify the actlr table and logic to use
   compatible string instead of sid, mask for device matching which is
   similar to qcom_smmu_client_of_match mechanism.
 - Expand the comment in qcom_smmu500_reset to document reason why CPRE bit is re-enabled again
   after arm_mmu500_reset resets the bit.
 - Rearrange the series in order to keep prefetch setting patches in the end.
 Link to v14:
 https://lore.kernel.org/all/20240816174259.2056829-1-quic_bibekkum@quicinc.com/

Changes in v14 from v13:
 Patch 6/6:
 - As discussed incorprate changes to carry out PRR implementation only for
   targets based on MMU-500 by using compat string based SMMU version detection.
 - Split the set_actlr interface into two separate interface namely set_prr_bit
   and set_prr_addr to set the prr enable bit and prr page address resepectively.
 Patch 3/6:
  - Fix a bug in gfx actlr_config which is uncovered while testing the gfx actlr setting in sc7280
    during PRR experiment which prevented clients on certain sids of gfx smmmu to be skipped during
    setting up of the ACTLR values : Fix involves swapping the arguments passed in smr_is_subset to make
     device smr <from devicetree> a subset of actlr_config table entries < defined in the driver>.
 Patch 4/6, 5/6:
  - Sort the actlr table values in increasing order of the SIDs.
 Link to v13:
 https://lore.kernel.org/all/20240628140435.1652374-1-quic_bibekkum@quicinc.com/

Changes in v13 from v12:
 - Fix the compilation issues reported by kernel test robot [1].
 [1]: https://lore.kernel.org/all/202406281241.xEX0TWjt-lkp@intel.com/#t
 Link to v12:
 https://lore.kernel.org/all/20240626143020.3682243-1-quic_bibekkum@quicinc.com/

Changes in v12 from v11:
 Changes to incorporate suggestion from Rob:
 - Fix the set and reset logic for prr bit as pointed out in v11-6/6.
 - Rename set_actlr_bit function name to set_prr.
 - Add extension for PRR name as Partially-Resident-Region in comments
   for set_prr function.
 - Add few missing sids for sc7280 in patch-5/6.
 Link to v11:
 https://lore.kernel.org/all/20240605121713.3596499-1-quic_bibekkum@quicinc.com/

Changes in v11 from v10:
 - Include a new patch 6/6 to add support for ACTLR PRR bit
   through adreno-smmu-priv interface as suggested by Rob and Dmitry.
 Link to v10:
 https://lore.kernel.org/all/20240524131800.2288259-1-quic_bibekkum@quicinc.com/

Changes in v10 from v9:
 - Added reviewed-by tags 1/5,2/5,3/5.
 Changes incorporated:
 - Remove redundant PRR bit setting from gfx actlr table(patch 4/5,5/5)
   as this bit needs special handling in the gfx driver along with
   the associated register settings.
 Link to discussion on PRR bit:
 https://lore.kernel.org/all/f2222714-1e00-424e-946d-c314d55541b8@quicinc.com/
 Link to v9:
 https://lore.kernel.org/all/20240123144543.9405-1-quic_bibekkum@quicinc.com/

Changes in v9 from v8:
 Changes to incorporate suggestions from Konrad as follows:
 - Re-wrap struct members of actlr_variant in patch 4/5,5/5
   in a cleaner way.
 - Move actlr_config members to the header.
 Link to v8:
 https://lore.kernel.org/all/20240116150411.23876-1-quic_bibekkum@quicinc.com/

Changes in v8 from v7:
 - Added reviewed-by tags on patch 1/5, 2/5.
 Changes to incorporate suggestions from Pavan and Konrad:
 - Remove non necessary extra lines.
 - Use num_smmu and num_actlrcfg to store the array size and use the
   same to traverse the table and save on sentinel space along with
   indentation levels.
 - Refactor blocks containing qcom_smmu_set_actlr to remove block
   repetition in patch 3/5.
 - Change copyright year from 2023 to 2022-2023 in patch 3/5.
 - Modify qcom_smmu_match_data.actlrvar and actlr_variant.actlrcfg to
   const pointer to a const resource.
 - use C99 designated initializers and put the address first.
 Link to v7:
 https://lore.kernel.org/all/20240109114220.30243-1-quic_bibekkum@quicinc.com/

Changes in v7 from v6:
 Changes to incorporate suggestions from Dmitry as follows:
 - Use io_start address instead of compatible string to identify the
   correct instance by comparing with smmu start address and check for
   which smmu the corresponding actlr table is to be picked.
Link to v6:
https://lore.kernel.org/all/20231220133808.5654-1-quic_bibekkum@quicinc.com/

Changes in v6 from v5:
 - Remove extra Suggested-by tags.
 - Add return check for arm_mmu500_reset in 1/5 as discussed.
Link to v5:
https://lore.kernel.org/all/20231219135947.1623-1-quic_bibekkum@quicinc.com/

Changes in v5 from v4:
 New addition:
 - Modify copyright year in arm-smmu-qcom.h to 2023 from 2022.
 Changes to incorporate suggestions from Dmitry as follows:
 - Modify the defines for prefetch in (foo << bar) format
   as suggested.(FIELD_PREP could not be used in defines
   is not inside any block/function)
 Changes to incorporate suggestions from Konrad as follows:
 - Shift context caching enablement patch as 1/5 instead of 5/5 to
   be picked up as independent patch.
 - Fix the codestyle to orient variables in reverse xmas tree format
   for patch 1/5.
 - Fix variable name in patch 1/5 as suggested.
 Link to v4:
https://lore.kernel.org/all/20231215101827.30549-1-quic_bibekkum@quicinc.com/

Changes in v4 from v3:
 New addition:
 - Remove actlrcfg_size and use NULL end element instead to traverse
   the actlr table, as this would be a cleaner approach by removing
   redundancy of actlrcfg_size.
 - Renaming of actlr set function to arm_smmu_qcom based proprietary
   convention.
 - break from loop once sid is found and ACTLR value is initialized
   in qcom_smmu_set_actlr.
 - Modify the GFX prefetch value separating into 2 sensible defines.
 - Modify comments for prefetch defines as per SMMU-500 TRM.
 Changes to incorporate suggestions from Konrad as follows:
 - Use Reverse-Christmas-tree sorting wherever applicable.
 - Pass arguments directly to arm_smmu_set_actlr instead of creating
   duplicate variables.
 - Use array indexing instead of direct pointer addressed by new
   addition of eliminating actlrcfg_size.
 - Switch the HEX value's case from upper to lower case in SC7280
   actlrcfg table.
 Changes to incorporate suggestions from Dmitry as follows:
 - Separate changes not related to ACTLR support to different commit
   with patch 5/5.
 - Using pointer to struct for arguments in smr_is_subset().
 Changes to incorporate suggestions from Bjorn as follows:
 - fix the commit message for patch 2/5 to properly document the
   value space to avoid confusion.
 Fixed build issues reported by kernel test robot [1] for
 arm64-allyesconfig [2].
 [1]: https://lore.kernel.org/all/202312011750.Pwca3TWE-lkp@intel.com/
 [2]:
https://download.01.org/0day-ci/archive/20231201/202312011750.Pwca3TWE-lkp@intel.com/config
 Link to v3:
https://lore.kernel.org/all/20231127145412.3981-1-quic_bibekkum@quicinc.com/

Changes in v3 from v2:
 New addition:
 - Include patch 3/4 for adding ACTLR support and data for SC7280.
 - Add driver changes for actlr support in gpu smmu.
 - Add target wise actlr data and implementation ops for gpu smmu.
 Changes to incorporate suggestions from Robin as follows:
 - Match the ACTLR values with individual corresponding SID instead
   of assuming that any SMR will be programmed to match a superset of
   the data.
 - Instead of replicating each elements from qcom_smmu_match_data to
   qcom_smmu structre during smmu device creation, replace the
   replicated members with qcom_smmu_match_data structure inside
   qcom_smmu structre and handle the dereference in places that
   requires them.
 Changes to incorporate suggestions from Dmitry and Konrad as follows:
 - Maintain actlr table inside a single structure instead of
   nested structure.
 - Rename prefetch defines to more appropriately describe their
   behavior.
 - Remove SM8550 specific implementation ops and roll back to default
   qcom_smmu_500_impl implementation ops.
 - Add back the removed comments which are NAK.
 - Fix commit description for patch 4/4.
 Link to v2:
https://lore.kernel.org/all/20231114135654.30475-1-quic_bibekkum@quicinc.com/

Changes in v2 from v1:
 - Incorporated suggestions on v1 from Dmitry,Konrad,Pratyush.
 - Added defines for ACTLR values.
 - Linked sm8550 implementation structure to corresponding
   compatible string.
 - Repackaged actlr value set implementation to separate function.
 - Fixed indentation errors.
 - Link to v1:
https://lore.kernel.org/all/20231103215124.1095-1-quic_bibekkum@quicinc.com/

Changes in v1 from RFC:
 - Incorporated suggestion form Robin on RFC
 - Moved the actlr data table into driver, instead of maintaining
   it inside soc specific DT and piggybacking on exisiting iommus
   property (iommu = <SID, MASK, ACTLR>) to set this value during
   smmu probe.
 - Link to RFC:
https://lore.kernel.org/all/a01e7e60-6ead-4a9e-ba90-22a8a6bbd03f@quicinc.com/

Bibek Kumar Patro (5):
  iommu/arm-smmu: re-enable context caching in smmu reset operation
  iommu/arm-smmu: refactor qcom_smmu structure to include single pointer
  iommu/arm-smmu: add support for PRR bit setup
  iommu/arm-smmu: introduction of ACTLR for custom prefetcher settings
  iommu/arm-smmu: add ACTLR data and support for qcom_smmu_500

 Documentation/arch/arm64/silicon-errata.rst   |   3 +-
 drivers/iommu/Kconfig                         |  12 ++
 drivers/iommu/arm/arm-smmu/arm-smmu-impl.c    |   5 +-
 .../iommu/arm/arm-smmu/arm-smmu-qcom-debug.c  |   2 +-
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c    | 105 +++++++++++++++++-
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h    |   3 +-
 drivers/iommu/arm/arm-smmu/arm-smmu.h         |   2 +
 include/linux/adreno-smmu-priv.h              |  14 +++
 8 files changed, 140 insertions(+), 6 deletions(-)

--
2.34.1

[PATCH RESEND v17 1/5] iommu/arm-smmu: re-enable context caching in smmu reset operation

[Bibek Kumar Patro]

Default MMU-500 reset operation disables context caching in
prefetch buffer. It is however expected for context banks using
the ACTLR register to retain their prefetch value during reset
and runtime suspend.

Add config 'ARM_SMMU_MMU_500_CPRE_ERRATA' to gate this errata
workaround in default MMU-500 reset operation which defaults to
'Y' and provide option to disable workaround for context caching
in prefetch buffer as and when needed.

Suggested-by: Will Deacon <will@kernel.org>
Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 Documentation/arch/arm64/silicon-errata.rst |  3 ++-
 drivers/iommu/Kconfig                       | 12 ++++++++++++
 drivers/iommu/arm/arm-smmu/arm-smmu-impl.c  |  5 +++--
 3 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
index 65bfab1b1861..92207d55fd1c 100644
--- a/Documentation/arch/arm64/silicon-errata.rst
+++ b/Documentation/arch/arm64/silicon-errata.rst
@@ -198,7 +198,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-V3     | #3312417        | ARM64_ERRATUM_3194386       |
 +----------------+-----------------+-----------------+-----------------------------+
-| ARM            | MMU-500         | #841119,826419  | N/A                         |
+| ARM            | MMU-500         | #841119,826419  | ARM_SMMU_MMU_500_CPRE_ERRATA|
+|                |                 | #562869,1047329 |                             |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | MMU-600         | #1076982,1209401| N/A                         |
 +----------------+-----------------+-----------------+-----------------------------+
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index b3aa1f5d5321..7eb67608a519 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -366,6 +366,18 @@ config ARM_SMMU_DISABLE_BYPASS_BY_DEFAULT
 	  'arm-smmu.disable_bypass' will continue to override this
 	  config.

+config ARM_SMMU_MMU_500_CPRE_ERRATA
+	bool "Enable errata workaround for CPRE in SMMU reset path"
+	depends on ARM_SMMU
+	default y
+	help
+	  Say Y here (by default) to apply workaround to disable
+	  MMU-500's next-page prefetcher for sake of 4 known errata.
+
+	  Say N here only when it is sure that any errata related to
+	  prefetch enablement are not applicable on the platform.
+	  Refer silicon-errata.rst for info on errata IDs.
+
 config ARM_SMMU_QCOM
 	def_tristate y
 	depends on ARM_SMMU && ARCH_QCOM
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-impl.c b/drivers/iommu/arm/arm-smmu/arm-smmu-impl.c
index 99030e6b16e7..db9b9a8e139c 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-impl.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-impl.c
@@ -110,7 +110,6 @@ static struct arm_smmu_device *cavium_smmu_impl_init(struct arm_smmu_device *smm
 int arm_mmu500_reset(struct arm_smmu_device *smmu)
 {
 	u32 reg, major;
-	int i;
 	/*
 	 * On MMU-500 r2p0 onwards we need to clear ACR.CACHE_LOCK before
 	 * writes to the context bank ACTLRs will stick. And we just hope that
@@ -128,11 +127,12 @@ int arm_mmu500_reset(struct arm_smmu_device *smmu)
 	reg |= ARM_MMU500_ACR_SMTNMB_TLBEN | ARM_MMU500_ACR_S2CRB_TLBEN;
 	arm_smmu_gr0_write(smmu, ARM_SMMU_GR0_sACR, reg);

+#ifdef CONFIG_ARM_SMMU_MMU_500_CPRE_ERRATA
 	/*
 	 * Disable MMU-500's not-particularly-beneficial next-page
 	 * prefetcher for the sake of at least 5 known errata.
 	 */
-	for (i = 0; i < smmu->num_context_banks; ++i) {
+	for (int i = 0; i < smmu->num_context_banks; ++i) {
 		reg = arm_smmu_cb_read(smmu, i, ARM_SMMU_CB_ACTLR);
 		reg &= ~ARM_MMU500_ACTLR_CPRE;
 		arm_smmu_cb_write(smmu, i, ARM_SMMU_CB_ACTLR, reg);
@@ -140,6 +140,7 @@ int arm_mmu500_reset(struct arm_smmu_device *smmu)
 		if (reg & ARM_MMU500_ACTLR_CPRE)
 			dev_warn_once(smmu->dev, "Failed to disable prefetcher for errata workarounds, check SACR.CACHE_LOCK\n");
 	}
+#endif

 	return 0;
 }
--
2.34.1

[PATCH RESEND v17 2/5] iommu/arm-smmu: refactor qcom_smmu structure to include single pointer

[Bibek Kumar Patro]

qcom_smmu_match_data is static and constant so refactor qcom_smmu
to store single pointer to qcom_smmu_match_data instead of
replicating multiple child members of the same and handle the further
dereferences in the places that want them.

Suggested-by: Robin Murphy <robin.murphy@arm.com>
Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c | 2 +-
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c       | 2 +-
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h       | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
index 548783f3f8e8..d03b2239baad 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
@@ -73,7 +73,7 @@ void qcom_smmu_tlb_sync_debug(struct arm_smmu_device *smmu)
 	if (__ratelimit(&rs)) {
 		dev_err(smmu->dev, "TLB sync timed out -- SMMU may be deadlocked\n");

-		cfg = qsmmu->cfg;
+		cfg = qsmmu->data->cfg;
 		if (!cfg)
 			return;

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index 6372f3e25c4b..d26f5aea248e 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -507,7 +507,7 @@ static struct arm_smmu_device *qcom_smmu_create(struct arm_smmu_device *smmu,
 		return ERR_PTR(-ENOMEM);

 	qsmmu->smmu.impl = impl;
-	qsmmu->cfg = data->cfg;
+	qsmmu->data = data;

 	return &qsmmu->smmu;
 }
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
index 3c134d1a6277..b55cd3e3ae48 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
@@ -8,7 +8,7 @@

 struct qcom_smmu {
 	struct arm_smmu_device smmu;
-	const struct qcom_smmu_config *cfg;
+	const struct qcom_smmu_match_data *data;
 	bool bypass_quirk;
 	u8 bypass_cbndx;
 	u32 stall_enabled;
--
2.34.1

[PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

Add an adreno-smmu-priv interface for drm/msm to call
into arm-smmu-qcom and initiate the PRR bit setup or reset
sequence as per request.

This will be used by GPU to setup the PRR bit and related
configuration registers through adreno-smmu private
interface instead of directly poking the smmu hardware.

Suggested-by: Rob Clark <robdclark@gmail.com>
Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
 drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
 include/linux/adreno-smmu-priv.h           | 14 ++++++++
 3 files changed, 53 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index d26f5aea248e..0e4f3fbda961 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -16,6 +16,8 @@

 #define QCOM_DUMMY_VAL	-1

+#define GFX_ACTLR_PRR          (1 << 5)
+
 static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
 {
 	return container_of(smmu, struct qcom_smmu, smmu);
@@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
 	arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
 }

+static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
+{
+	struct arm_smmu_domain *smmu_domain = (void *)cookie;
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
+	u32 reg = 0;
+
+	reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
+	reg &= ~GFX_ACTLR_PRR;
+	if (set)
+		reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
+	arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
+}
+
+static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
+{
+	struct arm_smmu_domain *smmu_domain = (void *)cookie;
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+
+	writel_relaxed(lower_32_bits(page_addr),
+				smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
+
+	writel_relaxed(upper_32_bits(page_addr),
+				smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
+}
+
 #define QCOM_ADRENO_SMMU_GPU_SID 0

 static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
@@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
 static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 		struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
 {
+	const struct device_node *np = smmu_domain->smmu->dev->of_node;
 	struct adreno_smmu_priv *priv;

 	smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
@@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 	priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
 	priv->set_stall = qcom_adreno_smmu_set_stall;
 	priv->resume_translation = qcom_adreno_smmu_resume_translation;
+	priv->set_prr_bit = NULL;
+	priv->set_prr_addr = NULL;
+
+	if (of_device_is_compatible(np, "qcom,smmu-500") &&
+			of_device_is_compatible(np, "qcom,adreno-smmu")) {
+		priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
+		priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
+	}

 	return 0;
 }
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
index e2aeb511ae90..2dbf3243b5ad 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
@@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
 #define ARM_SMMU_SCTLR_M		BIT(0)

 #define ARM_SMMU_CB_ACTLR		0x4
+#define ARM_SMMU_GFX_PRR_CFG_LADDR	0x6008
+#define ARM_SMMU_GFX_PRR_CFG_UADDR	0x600C

 #define ARM_SMMU_CB_RESUME		0x8
 #define ARM_SMMU_RESUME_TERMINATE	BIT(0)
diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
index c637e0997f6d..614665153b3e 100644
--- a/include/linux/adreno-smmu-priv.h
+++ b/include/linux/adreno-smmu-priv.h
@@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
  *                 the GPU driver must call resume_translation()
  * @resume_translation: Resume translation after a fault
  *
+ * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
+ *             targets without PRR support. Exercise caution and verify target
+ *             capabilities before invoking these callbacks to prevent potential
+ *             runtime errors or unexpected behavior.
+ *
+ * @set_prr_bit:   Extendible interface to be used by GPU to modify the
+ *		   ACTLR register bits, currently used to configure
+ *		   Partially-Resident-Region (PRR) bit for feature's
+ *		   setup and reset sequence as requested.
+ * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
+ *		   physical address of PRR page passed from
+ *		   GPU driver.
  *
  * The GPU driver (drm/msm) and adreno-smmu work together for controlling
  * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
@@ -67,6 +79,8 @@ struct adreno_smmu_priv {
     void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
     void (*set_stall)(const void *cookie, bool enabled);
     void (*resume_translation)(const void *cookie, bool terminate);
+    void (*set_prr_bit)(const void *cookie, bool set);
+    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
 };

 #endif /* __ADRENO_SMMU_PRIV_H */
--
2.34.1

[PATCH RESEND v17 4/5] iommu/arm-smmu: introduction of ACTLR for custom prefetcher settings

[Bibek Kumar Patro]

Currently in Qualcomm  SoCs the default prefetch is set to 1 which allows
the TLB to fetch just the next page table. MMU-500 features ACTLR
register which is implementation defined and is used for Qualcomm SoCs
to have a custom prefetch setting enabling TLB to prefetch the next set
of page tables accordingly allowing for faster translations.

ACTLR value is unique for each SMR (Stream matching register) and stored
in a pre-populated table. This value is set to the register during
context bank initialisation.

Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 33 ++++++++++++++++++++++
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h |  1 +
 2 files changed, 34 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index 0e4f3fbda961..b595fee23836 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -235,14 +235,37 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
 	return true;
 }

+static void qcom_smmu_set_actlr_dev(struct device *dev, struct arm_smmu_device *smmu, int cbndx,
+		const struct of_device_id *client_match)
+{
+	const struct of_device_id *match =
+			of_match_device(client_match, dev);
+
+	if (!match) {
+		dev_dbg(dev, "no ACTLR settings present\n");
+		return;
+	}
+
+	arm_smmu_cb_write(smmu, cbndx, ARM_SMMU_CB_ACTLR, (unsigned long)match->data);
+}
+
 static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 		struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
 {
 	const struct device_node *np = smmu_domain->smmu->dev->of_node;
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+	struct qcom_smmu *qsmmu = to_qcom_smmu(smmu);
+	const struct of_device_id *client_match;
+	int cbndx = smmu_domain->cfg.cbndx;
 	struct adreno_smmu_priv *priv;

 	smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;

+	client_match = qsmmu->data->client_match;
+
+	if (client_match)
+		qcom_smmu_set_actlr_dev(dev, smmu, cbndx, client_match);
+
 	/* Only enable split pagetables for the GPU device (SID 0) */
 	if (!qcom_adreno_smmu_is_gpu_device(dev))
 		return 0;
@@ -306,8 +329,18 @@ static const struct of_device_id qcom_smmu_client_of_match[] __maybe_unused = {
 static int qcom_smmu_init_context(struct arm_smmu_domain *smmu_domain,
 		struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
 {
+	struct arm_smmu_device *smmu = smmu_domain->smmu;
+	struct qcom_smmu *qsmmu = to_qcom_smmu(smmu);
+	const struct of_device_id *client_match;
+	int cbndx = smmu_domain->cfg.cbndx;
+
 	smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;

+	client_match = qsmmu->data->client_match;
+
+	if (client_match)
+		qcom_smmu_set_actlr_dev(dev, smmu, cbndx, client_match);
+
 	return 0;
 }

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
index b55cd3e3ae48..8addd453f5f1 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
@@ -28,6 +28,7 @@ struct qcom_smmu_match_data {
 	const struct qcom_smmu_config *cfg;
 	const struct arm_smmu_impl *impl;
 	const struct arm_smmu_impl *adreno_impl;
+	const struct of_device_id * const client_match;
 };

 irqreturn_t qcom_smmu_context_fault(int irq, void *dev);
--
2.34.1

[PATCH RESEND v17 5/5] iommu/arm-smmu: add ACTLR data and support for qcom_smmu_500

[Bibek Kumar Patro]

Add ACTLR data table for qcom_smmu_500 including
corresponding data entry and set prefetch value by
way of a list of compatible strings.

Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
---
 drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 33 ++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index b595fee23836..5106103574ab 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -16,8 +16,40 @@

 #define QCOM_DUMMY_VAL	-1

+/*
+ * SMMU-500 TRM defines BIT(0) as CMTLB (Enable context caching in the
+ * macro TLB) and BIT(1) as CPRE (Enable context caching in the prefetch
+ * buffer). The remaining bits are implementation defined and vary across
+ * SoCs.
+ */
+
+#define CPRE			(1 << 1)
+#define CMTLB			(1 << 0)
+#define PREFETCH_SHIFT		8
+#define PREFETCH_DEFAULT	0
+#define PREFETCH_SHALLOW	(1 << PREFETCH_SHIFT)
+#define PREFETCH_MODERATE	(2 << PREFETCH_SHIFT)
+#define PREFETCH_DEEP		(3 << PREFETCH_SHIFT)
 #define GFX_ACTLR_PRR          (1 << 5)

+static const struct of_device_id qcom_smmu_actlr_client_of_match[] = {
+	{ .compatible = "qcom,adreno",
+			.data = (const void *) (PREFETCH_DEEP | CPRE | CMTLB) },
+	{ .compatible = "qcom,adreno-gmu",
+			.data = (const void *) (PREFETCH_DEEP | CPRE | CMTLB) },
+	{ .compatible = "qcom,adreno-smmu",
+			.data = (const void *) (PREFETCH_DEEP | CPRE | CMTLB) },
+	{ .compatible = "qcom,fastrpc",
+			.data = (const void *) (PREFETCH_DEEP | CPRE | CMTLB) },
+	{ .compatible = "qcom,sc7280-mdss",
+			.data = (const void *) (PREFETCH_SHALLOW | CPRE | CMTLB) },
+	{ .compatible = "qcom,sc7280-venus",
+			.data = (const void *) (PREFETCH_SHALLOW | CPRE | CMTLB) },
+	{ .compatible = "qcom,sm8550-mdss",
+			.data = (const void *) (PREFETCH_DEFAULT | CMTLB) },
+	{ }
+};
+
 static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
 {
 	return container_of(smmu, struct qcom_smmu, smmu);
@@ -620,6 +652,7 @@ static const struct qcom_smmu_match_data qcom_smmu_500_impl0_data = {
 	.impl = &qcom_smmu_500_impl,
 	.adreno_impl = &qcom_adreno_smmu_500_impl,
 	.cfg = &qcom_smmu_impl0_cfg,
+	.client_match = qcom_smmu_actlr_client_of_match,
 };

 /*
--
2.34.1

Re: [PATCH RESEND v17 0/5] iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs

[Dmitry Baryshkov]

On Thu, Nov 14, 2024 at 09:37:16PM +0530, Bibek Kumar Patro wrote:
> This patch series consist of six parts and covers the following:
> 
> 1. Provide option to re-enable context caching to retain prefetcher
>    settings during reset and runtime suspend.
> 
> 2. Remove cfg inside qcom_smmu structure and replace it with single
>    pointer to qcom_smmu_match_data avoiding replication of multiple
>    members from same.
> 
> 3. Add support for ACTLR PRR bit setup via adreno-smmu-priv interface.
> 
> 4. Introduce intital set of driver changes to implement ACTLR register
>    for custom prefetcher settings in Qualcomm SoCs.
> 
> 5. Add ACTLR data and support for qcom_smmu_500.
> 
> Resend of v17:
>  Addition of minor fix of the build warning reported by kernel test robot [1] by powerpc_random config [2].
>  [1]:https://lore.kernel.org/all/202411140748.6mcFdJdO-lkp@intel.com/#t
>  [2]:https://download.01.org/0day-ci/archive/20241114/202411140748.6mcFdJdO-lkp@intel.com/config

Nit: then it's not a resend, but a new iteration. RESEND literally means
resending the same patchset.

>  
> Changes in v17 from v16:
>  Tags provided earlier not collected yet on patch 1/5, 3/5, 4/5, 5/5
>  due to the following revisions:
>  - 1/5 : Move the CPRE workaround out of qualcomm specific logic and gate with config
>          , update the silicon-errata.rst file
>  - 2/5 : No changes - reviewed-by tags collected
>  - 3/5 : Move the compatible check before assignment of callback as suggested.
>  - 4/5 : Add the actlr setting for *adreno variant* of MMU-500 as well.
>  - 5/5 : Due to changes in 1/5, minor refactoring had to be done before adding table.
>  Link to v16:
>  https://lore.kernel.org/all/20241008125410.3422512-1-quic_bibekkum@quicinc.com/
> 
> Changes in v16 from v15:
>  - Incorporate Dimitry's suggestion on patch 4/5 to use dev_dbg instead of dev_notice.
>  - Fix kernel test robot warning [1] coming for 32bit architecture configuration.
>  - Updatingthe tags
>  [1]: https://lore.kernel.org/all/202409230343.Q8KnYl2w-lkp@intel.com/
>  Link to v15:
>  https://lore.kernel.org/all/20240920155813.3434021-1-quic_bibekkum@quicinc.com/
> 
> Changes in v15 from v14:
>  - As discussed with Robin and Dmitry modify the actlr table and logic to use
>    compatible string instead of sid, mask for device matching which is
>    similar to qcom_smmu_client_of_match mechanism.
>  - Expand the comment in qcom_smmu500_reset to document reason why CPRE bit is re-enabled again
>    after arm_mmu500_reset resets the bit.
>  - Rearrange the series in order to keep prefetch setting patches in the end.
>  Link to v14:
>  https://lore.kernel.org/all/20240816174259.2056829-1-quic_bibekkum@quicinc.com/
> 
> Changes in v14 from v13:
>  Patch 6/6:
>  - As discussed incorprate changes to carry out PRR implementation only for
>    targets based on MMU-500 by using compat string based SMMU version detection.
>  - Split the set_actlr interface into two separate interface namely set_prr_bit
>    and set_prr_addr to set the prr enable bit and prr page address resepectively.
>  Patch 3/6:
>   - Fix a bug in gfx actlr_config which is uncovered while testing the gfx actlr setting in sc7280
>     during PRR experiment which prevented clients on certain sids of gfx smmmu to be skipped during
>     setting up of the ACTLR values : Fix involves swapping the arguments passed in smr_is_subset to make
>      device smr <from devicetree> a subset of actlr_config table entries < defined in the driver>.
>  Patch 4/6, 5/6:
>   - Sort the actlr table values in increasing order of the SIDs.
>  Link to v13:
>  https://lore.kernel.org/all/20240628140435.1652374-1-quic_bibekkum@quicinc.com/
> 
> Changes in v13 from v12:
>  - Fix the compilation issues reported by kernel test robot [1].
>  [1]: https://lore.kernel.org/all/202406281241.xEX0TWjt-lkp@intel.com/#t
>  Link to v12:
>  https://lore.kernel.org/all/20240626143020.3682243-1-quic_bibekkum@quicinc.com/
> 
> Changes in v12 from v11:
>  Changes to incorporate suggestion from Rob:
>  - Fix the set and reset logic for prr bit as pointed out in v11-6/6.
>  - Rename set_actlr_bit function name to set_prr.
>  - Add extension for PRR name as Partially-Resident-Region in comments
>    for set_prr function.
>  - Add few missing sids for sc7280 in patch-5/6.
>  Link to v11:
>  https://lore.kernel.org/all/20240605121713.3596499-1-quic_bibekkum@quicinc.com/
> 
> Changes in v11 from v10:
>  - Include a new patch 6/6 to add support for ACTLR PRR bit
>    through adreno-smmu-priv interface as suggested by Rob and Dmitry.
>  Link to v10:
>  https://lore.kernel.org/all/20240524131800.2288259-1-quic_bibekkum@quicinc.com/
> 
> Changes in v10 from v9:
>  - Added reviewed-by tags 1/5,2/5,3/5.
>  Changes incorporated:
>  - Remove redundant PRR bit setting from gfx actlr table(patch 4/5,5/5)
>    as this bit needs special handling in the gfx driver along with
>    the associated register settings.
>  Link to discussion on PRR bit:
>  https://lore.kernel.org/all/f2222714-1e00-424e-946d-c314d55541b8@quicinc.com/
>  Link to v9:
>  https://lore.kernel.org/all/20240123144543.9405-1-quic_bibekkum@quicinc.com/
> 
> Changes in v9 from v8:
>  Changes to incorporate suggestions from Konrad as follows:
>  - Re-wrap struct members of actlr_variant in patch 4/5,5/5
>    in a cleaner way.
>  - Move actlr_config members to the header.
>  Link to v8:
>  https://lore.kernel.org/all/20240116150411.23876-1-quic_bibekkum@quicinc.com/
> 
> Changes in v8 from v7:
>  - Added reviewed-by tags on patch 1/5, 2/5.
>  Changes to incorporate suggestions from Pavan and Konrad:
>  - Remove non necessary extra lines.
>  - Use num_smmu and num_actlrcfg to store the array size and use the
>    same to traverse the table and save on sentinel space along with
>    indentation levels.
>  - Refactor blocks containing qcom_smmu_set_actlr to remove block
>    repetition in patch 3/5.
>  - Change copyright year from 2023 to 2022-2023 in patch 3/5.
>  - Modify qcom_smmu_match_data.actlrvar and actlr_variant.actlrcfg to
>    const pointer to a const resource.
>  - use C99 designated initializers and put the address first.
>  Link to v7:
>  https://lore.kernel.org/all/20240109114220.30243-1-quic_bibekkum@quicinc.com/
> 
> Changes in v7 from v6:
>  Changes to incorporate suggestions from Dmitry as follows:
>  - Use io_start address instead of compatible string to identify the
>    correct instance by comparing with smmu start address and check for
>    which smmu the corresponding actlr table is to be picked.
> Link to v6:
> https://lore.kernel.org/all/20231220133808.5654-1-quic_bibekkum@quicinc.com/
> 
> Changes in v6 from v5:
>  - Remove extra Suggested-by tags.
>  - Add return check for arm_mmu500_reset in 1/5 as discussed.
> Link to v5:
> https://lore.kernel.org/all/20231219135947.1623-1-quic_bibekkum@quicinc.com/
> 
> Changes in v5 from v4:
>  New addition:
>  - Modify copyright year in arm-smmu-qcom.h to 2023 from 2022.
>  Changes to incorporate suggestions from Dmitry as follows:
>  - Modify the defines for prefetch in (foo << bar) format
>    as suggested.(FIELD_PREP could not be used in defines
>    is not inside any block/function)
>  Changes to incorporate suggestions from Konrad as follows:
>  - Shift context caching enablement patch as 1/5 instead of 5/5 to
>    be picked up as independent patch.
>  - Fix the codestyle to orient variables in reverse xmas tree format
>    for patch 1/5.
>  - Fix variable name in patch 1/5 as suggested.
>  Link to v4:
> https://lore.kernel.org/all/20231215101827.30549-1-quic_bibekkum@quicinc.com/
> 
> Changes in v4 from v3:
>  New addition:
>  - Remove actlrcfg_size and use NULL end element instead to traverse
>    the actlr table, as this would be a cleaner approach by removing
>    redundancy of actlrcfg_size.
>  - Renaming of actlr set function to arm_smmu_qcom based proprietary
>    convention.
>  - break from loop once sid is found and ACTLR value is initialized
>    in qcom_smmu_set_actlr.
>  - Modify the GFX prefetch value separating into 2 sensible defines.
>  - Modify comments for prefetch defines as per SMMU-500 TRM.
>  Changes to incorporate suggestions from Konrad as follows:
>  - Use Reverse-Christmas-tree sorting wherever applicable.
>  - Pass arguments directly to arm_smmu_set_actlr instead of creating
>    duplicate variables.
>  - Use array indexing instead of direct pointer addressed by new
>    addition of eliminating actlrcfg_size.
>  - Switch the HEX value's case from upper to lower case in SC7280
>    actlrcfg table.
>  Changes to incorporate suggestions from Dmitry as follows:
>  - Separate changes not related to ACTLR support to different commit
>    with patch 5/5.
>  - Using pointer to struct for arguments in smr_is_subset().
>  Changes to incorporate suggestions from Bjorn as follows:
>  - fix the commit message for patch 2/5 to properly document the
>    value space to avoid confusion.
>  Fixed build issues reported by kernel test robot [1] for
>  arm64-allyesconfig [2].
>  [1]: https://lore.kernel.org/all/202312011750.Pwca3TWE-lkp@intel.com/
>  [2]:
> https://download.01.org/0day-ci/archive/20231201/202312011750.Pwca3TWE-lkp@intel.com/config
>  Link to v3:
> https://lore.kernel.org/all/20231127145412.3981-1-quic_bibekkum@quicinc.com/
> 
> Changes in v3 from v2:
>  New addition:
>  - Include patch 3/4 for adding ACTLR support and data for SC7280.
>  - Add driver changes for actlr support in gpu smmu.
>  - Add target wise actlr data and implementation ops for gpu smmu.
>  Changes to incorporate suggestions from Robin as follows:
>  - Match the ACTLR values with individual corresponding SID instead
>    of assuming that any SMR will be programmed to match a superset of
>    the data.
>  - Instead of replicating each elements from qcom_smmu_match_data to
>    qcom_smmu structre during smmu device creation, replace the
>    replicated members with qcom_smmu_match_data structure inside
>    qcom_smmu structre and handle the dereference in places that
>    requires them.
>  Changes to incorporate suggestions from Dmitry and Konrad as follows:
>  - Maintain actlr table inside a single structure instead of
>    nested structure.
>  - Rename prefetch defines to more appropriately describe their
>    behavior.
>  - Remove SM8550 specific implementation ops and roll back to default
>    qcom_smmu_500_impl implementation ops.
>  - Add back the removed comments which are NAK.
>  - Fix commit description for patch 4/4.
>  Link to v2:
> https://lore.kernel.org/all/20231114135654.30475-1-quic_bibekkum@quicinc.com/
> 
> Changes in v2 from v1:
>  - Incorporated suggestions on v1 from Dmitry,Konrad,Pratyush.
>  - Added defines for ACTLR values.
>  - Linked sm8550 implementation structure to corresponding
>    compatible string.
>  - Repackaged actlr value set implementation to separate function.
>  - Fixed indentation errors.
>  - Link to v1:
> https://lore.kernel.org/all/20231103215124.1095-1-quic_bibekkum@quicinc.com/
> 
> Changes in v1 from RFC:
>  - Incorporated suggestion form Robin on RFC
>  - Moved the actlr data table into driver, instead of maintaining
>    it inside soc specific DT and piggybacking on exisiting iommus
>    property (iommu = <SID, MASK, ACTLR>) to set this value during
>    smmu probe.
>  - Link to RFC:
> https://lore.kernel.org/all/a01e7e60-6ead-4a9e-ba90-22a8a6bbd03f@quicinc.com/
> 
> Bibek Kumar Patro (5):
>   iommu/arm-smmu: re-enable context caching in smmu reset operation
>   iommu/arm-smmu: refactor qcom_smmu structure to include single pointer
>   iommu/arm-smmu: add support for PRR bit setup
>   iommu/arm-smmu: introduction of ACTLR for custom prefetcher settings
>   iommu/arm-smmu: add ACTLR data and support for qcom_smmu_500
> 
>  Documentation/arch/arm64/silicon-errata.rst   |   3 +-
>  drivers/iommu/Kconfig                         |  12 ++
>  drivers/iommu/arm/arm-smmu/arm-smmu-impl.c    |   5 +-
>  .../iommu/arm/arm-smmu/arm-smmu-qcom-debug.c  |   2 +-
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c    | 105 +++++++++++++++++-
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h    |   3 +-
>  drivers/iommu/arm/arm-smmu/arm-smmu.h         |   2 +
>  include/linux/adreno-smmu-priv.h              |  14 +++
>  8 files changed, 140 insertions(+), 6 deletions(-)
> 
> --
> 2.34.1
> 

-- 
With best wishes
Dmitry

Re: [PATCH RESEND v17 0/5] iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs

[Bibek Kumar Patro]

On 11/15/2024 4:26 AM, Dmitry Baryshkov wrote:
> On Thu, Nov 14, 2024 at 09:37:16PM +0530, Bibek Kumar Patro wrote:
>> This patch series consist of six parts and covers the following:
>>
>> 1. Provide option to re-enable context caching to retain prefetcher
>>     settings during reset and runtime suspend.
>>
>> 2. Remove cfg inside qcom_smmu structure and replace it with single
>>     pointer to qcom_smmu_match_data avoiding replication of multiple
>>     members from same.
>>
>> 3. Add support for ACTLR PRR bit setup via adreno-smmu-priv interface.
>>
>> 4. Introduce intital set of driver changes to implement ACTLR register
>>     for custom prefetcher settings in Qualcomm SoCs.
>>
>> 5. Add ACTLR data and support for qcom_smmu_500.
>>
>> Resend of v17:
>>   Addition of minor fix of the build warning reported by kernel test robot [1] by powerpc_random config [2].
>>   [1]:https://lore.kernel.org/all/202411140748.6mcFdJdO-lkp@intel.com/#t
>>   [2]:https://download.01.org/0day-ci/archive/20241114/202411140748.6mcFdJdO-lkp@intel.com/config
> 
> Nit: then it's not a resend, but a new iteration. RESEND literally means
> resending the same patchset.

I see. Since it is a simple "int i" and so did not feel worth a new 
iteration.

Thanks & regards,
Bibek

> 
>>   
>> Changes in v17 from v16:
>>   Tags provided earlier not collected yet on patch 1/5, 3/5, 4/5, 5/5
>>   due to the following revisions:
>>   - 1/5 : Move the CPRE workaround out of qualcomm specific logic and gate with config
>>           , update the silicon-errata.rst file
>>   - 2/5 : No changes - reviewed-by tags collected
>>   - 3/5 : Move the compatible check before assignment of callback as suggested.
>>   - 4/5 : Add the actlr setting for *adreno variant* of MMU-500 as well.
>>   - 5/5 : Due to changes in 1/5, minor refactoring had to be done before adding table.
>>   Link to v16:
>>   https://lore.kernel.org/all/20241008125410.3422512-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v16 from v15:
>>   - Incorporate Dimitry's suggestion on patch 4/5 to use dev_dbg instead of dev_notice.
>>   - Fix kernel test robot warning [1] coming for 32bit architecture configuration.
>>   - Updatingthe tags
>>   [1]: https://lore.kernel.org/all/202409230343.Q8KnYl2w-lkp@intel.com/
>>   Link to v15:
>>   https://lore.kernel.org/all/20240920155813.3434021-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v15 from v14:
>>   - As discussed with Robin and Dmitry modify the actlr table and logic to use
>>     compatible string instead of sid, mask for device matching which is
>>     similar to qcom_smmu_client_of_match mechanism.
>>   - Expand the comment in qcom_smmu500_reset to document reason why CPRE bit is re-enabled again
>>     after arm_mmu500_reset resets the bit.
>>   - Rearrange the series in order to keep prefetch setting patches in the end.
>>   Link to v14:
>>   https://lore.kernel.org/all/20240816174259.2056829-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v14 from v13:
>>   Patch 6/6:
>>   - As discussed incorprate changes to carry out PRR implementation only for
>>     targets based on MMU-500 by using compat string based SMMU version detection.
>>   - Split the set_actlr interface into two separate interface namely set_prr_bit
>>     and set_prr_addr to set the prr enable bit and prr page address resepectively.
>>   Patch 3/6:
>>    - Fix a bug in gfx actlr_config which is uncovered while testing the gfx actlr setting in sc7280
>>      during PRR experiment which prevented clients on certain sids of gfx smmmu to be skipped during
>>      setting up of the ACTLR values : Fix involves swapping the arguments passed in smr_is_subset to make
>>       device smr <from devicetree> a subset of actlr_config table entries < defined in the driver>.
>>   Patch 4/6, 5/6:
>>    - Sort the actlr table values in increasing order of the SIDs.
>>   Link to v13:
>>   https://lore.kernel.org/all/20240628140435.1652374-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v13 from v12:
>>   - Fix the compilation issues reported by kernel test robot [1].
>>   [1]: https://lore.kernel.org/all/202406281241.xEX0TWjt-lkp@intel.com/#t
>>   Link to v12:
>>   https://lore.kernel.org/all/20240626143020.3682243-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v12 from v11:
>>   Changes to incorporate suggestion from Rob:
>>   - Fix the set and reset logic for prr bit as pointed out in v11-6/6.
>>   - Rename set_actlr_bit function name to set_prr.
>>   - Add extension for PRR name as Partially-Resident-Region in comments
>>     for set_prr function.
>>   - Add few missing sids for sc7280 in patch-5/6.
>>   Link to v11:
>>   https://lore.kernel.org/all/20240605121713.3596499-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v11 from v10:
>>   - Include a new patch 6/6 to add support for ACTLR PRR bit
>>     through adreno-smmu-priv interface as suggested by Rob and Dmitry.
>>   Link to v10:
>>   https://lore.kernel.org/all/20240524131800.2288259-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v10 from v9:
>>   - Added reviewed-by tags 1/5,2/5,3/5.
>>   Changes incorporated:
>>   - Remove redundant PRR bit setting from gfx actlr table(patch 4/5,5/5)
>>     as this bit needs special handling in the gfx driver along with
>>     the associated register settings.
>>   Link to discussion on PRR bit:
>>   https://lore.kernel.org/all/f2222714-1e00-424e-946d-c314d55541b8@quicinc.com/
>>   Link to v9:
>>   https://lore.kernel.org/all/20240123144543.9405-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v9 from v8:
>>   Changes to incorporate suggestions from Konrad as follows:
>>   - Re-wrap struct members of actlr_variant in patch 4/5,5/5
>>     in a cleaner way.
>>   - Move actlr_config members to the header.
>>   Link to v8:
>>   https://lore.kernel.org/all/20240116150411.23876-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v8 from v7:
>>   - Added reviewed-by tags on patch 1/5, 2/5.
>>   Changes to incorporate suggestions from Pavan and Konrad:
>>   - Remove non necessary extra lines.
>>   - Use num_smmu and num_actlrcfg to store the array size and use the
>>     same to traverse the table and save on sentinel space along with
>>     indentation levels.
>>   - Refactor blocks containing qcom_smmu_set_actlr to remove block
>>     repetition in patch 3/5.
>>   - Change copyright year from 2023 to 2022-2023 in patch 3/5.
>>   - Modify qcom_smmu_match_data.actlrvar and actlr_variant.actlrcfg to
>>     const pointer to a const resource.
>>   - use C99 designated initializers and put the address first.
>>   Link to v7:
>>   https://lore.kernel.org/all/20240109114220.30243-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v7 from v6:
>>   Changes to incorporate suggestions from Dmitry as follows:
>>   - Use io_start address instead of compatible string to identify the
>>     correct instance by comparing with smmu start address and check for
>>     which smmu the corresponding actlr table is to be picked.
>> Link to v6:
>> https://lore.kernel.org/all/20231220133808.5654-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v6 from v5:
>>   - Remove extra Suggested-by tags.
>>   - Add return check for arm_mmu500_reset in 1/5 as discussed.
>> Link to v5:
>> https://lore.kernel.org/all/20231219135947.1623-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v5 from v4:
>>   New addition:
>>   - Modify copyright year in arm-smmu-qcom.h to 2023 from 2022.
>>   Changes to incorporate suggestions from Dmitry as follows:
>>   - Modify the defines for prefetch in (foo << bar) format
>>     as suggested.(FIELD_PREP could not be used in defines
>>     is not inside any block/function)
>>   Changes to incorporate suggestions from Konrad as follows:
>>   - Shift context caching enablement patch as 1/5 instead of 5/5 to
>>     be picked up as independent patch.
>>   - Fix the codestyle to orient variables in reverse xmas tree format
>>     for patch 1/5.
>>   - Fix variable name in patch 1/5 as suggested.
>>   Link to v4:
>> https://lore.kernel.org/all/20231215101827.30549-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v4 from v3:
>>   New addition:
>>   - Remove actlrcfg_size and use NULL end element instead to traverse
>>     the actlr table, as this would be a cleaner approach by removing
>>     redundancy of actlrcfg_size.
>>   - Renaming of actlr set function to arm_smmu_qcom based proprietary
>>     convention.
>>   - break from loop once sid is found and ACTLR value is initialized
>>     in qcom_smmu_set_actlr.
>>   - Modify the GFX prefetch value separating into 2 sensible defines.
>>   - Modify comments for prefetch defines as per SMMU-500 TRM.
>>   Changes to incorporate suggestions from Konrad as follows:
>>   - Use Reverse-Christmas-tree sorting wherever applicable.
>>   - Pass arguments directly to arm_smmu_set_actlr instead of creating
>>     duplicate variables.
>>   - Use array indexing instead of direct pointer addressed by new
>>     addition of eliminating actlrcfg_size.
>>   - Switch the HEX value's case from upper to lower case in SC7280
>>     actlrcfg table.
>>   Changes to incorporate suggestions from Dmitry as follows:
>>   - Separate changes not related to ACTLR support to different commit
>>     with patch 5/5.
>>   - Using pointer to struct for arguments in smr_is_subset().
>>   Changes to incorporate suggestions from Bjorn as follows:
>>   - fix the commit message for patch 2/5 to properly document the
>>     value space to avoid confusion.
>>   Fixed build issues reported by kernel test robot [1] for
>>   arm64-allyesconfig [2].
>>   [1]: https://lore.kernel.org/all/202312011750.Pwca3TWE-lkp@intel.com/
>>   [2]:
>> https://download.01.org/0day-ci/archive/20231201/202312011750.Pwca3TWE-lkp@intel.com/config
>>   Link to v3:
>> https://lore.kernel.org/all/20231127145412.3981-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v3 from v2:
>>   New addition:
>>   - Include patch 3/4 for adding ACTLR support and data for SC7280.
>>   - Add driver changes for actlr support in gpu smmu.
>>   - Add target wise actlr data and implementation ops for gpu smmu.
>>   Changes to incorporate suggestions from Robin as follows:
>>   - Match the ACTLR values with individual corresponding SID instead
>>     of assuming that any SMR will be programmed to match a superset of
>>     the data.
>>   - Instead of replicating each elements from qcom_smmu_match_data to
>>     qcom_smmu structre during smmu device creation, replace the
>>     replicated members with qcom_smmu_match_data structure inside
>>     qcom_smmu structre and handle the dereference in places that
>>     requires them.
>>   Changes to incorporate suggestions from Dmitry and Konrad as follows:
>>   - Maintain actlr table inside a single structure instead of
>>     nested structure.
>>   - Rename prefetch defines to more appropriately describe their
>>     behavior.
>>   - Remove SM8550 specific implementation ops and roll back to default
>>     qcom_smmu_500_impl implementation ops.
>>   - Add back the removed comments which are NAK.
>>   - Fix commit description for patch 4/4.
>>   Link to v2:
>> https://lore.kernel.org/all/20231114135654.30475-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v2 from v1:
>>   - Incorporated suggestions on v1 from Dmitry,Konrad,Pratyush.
>>   - Added defines for ACTLR values.
>>   - Linked sm8550 implementation structure to corresponding
>>     compatible string.
>>   - Repackaged actlr value set implementation to separate function.
>>   - Fixed indentation errors.
>>   - Link to v1:
>> https://lore.kernel.org/all/20231103215124.1095-1-quic_bibekkum@quicinc.com/
>>
>> Changes in v1 from RFC:
>>   - Incorporated suggestion form Robin on RFC
>>   - Moved the actlr data table into driver, instead of maintaining
>>     it inside soc specific DT and piggybacking on exisiting iommus
>>     property (iommu = <SID, MASK, ACTLR>) to set this value during
>>     smmu probe.
>>   - Link to RFC:
>> https://lore.kernel.org/all/a01e7e60-6ead-4a9e-ba90-22a8a6bbd03f@quicinc.com/
>>
>> Bibek Kumar Patro (5):
>>    iommu/arm-smmu: re-enable context caching in smmu reset operation
>>    iommu/arm-smmu: refactor qcom_smmu structure to include single pointer
>>    iommu/arm-smmu: add support for PRR bit setup
>>    iommu/arm-smmu: introduction of ACTLR for custom prefetcher settings
>>    iommu/arm-smmu: add ACTLR data and support for qcom_smmu_500
>>
>>   Documentation/arch/arm64/silicon-errata.rst   |   3 +-
>>   drivers/iommu/Kconfig                         |  12 ++
>>   drivers/iommu/arm/arm-smmu/arm-smmu-impl.c    |   5 +-
>>   .../iommu/arm/arm-smmu/arm-smmu-qcom-debug.c  |   2 +-
>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c    | 105 +++++++++++++++++-
>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h    |   3 +-
>>   drivers/iommu/arm/arm-smmu/arm-smmu.h         |   2 +
>>   include/linux/adreno-smmu-priv.h              |  14 +++
>>   8 files changed, 140 insertions(+), 6 deletions(-)
>>
>> --
>> 2.34.1
>>
> 

Re: [PATCH RESEND v17 0/5] iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs

[Bibek Kumar Patro]

On 11/15/2024 10:43 PM, Bibek Kumar Patro wrote:
> 
> 
> On 11/15/2024 4:26 AM, Dmitry Baryshkov wrote:
>> On Thu, Nov 14, 2024 at 09:37:16PM +0530, Bibek Kumar Patro wrote:
>>> This patch series consist of six parts and covers the following:
>>>
>>> 1. Provide option to re-enable context caching to retain prefetcher
>>>     settings during reset and runtime suspend.
>>>
>>> 2. Remove cfg inside qcom_smmu structure and replace it with single
>>>     pointer to qcom_smmu_match_data avoiding replication of multiple
>>>     members from same.
>>>
>>> 3. Add support for ACTLR PRR bit setup via adreno-smmu-priv interface.
>>>
>>> 4. Introduce intital set of driver changes to implement ACTLR register
>>>     for custom prefetcher settings in Qualcomm SoCs.
>>>
>>> 5. Add ACTLR data and support for qcom_smmu_500.
>>>
>>> Resend of v17:
>>>   Addition of minor fix of the build warning reported by kernel test 
>>> robot [1] by powerpc_random config [2].
>>>   [1]:https://lore.kernel.org/all/202411140748.6mcFdJdO-lkp@intel.com/#t
>>>   [2]:https://download.01.org/0day-ci/ 
>>> archive/20241114/202411140748.6mcFdJdO-lkp@intel.com/config
>>
>> Nit: then it's not a resend, but a new iteration. RESEND literally means
>> resending the same patchset.
> 
> I see. Since it is a simple "int i" and so did not feel worth a new 
> iteration.
> 
> Thanks & regards,
> Bibek
> 

Let me know if you still feel this should be a new patch, I will send 
the new iteration then in that case or I'll wait till further reviews on 
this series.

Thanks & regards,
Bibek

>>
>>> Changes in v17 from v16:
>>>   Tags provided earlier not collected yet on patch 1/5, 3/5, 4/5, 5/5
>>>   due to the following revisions:
>>>   - 1/5 : Move the CPRE workaround out of qualcomm specific logic and 
>>> gate with config
>>>           , update the silicon-errata.rst file
>>>   - 2/5 : No changes - reviewed-by tags collected
>>>   - 3/5 : Move the compatible check before assignment of callback as 
>>> suggested.
>>>   - 4/5 : Add the actlr setting for *adreno variant* of MMU-500 as well.
>>>   - 5/5 : Due to changes in 1/5, minor refactoring had to be done 
>>> before adding table.
>>>   Link to v16:
>>>   https://lore.kernel.org/all/20241008125410.3422512-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v16 from v15:
>>>   - Incorporate Dimitry's suggestion on patch 4/5 to use dev_dbg 
>>> instead of dev_notice.
>>>   - Fix kernel test robot warning [1] coming for 32bit architecture 
>>> configuration.
>>>   - Updatingthe tags
>>>   [1]: https://lore.kernel.org/all/202409230343.Q8KnYl2w-lkp@intel.com/
>>>   Link to v15:
>>>   https://lore.kernel.org/all/20240920155813.3434021-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v15 from v14:
>>>   - As discussed with Robin and Dmitry modify the actlr table and 
>>> logic to use
>>>     compatible string instead of sid, mask for device matching which is
>>>     similar to qcom_smmu_client_of_match mechanism.
>>>   - Expand the comment in qcom_smmu500_reset to document reason why 
>>> CPRE bit is re-enabled again
>>>     after arm_mmu500_reset resets the bit.
>>>   - Rearrange the series in order to keep prefetch setting patches in 
>>> the end.
>>>   Link to v14:
>>>   https://lore.kernel.org/all/20240816174259.2056829-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v14 from v13:
>>>   Patch 6/6:
>>>   - As discussed incorprate changes to carry out PRR implementation 
>>> only for
>>>     targets based on MMU-500 by using compat string based SMMU 
>>> version detection.
>>>   - Split the set_actlr interface into two separate interface namely 
>>> set_prr_bit
>>>     and set_prr_addr to set the prr enable bit and prr page address 
>>> resepectively.
>>>   Patch 3/6:
>>>    - Fix a bug in gfx actlr_config which is uncovered while testing 
>>> the gfx actlr setting in sc7280
>>>      during PRR experiment which prevented clients on certain sids of 
>>> gfx smmmu to be skipped during
>>>      setting up of the ACTLR values : Fix involves swapping the 
>>> arguments passed in smr_is_subset to make
>>>       device smr <from devicetree> a subset of actlr_config table 
>>> entries < defined in the driver>.
>>>   Patch 4/6, 5/6:
>>>    - Sort the actlr table values in increasing order of the SIDs.
>>>   Link to v13:
>>>   https://lore.kernel.org/all/20240628140435.1652374-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v13 from v12:
>>>   - Fix the compilation issues reported by kernel test robot [1].
>>>   [1]: https://lore.kernel.org/all/202406281241.xEX0TWjt- 
>>> lkp@intel.com/#t
>>>   Link to v12:
>>>   https://lore.kernel.org/all/20240626143020.3682243-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v12 from v11:
>>>   Changes to incorporate suggestion from Rob:
>>>   - Fix the set and reset logic for prr bit as pointed out in v11-6/6.
>>>   - Rename set_actlr_bit function name to set_prr.
>>>   - Add extension for PRR name as Partially-Resident-Region in comments
>>>     for set_prr function.
>>>   - Add few missing sids for sc7280 in patch-5/6.
>>>   Link to v11:
>>>   https://lore.kernel.org/all/20240605121713.3596499-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v11 from v10:
>>>   - Include a new patch 6/6 to add support for ACTLR PRR bit
>>>     through adreno-smmu-priv interface as suggested by Rob and Dmitry.
>>>   Link to v10:
>>>   https://lore.kernel.org/all/20240524131800.2288259-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v10 from v9:
>>>   - Added reviewed-by tags 1/5,2/5,3/5.
>>>   Changes incorporated:
>>>   - Remove redundant PRR bit setting from gfx actlr table(patch 4/5,5/5)
>>>     as this bit needs special handling in the gfx driver along with
>>>     the associated register settings.
>>>   Link to discussion on PRR bit:
>>>   https://lore.kernel.org/all/f2222714-1e00-424e-946d- 
>>> c314d55541b8@quicinc.com/
>>>   Link to v9:
>>>   https://lore.kernel.org/all/20240123144543.9405-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v9 from v8:
>>>   Changes to incorporate suggestions from Konrad as follows:
>>>   - Re-wrap struct members of actlr_variant in patch 4/5,5/5
>>>     in a cleaner way.
>>>   - Move actlr_config members to the header.
>>>   Link to v8:
>>>   https://lore.kernel.org/all/20240116150411.23876-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v8 from v7:
>>>   - Added reviewed-by tags on patch 1/5, 2/5.
>>>   Changes to incorporate suggestions from Pavan and Konrad:
>>>   - Remove non necessary extra lines.
>>>   - Use num_smmu and num_actlrcfg to store the array size and use the
>>>     same to traverse the table and save on sentinel space along with
>>>     indentation levels.
>>>   - Refactor blocks containing qcom_smmu_set_actlr to remove block
>>>     repetition in patch 3/5.
>>>   - Change copyright year from 2023 to 2022-2023 in patch 3/5.
>>>   - Modify qcom_smmu_match_data.actlrvar and actlr_variant.actlrcfg to
>>>     const pointer to a const resource.
>>>   - use C99 designated initializers and put the address first.
>>>   Link to v7:
>>>   https://lore.kernel.org/all/20240109114220.30243-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v7 from v6:
>>>   Changes to incorporate suggestions from Dmitry as follows:
>>>   - Use io_start address instead of compatible string to identify the
>>>     correct instance by comparing with smmu start address and check for
>>>     which smmu the corresponding actlr table is to be picked.
>>> Link to v6:
>>> https://lore.kernel.org/all/20231220133808.5654-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v6 from v5:
>>>   - Remove extra Suggested-by tags.
>>>   - Add return check for arm_mmu500_reset in 1/5 as discussed.
>>> Link to v5:
>>> https://lore.kernel.org/all/20231219135947.1623-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v5 from v4:
>>>   New addition:
>>>   - Modify copyright year in arm-smmu-qcom.h to 2023 from 2022.
>>>   Changes to incorporate suggestions from Dmitry as follows:
>>>   - Modify the defines for prefetch in (foo << bar) format
>>>     as suggested.(FIELD_PREP could not be used in defines
>>>     is not inside any block/function)
>>>   Changes to incorporate suggestions from Konrad as follows:
>>>   - Shift context caching enablement patch as 1/5 instead of 5/5 to
>>>     be picked up as independent patch.
>>>   - Fix the codestyle to orient variables in reverse xmas tree format
>>>     for patch 1/5.
>>>   - Fix variable name in patch 1/5 as suggested.
>>>   Link to v4:
>>> https://lore.kernel.org/all/20231215101827.30549-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v4 from v3:
>>>   New addition:
>>>   - Remove actlrcfg_size and use NULL end element instead to traverse
>>>     the actlr table, as this would be a cleaner approach by removing
>>>     redundancy of actlrcfg_size.
>>>   - Renaming of actlr set function to arm_smmu_qcom based proprietary
>>>     convention.
>>>   - break from loop once sid is found and ACTLR value is initialized
>>>     in qcom_smmu_set_actlr.
>>>   - Modify the GFX prefetch value separating into 2 sensible defines.
>>>   - Modify comments for prefetch defines as per SMMU-500 TRM.
>>>   Changes to incorporate suggestions from Konrad as follows:
>>>   - Use Reverse-Christmas-tree sorting wherever applicable.
>>>   - Pass arguments directly to arm_smmu_set_actlr instead of creating
>>>     duplicate variables.
>>>   - Use array indexing instead of direct pointer addressed by new
>>>     addition of eliminating actlrcfg_size.
>>>   - Switch the HEX value's case from upper to lower case in SC7280
>>>     actlrcfg table.
>>>   Changes to incorporate suggestions from Dmitry as follows:
>>>   - Separate changes not related to ACTLR support to different commit
>>>     with patch 5/5.
>>>   - Using pointer to struct for arguments in smr_is_subset().
>>>   Changes to incorporate suggestions from Bjorn as follows:
>>>   - fix the commit message for patch 2/5 to properly document the
>>>     value space to avoid confusion.
>>>   Fixed build issues reported by kernel test robot [1] for
>>>   arm64-allyesconfig [2].
>>>   [1]: https://lore.kernel.org/all/202312011750.Pwca3TWE-lkp@intel.com/
>>>   [2]:
>>> https://download.01.org/0day-ci/ 
>>> archive/20231201/202312011750.Pwca3TWE-lkp@intel.com/config
>>>   Link to v3:
>>> https://lore.kernel.org/all/20231127145412.3981-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v3 from v2:
>>>   New addition:
>>>   - Include patch 3/4 for adding ACTLR support and data for SC7280.
>>>   - Add driver changes for actlr support in gpu smmu.
>>>   - Add target wise actlr data and implementation ops for gpu smmu.
>>>   Changes to incorporate suggestions from Robin as follows:
>>>   - Match the ACTLR values with individual corresponding SID instead
>>>     of assuming that any SMR will be programmed to match a superset of
>>>     the data.
>>>   - Instead of replicating each elements from qcom_smmu_match_data to
>>>     qcom_smmu structre during smmu device creation, replace the
>>>     replicated members with qcom_smmu_match_data structure inside
>>>     qcom_smmu structre and handle the dereference in places that
>>>     requires them.
>>>   Changes to incorporate suggestions from Dmitry and Konrad as follows:
>>>   - Maintain actlr table inside a single structure instead of
>>>     nested structure.
>>>   - Rename prefetch defines to more appropriately describe their
>>>     behavior.
>>>   - Remove SM8550 specific implementation ops and roll back to default
>>>     qcom_smmu_500_impl implementation ops.
>>>   - Add back the removed comments which are NAK.
>>>   - Fix commit description for patch 4/4.
>>>   Link to v2:
>>> https://lore.kernel.org/all/20231114135654.30475-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v2 from v1:
>>>   - Incorporated suggestions on v1 from Dmitry,Konrad,Pratyush.
>>>   - Added defines for ACTLR values.
>>>   - Linked sm8550 implementation structure to corresponding
>>>     compatible string.
>>>   - Repackaged actlr value set implementation to separate function.
>>>   - Fixed indentation errors.
>>>   - Link to v1:
>>> https://lore.kernel.org/all/20231103215124.1095-1- 
>>> quic_bibekkum@quicinc.com/
>>>
>>> Changes in v1 from RFC:
>>>   - Incorporated suggestion form Robin on RFC
>>>   - Moved the actlr data table into driver, instead of maintaining
>>>     it inside soc specific DT and piggybacking on exisiting iommus
>>>     property (iommu = <SID, MASK, ACTLR>) to set this value during
>>>     smmu probe.
>>>   - Link to RFC:
>>> https://lore.kernel.org/all/a01e7e60-6ead-4a9e- 
>>> ba90-22a8a6bbd03f@quicinc.com/
>>>
>>> Bibek Kumar Patro (5):
>>>    iommu/arm-smmu: re-enable context caching in smmu reset operation
>>>    iommu/arm-smmu: refactor qcom_smmu structure to include single 
>>> pointer
>>>    iommu/arm-smmu: add support for PRR bit setup
>>>    iommu/arm-smmu: introduction of ACTLR for custom prefetcher settings
>>>    iommu/arm-smmu: add ACTLR data and support for qcom_smmu_500
>>>
>>>   Documentation/arch/arm64/silicon-errata.rst   |   3 +-
>>>   drivers/iommu/Kconfig                         |  12 ++
>>>   drivers/iommu/arm/arm-smmu/arm-smmu-impl.c    |   5 +-
>>>   .../iommu/arm/arm-smmu/arm-smmu-qcom-debug.c  |   2 +-
>>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c    | 105 +++++++++++++++++-
>>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h    |   3 +-
>>>   drivers/iommu/arm/arm-smmu/arm-smmu.h         |   2 +
>>>   include/linux/adreno-smmu-priv.h              |  14 +++
>>>   8 files changed, 140 insertions(+), 6 deletions(-)
>>>
>>> -- 
>>> 2.34.1
>>>
>>
> 
> 

Re: [PATCH RESEND v17 0/5] iommu/arm-smmu: introduction of ACTLR implementation for Qualcomm SoCs

[Dmitry Baryshkov]

On Mon, 18 Nov 2024 at 14:12, Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 11/15/2024 10:43 PM, Bibek Kumar Patro wrote:
> >
> >
> > On 11/15/2024 4:26 AM, Dmitry Baryshkov wrote:
> >> On Thu, Nov 14, 2024 at 09:37:16PM +0530, Bibek Kumar Patro wrote:
> >>> This patch series consist of six parts and covers the following:
> >>>
> >>> 1. Provide option to re-enable context caching to retain prefetcher
> >>>     settings during reset and runtime suspend.
> >>>
> >>> 2. Remove cfg inside qcom_smmu structure and replace it with single
> >>>     pointer to qcom_smmu_match_data avoiding replication of multiple
> >>>     members from same.
> >>>
> >>> 3. Add support for ACTLR PRR bit setup via adreno-smmu-priv interface.
> >>>
> >>> 4. Introduce intital set of driver changes to implement ACTLR register
> >>>     for custom prefetcher settings in Qualcomm SoCs.
> >>>
> >>> 5. Add ACTLR data and support for qcom_smmu_500.
> >>>
> >>> Resend of v17:
> >>>   Addition of minor fix of the build warning reported by kernel test
> >>> robot [1] by powerpc_random config [2].
> >>>   [1]:https://lore.kernel.org/all/202411140748.6mcFdJdO-lkp@intel.com/#t
> >>>   [2]:https://download.01.org/0day-ci/
> >>> archive/20241114/202411140748.6mcFdJdO-lkp@intel.com/config
> >>
> >> Nit: then it's not a resend, but a new iteration. RESEND literally means
> >> resending the same patchset.
> >
> > I see. Since it is a simple "int i" and so did not feel worth a new
> > iteration.
> >
> > Thanks & regards,
> > Bibek
> >
>
> Let me know if you still feel this should be a new patch, I will send
> the new iteration then in that case or I'll wait till further reviews on
> this series.

I think it should be fine like it is now. Thus I marked the comment as
"nit", "not important thing"

-- 
With best wishes
Dmitry

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
> Add an adreno-smmu-priv interface for drm/msm to call
> into arm-smmu-qcom and initiate the PRR bit setup or reset
> sequence as per request.
>
> This will be used by GPU to setup the PRR bit and related
> configuration registers through adreno-smmu private
> interface instead of directly poking the smmu hardware.
>
> Suggested-by: Rob Clark <robdclark@gmail.com>
> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> ---
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
>  drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>  include/linux/adreno-smmu-priv.h           | 14 ++++++++
>  3 files changed, 53 insertions(+)
>
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> index d26f5aea248e..0e4f3fbda961 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> @@ -16,6 +16,8 @@
>
>  #define QCOM_DUMMY_VAL -1
>
> +#define GFX_ACTLR_PRR          (1 << 5)
> +
>  static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>  {
>         return container_of(smmu, struct qcom_smmu, smmu);
> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>         arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>  }
>
> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
> +{
> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> +       u32 reg = 0;
> +
> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> +       reg &= ~GFX_ACTLR_PRR;
> +       if (set)
> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> +}
> +
> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
> +{
> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> +
> +       writel_relaxed(lower_32_bits(page_addr),
> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> +
> +       writel_relaxed(upper_32_bits(page_addr),
> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> +}
> +
>  #define QCOM_ADRENO_SMMU_GPU_SID 0
>
>  static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
>  static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>                 struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>  {
> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
>         struct adreno_smmu_priv *priv;
>
>         smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>         priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>         priv->set_stall = qcom_adreno_smmu_set_stall;
>         priv->resume_translation = qcom_adreno_smmu_resume_translation;
> +       priv->set_prr_bit = NULL;
> +       priv->set_prr_addr = NULL;
> +
> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {

fwiw, it seems like PRR actually works on sc7180, which is _not_
mmu-500, so I guess the support actually goes back further.

I'm curious if we can just rely on this being supported by any hw that
has a6xx or newer?

BR,
-R

> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
> +       }
>
>         return 0;
>  }
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> index e2aeb511ae90..2dbf3243b5ad 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>  #define ARM_SMMU_SCTLR_M               BIT(0)
>
>  #define ARM_SMMU_CB_ACTLR              0x4
> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>
>  #define ARM_SMMU_CB_RESUME             0x8
>  #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> index c637e0997f6d..614665153b3e 100644
> --- a/include/linux/adreno-smmu-priv.h
> +++ b/include/linux/adreno-smmu-priv.h
> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
>   *                 the GPU driver must call resume_translation()
>   * @resume_translation: Resume translation after a fault
>   *
> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
> + *             targets without PRR support. Exercise caution and verify target
> + *             capabilities before invoking these callbacks to prevent potential
> + *             runtime errors or unexpected behavior.
> + *
> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
> + *                ACTLR register bits, currently used to configure
> + *                Partially-Resident-Region (PRR) bit for feature's
> + *                setup and reset sequence as requested.
> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
> + *                physical address of PRR page passed from
> + *                GPU driver.
>   *
>   * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>   * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
>      void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>      void (*set_stall)(const void *cookie, bool enabled);
>      void (*resume_translation)(const void *cookie, bool terminate);
> +    void (*set_prr_bit)(const void *cookie, bool set);
> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
>  };
>
>  #endif /* __ADRENO_SMMU_PRIV_H */
> --
> 2.34.1
>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
>
> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
> >
> > Add an adreno-smmu-priv interface for drm/msm to call
> > into arm-smmu-qcom and initiate the PRR bit setup or reset
> > sequence as per request.
> >
> > This will be used by GPU to setup the PRR bit and related
> > configuration registers through adreno-smmu private
> > interface instead of directly poking the smmu hardware.
> >
> > Suggested-by: Rob Clark <robdclark@gmail.com>
> > Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> > ---
> >  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
> >  drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >  include/linux/adreno-smmu-priv.h           | 14 ++++++++
> >  3 files changed, 53 insertions(+)
> >
> > diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > index d26f5aea248e..0e4f3fbda961 100644
> > --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> > @@ -16,6 +16,8 @@
> >
> >  #define QCOM_DUMMY_VAL -1
> >
> > +#define GFX_ACTLR_PRR          (1 << 5)
> > +
> >  static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
> >  {
> >         return container_of(smmu, struct qcom_smmu, smmu);
> > @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >         arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >  }
> >
> > +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
> > +{
> > +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> > +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> > +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> > +       u32 reg = 0;
> > +
> > +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> > +       reg &= ~GFX_ACTLR_PRR;
> > +       if (set)
> > +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> > +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> > +}
> > +
> > +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
> > +{
> > +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> > +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> > +
> > +       writel_relaxed(lower_32_bits(page_addr),
> > +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> > +
> > +       writel_relaxed(upper_32_bits(page_addr),
> > +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> > +}
> > +
> >  #define QCOM_ADRENO_SMMU_GPU_SID 0
> >
> >  static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> > @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
> >  static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >                 struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
> >  {
> > +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
> >         struct adreno_smmu_priv *priv;
> >
> >         smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
> > @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >         priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >         priv->set_stall = qcom_adreno_smmu_set_stall;
> >         priv->resume_translation = qcom_adreno_smmu_resume_translation;
> > +       priv->set_prr_bit = NULL;
> > +       priv->set_prr_addr = NULL;
> > +
> > +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
> > +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
>
> fwiw, it seems like PRR actually works on sc7180, which is _not_
> mmu-500, so I guess the support actually goes back further.
>
> I'm curious if we can just rely on this being supported by any hw that
> has a6xx or newer?


Also, unrelated, but we can't assume the smmu is powered when drm
driver calls set_prr_bit/addr, could you add in runpm get/put around
the register access?

Otherwise Conner and I have vk sparse residency working now

BR,
-R

>
> > +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
> > +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
> > +       }
> >
> >         return 0;
> >  }
> > diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> > index e2aeb511ae90..2dbf3243b5ad 100644
> > --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> > +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> > @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >  #define ARM_SMMU_SCTLR_M               BIT(0)
> >
> >  #define ARM_SMMU_CB_ACTLR              0x4
> > +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> > +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >
> >  #define ARM_SMMU_CB_RESUME             0x8
> >  #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> > diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> > index c637e0997f6d..614665153b3e 100644
> > --- a/include/linux/adreno-smmu-priv.h
> > +++ b/include/linux/adreno-smmu-priv.h
> > @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
> >   *                 the GPU driver must call resume_translation()
> >   * @resume_translation: Resume translation after a fault
> >   *
> > + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
> > + *             targets without PRR support. Exercise caution and verify target
> > + *             capabilities before invoking these callbacks to prevent potential
> > + *             runtime errors or unexpected behavior.
> > + *
> > + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
> > + *                ACTLR register bits, currently used to configure
> > + *                Partially-Resident-Region (PRR) bit for feature's
> > + *                setup and reset sequence as requested.
> > + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
> > + *                physical address of PRR page passed from
> > + *                GPU driver.
> >   *
> >   * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >   * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> > @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
> >      void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >      void (*set_stall)(const void *cookie, bool enabled);
> >      void (*resume_translation)(const void *cookie, bool terminate);
> > +    void (*set_prr_bit)(const void *cookie, bool set);
> > +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
> >  };
> >
> >  #endif /* __ADRENO_SMMU_PRIV_H */
> > --
> > 2.34.1
> >

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 11/20/2024 10:47 PM, Rob Clark wrote:
> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>> Add an adreno-smmu-priv interface for drm/msm to call
>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>> sequence as per request.
>>
>> This will be used by GPU to setup the PRR bit and related
>> configuration registers through adreno-smmu private
>> interface instead of directly poking the smmu hardware.
>>
>> Suggested-by: Rob Clark <robdclark@gmail.com>
>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>> ---
>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
>>   drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>   include/linux/adreno-smmu-priv.h           | 14 ++++++++
>>   3 files changed, 53 insertions(+)
>>
>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> index d26f5aea248e..0e4f3fbda961 100644
>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>> @@ -16,6 +16,8 @@
>>
>>   #define QCOM_DUMMY_VAL -1
>>
>> +#define GFX_ACTLR_PRR          (1 << 5)
>> +
>>   static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>   {
>>          return container_of(smmu, struct qcom_smmu, smmu);
>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>          arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>   }
>>
>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
>> +{
>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>> +       u32 reg = 0;
>> +
>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>> +       reg &= ~GFX_ACTLR_PRR;
>> +       if (set)
>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>> +}
>> +
>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
>> +{
>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>> +
>> +       writel_relaxed(lower_32_bits(page_addr),
>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>> +
>> +       writel_relaxed(upper_32_bits(page_addr),
>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>> +}
>> +
>>   #define QCOM_ADRENO_SMMU_GPU_SID 0
>>
>>   static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
>>   static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>                  struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>>   {
>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
>>          struct adreno_smmu_priv *priv;
>>
>>          smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>          priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>          priv->set_stall = qcom_adreno_smmu_set_stall;
>>          priv->resume_translation = qcom_adreno_smmu_resume_translation;
>> +       priv->set_prr_bit = NULL;
>> +       priv->set_prr_addr = NULL;
>> +
>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
> 
> fwiw, it seems like PRR actually works on sc7180, which is _not_
> mmu-500, so I guess the support actually goes back further.
> 

As I checked sc7180 was on previous variant of SMMU,
so targets on this variant have different steps to set PRR bit.
<Do not have both prr bit and PRR page registers>.

It's MMU-500 targets only where PRR support is with both PRR bit
and PRR page addr registers. As I was re-visiting our discussions on v13
regarding this - I remember that's why we started using the SMMU-
compatible string based PRR procedure selection instead of the reserved-
memory node. [1] i.e Based on SMMU variant (as selected by compatible
string), specific sequence will be selected.

So for now only MMU-500 based selection has been supported as part of
this series and will add subsequent support for other SMMU-variants
thereafter.

> I'm curious if we can just rely on this being supported by any hw that
> has a6xx or newer?
> 

I'd need to check on targets which will be based on a6xx onwards, on
what will be the procedure planned to support PRR feature. I'll update
the information over this space.

[1]: 
https://lore.kernel.org/all/5790afa3-f9c0-4720-9804-8a7ff3d91854@quicinc.com/#:~:text=%3E%20I%20guess%20if,part%20as%20well.

Thanks & regards,
Bibek

> BR,
> -R
> 
>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
>> +       }
>>
>>          return 0;
>>   }
>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>> index e2aeb511ae90..2dbf3243b5ad 100644
>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>   #define ARM_SMMU_SCTLR_M               BIT(0)
>>
>>   #define ARM_SMMU_CB_ACTLR              0x4
>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>
>>   #define ARM_SMMU_CB_RESUME             0x8
>>   #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>> index c637e0997f6d..614665153b3e 100644
>> --- a/include/linux/adreno-smmu-priv.h
>> +++ b/include/linux/adreno-smmu-priv.h
>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
>>    *                 the GPU driver must call resume_translation()
>>    * @resume_translation: Resume translation after a fault
>>    *
>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
>> + *             targets without PRR support. Exercise caution and verify target
>> + *             capabilities before invoking these callbacks to prevent potential
>> + *             runtime errors or unexpected behavior.
>> + *
>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
>> + *                ACTLR register bits, currently used to configure
>> + *                Partially-Resident-Region (PRR) bit for feature's
>> + *                setup and reset sequence as requested.
>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
>> + *                physical address of PRR page passed from
>> + *                GPU driver.
>>    *
>>    * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>    * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
>>       void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>       void (*set_stall)(const void *cookie, bool enabled);
>>       void (*resume_translation)(const void *cookie, bool terminate);
>> +    void (*set_prr_bit)(const void *cookie, bool set);
>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
>>   };
>>
>>   #endif /* __ADRENO_SMMU_PRIV_H */
>> --
>> 2.34.1
>>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 11/21/2024 3:40 AM, Rob Clark wrote:
> On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
>>
>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
>> <quic_bibekkum@quicinc.com> wrote:
>>>
>>> Add an adreno-smmu-priv interface for drm/msm to call
>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>> sequence as per request.
>>>
>>> This will be used by GPU to setup the PRR bit and related
>>> configuration registers through adreno-smmu private
>>> interface instead of directly poking the smmu hardware.
>>>
>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>> ---
>>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
>>>   drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>   include/linux/adreno-smmu-priv.h           | 14 ++++++++
>>>   3 files changed, 53 insertions(+)
>>>
>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>> index d26f5aea248e..0e4f3fbda961 100644
>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>> @@ -16,6 +16,8 @@
>>>
>>>   #define QCOM_DUMMY_VAL -1
>>>
>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>> +
>>>   static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>>   {
>>>          return container_of(smmu, struct qcom_smmu, smmu);
>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>          arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>   }
>>>
>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
>>> +{
>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>> +       u32 reg = 0;
>>> +
>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>> +       reg &= ~GFX_ACTLR_PRR;
>>> +       if (set)
>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>> +}
>>> +
>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
>>> +{
>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>> +
>>> +       writel_relaxed(lower_32_bits(page_addr),
>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>> +
>>> +       writel_relaxed(upper_32_bits(page_addr),
>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>> +}
>>> +
>>>   #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>
>>>   static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
>>>   static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>                  struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>>>   {
>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
>>>          struct adreno_smmu_priv *priv;
>>>
>>>          smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>          priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>          priv->set_stall = qcom_adreno_smmu_set_stall;
>>>          priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>> +       priv->set_prr_bit = NULL;
>>> +       priv->set_prr_addr = NULL;
>>> +
>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
>>
>> fwiw, it seems like PRR actually works on sc7180, which is _not_
>> mmu-500, so I guess the support actually goes back further.
>>
>> I'm curious if we can just rely on this being supported by any hw that
>> has a6xx or newer?
> 
> 
> Also, unrelated, but we can't assume the smmu is powered when drm
> driver calls set_prr_bit/addr, could you add in runpm get/put around
> the register access?
> 

I see, thanks for this observation.
It's surely a possible case, when they access these registers
SMMU state is off.
I will add the suggested runpm ops around the register access.

> Otherwise Conner and I have vk sparse residency working now
> 

Sorry, could not get this. Did you mean that vk sparse residency
is working now using this patch?

Thanks & regards,
Bibek

> BR,
> -R
> 
>>
>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
>>> +       }
>>>
>>>          return 0;
>>>   }
>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>> index e2aeb511ae90..2dbf3243b5ad 100644
>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>   #define ARM_SMMU_SCTLR_M               BIT(0)
>>>
>>>   #define ARM_SMMU_CB_ACTLR              0x4
>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>
>>>   #define ARM_SMMU_CB_RESUME             0x8
>>>   #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>> index c637e0997f6d..614665153b3e 100644
>>> --- a/include/linux/adreno-smmu-priv.h
>>> +++ b/include/linux/adreno-smmu-priv.h
>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
>>>    *                 the GPU driver must call resume_translation()
>>>    * @resume_translation: Resume translation after a fault
>>>    *
>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
>>> + *             targets without PRR support. Exercise caution and verify target
>>> + *             capabilities before invoking these callbacks to prevent potential
>>> + *             runtime errors or unexpected behavior.
>>> + *
>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
>>> + *                ACTLR register bits, currently used to configure
>>> + *                Partially-Resident-Region (PRR) bit for feature's
>>> + *                setup and reset sequence as requested.
>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
>>> + *                physical address of PRR page passed from
>>> + *                GPU driver.
>>>    *
>>>    * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>    * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
>>>       void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>       void (*set_stall)(const void *cookie, bool enabled);
>>>       void (*resume_translation)(const void *cookie, bool terminate);
>>> +    void (*set_prr_bit)(const void *cookie, bool set);
>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
>>>   };
>>>
>>>   #endif /* __ADRENO_SMMU_PRIV_H */
>>> --
>>> 2.34.1
>>>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Fri, Nov 22, 2024 at 8:20 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 11/21/2024 3:40 AM, Rob Clark wrote:
> > On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
> >>
> >> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
> >> <quic_bibekkum@quicinc.com> wrote:
> >>>
> >>> Add an adreno-smmu-priv interface for drm/msm to call
> >>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>> sequence as per request.
> >>>
> >>> This will be used by GPU to setup the PRR bit and related
> >>> configuration registers through adreno-smmu private
> >>> interface instead of directly poking the smmu hardware.
> >>>
> >>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>> ---
> >>>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
> >>>   drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>   include/linux/adreno-smmu-priv.h           | 14 ++++++++
> >>>   3 files changed, 53 insertions(+)
> >>>
> >>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>> index d26f5aea248e..0e4f3fbda961 100644
> >>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>> @@ -16,6 +16,8 @@
> >>>
> >>>   #define QCOM_DUMMY_VAL -1
> >>>
> >>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>> +
> >>>   static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
> >>>   {
> >>>          return container_of(smmu, struct qcom_smmu, smmu);
> >>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>          arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>   }
> >>>
> >>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
> >>> +{
> >>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>> +       u32 reg = 0;
> >>> +
> >>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>> +       reg &= ~GFX_ACTLR_PRR;
> >>> +       if (set)
> >>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>> +}
> >>> +
> >>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
> >>> +{
> >>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>> +
> >>> +       writel_relaxed(lower_32_bits(page_addr),
> >>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>> +
> >>> +       writel_relaxed(upper_32_bits(page_addr),
> >>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>> +}
> >>> +
> >>>   #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>
> >>>   static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
> >>>   static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>                  struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
> >>>   {
> >>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
> >>>          struct adreno_smmu_priv *priv;
> >>>
> >>>          smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
> >>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>          priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>          priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>          priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>> +       priv->set_prr_bit = NULL;
> >>> +       priv->set_prr_addr = NULL;
> >>> +
> >>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
> >>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
> >>
> >> fwiw, it seems like PRR actually works on sc7180, which is _not_
> >> mmu-500, so I guess the support actually goes back further.
> >>
> >> I'm curious if we can just rely on this being supported by any hw that
> >> has a6xx or newer?
> >
> >
> > Also, unrelated, but we can't assume the smmu is powered when drm
> > driver calls set_prr_bit/addr, could you add in runpm get/put around
> > the register access?
> >
>
> I see, thanks for this observation.
> It's surely a possible case, when they access these registers
> SMMU state is off.
> I will add the suggested runpm ops around the register access.
>
> > Otherwise Conner and I have vk sparse residency working now
> >
>
> Sorry, could not get this. Did you mean that vk sparse residency
> is working now using this patch?

Yes, vk sparse residency is working with this patch (plus addition of
runpm get/put, plus drm/msm patches plus turnip patches) ;-)

BR,
-R

> Thanks & regards,
> Bibek
>
> > BR,
> > -R
> >
> >>
> >>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
> >>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
> >>> +       }
> >>>
> >>>          return 0;
> >>>   }
> >>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>> index e2aeb511ae90..2dbf3243b5ad 100644
> >>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>>   #define ARM_SMMU_SCTLR_M               BIT(0)
> >>>
> >>>   #define ARM_SMMU_CB_ACTLR              0x4
> >>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>>
> >>>   #define ARM_SMMU_CB_RESUME             0x8
> >>>   #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >>> index c637e0997f6d..614665153b3e 100644
> >>> --- a/include/linux/adreno-smmu-priv.h
> >>> +++ b/include/linux/adreno-smmu-priv.h
> >>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
> >>>    *                 the GPU driver must call resume_translation()
> >>>    * @resume_translation: Resume translation after a fault
> >>>    *
> >>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
> >>> + *             targets without PRR support. Exercise caution and verify target
> >>> + *             capabilities before invoking these callbacks to prevent potential
> >>> + *             runtime errors or unexpected behavior.
> >>> + *
> >>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
> >>> + *                ACTLR register bits, currently used to configure
> >>> + *                Partially-Resident-Region (PRR) bit for feature's
> >>> + *                setup and reset sequence as requested.
> >>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
> >>> + *                physical address of PRR page passed from
> >>> + *                GPU driver.
> >>>    *
> >>>    * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>>    * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
> >>>       void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>>       void (*set_stall)(const void *cookie, bool enabled);
> >>>       void (*resume_translation)(const void *cookie, bool terminate);
> >>> +    void (*set_prr_bit)(const void *cookie, bool set);
> >>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
> >>>   };
> >>>
> >>>   #endif /* __ADRENO_SMMU_PRIV_H */
> >>> --
> >>> 2.34.1
> >>>
>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Fri, Nov 22, 2024 at 8:19 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 11/20/2024 10:47 PM, Rob Clark wrote:
> > On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >> Add an adreno-smmu-priv interface for drm/msm to call
> >> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >> sequence as per request.
> >>
> >> This will be used by GPU to setup the PRR bit and related
> >> configuration registers through adreno-smmu private
> >> interface instead of directly poking the smmu hardware.
> >>
> >> Suggested-by: Rob Clark <robdclark@gmail.com>
> >> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >> ---
> >>   drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
> >>   drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>   include/linux/adreno-smmu-priv.h           | 14 ++++++++
> >>   3 files changed, 53 insertions(+)
> >>
> >> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> index d26f5aea248e..0e4f3fbda961 100644
> >> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >> @@ -16,6 +16,8 @@
> >>
> >>   #define QCOM_DUMMY_VAL -1
> >>
> >> +#define GFX_ACTLR_PRR          (1 << 5)
> >> +
> >>   static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
> >>   {
> >>          return container_of(smmu, struct qcom_smmu, smmu);
> >> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>          arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>   }
> >>
> >> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
> >> +{
> >> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >> +       u32 reg = 0;
> >> +
> >> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >> +       reg &= ~GFX_ACTLR_PRR;
> >> +       if (set)
> >> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >> +}
> >> +
> >> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
> >> +{
> >> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >> +
> >> +       writel_relaxed(lower_32_bits(page_addr),
> >> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >> +
> >> +       writel_relaxed(upper_32_bits(page_addr),
> >> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >> +}
> >> +
> >>   #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>
> >>   static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
> >>   static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>                  struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
> >>   {
> >> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
> >>          struct adreno_smmu_priv *priv;
> >>
> >>          smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
> >> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>          priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>          priv->set_stall = qcom_adreno_smmu_set_stall;
> >>          priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >> +       priv->set_prr_bit = NULL;
> >> +       priv->set_prr_addr = NULL;
> >> +
> >> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
> >> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
> >
> > fwiw, it seems like PRR actually works on sc7180, which is _not_
> > mmu-500, so I guess the support actually goes back further.
> >
>
> As I checked sc7180 was on previous variant of SMMU,
> so targets on this variant have different steps to set PRR bit.
> <Do not have both prr bit and PRR page registers>.

Experimentally, I have to set both the PRR LADDR/UADDR regs and
ACTLR.PRR bit on sc7180 to get the sparse-residency tests passing.  So
the underlying hw seems to work in the same way as mmu-500.  _But_
this is on a sc7180 chromebook, the situation might be different
depending on fw on things that have QC hyp.

> It's MMU-500 targets only where PRR support is with both PRR bit
> and PRR page addr registers. As I was re-visiting our discussions on v13
> regarding this - I remember that's why we started using the SMMU-
> compatible string based PRR procedure selection instead of the reserved-
> memory node. [1] i.e Based on SMMU variant (as selected by compatible
> string), specific sequence will be selected.
>
> So for now only MMU-500 based selection has been supported as part of
> this series and will add subsequent support for other SMMU-variants
> thereafter.
>
> > I'm curious if we can just rely on this being supported by any hw that
> > has a6xx or newer?
> >
>
> I'd need to check on targets which will be based on a6xx onwards, on
> what will be the procedure planned to support PRR feature. I'll update
> the information over this space.

One of the open questions about the drm/msm sparse-memory patchset is
whether we need to expose to userspace whether PRR is supported, or if
we can just rely on sparse-binding support implying sparse residency
(ie, PRR) support.  All a6xx devices support per-process pgtables,
which is the only requirement for basic sparseBinding support.  But
PRR is required to additionally expose
sparseResidencyBuffer/sparseResidencyImage2D.

I think, long term, turnip probably will want to drop support for
older kernels and remove support for legacy buffer mapping.  But if we
have some a6xx devices without PRR, then to do that we'd need to
decouple sparse binding and sparse residency.  (Vulkan allows a driver
to expose the former without the latter.)

BR,
-R

> [1]:
> https://lore.kernel.org/all/5790afa3-f9c0-4720-9804-8a7ff3d91854@quicinc.com/#:~:text=%3E%20I%20guess%20if,part%20as%20well.
>
> Thanks & regards,
> Bibek
>
> > BR,
> > -R
> >
> >> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
> >> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
> >> +       }
> >>
> >>          return 0;
> >>   }
> >> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >> index e2aeb511ae90..2dbf3243b5ad 100644
> >> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>   #define ARM_SMMU_SCTLR_M               BIT(0)
> >>
> >>   #define ARM_SMMU_CB_ACTLR              0x4
> >> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>
> >>   #define ARM_SMMU_CB_RESUME             0x8
> >>   #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >> index c637e0997f6d..614665153b3e 100644
> >> --- a/include/linux/adreno-smmu-priv.h
> >> +++ b/include/linux/adreno-smmu-priv.h
> >> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
> >>    *                 the GPU driver must call resume_translation()
> >>    * @resume_translation: Resume translation after a fault
> >>    *
> >> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
> >> + *             targets without PRR support. Exercise caution and verify target
> >> + *             capabilities before invoking these callbacks to prevent potential
> >> + *             runtime errors or unexpected behavior.
> >> + *
> >> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
> >> + *                ACTLR register bits, currently used to configure
> >> + *                Partially-Resident-Region (PRR) bit for feature's
> >> + *                setup and reset sequence as requested.
> >> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
> >> + *                physical address of PRR page passed from
> >> + *                GPU driver.
> >>    *
> >>    * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>    * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
> >>       void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>       void (*set_stall)(const void *cookie, bool enabled);
> >>       void (*resume_translation)(const void *cookie, bool terminate);
> >> +    void (*set_prr_bit)(const void *cookie, bool set);
> >> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
> >>   };
> >>
> >>   #endif /* __ADRENO_SMMU_PRIV_H */
> >> --
> >> 2.34.1
> >>
>

Re: [PATCH RESEND v17 4/5] iommu/arm-smmu: introduction of ACTLR for custom prefetcher settings

[Dmitry Baryshkov]

On Thu, Nov 14, 2024 at 09:37:20PM +0530, Bibek Kumar Patro wrote:
> Currently in Qualcomm  SoCs the default prefetch is set to 1 which allows
> the TLB to fetch just the next page table. MMU-500 features ACTLR
> register which is implementation defined and is used for Qualcomm SoCs
> to have a custom prefetch setting enabling TLB to prefetch the next set
> of page tables accordingly allowing for faster translations.
> 
> ACTLR value is unique for each SMR (Stream matching register) and stored
> in a pre-populated table. This value is set to the register during
> context bank initialisation.
> 
> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> ---
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 33 ++++++++++++++++++++++
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h |  1 +
>  2 files changed, 34 insertions(+)
> 

Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>

-- 
With best wishes
Dmitry

Re: [PATCH RESEND v17 5/5] iommu/arm-smmu: add ACTLR data and support for qcom_smmu_500

[Dmitry Baryshkov]

On Thu, Nov 14, 2024 at 09:37:21PM +0530, Bibek Kumar Patro wrote:
> Add ACTLR data table for qcom_smmu_500 including
> corresponding data entry and set prefetch value by
> way of a list of compatible strings.
> 
> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> ---
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 33 ++++++++++++++++++++++
>  1 file changed, 33 insertions(+)

Reviewed-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>

-- 
With best wishes
Dmitry

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 11/22/2024 10:33 PM, Rob Clark wrote:
> On Fri, Nov 22, 2024 at 8:19 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 11/20/2024 10:47 PM, Rob Clark wrote:
>>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
>>> <quic_bibekkum@quicinc.com> wrote:
>>>>
>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>> sequence as per request.
>>>>
>>>> This will be used by GPU to setup the PRR bit and related
>>>> configuration registers through adreno-smmu private
>>>> interface instead of directly poking the smmu hardware.
>>>>
>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>> ---
>>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
>>>>    drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>    include/linux/adreno-smmu-priv.h           | 14 ++++++++
>>>>    3 files changed, 53 insertions(+)
>>>>
>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> index d26f5aea248e..0e4f3fbda961 100644
>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>> @@ -16,6 +16,8 @@
>>>>
>>>>    #define QCOM_DUMMY_VAL -1
>>>>
>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>> +
>>>>    static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>>>    {
>>>>           return container_of(smmu, struct qcom_smmu, smmu);
>>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>           arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>    }
>>>>
>>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
>>>> +{
>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>> +       u32 reg = 0;
>>>> +
>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>> +       if (set)
>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>> +}
>>>> +
>>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
>>>> +{
>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>> +
>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>> +
>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>> +}
>>>> +
>>>>    #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>
>>>>    static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
>>>>    static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>                   struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>>>>    {
>>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
>>>>           struct adreno_smmu_priv *priv;
>>>>
>>>>           smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
>>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>           priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>           priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>           priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>> +       priv->set_prr_bit = NULL;
>>>> +       priv->set_prr_addr = NULL;
>>>> +
>>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
>>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
>>>
>>> fwiw, it seems like PRR actually works on sc7180, which is _not_
>>> mmu-500, so I guess the support actually goes back further.
>>>
>>
>> As I checked sc7180 was on previous variant of SMMU,
>> so targets on this variant have different steps to set PRR bit.
>> <Do not have both prr bit and PRR page registers>.
> 
> Experimentally, I have to set both the PRR LADDR/UADDR regs and
> ACTLR.PRR bit on sc7180 to get the sparse-residency tests passing.  So
> the underlying hw seems to work in the same way as mmu-500.  _But_
> this is on a sc7180 chromebook, the situation might be different
> depending on fw on things that have QC hyp.
> 

I checked on sc7180 chipset which is based on the smmu-v2,
this time by looking for these offsets specifically. I can see the 
nomenclature of the PRR related registers are a bit different
compared to MMU-500 variant.
Also the implementation register is 64 bit instead of
dual 32 bit as in case of MMU-500. and PRR bit is not marked in
ACTLR register offset.

So turns out PRR is supported but with some modification and
can be carried out with same compatible based approach only - as per
my understanding.

In current patch plan was to provide support for MMU-500 based targets
and won't break any legacy targets, so we can take the PRR support
for legacy targets in different series once our evaluation is done on 
smmu-v2 ?
We would explore more on this PRR feature for smmu-v2 based targets,
before supporting it.

Thanks & regards,
Bibek

>> It's MMU-500 targets only where PRR support is with both PRR bit
>> and PRR page addr registers. As I was re-visiting our discussions on v13
>> regarding this - I remember that's why we started using the SMMU-
>> compatible string based PRR procedure selection instead of the reserved-
>> memory node. [1] i.e Based on SMMU variant (as selected by compatible
>> string), specific sequence will be selected.
>>
>> So for now only MMU-500 based selection has been supported as part of
>> this series and will add subsequent support for other SMMU-variants
>> thereafter.
>>
>>> I'm curious if we can just rely on this being supported by any hw that
>>> has a6xx or newer?
>>>
>>
>> I'd need to check on targets which will be based on a6xx onwards, on
>> what will be the procedure planned to support PRR feature. I'll update
>> the information over this space.
> 
> One of the open questions about the drm/msm sparse-memory patchset is
> whether we need to expose to userspace whether PRR is supported, or if
> we can just rely on sparse-binding support implying sparse residency
> (ie, PRR) support. All a6xx devices support per-process pgtables,
> which is the only requirement for basic sparseBinding support.  But
> PRR is required to additionally expose
> sparseResidencyBuffer/sparseResidencyImage2D.
> 
> I think, long term, turnip probably will want to drop support for
> older kernels and remove support for legacy buffer mapping.  But if we
> have some a6xx devices without PRR, then to do that we'd need to
> decouple sparse binding and sparse residency.  (Vulkan allows a driver
> to expose the former without the latter.)
> 
> BR,
> -R
> 
>> [1]:
>> https://lore.kernel.org/all/5790afa3-f9c0-4720-9804-8a7ff3d91854@quicinc.com/#:~:text=%3E%20I%20guess%20if,part%20as%20well.
>>
>> Thanks & regards,
>> Bibek
>>
>>> BR,
>>> -R
>>>
>>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
>>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
>>>> +       }
>>>>
>>>>           return 0;
>>>>    }
>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>> index e2aeb511ae90..2dbf3243b5ad 100644
>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>    #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>
>>>>    #define ARM_SMMU_CB_ACTLR              0x4
>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>
>>>>    #define ARM_SMMU_CB_RESUME             0x8
>>>>    #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>> index c637e0997f6d..614665153b3e 100644
>>>> --- a/include/linux/adreno-smmu-priv.h
>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
>>>>     *                 the GPU driver must call resume_translation()
>>>>     * @resume_translation: Resume translation after a fault
>>>>     *
>>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
>>>> + *             targets without PRR support. Exercise caution and verify target
>>>> + *             capabilities before invoking these callbacks to prevent potential
>>>> + *             runtime errors or unexpected behavior.
>>>> + *
>>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
>>>> + *                ACTLR register bits, currently used to configure
>>>> + *                Partially-Resident-Region (PRR) bit for feature's
>>>> + *                setup and reset sequence as requested.
>>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
>>>> + *                physical address of PRR page passed from
>>>> + *                GPU driver.
>>>>     *
>>>>     * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>     * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
>>>>        void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>        void (*set_stall)(const void *cookie, bool enabled);
>>>>        void (*resume_translation)(const void *cookie, bool terminate);
>>>> +    void (*set_prr_bit)(const void *cookie, bool set);
>>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
>>>>    };
>>>>
>>>>    #endif /* __ADRENO_SMMU_PRIV_H */
>>>> --
>>>> 2.34.1
>>>>
>>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 11/22/2024 10:04 PM, Rob Clark wrote:
> On Fri, Nov 22, 2024 at 8:20 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 11/21/2024 3:40 AM, Rob Clark wrote:
>>> On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
>>>>
>>>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>
>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>> sequence as per request.
>>>>>
>>>>> This will be used by GPU to setup the PRR bit and related
>>>>> configuration registers through adreno-smmu private
>>>>> interface instead of directly poking the smmu hardware.
>>>>>
>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>> ---
>>>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
>>>>>    drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>    include/linux/adreno-smmu-priv.h           | 14 ++++++++
>>>>>    3 files changed, 53 insertions(+)
>>>>>
>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>> index d26f5aea248e..0e4f3fbda961 100644
>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>> @@ -16,6 +16,8 @@
>>>>>
>>>>>    #define QCOM_DUMMY_VAL -1
>>>>>
>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>> +
>>>>>    static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>>>>    {
>>>>>           return container_of(smmu, struct qcom_smmu, smmu);
>>>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>           arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>    }
>>>>>
>>>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
>>>>> +{
>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>> +       u32 reg = 0;
>>>>> +
>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>> +       if (set)
>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>> +}
>>>>> +
>>>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
>>>>> +{
>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>> +
>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>> +
>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>> +}
>>>>> +
>>>>>    #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>>
>>>>>    static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
>>>>>    static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>                   struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>>>>>    {
>>>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
>>>>>           struct adreno_smmu_priv *priv;
>>>>>
>>>>>           smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
>>>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>           priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>>           priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>>           priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>>> +       priv->set_prr_bit = NULL;
>>>>> +       priv->set_prr_addr = NULL;
>>>>> +
>>>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
>>>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
>>>>
>>>> fwiw, it seems like PRR actually works on sc7180, which is _not_
>>>> mmu-500, so I guess the support actually goes back further.
>>>>
>>>> I'm curious if we can just rely on this being supported by any hw that
>>>> has a6xx or newer?
>>>
>>>
>>> Also, unrelated, but we can't assume the smmu is powered when drm
>>> driver calls set_prr_bit/addr, could you add in runpm get/put around
>>> the register access?
>>>
>>
>> I see, thanks for this observation.
>> It's surely a possible case, when they access these registers
>> SMMU state is off.
>> I will add the suggested runpm ops around the register access.
>>
>>> Otherwise Conner and I have vk sparse residency working now
>>>
>>
>> Sorry, could not get this. Did you mean that vk sparse residency
>> is working now using this patch?
> 
> Yes, vk sparse residency is working with this patch (plus addition of
> runpm get/put, plus drm/msm patches plus turnip patches) ;-)
> 

Thanks for testing the sparse residency feature with our patch Rob,
I have an additional query regarding the adreno private interfaces. 
Specifically, I was referring to other interfaces such as 
qcom_adreno_smmu_get_fault_info [1]. It appears that we do not have a 
runpm get/put around the register access in this context.

Could you please clarify whether we need an SMMU vote around register 
access in the case of PRR? IMO, should the users of this callback ensure 
they put a vote before accessing the cb?

[1]: 
https://lore.kernel.org/all/20210610214431.539029-1-robdclark@gmail.com/

Thanks & regards,
Bibek

> BR,
> -R
> 
>> Thanks & regards,
>> Bibek
>>
>>> BR,
>>> -R
>>>
>>>>
>>>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
>>>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
>>>>> +       }
>>>>>
>>>>>           return 0;
>>>>>    }
>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>> index e2aeb511ae90..2dbf3243b5ad 100644
>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>>    #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>>
>>>>>    #define ARM_SMMU_CB_ACTLR              0x4
>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>>
>>>>>    #define ARM_SMMU_CB_RESUME             0x8
>>>>>    #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>>> index c637e0997f6d..614665153b3e 100644
>>>>> --- a/include/linux/adreno-smmu-priv.h
>>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
>>>>>     *                 the GPU driver must call resume_translation()
>>>>>     * @resume_translation: Resume translation after a fault
>>>>>     *
>>>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
>>>>> + *             targets without PRR support. Exercise caution and verify target
>>>>> + *             capabilities before invoking these callbacks to prevent potential
>>>>> + *             runtime errors or unexpected behavior.
>>>>> + *
>>>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
>>>>> + *                ACTLR register bits, currently used to configure
>>>>> + *                Partially-Resident-Region (PRR) bit for feature's
>>>>> + *                setup and reset sequence as requested.
>>>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
>>>>> + *                physical address of PRR page passed from
>>>>> + *                GPU driver.
>>>>>     *
>>>>>     * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>>     * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
>>>>>        void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>>        void (*set_stall)(const void *cookie, bool enabled);
>>>>>        void (*resume_translation)(const void *cookie, bool terminate);
>>>>> +    void (*set_prr_bit)(const void *cookie, bool set);
>>>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
>>>>>    };
>>>>>
>>>>>    #endif /* __ADRENO_SMMU_PRIV_H */
>>>>> --
>>>>> 2.34.1
>>>>>
>>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Wed, Dec 4, 2024 at 3:27 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 11/22/2024 10:33 PM, Rob Clark wrote:
> > On Fri, Nov 22, 2024 at 8:19 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >>
> >>
> >> On 11/20/2024 10:47 PM, Rob Clark wrote:
> >>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
> >>> <quic_bibekkum@quicinc.com> wrote:
> >>>>
> >>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>> sequence as per request.
> >>>>
> >>>> This will be used by GPU to setup the PRR bit and related
> >>>> configuration registers through adreno-smmu private
> >>>> interface instead of directly poking the smmu hardware.
> >>>>
> >>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>> ---
> >>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
> >>>>    drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>    include/linux/adreno-smmu-priv.h           | 14 ++++++++
> >>>>    3 files changed, 53 insertions(+)
> >>>>
> >>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> index d26f5aea248e..0e4f3fbda961 100644
> >>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>> @@ -16,6 +16,8 @@
> >>>>
> >>>>    #define QCOM_DUMMY_VAL -1
> >>>>
> >>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>> +
> >>>>    static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
> >>>>    {
> >>>>           return container_of(smmu, struct qcom_smmu, smmu);
> >>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>           arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>    }
> >>>>
> >>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
> >>>> +{
> >>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>> +       u32 reg = 0;
> >>>> +
> >>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>> +       if (set)
> >>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>> +}
> >>>> +
> >>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
> >>>> +{
> >>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>> +
> >>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>> +
> >>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>> +}
> >>>> +
> >>>>    #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>>
> >>>>    static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
> >>>>    static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>                   struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
> >>>>    {
> >>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
> >>>>           struct adreno_smmu_priv *priv;
> >>>>
> >>>>           smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
> >>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>           priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>>           priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>>           priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>>> +       priv->set_prr_bit = NULL;
> >>>> +       priv->set_prr_addr = NULL;
> >>>> +
> >>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
> >>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
> >>>
> >>> fwiw, it seems like PRR actually works on sc7180, which is _not_
> >>> mmu-500, so I guess the support actually goes back further.
> >>>
> >>
> >> As I checked sc7180 was on previous variant of SMMU,
> >> so targets on this variant have different steps to set PRR bit.
> >> <Do not have both prr bit and PRR page registers>.
> >
> > Experimentally, I have to set both the PRR LADDR/UADDR regs and
> > ACTLR.PRR bit on sc7180 to get the sparse-residency tests passing.  So
> > the underlying hw seems to work in the same way as mmu-500.  _But_
> > this is on a sc7180 chromebook, the situation might be different
> > depending on fw on things that have QC hyp.
> >
>
> I checked on sc7180 chipset which is based on the smmu-v2,
> this time by looking for these offsets specifically. I can see the
> nomenclature of the PRR related registers are a bit different
> compared to MMU-500 variant.
> Also the implementation register is 64 bit instead of
> dual 32 bit as in case of MMU-500. and PRR bit is not marked in
> ACTLR register offset.

Interesting.. in my experiments it needed both the ACTLR.PRR bit set
and the LADDR/UADDR.  Maybe it was just happy coincidence that two 32b
writes worked?

> So turns out PRR is supported but with some modification and
> can be carried out with same compatible based approach only - as per
> my understanding.
>
> In current patch plan was to provide support for MMU-500 based targets
> and won't break any legacy targets, so we can take the PRR support
> for legacy targets in different series once our evaluation is done on
> smmu-v2 ?

I guess it wouldn't be the end of the world, but it would mean that
drm would need to expose PRR support to userspace separately from
sparse binding support.  Maybe we need to do that anyways.  (I'm not
really sure how many different a6xx+smmu-v2 devices are out there, but
I guess they are all based on the same generation of snapdragon?)

BR,
-R

> We would explore more on this PRR feature for smmu-v2 based targets,
> before supporting it.
>
> Thanks & regards,
> Bibek
>
> >> It's MMU-500 targets only where PRR support is with both PRR bit
> >> and PRR page addr registers. As I was re-visiting our discussions on v13
> >> regarding this - I remember that's why we started using the SMMU-
> >> compatible string based PRR procedure selection instead of the reserved-
> >> memory node. [1] i.e Based on SMMU variant (as selected by compatible
> >> string), specific sequence will be selected.
> >>
> >> So for now only MMU-500 based selection has been supported as part of
> >> this series and will add subsequent support for other SMMU-variants
> >> thereafter.
> >>
> >>> I'm curious if we can just rely on this being supported by any hw that
> >>> has a6xx or newer?
> >>>
> >>
> >> I'd need to check on targets which will be based on a6xx onwards, on
> >> what will be the procedure planned to support PRR feature. I'll update
> >> the information over this space.
> >
> > One of the open questions about the drm/msm sparse-memory patchset is
> > whether we need to expose to userspace whether PRR is supported, or if
> > we can just rely on sparse-binding support implying sparse residency
> > (ie, PRR) support. All a6xx devices support per-process pgtables,
> > which is the only requirement for basic sparseBinding support.  But
> > PRR is required to additionally expose
> > sparseResidencyBuffer/sparseResidencyImage2D.
> >
> > I think, long term, turnip probably will want to drop support for
> > older kernels and remove support for legacy buffer mapping.  But if we
> > have some a6xx devices without PRR, then to do that we'd need to
> > decouple sparse binding and sparse residency.  (Vulkan allows a driver
> > to expose the former without the latter.)
> >
> > BR,
> > -R
> >
> >> [1]:
> >> https://lore.kernel.org/all/5790afa3-f9c0-4720-9804-8a7ff3d91854@quicinc.com/#:~:text=%3E%20I%20guess%20if,part%20as%20well.
> >>
> >> Thanks & regards,
> >> Bibek
> >>
> >>> BR,
> >>> -R
> >>>
> >>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
> >>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
> >>>> +       }
> >>>>
> >>>>           return 0;
> >>>>    }
> >>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>> index e2aeb511ae90..2dbf3243b5ad 100644
> >>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>>>    #define ARM_SMMU_SCTLR_M               BIT(0)
> >>>>
> >>>>    #define ARM_SMMU_CB_ACTLR              0x4
> >>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>>>
> >>>>    #define ARM_SMMU_CB_RESUME             0x8
> >>>>    #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >>>> index c637e0997f6d..614665153b3e 100644
> >>>> --- a/include/linux/adreno-smmu-priv.h
> >>>> +++ b/include/linux/adreno-smmu-priv.h
> >>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
> >>>>     *                 the GPU driver must call resume_translation()
> >>>>     * @resume_translation: Resume translation after a fault
> >>>>     *
> >>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
> >>>> + *             targets without PRR support. Exercise caution and verify target
> >>>> + *             capabilities before invoking these callbacks to prevent potential
> >>>> + *             runtime errors or unexpected behavior.
> >>>> + *
> >>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
> >>>> + *                ACTLR register bits, currently used to configure
> >>>> + *                Partially-Resident-Region (PRR) bit for feature's
> >>>> + *                setup and reset sequence as requested.
> >>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
> >>>> + *                physical address of PRR page passed from
> >>>> + *                GPU driver.
> >>>>     *
> >>>>     * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>>>     * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
> >>>>        void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>>>        void (*set_stall)(const void *cookie, bool enabled);
> >>>>        void (*resume_translation)(const void *cookie, bool terminate);
> >>>> +    void (*set_prr_bit)(const void *cookie, bool set);
> >>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
> >>>>    };
> >>>>
> >>>>    #endif /* __ADRENO_SMMU_PRIV_H */
> >>>> --
> >>>> 2.34.1
> >>>>
> >>
>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Wed, Dec 4, 2024 at 3:28 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 11/22/2024 10:04 PM, Rob Clark wrote:
> > On Fri, Nov 22, 2024 at 8:20 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >>
> >>
> >> On 11/21/2024 3:40 AM, Rob Clark wrote:
> >>> On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
> >>>>
> >>>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
> >>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>
> >>>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>>> sequence as per request.
> >>>>>
> >>>>> This will be used by GPU to setup the PRR bit and related
> >>>>> configuration registers through adreno-smmu private
> >>>>> interface instead of directly poking the smmu hardware.
> >>>>>
> >>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>>> ---
> >>>>>    drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
> >>>>>    drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>>    include/linux/adreno-smmu-priv.h           | 14 ++++++++
> >>>>>    3 files changed, 53 insertions(+)
> >>>>>
> >>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>> index d26f5aea248e..0e4f3fbda961 100644
> >>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>> @@ -16,6 +16,8 @@
> >>>>>
> >>>>>    #define QCOM_DUMMY_VAL -1
> >>>>>
> >>>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>>> +
> >>>>>    static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
> >>>>>    {
> >>>>>           return container_of(smmu, struct qcom_smmu, smmu);
> >>>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>>           arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>>    }
> >>>>>
> >>>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
> >>>>> +{
> >>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>>> +       u32 reg = 0;
> >>>>> +
> >>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>>> +       if (set)
> >>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>> +}
> >>>>> +
> >>>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
> >>>>> +{
> >>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>> +
> >>>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>> +
> >>>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>> +}
> >>>>> +
> >>>>>    #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>>>
> >>>>>    static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
> >>>>>    static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>                   struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
> >>>>>    {
> >>>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
> >>>>>           struct adreno_smmu_priv *priv;
> >>>>>
> >>>>>           smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
> >>>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>           priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>>>           priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>>>           priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>>>> +       priv->set_prr_bit = NULL;
> >>>>> +       priv->set_prr_addr = NULL;
> >>>>> +
> >>>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
> >>>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
> >>>>
> >>>> fwiw, it seems like PRR actually works on sc7180, which is _not_
> >>>> mmu-500, so I guess the support actually goes back further.
> >>>>
> >>>> I'm curious if we can just rely on this being supported by any hw that
> >>>> has a6xx or newer?
> >>>
> >>>
> >>> Also, unrelated, but we can't assume the smmu is powered when drm
> >>> driver calls set_prr_bit/addr, could you add in runpm get/put around
> >>> the register access?
> >>>
> >>
> >> I see, thanks for this observation.
> >> It's surely a possible case, when they access these registers
> >> SMMU state is off.
> >> I will add the suggested runpm ops around the register access.
> >>
> >>> Otherwise Conner and I have vk sparse residency working now
> >>>
> >>
> >> Sorry, could not get this. Did you mean that vk sparse residency
> >> is working now using this patch?
> >
> > Yes, vk sparse residency is working with this patch (plus addition of
> > runpm get/put, plus drm/msm patches plus turnip patches) ;-)
> >
>
> Thanks for testing the sparse residency feature with our patch Rob,
> I have an additional query regarding the adreno private interfaces.
> Specifically, I was referring to other interfaces such as
> qcom_adreno_smmu_get_fault_info [1]. It appears that we do not have a
> runpm get/put around the register access in this context.

get_fault_info() is called from the iommu fault handler callback, so
from the fault irq handler, which is why it didn't need the runpm
get/put.  Maybe it is bad to make this assumption about usage, but
then again adreno_smmu_priv isn't really a general purpose interface.

> Could you please clarify whether we need an SMMU vote around register
> access in the case of PRR? IMO, should the users of this callback ensure
> they put a vote before accessing the cb?

How can drm vote for the smmu device?  I guess it could power up
itself and rely on device-link.. but that is pretty overkill to power
up the entire gpu in this case.  I think it is best for the vote to
happen in the PRR callbacks.

BR,
-R

> [1]:
> https://lore.kernel.org/all/20210610214431.539029-1-robdclark@gmail.com/
>
> Thanks & regards,
> Bibek
>
> > BR,
> > -R
> >
> >> Thanks & regards,
> >> Bibek
> >>
> >>> BR,
> >>> -R
> >>>
> >>>>
> >>>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
> >>>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
> >>>>> +       }
> >>>>>
> >>>>>           return 0;
> >>>>>    }
> >>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>> index e2aeb511ae90..2dbf3243b5ad 100644
> >>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>>>>    #define ARM_SMMU_SCTLR_M               BIT(0)
> >>>>>
> >>>>>    #define ARM_SMMU_CB_ACTLR              0x4
> >>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>>>>
> >>>>>    #define ARM_SMMU_CB_RESUME             0x8
> >>>>>    #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >>>>> index c637e0997f6d..614665153b3e 100644
> >>>>> --- a/include/linux/adreno-smmu-priv.h
> >>>>> +++ b/include/linux/adreno-smmu-priv.h
> >>>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
> >>>>>     *                 the GPU driver must call resume_translation()
> >>>>>     * @resume_translation: Resume translation after a fault
> >>>>>     *
> >>>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
> >>>>> + *             targets without PRR support. Exercise caution and verify target
> >>>>> + *             capabilities before invoking these callbacks to prevent potential
> >>>>> + *             runtime errors or unexpected behavior.
> >>>>> + *
> >>>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
> >>>>> + *                ACTLR register bits, currently used to configure
> >>>>> + *                Partially-Resident-Region (PRR) bit for feature's
> >>>>> + *                setup and reset sequence as requested.
> >>>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
> >>>>> + *                physical address of PRR page passed from
> >>>>> + *                GPU driver.
> >>>>>     *
> >>>>>     * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>>>>     * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >>>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
> >>>>>        void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>>>>        void (*set_stall)(const void *cookie, bool enabled);
> >>>>>        void (*resume_translation)(const void *cookie, bool terminate);
> >>>>> +    void (*set_prr_bit)(const void *cookie, bool set);
> >>>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
> >>>>>    };
> >>>>>
> >>>>>    #endif /* __ADRENO_SMMU_PRIV_H */
> >>>>> --
> >>>>> 2.34.1
> >>>>>
> >>
>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 12/4/2024 8:51 PM, Rob Clark wrote:
> On Wed, Dec 4, 2024 at 3:27 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 11/22/2024 10:33 PM, Rob Clark wrote:
>>> On Fri, Nov 22, 2024 at 8:19 AM Bibek Kumar Patro
>>> <quic_bibekkum@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 11/20/2024 10:47 PM, Rob Clark wrote:
>>>>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>
>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>>> sequence as per request.
>>>>>>
>>>>>> This will be used by GPU to setup the PRR bit and related
>>>>>> configuration registers through adreno-smmu private
>>>>>> interface instead of directly poking the smmu hardware.
>>>>>>
>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>>> ---
>>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
>>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>>     include/linux/adreno-smmu-priv.h           | 14 ++++++++
>>>>>>     3 files changed, 53 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>> index d26f5aea248e..0e4f3fbda961 100644
>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>> @@ -16,6 +16,8 @@
>>>>>>
>>>>>>     #define QCOM_DUMMY_VAL -1
>>>>>>
>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>>> +
>>>>>>     static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>>>>>     {
>>>>>>            return container_of(smmu, struct qcom_smmu, smmu);
>>>>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>>            arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>>     }
>>>>>>
>>>>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
>>>>>> +{
>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>>> +       u32 reg = 0;
>>>>>> +
>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>>> +       if (set)
>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>> +}
>>>>>> +
>>>>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
>>>>>> +{
>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>> +
>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>> +
>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>> +}
>>>>>> +
>>>>>>     #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>>>
>>>>>>     static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
>>>>>>     static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>                    struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>>>>>>     {
>>>>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
>>>>>>            struct adreno_smmu_priv *priv;
>>>>>>
>>>>>>            smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
>>>>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>            priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>>>            priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>>>            priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>>>> +       priv->set_prr_bit = NULL;
>>>>>> +       priv->set_prr_addr = NULL;
>>>>>> +
>>>>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
>>>>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
>>>>>
>>>>> fwiw, it seems like PRR actually works on sc7180, which is _not_
>>>>> mmu-500, so I guess the support actually goes back further.
>>>>>
>>>>
>>>> As I checked sc7180 was on previous variant of SMMU,
>>>> so targets on this variant have different steps to set PRR bit.
>>>> <Do not have both prr bit and PRR page registers>.
>>>
>>> Experimentally, I have to set both the PRR LADDR/UADDR regs and
>>> ACTLR.PRR bit on sc7180 to get the sparse-residency tests passing.  So
>>> the underlying hw seems to work in the same way as mmu-500.  _But_
>>> this is on a sc7180 chromebook, the situation might be different
>>> depending on fw on things that have QC hyp.
>>>
>>
>> I checked on sc7180 chipset which is based on the smmu-v2,
>> this time by looking for these offsets specifically. I can see the
>> nomenclature of the PRR related registers are a bit different
>> compared to MMU-500 variant.
>> Also the implementation register is 64 bit instead of
>> dual 32 bit as in case of MMU-500. and PRR bit is not marked in
>> ACTLR register offset.
> 
> Interesting.. in my experiments it needed both the ACTLR.PRR bit set
> and the LADDR/UADDR.  Maybe it was just happy coincidence that two 32b
> writes worked?
> 

I think it's similar to case we discussed [1] in v11
on why not good idea to use writeq on 32bit regs.
So some corner cases for unaligned 32b addresses might get truncated.

[1]: 
https://lore.kernel.org/all/ae35bf9b-4401-4a99-abd7-c0d9d399a46b@quicinc.com/#:~:text=So%20I%20think-,writeq,-for%2064bit%20write


>> So turns out PRR is supported but with some modification and
>> can be carried out with same compatible based approach only - as per
>> my understanding.
>>
>> In current patch plan was to provide support for MMU-500 based targets
>> and won't break any legacy targets, so we can take the PRR support
>> for legacy targets in different series once our evaluation is done on
>> smmu-v2 ?
> 
> I guess it wouldn't be the end of the world, but it would mean that
> drm would need to expose PRR support to userspace separately from
> sparse binding support.  Maybe we need to do that anyways.  (I'm not
> really sure how many different a6xx+smmu-v2 devices are out there, but
> I guess they are all based on the same generation of snapdragon?)
> 

We would need to audit on all the smmu-v2 based devices using a6xx
version before I can comment on this, but mostly it should be in the 
same generation of snapdragon (scxx/smxx).

Thanks & regards,
bibek

> BR,
> -R
> 
>> We would explore more on this PRR feature for smmu-v2 based targets,
>> before supporting it.
>>
>> Thanks & regards,
>> Bibek
>>
>>>> It's MMU-500 targets only where PRR support is with both PRR bit
>>>> and PRR page addr registers. As I was re-visiting our discussions on v13
>>>> regarding this - I remember that's why we started using the SMMU-
>>>> compatible string based PRR procedure selection instead of the reserved-
>>>> memory node. [1] i.e Based on SMMU variant (as selected by compatible
>>>> string), specific sequence will be selected.
>>>>
>>>> So for now only MMU-500 based selection has been supported as part of
>>>> this series and will add subsequent support for other SMMU-variants
>>>> thereafter.
>>>>
>>>>> I'm curious if we can just rely on this being supported by any hw that
>>>>> has a6xx or newer?
>>>>>
>>>>
>>>> I'd need to check on targets which will be based on a6xx onwards, on
>>>> what will be the procedure planned to support PRR feature. I'll update
>>>> the information over this space.
>>>
>>> One of the open questions about the drm/msm sparse-memory patchset is
>>> whether we need to expose to userspace whether PRR is supported, or if
>>> we can just rely on sparse-binding support implying sparse residency
>>> (ie, PRR) support. All a6xx devices support per-process pgtables,
>>> which is the only requirement for basic sparseBinding support.  But
>>> PRR is required to additionally expose
>>> sparseResidencyBuffer/sparseResidencyImage2D.
>>>
>>> I think, long term, turnip probably will want to drop support for
>>> older kernels and remove support for legacy buffer mapping.  But if we
>>> have some a6xx devices without PRR, then to do that we'd need to
>>> decouple sparse binding and sparse residency.  (Vulkan allows a driver
>>> to expose the former without the latter.)
>>>
>>> BR,
>>> -R
>>>
>>>> [1]:
>>>> https://lore.kernel.org/all/5790afa3-f9c0-4720-9804-8a7ff3d91854@quicinc.com/#:~:text=%3E%20I%20guess%20if,part%20as%20well.
>>>>
>>>> Thanks & regards,
>>>> Bibek
>>>>
>>>>> BR,
>>>>> -R
>>>>>
>>>>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
>>>>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
>>>>>> +       }
>>>>>>
>>>>>>            return 0;
>>>>>>     }
>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>> index e2aeb511ae90..2dbf3243b5ad 100644
>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>>>     #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>>>
>>>>>>     #define ARM_SMMU_CB_ACTLR              0x4
>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>>>
>>>>>>     #define ARM_SMMU_CB_RESUME             0x8
>>>>>>     #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>>>> index c637e0997f6d..614665153b3e 100644
>>>>>> --- a/include/linux/adreno-smmu-priv.h
>>>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
>>>>>>      *                 the GPU driver must call resume_translation()
>>>>>>      * @resume_translation: Resume translation after a fault
>>>>>>      *
>>>>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
>>>>>> + *             targets without PRR support. Exercise caution and verify target
>>>>>> + *             capabilities before invoking these callbacks to prevent potential
>>>>>> + *             runtime errors or unexpected behavior.
>>>>>> + *
>>>>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
>>>>>> + *                ACTLR register bits, currently used to configure
>>>>>> + *                Partially-Resident-Region (PRR) bit for feature's
>>>>>> + *                setup and reset sequence as requested.
>>>>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
>>>>>> + *                physical address of PRR page passed from
>>>>>> + *                GPU driver.
>>>>>>      *
>>>>>>      * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>>>      * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
>>>>>>         void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>>>         void (*set_stall)(const void *cookie, bool enabled);
>>>>>>         void (*resume_translation)(const void *cookie, bool terminate);
>>>>>> +    void (*set_prr_bit)(const void *cookie, bool set);
>>>>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
>>>>>>     };
>>>>>>
>>>>>>     #endif /* __ADRENO_SMMU_PRIV_H */
>>>>>> --
>>>>>> 2.34.1
>>>>>>
>>>>
>>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 12/4/2024 8:54 PM, Rob Clark wrote:
> On Wed, Dec 4, 2024 at 3:28 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 11/22/2024 10:04 PM, Rob Clark wrote:
>>> On Fri, Nov 22, 2024 at 8:20 AM Bibek Kumar Patro
>>> <quic_bibekkum@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 11/21/2024 3:40 AM, Rob Clark wrote:
>>>>> On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
>>>>>>
>>>>>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>
>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>>>> sequence as per request.
>>>>>>>
>>>>>>> This will be used by GPU to setup the PRR bit and related
>>>>>>> configuration registers through adreno-smmu private
>>>>>>> interface instead of directly poking the smmu hardware.
>>>>>>>
>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>>>> ---
>>>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
>>>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>>>     include/linux/adreno-smmu-priv.h           | 14 ++++++++
>>>>>>>     3 files changed, 53 insertions(+)
>>>>>>>
>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>> index d26f5aea248e..0e4f3fbda961 100644
>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>> @@ -16,6 +16,8 @@
>>>>>>>
>>>>>>>     #define QCOM_DUMMY_VAL -1
>>>>>>>
>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>>>> +
>>>>>>>     static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>>>>>>     {
>>>>>>>            return container_of(smmu, struct qcom_smmu, smmu);
>>>>>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>>>            arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>>>     }
>>>>>>>
>>>>>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
>>>>>>> +{
>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>>>> +       u32 reg = 0;
>>>>>>> +
>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>>>> +       if (set)
>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>> +}
>>>>>>> +
>>>>>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
>>>>>>> +{
>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>>> +
>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>> +
>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>> +}
>>>>>>> +
>>>>>>>     #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>>>>
>>>>>>>     static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>>>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
>>>>>>>     static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>>                    struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>>>>>>>     {
>>>>>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
>>>>>>>            struct adreno_smmu_priv *priv;
>>>>>>>
>>>>>>>            smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
>>>>>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>>            priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>>>>            priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>>>>            priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>>>>> +       priv->set_prr_bit = NULL;
>>>>>>> +       priv->set_prr_addr = NULL;
>>>>>>> +
>>>>>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
>>>>>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
>>>>>>
>>>>>> fwiw, it seems like PRR actually works on sc7180, which is _not_
>>>>>> mmu-500, so I guess the support actually goes back further.
>>>>>>
>>>>>> I'm curious if we can just rely on this being supported by any hw that
>>>>>> has a6xx or newer?
>>>>>
>>>>>
>>>>> Also, unrelated, but we can't assume the smmu is powered when drm
>>>>> driver calls set_prr_bit/addr, could you add in runpm get/put around
>>>>> the register access?
>>>>>
>>>>
>>>> I see, thanks for this observation.
>>>> It's surely a possible case, when they access these registers
>>>> SMMU state is off.
>>>> I will add the suggested runpm ops around the register access.
>>>>
>>>>> Otherwise Conner and I have vk sparse residency working now
>>>>>
>>>>
>>>> Sorry, could not get this. Did you mean that vk sparse residency
>>>> is working now using this patch?
>>>
>>> Yes, vk sparse residency is working with this patch (plus addition of
>>> runpm get/put, plus drm/msm patches plus turnip patches) ;-)
>>>
>>
>> Thanks for testing the sparse residency feature with our patch Rob,
>> I have an additional query regarding the adreno private interfaces.
>> Specifically, I was referring to other interfaces such as
>> qcom_adreno_smmu_get_fault_info [1]. It appears that we do not have a
>> runpm get/put around the register access in this context.
> 
> get_fault_info() is called from the iommu fault handler callback, so
> from the fault irq handler, which is why it didn't need the runpm
> get/put.  Maybe it is bad to make this assumption about usage, but
> then again adreno_smmu_priv isn't really a general purpose interface.
> 

Ah okay, got it.
I was just going through all the adreno_smmmu_priv interfaces just
to get a better understanding of it's interaction with smmu and it seems
like apart from PRR and get_fault_info(), set_ttbr0_cfg(),
resume_translation() is also having reg access but not voting.
Should we put runpm_put/get here as well or these can be excluded.
<Just curious about the thought process behind this, is it because of
some sequence when to put a vote, like reg access in middle of smmu
power cycle and not in beginning or end.>

>> Could you please clarify whether we need an SMMU vote around register
>> access in the case of PRR? IMO, should the users of this callback ensure
>> they put a vote before accessing the cb?
> 
> How can drm vote for the smmu device?  I guess it could power up
> itself and rely on device-link.. but that is pretty overkill to power
> up the entire gpu in this case.  I think it is best for the vote to
> happen in the PRR callbacks.
> 

Okay I see, GPU can only power itself up through <gpu_state_get I
assume> but won't have exclusive access to power on the smmu only.

Incase we go forward to put vote in PRR callback for SMMU, I was
planning that we can refactor existing arm_smmu_rpm_put/get() from
arm_smmu.c to it's header file so that the same can be used in
arm_smmu_qcom.c ? What's your thoughts on this?

Thanks & regards,
Bibek

> BR,
> -R
> 
>> [1]:
>> https://lore.kernel.org/all/20210610214431.539029-1-robdclark@gmail.com/
>>
>> Thanks & regards,
>> Bibek
>>
>>> BR,
>>> -R
>>>
>>>> Thanks & regards,
>>>> Bibek
>>>>
>>>>> BR,
>>>>> -R
>>>>>
>>>>>>
>>>>>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
>>>>>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
>>>>>>> +       }
>>>>>>>
>>>>>>>            return 0;
>>>>>>>     }
>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>> index e2aeb511ae90..2dbf3243b5ad 100644
>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>>>>     #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>>>>
>>>>>>>     #define ARM_SMMU_CB_ACTLR              0x4
>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>>>>
>>>>>>>     #define ARM_SMMU_CB_RESUME             0x8
>>>>>>>     #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>>>>> index c637e0997f6d..614665153b3e 100644
>>>>>>> --- a/include/linux/adreno-smmu-priv.h
>>>>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>>>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
>>>>>>>      *                 the GPU driver must call resume_translation()
>>>>>>>      * @resume_translation: Resume translation after a fault
>>>>>>>      *
>>>>>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
>>>>>>> + *             targets without PRR support. Exercise caution and verify target
>>>>>>> + *             capabilities before invoking these callbacks to prevent potential
>>>>>>> + *             runtime errors or unexpected behavior.
>>>>>>> + *
>>>>>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
>>>>>>> + *                ACTLR register bits, currently used to configure
>>>>>>> + *                Partially-Resident-Region (PRR) bit for feature's
>>>>>>> + *                setup and reset sequence as requested.
>>>>>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
>>>>>>> + *                physical address of PRR page passed from
>>>>>>> + *                GPU driver.
>>>>>>>      *
>>>>>>>      * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>>>>      * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>>>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
>>>>>>>         void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>>>>         void (*set_stall)(const void *cookie, bool enabled);
>>>>>>>         void (*resume_translation)(const void *cookie, bool terminate);
>>>>>>> +    void (*set_prr_bit)(const void *cookie, bool set);
>>>>>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
>>>>>>>     };
>>>>>>>
>>>>>>>     #endif /* __ADRENO_SMMU_PRIV_H */
>>>>>>> --
>>>>>>> 2.34.1
>>>>>>>
>>>>
>>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Fri, Dec 6, 2024 at 4:36 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 12/4/2024 8:54 PM, Rob Clark wrote:
> > On Wed, Dec 4, 2024 at 3:28 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >>
> >>
> >> On 11/22/2024 10:04 PM, Rob Clark wrote:
> >>> On Fri, Nov 22, 2024 at 8:20 AM Bibek Kumar Patro
> >>> <quic_bibekkum@quicinc.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 11/21/2024 3:40 AM, Rob Clark wrote:
> >>>>> On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
> >>>>>>
> >>>>>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
> >>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>
> >>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>>>>> sequence as per request.
> >>>>>>>
> >>>>>>> This will be used by GPU to setup the PRR bit and related
> >>>>>>> configuration registers through adreno-smmu private
> >>>>>>> interface instead of directly poking the smmu hardware.
> >>>>>>>
> >>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>>>>> ---
> >>>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
> >>>>>>>     drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>>>>     include/linux/adreno-smmu-priv.h           | 14 ++++++++
> >>>>>>>     3 files changed, 53 insertions(+)
> >>>>>>>
> >>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>> index d26f5aea248e..0e4f3fbda961 100644
> >>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>> @@ -16,6 +16,8 @@
> >>>>>>>
> >>>>>>>     #define QCOM_DUMMY_VAL -1
> >>>>>>>
> >>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>>>>> +
> >>>>>>>     static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
> >>>>>>>     {
> >>>>>>>            return container_of(smmu, struct qcom_smmu, smmu);
> >>>>>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>>>>            arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>>>>     }
> >>>>>>>
> >>>>>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
> >>>>>>> +{
> >>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>>>>> +       u32 reg = 0;
> >>>>>>> +
> >>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>>>>> +       if (set)
> >>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>> +}
> >>>>>>> +
> >>>>>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
> >>>>>>> +{
> >>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>>>> +
> >>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>> +
> >>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>> +}
> >>>>>>> +
> >>>>>>>     #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>>>>>
> >>>>>>>     static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>>>>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
> >>>>>>>     static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>>>                    struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
> >>>>>>>     {
> >>>>>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
> >>>>>>>            struct adreno_smmu_priv *priv;
> >>>>>>>
> >>>>>>>            smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
> >>>>>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>>>            priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>>>>>            priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>>>>>            priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>>>>>> +       priv->set_prr_bit = NULL;
> >>>>>>> +       priv->set_prr_addr = NULL;
> >>>>>>> +
> >>>>>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
> >>>>>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
> >>>>>>
> >>>>>> fwiw, it seems like PRR actually works on sc7180, which is _not_
> >>>>>> mmu-500, so I guess the support actually goes back further.
> >>>>>>
> >>>>>> I'm curious if we can just rely on this being supported by any hw that
> >>>>>> has a6xx or newer?
> >>>>>
> >>>>>
> >>>>> Also, unrelated, but we can't assume the smmu is powered when drm
> >>>>> driver calls set_prr_bit/addr, could you add in runpm get/put around
> >>>>> the register access?
> >>>>>
> >>>>
> >>>> I see, thanks for this observation.
> >>>> It's surely a possible case, when they access these registers
> >>>> SMMU state is off.
> >>>> I will add the suggested runpm ops around the register access.
> >>>>
> >>>>> Otherwise Conner and I have vk sparse residency working now
> >>>>>
> >>>>
> >>>> Sorry, could not get this. Did you mean that vk sparse residency
> >>>> is working now using this patch?
> >>>
> >>> Yes, vk sparse residency is working with this patch (plus addition of
> >>> runpm get/put, plus drm/msm patches plus turnip patches) ;-)
> >>>
> >>
> >> Thanks for testing the sparse residency feature with our patch Rob,
> >> I have an additional query regarding the adreno private interfaces.
> >> Specifically, I was referring to other interfaces such as
> >> qcom_adreno_smmu_get_fault_info [1]. It appears that we do not have a
> >> runpm get/put around the register access in this context.
> >
> > get_fault_info() is called from the iommu fault handler callback, so
> > from the fault irq handler, which is why it didn't need the runpm
> > get/put.  Maybe it is bad to make this assumption about usage, but
> > then again adreno_smmu_priv isn't really a general purpose interface.
> >
>
> Ah okay, got it.
> I was just going through all the adreno_smmmu_priv interfaces just
> to get a better understanding of it's interaction with smmu and it seems
> like apart from PRR and get_fault_info(), set_ttbr0_cfg(),
> resume_translation() is also having reg access but not voting.
> Should we put runpm_put/get here as well or these can be excluded.
> <Just curious about the thought process behind this, is it because of
> some sequence when to put a vote, like reg access in middle of smmu
> power cycle and not in beginning or end.>

I think we just haven't needed it, or noticed that we needed it,
outside of setting up prr.

As I mentioned, get_fault_info() is called from the fault irq, so we
know the smmu is already powered.

As far as set_ttbr0_cfg(), it probably works just because
arm_smmu_write_context_bank() ends up getting called again in the
resume path, so if the smmu is suspended when set_ttbr0_cfg() is
called, the writes just get ignored.  But the updated cfg is
re-applied to the hw when it is resumed.  Probably the same situation
with resume_translation().. ie. if the smmu is suspended there are no
translations to resume.

Maybe it would be more correct in set_ttbr0_cfg() and
resume_translation() to do a pm_runtime_get_if_in_use() and skip the
hw writes if the smmu is suspended.

>
> >> Could you please clarify whether we need an SMMU vote around register
> >> access in the case of PRR? IMO, should the users of this callback ensure
> >> they put a vote before accessing the cb?
> >
> > How can drm vote for the smmu device?  I guess it could power up
> > itself and rely on device-link.. but that is pretty overkill to power
> > up the entire gpu in this case.  I think it is best for the vote to
> > happen in the PRR callbacks.
> >
>
> Okay I see, GPU can only power itself up through <gpu_state_get I
> assume> but won't have exclusive access to power on the smmu only.
>
> Incase we go forward to put vote in PRR callback for SMMU, I was
> planning that we can refactor existing arm_smmu_rpm_put/get() from
> arm_smmu.c to it's header file so that the same can be used in
> arm_smmu_qcom.c ? What's your thoughts on this?

I had briefly thought of doing the same.  But the main reason for
those helpers is common arm-smmu code that is used on non-qcom
platforms where runpm is not enabled.  In arm-smmu-qcom.c, we know
that runpm is enabled, so we could just use  return
pm_runtime_resume_and_get()/pm_runtime_put_autosuspend() directly.

BR,
-R

> Thanks & regards,
> Bibek
>
> > BR,
> > -R
> >
> >> [1]:
> >> https://lore.kernel.org/all/20210610214431.539029-1-robdclark@gmail.com/
> >>
> >> Thanks & regards,
> >> Bibek
> >>
> >>> BR,
> >>> -R
> >>>
> >>>> Thanks & regards,
> >>>> Bibek
> >>>>
> >>>>> BR,
> >>>>> -R
> >>>>>
> >>>>>>
> >>>>>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
> >>>>>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
> >>>>>>> +       }
> >>>>>>>
> >>>>>>>            return 0;
> >>>>>>>     }
> >>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>> index e2aeb511ae90..2dbf3243b5ad 100644
> >>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
> >>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
> >>>>>>>     #define ARM_SMMU_SCTLR_M               BIT(0)
> >>>>>>>
> >>>>>>>     #define ARM_SMMU_CB_ACTLR              0x4
> >>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
> >>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
> >>>>>>>
> >>>>>>>     #define ARM_SMMU_CB_RESUME             0x8
> >>>>>>>     #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
> >>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
> >>>>>>> index c637e0997f6d..614665153b3e 100644
> >>>>>>> --- a/include/linux/adreno-smmu-priv.h
> >>>>>>> +++ b/include/linux/adreno-smmu-priv.h
> >>>>>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
> >>>>>>>      *                 the GPU driver must call resume_translation()
> >>>>>>>      * @resume_translation: Resume translation after a fault
> >>>>>>>      *
> >>>>>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
> >>>>>>> + *             targets without PRR support. Exercise caution and verify target
> >>>>>>> + *             capabilities before invoking these callbacks to prevent potential
> >>>>>>> + *             runtime errors or unexpected behavior.
> >>>>>>> + *
> >>>>>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
> >>>>>>> + *                ACTLR register bits, currently used to configure
> >>>>>>> + *                Partially-Resident-Region (PRR) bit for feature's
> >>>>>>> + *                setup and reset sequence as requested.
> >>>>>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
> >>>>>>> + *                physical address of PRR page passed from
> >>>>>>> + *                GPU driver.
> >>>>>>>      *
> >>>>>>>      * The GPU driver (drm/msm) and adreno-smmu work together for controlling
> >>>>>>>      * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
> >>>>>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
> >>>>>>>         void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
> >>>>>>>         void (*set_stall)(const void *cookie, bool enabled);
> >>>>>>>         void (*resume_translation)(const void *cookie, bool terminate);
> >>>>>>> +    void (*set_prr_bit)(const void *cookie, bool set);
> >>>>>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
> >>>>>>>     };
> >>>>>>>
> >>>>>>>     #endif /* __ADRENO_SMMU_PRIV_H */
> >>>>>>> --
> >>>>>>> 2.34.1
> >>>>>>>
> >>>>
> >>
>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Bibek Kumar Patro]

On 12/6/2024 8:48 PM, Rob Clark wrote:
> On Fri, Dec 6, 2024 at 4:36 AM Bibek Kumar Patro
> <quic_bibekkum@quicinc.com> wrote:
>>
>>
>>
>> On 12/4/2024 8:54 PM, Rob Clark wrote:
>>> On Wed, Dec 4, 2024 at 3:28 AM Bibek Kumar Patro
>>> <quic_bibekkum@quicinc.com> wrote:
>>>>
>>>>
>>>>
>>>> On 11/22/2024 10:04 PM, Rob Clark wrote:
>>>>> On Fri, Nov 22, 2024 at 8:20 AM Bibek Kumar Patro
>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 11/21/2024 3:40 AM, Rob Clark wrote:
>>>>>>> On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
>>>>>>>>
>>>>>>>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
>>>>>>>> <quic_bibekkum@quicinc.com> wrote:
>>>>>>>>>
>>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
>>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
>>>>>>>>> sequence as per request.
>>>>>>>>>
>>>>>>>>> This will be used by GPU to setup the PRR bit and related
>>>>>>>>> configuration registers through adreno-smmu private
>>>>>>>>> interface instead of directly poking the smmu hardware.
>>>>>>>>>
>>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
>>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
>>>>>>>>> ---
>>>>>>>>>      drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
>>>>>>>>>      drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
>>>>>>>>>      include/linux/adreno-smmu-priv.h           | 14 ++++++++
>>>>>>>>>      3 files changed, 53 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>> index d26f5aea248e..0e4f3fbda961 100644
>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
>>>>>>>>> @@ -16,6 +16,8 @@
>>>>>>>>>
>>>>>>>>>      #define QCOM_DUMMY_VAL -1
>>>>>>>>>
>>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
>>>>>>>>> +
>>>>>>>>>      static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
>>>>>>>>>      {
>>>>>>>>>             return container_of(smmu, struct qcom_smmu, smmu);
>>>>>>>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
>>>>>>>>>             arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
>>>>>>>>>      }
>>>>>>>>>
>>>>>>>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
>>>>>>>>> +{
>>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
>>>>>>>>> +       u32 reg = 0;
>>>>>>>>> +
>>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
>>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
>>>>>>>>> +       if (set)
>>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
>>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
>>>>>>>>> +{
>>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
>>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
>>>>>>>>> +
>>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
>>>>>>>>> +
>>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
>>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>>      #define QCOM_ADRENO_SMMU_GPU_SID 0
>>>>>>>>>
>>>>>>>>>      static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
>>>>>>>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
>>>>>>>>>      static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>>>>                     struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
>>>>>>>>>      {
>>>>>>>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
>>>>>>>>>             struct adreno_smmu_priv *priv;
>>>>>>>>>
>>>>>>>>>             smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
>>>>>>>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>>>>>>>>>             priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
>>>>>>>>>             priv->set_stall = qcom_adreno_smmu_set_stall;
>>>>>>>>>             priv->resume_translation = qcom_adreno_smmu_resume_translation;
>>>>>>>>> +       priv->set_prr_bit = NULL;
>>>>>>>>> +       priv->set_prr_addr = NULL;
>>>>>>>>> +
>>>>>>>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
>>>>>>>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
>>>>>>>>
>>>>>>>> fwiw, it seems like PRR actually works on sc7180, which is _not_
>>>>>>>> mmu-500, so I guess the support actually goes back further.
>>>>>>>>
>>>>>>>> I'm curious if we can just rely on this being supported by any hw that
>>>>>>>> has a6xx or newer?
>>>>>>>
>>>>>>>
>>>>>>> Also, unrelated, but we can't assume the smmu is powered when drm
>>>>>>> driver calls set_prr_bit/addr, could you add in runpm get/put around
>>>>>>> the register access?
>>>>>>>
>>>>>>
>>>>>> I see, thanks for this observation.
>>>>>> It's surely a possible case, when they access these registers
>>>>>> SMMU state is off.
>>>>>> I will add the suggested runpm ops around the register access.
>>>>>>
>>>>>>> Otherwise Conner and I have vk sparse residency working now
>>>>>>>
>>>>>>
>>>>>> Sorry, could not get this. Did you mean that vk sparse residency
>>>>>> is working now using this patch?
>>>>>
>>>>> Yes, vk sparse residency is working with this patch (plus addition of
>>>>> runpm get/put, plus drm/msm patches plus turnip patches) ;-)
>>>>>
>>>>
>>>> Thanks for testing the sparse residency feature with our patch Rob,
>>>> I have an additional query regarding the adreno private interfaces.
>>>> Specifically, I was referring to other interfaces such as
>>>> qcom_adreno_smmu_get_fault_info [1]. It appears that we do not have a
>>>> runpm get/put around the register access in this context.
>>>
>>> get_fault_info() is called from the iommu fault handler callback, so
>>> from the fault irq handler, which is why it didn't need the runpm
>>> get/put.  Maybe it is bad to make this assumption about usage, but
>>> then again adreno_smmu_priv isn't really a general purpose interface.
>>>
>>
>> Ah okay, got it.
>> I was just going through all the adreno_smmmu_priv interfaces just
>> to get a better understanding of it's interaction with smmu and it seems
>> like apart from PRR and get_fault_info(), set_ttbr0_cfg(),
>> resume_translation() is also having reg access but not voting.
>> Should we put runpm_put/get here as well or these can be excluded.
>> <Just curious about the thought process behind this, is it because of
>> some sequence when to put a vote, like reg access in middle of smmu
>> power cycle and not in beginning or end.>
> 
> I think we just haven't needed it, or noticed that we needed it,
> outside of setting up prr.
> 
> As I mentioned, get_fault_info() is called from the fault irq, so we
> know the smmu is already powered.
> 

okay got it, that makes sense.

> As far as set_ttbr0_cfg(), it probably works just because
> arm_smmu_write_context_bank() ends up getting called again in the
> resume path, so if the smmu is suspended when set_ttbr0_cfg() is
> called, the writes just get ignored.  But the updated cfg is
> re-applied to the hw when it is resumed.  Probably the same situation
> with resume_translation().. ie. if the smmu is suspended there are no
> translations to resume.
> 
> Maybe it would be more correct in set_ttbr0_cfg() and
> resume_translation() to do a pm_runtime_get_if_in_use() and skip the
> hw writes if the smmu is suspended.
> 
>>
>>>> Could you please clarify whether we need an SMMU vote around register
>>>> access in the case of PRR? IMO, should the users of this callback ensure
>>>> they put a vote before accessing the cb?
>>>
>>> How can drm vote for the smmu device?  I guess it could power up
>>> itself and rely on device-link.. but that is pretty overkill to power
>>> up the entire gpu in this case.  I think it is best for the vote to
>>> happen in the PRR callbacks.
>>>
>>
>> Okay I see, GPU can only power itself up through <gpu_state_get I
>> assume> but won't have exclusive access to power on the smmu only.
>>
>> Incase we go forward to put vote in PRR callback for SMMU, I was
>> planning that we can refactor existing arm_smmu_rpm_put/get() from
>> arm_smmu.c to it's header file so that the same can be used in
>> arm_smmu_qcom.c ? What's your thoughts on this?
> 
> I had briefly thought of doing the same.  But the main reason for
> those helpers is common arm-smmu code that is used on non-qcom
> platforms where runpm is not enabled.  In arm-smmu-qcom.c, we know
> that runpm is enabled, so we could just use  return
> pm_runtime_resume_and_get()/pm_runtime_put_autosuspend() directly.
> 

Ohkay I see, we then do not need pm_runtime_enabled() check for qcom 
platforms before putting the vote.
I am currently modifying this patch only to directly add
pm_runtime_resume_and_get()/pm_runtime_put_autosuspend()
around the register access of PRR related adreno private interfaces.
I will send this updated patch as part of v18 shortly.

Additionally, we can evaluate the use of pm_runtime operations for 
set_ttbr0_cfg() and resume_translation() in a separate series ?

Thanks & regards,
Bibek

> BR,
> -R
> 
>> Thanks & regards,
>> Bibek
>>
>>> BR,
>>> -R
>>>
>>>> [1]:
>>>> https://lore.kernel.org/all/20210610214431.539029-1-robdclark@gmail.com/
>>>>
>>>> Thanks & regards,
>>>> Bibek
>>>>
>>>>> BR,
>>>>> -R
>>>>>
>>>>>> Thanks & regards,
>>>>>> Bibek
>>>>>>
>>>>>>> BR,
>>>>>>> -R
>>>>>>>
>>>>>>>>
>>>>>>>>> +               priv->set_prr_bit = qcom_adreno_smmu_set_prr_bit;
>>>>>>>>> +               priv->set_prr_addr = qcom_adreno_smmu_set_prr_addr;
>>>>>>>>> +       }
>>>>>>>>>
>>>>>>>>>             return 0;
>>>>>>>>>      }
>>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>> index e2aeb511ae90..2dbf3243b5ad 100644
>>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h
>>>>>>>>> @@ -154,6 +154,8 @@ enum arm_smmu_cbar_type {
>>>>>>>>>      #define ARM_SMMU_SCTLR_M               BIT(0)
>>>>>>>>>
>>>>>>>>>      #define ARM_SMMU_CB_ACTLR              0x4
>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_LADDR     0x6008
>>>>>>>>> +#define ARM_SMMU_GFX_PRR_CFG_UADDR     0x600C
>>>>>>>>>
>>>>>>>>>      #define ARM_SMMU_CB_RESUME             0x8
>>>>>>>>>      #define ARM_SMMU_RESUME_TERMINATE      BIT(0)
>>>>>>>>> diff --git a/include/linux/adreno-smmu-priv.h b/include/linux/adreno-smmu-priv.h
>>>>>>>>> index c637e0997f6d..614665153b3e 100644
>>>>>>>>> --- a/include/linux/adreno-smmu-priv.h
>>>>>>>>> +++ b/include/linux/adreno-smmu-priv.h
>>>>>>>>> @@ -50,6 +50,18 @@ struct adreno_smmu_fault_info {
>>>>>>>>>       *                 the GPU driver must call resume_translation()
>>>>>>>>>       * @resume_translation: Resume translation after a fault
>>>>>>>>>       *
>>>>>>>>> + * *CAUTION* : PRR callbacks (set_prr_bit/set_prr_addr) are NULL terminated for
>>>>>>>>> + *             targets without PRR support. Exercise caution and verify target
>>>>>>>>> + *             capabilities before invoking these callbacks to prevent potential
>>>>>>>>> + *             runtime errors or unexpected behavior.
>>>>>>>>> + *
>>>>>>>>> + * @set_prr_bit:   Extendible interface to be used by GPU to modify the
>>>>>>>>> + *                ACTLR register bits, currently used to configure
>>>>>>>>> + *                Partially-Resident-Region (PRR) bit for feature's
>>>>>>>>> + *                setup and reset sequence as requested.
>>>>>>>>> + * @set_prr_addr:  Configure the PRR_CFG_*ADDR register with the
>>>>>>>>> + *                physical address of PRR page passed from
>>>>>>>>> + *                GPU driver.
>>>>>>>>>       *
>>>>>>>>>       * The GPU driver (drm/msm) and adreno-smmu work together for controlling
>>>>>>>>>       * the GPU's SMMU instance.  This is by necessity, as the GPU is directly
>>>>>>>>> @@ -67,6 +79,8 @@ struct adreno_smmu_priv {
>>>>>>>>>          void (*get_fault_info)(const void *cookie, struct adreno_smmu_fault_info *info);
>>>>>>>>>          void (*set_stall)(const void *cookie, bool enabled);
>>>>>>>>>          void (*resume_translation)(const void *cookie, bool terminate);
>>>>>>>>> +    void (*set_prr_bit)(const void *cookie, bool set);
>>>>>>>>> +    void (*set_prr_addr)(const void *cookie, phys_addr_t page_addr);
>>>>>>>>>      };
>>>>>>>>>
>>>>>>>>>      #endif /* __ADRENO_SMMU_PRIV_H */
>>>>>>>>> --
>>>>>>>>> 2.34.1
>>>>>>>>>
>>>>>>
>>>>
>>

Re: [PATCH RESEND v17 3/5] iommu/arm-smmu: add support for PRR bit setup

[Rob Clark]

On Wed, Dec 11, 2024 at 5:30 AM Bibek Kumar Patro
<quic_bibekkum@quicinc.com> wrote:
>
>
>
> On 12/6/2024 8:48 PM, Rob Clark wrote:
> > On Fri, Dec 6, 2024 at 4:36 AM Bibek Kumar Patro
> > <quic_bibekkum@quicinc.com> wrote:
> >>
> >>
> >>
> >> On 12/4/2024 8:54 PM, Rob Clark wrote:
> >>> On Wed, Dec 4, 2024 at 3:28 AM Bibek Kumar Patro
> >>> <quic_bibekkum@quicinc.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 11/22/2024 10:04 PM, Rob Clark wrote:
> >>>>> On Fri, Nov 22, 2024 at 8:20 AM Bibek Kumar Patro
> >>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 11/21/2024 3:40 AM, Rob Clark wrote:
> >>>>>>> On Wed, Nov 20, 2024 at 9:17 AM Rob Clark <robdclark@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> On Thu, Nov 14, 2024 at 8:10 AM Bibek Kumar Patro
> >>>>>>>> <quic_bibekkum@quicinc.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Add an adreno-smmu-priv interface for drm/msm to call
> >>>>>>>>> into arm-smmu-qcom and initiate the PRR bit setup or reset
> >>>>>>>>> sequence as per request.
> >>>>>>>>>
> >>>>>>>>> This will be used by GPU to setup the PRR bit and related
> >>>>>>>>> configuration registers through adreno-smmu private
> >>>>>>>>> interface instead of directly poking the smmu hardware.
> >>>>>>>>>
> >>>>>>>>> Suggested-by: Rob Clark <robdclark@gmail.com>
> >>>>>>>>> Signed-off-by: Bibek Kumar Patro <quic_bibekkum@quicinc.com>
> >>>>>>>>> ---
> >>>>>>>>>      drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 37 ++++++++++++++++++++++
> >>>>>>>>>      drivers/iommu/arm/arm-smmu/arm-smmu.h      |  2 ++
> >>>>>>>>>      include/linux/adreno-smmu-priv.h           | 14 ++++++++
> >>>>>>>>>      3 files changed, 53 insertions(+)
> >>>>>>>>>
> >>>>>>>>> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>> index d26f5aea248e..0e4f3fbda961 100644
> >>>>>>>>> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> >>>>>>>>> @@ -16,6 +16,8 @@
> >>>>>>>>>
> >>>>>>>>>      #define QCOM_DUMMY_VAL -1
> >>>>>>>>>
> >>>>>>>>> +#define GFX_ACTLR_PRR          (1 << 5)
> >>>>>>>>> +
> >>>>>>>>>      static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
> >>>>>>>>>      {
> >>>>>>>>>             return container_of(smmu, struct qcom_smmu, smmu);
> >>>>>>>>> @@ -99,6 +101,32 @@ static void qcom_adreno_smmu_resume_translation(const void *cookie, bool termina
> >>>>>>>>>             arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
> >>>>>>>>>      }
> >>>>>>>>>
> >>>>>>>>> +static void qcom_adreno_smmu_set_prr_bit(const void *cookie, bool set)
> >>>>>>>>> +{
> >>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>>>>>> +       struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
> >>>>>>>>> +       u32 reg = 0;
> >>>>>>>>> +
> >>>>>>>>> +       reg =  arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR);
> >>>>>>>>> +       reg &= ~GFX_ACTLR_PRR;
> >>>>>>>>> +       if (set)
> >>>>>>>>> +               reg |= FIELD_PREP(GFX_ACTLR_PRR, 1);
> >>>>>>>>> +       arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_ACTLR, reg);
> >>>>>>>>> +}
> >>>>>>>>> +
> >>>>>>>>> +static void qcom_adreno_smmu_set_prr_addr(const void *cookie, phys_addr_t page_addr)
> >>>>>>>>> +{
> >>>>>>>>> +       struct arm_smmu_domain *smmu_domain = (void *)cookie;
> >>>>>>>>> +       struct arm_smmu_device *smmu = smmu_domain->smmu;
> >>>>>>>>> +
> >>>>>>>>> +       writel_relaxed(lower_32_bits(page_addr),
> >>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_LADDR);
> >>>>>>>>> +
> >>>>>>>>> +       writel_relaxed(upper_32_bits(page_addr),
> >>>>>>>>> +                               smmu->base + ARM_SMMU_GFX_PRR_CFG_UADDR);
> >>>>>>>>> +}
> >>>>>>>>> +
> >>>>>>>>>      #define QCOM_ADRENO_SMMU_GPU_SID 0
> >>>>>>>>>
> >>>>>>>>>      static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
> >>>>>>>>> @@ -210,6 +238,7 @@ static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
> >>>>>>>>>      static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>>>>>                     struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
> >>>>>>>>>      {
> >>>>>>>>> +       const struct device_node *np = smmu_domain->smmu->dev->of_node;
> >>>>>>>>>             struct adreno_smmu_priv *priv;
> >>>>>>>>>
> >>>>>>>>>             smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
> >>>>>>>>> @@ -239,6 +268,14 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> >>>>>>>>>             priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
> >>>>>>>>>             priv->set_stall = qcom_adreno_smmu_set_stall;
> >>>>>>>>>             priv->resume_translation = qcom_adreno_smmu_resume_translation;
> >>>>>>>>> +       priv->set_prr_bit = NULL;
> >>>>>>>>> +       priv->set_prr_addr = NULL;
> >>>>>>>>> +
> >>>>>>>>> +       if (of_device_is_compatible(np, "qcom,smmu-500") &&
> >>>>>>>>> +                       of_device_is_compatible(np, "qcom,adreno-smmu")) {
> >>>>>>>>
> >>>>>>>> fwiw, it seems like PRR actually works on sc7180, which is _not_
> >>>>>>>> mmu-500, so I guess the support actually goes back further.
> >>>>>>>>
> >>>>>>>> I'm curious if we can just rely on this being supported by any hw that
> >>>>>>>> has a6xx or newer?
> >>>>>>>
> >>>>>>>
> >>>>>>> Also, unrelated, but we can't assume the smmu is powered when drm
> >>>>>>> driver calls set_prr_bit/addr, could you add in runpm get/put around
> >>>>>>> the register access?
> >>>>>>>
> >>>>>>
> >>>>>> I see, thanks for this observation.
> >>>>>> It's surely a possible case, when they access these registers
> >>>>>> SMMU state is off.
> >>>>>> I will add the suggested runpm ops around the register access.
> >>>>>>
> >>>>>>> Otherwise Conner and I have vk sparse residency working now
> >>>>>>>
> >>>>>>
> >>>>>> Sorry, could not get this. Did you mean that vk sparse residency
> >>>>>> is working now using this patch?
> >>>>>
> >>>>> Yes, vk sparse residency is working with this patch (plus addition of
> >>>>> runpm get/put, plus drm/msm patches plus turnip patches) ;-)
> >>>>>
> >>>>
> >>>> Thanks for testing the sparse residency feature with our patch Rob,
> >>>> I have an additional query regarding the adreno private interfaces.
> >>>> Specifically, I was referring to other interfaces such as
> >>>> qcom_adreno_smmu_get_fault_info [1]. It appears that we do not have a
> >>>> runpm get/put around the register access in this context.
> >>>
> >>> get_fault_info() is called from the iommu fault handler callback, so
> >>> from the fault irq handler, which is why it didn't need the runpm
> >>> get/put.  Maybe it is bad to make this assumption about usage, but
> >>> then again adreno_smmu_priv isn't really a general purpose interface.
> >>>
> >>
> >> Ah okay, got it.
> >> I was just going through all the adreno_smmmu_priv interfaces just
> >> to get a better understanding of it's interaction with smmu and it seems
> >> like apart from PRR and get_fault_info(), set_ttbr0_cfg(),
> >> resume_translation() is also having reg access but not voting.
> >> Should we put runpm_put/get here as well or these can be excluded.
> >> <Just curious about the thought process behind this, is it because of
> >> some sequence when to put a vote, like reg access in middle of smmu
> >> power cycle and not in beginning or end.>
> >
> > I think we just haven't needed it, or noticed that we needed it,
> > outside of setting up prr.
> >
> > As I mentioned, get_fault_info() is called from the fault irq, so we
> > know the smmu is already powered.
> >
>
> okay got it, that makes sense.
>
> > As far as set_ttbr0_cfg(), it probably works just because
> > arm_smmu_write_context_bank() ends up getting called again in the
> > resume path, so if the smmu is suspended when set_ttbr0_cfg() is
> > called, the writes just get ignored.  But the updated cfg is
> > re-applied to the hw when it is resumed.  Probably the same situation
> > with resume_translation().. ie. if the smmu is suspended there are no
> > translations to resume.
> >
> > Maybe it would be more correct in set_ttbr0_cfg() and
> > resume_translation() to do a pm_runtime_get_if_in_use() and skip the
> > hw writes if the smmu is suspended.
> >
> >>
> >>>> Could you please clarify whether we need an SMMU vote around register
> >>>> access in the case of PRR? IMO, should the users of this callback ensure
> >>>> they put a vote before accessing the cb?
> >>>
> >>> How can drm vote for the smmu device?  I guess it could power up
> >>> itself and rely on device-link.. but that is pretty overkill to power
> >>> up the entire gpu in this case.  I think it is best for the vote to
> >>> happen in the PRR callbacks.
> >>>
> >>
> >> Okay I see, GPU can only power itself up through <gpu_state_get I
> >> assume> but won't have exclusive access to power on the smmu only.
> >>
> >> Incase we go forward to put vote in PRR callback for SMMU, I was
> >> planning that we can refactor existing arm_smmu_rpm_put/get() from
> >> arm_smmu.c to it's header file so that the same can be used in
> >> arm_smmu_qcom.c ? What's your thoughts on this?
> >
> > I had briefly thought of doing the same.  But the main reason for
> > those helpers is common arm-smmu code that is used on non-qcom
> > platforms where runpm is not enabled.  In arm-smmu-qcom.c, we know
> > that runpm is enabled, so we could just use  return
> > pm_runtime_resume_and_get()/pm_runtime_put_autosuspend() directly.
> >
>
> Ohkay I see, we then do not need pm_runtime_enabled() check for qcom
> platforms before putting the vote.
> I am currently modifying this patch only to directly add
> pm_runtime_resume_and_get()/pm_runtime_put_autosuspend()
> around the register access of PRR related adreno private interfaces.
> I will send this updated patch as part of v18 shortly.
>
> Additionally, we can evaluate the use of pm_runtime operations for
> set_ttbr0_cfg() and resume_translation() in a separate series ?

Yup, sounds good

BR,
-R