* [PATCH v5 1/9] iommu/arm-smmu-v3: Handle ARM erratum for CONT under invalidation with SVA
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-07 14:21 ` Mostafa Saleh
2026-09-01 17:49 ` [PATCH v5 2/9] iommu/arm-smmu-v3: Pass the parameters for the invalidation in a struct Jason Gunthorpe
` (7 subsequent siblings)
8 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
The erratum (MMU-700: #3777127, S3: #3673557) deals with under
invalidation of a CONT PTE grouping in the SMMU. The recommended work
around is to use a Range Invalidate (RIL) that spans the entire CONT. The
only user of CONT in the kernel right now is through SVA sharing a CPU
page table that contains a CONT created by the mm.
Previously it was thought that this errata was dealt with because the
driver always uses RIL. However, there is a subtle detail in the errata
that the RIL range must fully enclose the entire CONT for it to work.
It seems that two sequential RILs, with a split point falling inside a
CONT grouping, will not prevent the errata.
The SMMU's RIL generation algorithm does not produce a single RIL for a
single SVA invalidation request, nor does the mm carefully align the SVA
invalidation ranges to accommodate the RIL splitting.
Thus, when processing a SVA invalidation, the RIL splitting routine can
generate a RIL that is split in the middle of the CONT and risk under
invalidation from this errata. This condition could be triggered by a
malicious userspace manipulating the TLB gathers via mmap/mprotect/munmap.
Update the errata list to the include the S3 variation, detect the IOMMUs
that have it, and then have SVA invalidations use a simplified version of
the over invalidation algorithm from the tlbi rework series. This ensures
that a single RIL is issued for a single MMU notifier callback and now the
RIL is guarenteed to cover any posible CONT.
Future work to add CONT to iommu_domain page tables should either use this
one-invalidate/one-RIL algorithm or disable CONT support in the
iommu_domain.
Cc: stable@vger.kernel.org
Fixes: 3f1ce8e85ee0 ("iommu/arm-smmu-v3: Share process page tables")
Cc: Vijayanand Jitta <vijayanand.jitta@oss.qualcomm.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
Documentation/arch/arm64/silicon-errata.rst | 3 +-
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 7 ++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 87 +++++++++++++++----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +
4 files changed, 81 insertions(+), 18 deletions(-)
diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
index ac3248b9f2f3bb..68018bf75b7910 100644
--- a/Documentation/arch/arm64/silicon-errata.rst
+++ b/Documentation/arch/arm64/silicon-errata.rst
@@ -271,7 +271,8 @@ stable kernels.
+----------------+-----------------+-----------------+-----------------------------+
| ARM | MMU L1 | #3878312 | N/A |
+----------------+-----------------+-----------------+-----------------------------+
-| ARM | MMU S3 | #3995052 | N/A |
+| ARM | MMU S3 | #3995052, | N/A |
+| | | #3673557 | |
+----------------+-----------------+-----------------+-----------------------------+
| ARM | GIC-700 | #2941627 | ARM64_ERRATUM_2941627 |
+----------------+-----------------+-----------------+-----------------------------+
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 0a429c64fbf3e7..a0c9078646fb55 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -215,6 +215,13 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
if (system_supports_haft())
feat_mask |= ARM_SMMU_FEAT_HAFT;
+ /*
+ * The workaround for ARM_SMMU_OPT_FULL_CONT_RIL requires range
+ * invalidation support.
+ */
+ if (smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)
+ feat_mask |= ARM_SMMU_FEAT_RANGE_INV;
+
if ((smmu->features & feat_mask) != feat_mask)
return false;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 5732f3ba0122d6..d6896e25b6632a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2538,6 +2538,36 @@ static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
}
}
+/*
+ * Generate a RIL for ARM_SMMU_OPT_FULL_CONT_RIL by ensuring the entire SVA
+ * requested range is covered with a single RIL command. The scale is adjusted
+ * so that the RIL may extend past the end of the requested range. This ensures
+ * that any CONT the MM is invalidating is covered by a single RIL. TTL and LEAF
+ * are always 0 because this is only used by SVA.
+ */
+static bool arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
+ struct arm_smmu_cmdq_batch *cmds,
+ struct arm_smmu_cmd *cmd,
+ unsigned long iova, size_t size,
+ u8 tgsz_lg2)
+{
+ u64 cur_tg = iova >> tgsz_lg2;
+ u64 num_tg = ((iova + size - 1) >> tgsz_lg2) - cur_tg + 1;
+ unsigned int scale = fls64((num_tg - 1) / 32);
+
+ if (scale > 31)
+ return false;
+
+ cmd->data[0] |=
+ FIELD_PREP(CMDQ_TLBI_0_NUM,
+ DIV_ROUND_UP_ULL(num_tg, 1ULL << scale) - 1) |
+ FIELD_PREP(CMDQ_TLBI_0_SCALE, scale);
+ cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_TG, (tgsz_lg2 - 10) / 2) |
+ (cur_tg << tgsz_lg2);
+ arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
+ return true;
+}
+
static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t size,
size_t granule)
{
@@ -2565,21 +2595,30 @@ static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t size,
static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
struct arm_smmu_cmdq_batch *cmds,
struct arm_smmu_cmd *cmd,
- bool leaf,
+ bool single_ril, bool leaf,
unsigned long iova, size_t size,
unsigned int granule)
{
- if (arm_smmu_inv_size_too_big(inv->smmu, size, granule)) {
- struct arm_smmu_cmd nsize_cmd = *cmd;
+ struct arm_smmu_cmd nsize_cmd;
- u64p_replace_bits(&nsize_cmd.data[0], inv->nsize_opcode,
- CMDQ_0_OP);
- arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, &nsize_cmd);
+ if (arm_smmu_inv_size_too_big(inv->smmu, size, granule))
+ goto full_inv;
+
+ if (single_ril && size > granule) {
+ if (!arm_smmu_cmdq_batch_add_ril(inv->smmu, cmds, cmd, iova,
+ size, inv->pgsize))
+ goto full_inv;
return;
}
- arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, leaf,
- iova, size, granule, inv->pgsize);
+ arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, leaf, iova, size,
+ granule, inv->pgsize);
+ return;
+
+full_inv:
+ nsize_cmd = *cmd;
+ u64p_replace_bits(&nsize_cmd.data[0], inv->nsize_opcode, CMDQ_0_OP);
+ arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, &nsize_cmd);
}
static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
@@ -2600,7 +2639,8 @@ static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,
unsigned long iova, size_t size,
- unsigned int granule, bool leaf)
+ unsigned int granule, bool single_ril,
+ bool leaf)
{
struct arm_smmu_cmdq_batch cmds = {};
struct arm_smmu_inv *cur;
@@ -2630,14 +2670,14 @@ static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,
case INV_TYPE_S1_ASID:
cmd = arm_smmu_make_cmd_tlbi(cur->size_opcode,
cur->id, 0);
- arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, leaf,
- iova, size, granule);
+ arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, single_ril,
+ leaf, iova, size, granule);
break;
case INV_TYPE_S2_VMID:
cmd = arm_smmu_make_cmd_tlbi(cur->size_opcode,
0, cur->id);
- arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, leaf,
- iova, size, granule);
+ arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, single_ril,
+ leaf, iova, size, granule);
break;
case INV_TYPE_S2_VMID_S1_CLEAR:
/* CMDQ_OP_TLBI_S12_VMALL already flushed S1 entries */
@@ -2684,6 +2724,9 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
unsigned int granule, bool leaf)
{
struct arm_smmu_invs *invs;
+ bool single_ril =
+ smmu_domain->stage == ARM_SMMU_DOMAIN_SVA &&
+ (smmu_domain->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL);
/*
* An invalidation request must follow some IOPTE change and then load
@@ -2723,10 +2766,12 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
unsigned long flags;
read_lock_irqsave(&invs->rwlock, flags);
- __arm_smmu_domain_inv_range(invs, iova, size, granule, leaf);
+ __arm_smmu_domain_inv_range(invs, iova, size, granule,
+ single_ril, leaf);
read_unlock_irqrestore(&invs->rwlock, flags);
} else {
- __arm_smmu_domain_inv_range(invs, iova, size, granule, leaf);
+ __arm_smmu_domain_inv_range(invs, iova, size, granule,
+ single_ril, leaf);
}
rcu_read_unlock();
@@ -5009,12 +5054,20 @@ static void arm_smmu_device_iidr_probe(struct arm_smmu_device *smmu)
/* Arm errata 2268618, 2812531 */
smmu->features &= ~ARM_SMMU_FEAT_NESTING;
}
+ /* Arm errata 3777127 */
+ smmu->options |= ARM_SMMU_OPT_FULL_CONT_RIL;
break;
case IIDR_PRODUCTID_ARM_MMU_L1:
- case IIDR_PRODUCTID_ARM_MMU_S3:
- /* Arm errata 3878312/3995052 */
+ /* Arm errata 3878312 */
smmu->features &= ~ARM_SMMU_FEAT_BTM;
break;
+ case IIDR_PRODUCTID_ARM_MMU_S3:
+ /* Arm errata 3995052 */
+ smmu->features &= ~ARM_SMMU_FEAT_BTM;
+ /* Arm errata 3673557 */
+ if (variant < 1 || (variant == 1 && revision < 1))
+ smmu->options |= ARM_SMMU_OPT_FULL_CONT_RIL;
+ break;
}
break;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 50f8321e979cef..065d76eb148119 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -934,6 +934,8 @@ struct arm_smmu_device {
#define ARM_SMMU_OPT_MSIPOLL (1 << 2)
#define ARM_SMMU_OPT_CMDQ_FORCE_SYNC (1 << 3)
#define ARM_SMMU_OPT_TEGRA241_CMDQV (1 << 4)
+/* RANGE_INV is mandatory and one RIL must fully span an invalidated CONT */
+#define ARM_SMMU_OPT_FULL_CONT_RIL (1 << 5)
u32 options;
struct arm_smmu_cmdq cmdq;
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v5 1/9] iommu/arm-smmu-v3: Handle ARM erratum for CONT under invalidation with SVA
2026-09-01 17:49 ` [PATCH v5 1/9] iommu/arm-smmu-v3: Handle ARM erratum for CONT under invalidation with SVA Jason Gunthorpe
@ 2026-09-07 14:21 ` Mostafa Saleh
2026-09-07 15:11 ` Jason Gunthorpe
0 siblings, 1 reply; 25+ messages in thread
From: Mostafa Saleh @ 2026-09-07 14:21 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Tue, Sep 01, 2026 at 02:49:50PM -0300, Jason Gunthorpe wrote:
> The erratum (MMU-700: #3777127, S3: #3673557) deals with under
> invalidation of a CONT PTE grouping in the SMMU. The recommended work
> around is to use a Range Invalidate (RIL) that spans the entire CONT. The
> only user of CONT in the kernel right now is through SVA sharing a CPU
> page table that contains a CONT created by the mm.
>
> Previously it was thought that this errata was dealt with because the
> driver always uses RIL. However, there is a subtle detail in the errata
> that the RIL range must fully enclose the entire CONT for it to work.
>
> It seems that two sequential RILs, with a split point falling inside a
> CONT grouping, will not prevent the errata.
>
> The SMMU's RIL generation algorithm does not produce a single RIL for a
> single SVA invalidation request, nor does the mm carefully align the SVA
> invalidation ranges to accommodate the RIL splitting.
>
> Thus, when processing a SVA invalidation, the RIL splitting routine can
> generate a RIL that is split in the middle of the CONT and risk under
> invalidation from this errata. This condition could be triggered by a
> malicious userspace manipulating the TLB gathers via mmap/mprotect/munmap.
>
> Update the errata list to the include the S3 variation, detect the IOMMUs
> that have it, and then have SVA invalidations use a simplified version of
> the over invalidation algorithm from the tlbi rework series. This ensures
> that a single RIL is issued for a single MMU notifier callback and now the
> RIL is guarenteed to cover any posible CONT.
>
> Future work to add CONT to iommu_domain page tables should either use this
> one-invalidate/one-RIL algorithm or disable CONT support in the
> iommu_domain.
>
> Cc: stable@vger.kernel.org
> Fixes: 3f1ce8e85ee0 ("iommu/arm-smmu-v3: Share process page tables")
> Cc: Vijayanand Jitta <vijayanand.jitta@oss.qualcomm.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> Documentation/arch/arm64/silicon-errata.rst | 3 +-
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 7 ++
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 87 +++++++++++++++----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 +
> 4 files changed, 81 insertions(+), 18 deletions(-)
>
> diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst
> index ac3248b9f2f3bb..68018bf75b7910 100644
> --- a/Documentation/arch/arm64/silicon-errata.rst
> +++ b/Documentation/arch/arm64/silicon-errata.rst
> @@ -271,7 +271,8 @@ stable kernels.
> +----------------+-----------------+-----------------+-----------------------------+
> | ARM | MMU L1 | #3878312 | N/A |
> +----------------+-----------------+-----------------+-----------------------------+
> -| ARM | MMU S3 | #3995052 | N/A |
> +| ARM | MMU S3 | #3995052, | N/A |
> +| | | #3673557 | |
> +----------------+-----------------+-----------------+-----------------------------+
> | ARM | GIC-700 | #2941627 | ARM64_ERRATUM_2941627 |
> +----------------+-----------------+-----------------+-----------------------------+
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 0a429c64fbf3e7..a0c9078646fb55 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -215,6 +215,13 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
> if (system_supports_haft())
> feat_mask |= ARM_SMMU_FEAT_HAFT;
>
> + /*
> + * The workaround for ARM_SMMU_OPT_FULL_CONT_RIL requires range
> + * invalidation support.
> + */
> + if (smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)
> + feat_mask |= ARM_SMMU_FEAT_RANGE_INV;
> +
> if ((smmu->features & feat_mask) != feat_mask)
> return false;
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 5732f3ba0122d6..d6896e25b6632a 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2538,6 +2538,36 @@ static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
> }
> }
>
> +/*
> + * Generate a RIL for ARM_SMMU_OPT_FULL_CONT_RIL by ensuring the entire SVA
> + * requested range is covered with a single RIL command. The scale is adjusted
> + * so that the RIL may extend past the end of the requested range. This ensures
> + * that any CONT the MM is invalidating is covered by a single RIL. TTL and LEAF
> + * are always 0 because this is only used by SVA.
> + */
> +static bool arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> + struct arm_smmu_cmdq_batch *cmds,
> + struct arm_smmu_cmd *cmd,
> + unsigned long iova, size_t size,
> + u8 tgsz_lg2)
> +{
> + u64 cur_tg = iova >> tgsz_lg2;
Nit: tg suffix is a bit confusing, I guess pfn is more accurate, but
no strong opinion.
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
> + u64 num_tg = ((iova + size - 1) >> tgsz_lg2) - cur_tg + 1;
> + unsigned int scale = fls64((num_tg - 1) / 32);
> +
> + if (scale > 31)
> + return false;
> +
> + cmd->data[0] |=
> + FIELD_PREP(CMDQ_TLBI_0_NUM,
> + DIV_ROUND_UP_ULL(num_tg, 1ULL << scale) - 1) |
> + FIELD_PREP(CMDQ_TLBI_0_SCALE, scale);
> + cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_TG, (tgsz_lg2 - 10) / 2) |
> + (cur_tg << tgsz_lg2);
> + arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
> + return true;
> +}
> +
> static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t size,
> size_t granule)
> {
> @@ -2565,21 +2595,30 @@ static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t size,
> static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
> struct arm_smmu_cmdq_batch *cmds,
> struct arm_smmu_cmd *cmd,
> - bool leaf,
> + bool single_ril, bool leaf,
> unsigned long iova, size_t size,
> unsigned int granule)
> {
> - if (arm_smmu_inv_size_too_big(inv->smmu, size, granule)) {
> - struct arm_smmu_cmd nsize_cmd = *cmd;
> + struct arm_smmu_cmd nsize_cmd;
>
> - u64p_replace_bits(&nsize_cmd.data[0], inv->nsize_opcode,
> - CMDQ_0_OP);
> - arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, &nsize_cmd);
> + if (arm_smmu_inv_size_too_big(inv->smmu, size, granule))
> + goto full_inv;
> +
> + if (single_ril && size > granule) {
> + if (!arm_smmu_cmdq_batch_add_ril(inv->smmu, cmds, cmd, iova,
> + size, inv->pgsize))
> + goto full_inv;
> return;
> }
>
> - arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, leaf,
> - iova, size, granule, inv->pgsize);
> + arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, leaf, iova, size,
> + granule, inv->pgsize);
> + return;
> +
> +full_inv:
> + nsize_cmd = *cmd;
> + u64p_replace_bits(&nsize_cmd.data[0], inv->nsize_opcode, CMDQ_0_OP);
> + arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, &nsize_cmd);
> }
>
> static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
> @@ -2600,7 +2639,8 @@ static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
>
> static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,
> unsigned long iova, size_t size,
> - unsigned int granule, bool leaf)
> + unsigned int granule, bool single_ril,
> + bool leaf)
> {
> struct arm_smmu_cmdq_batch cmds = {};
> struct arm_smmu_inv *cur;
> @@ -2630,14 +2670,14 @@ static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,
> case INV_TYPE_S1_ASID:
> cmd = arm_smmu_make_cmd_tlbi(cur->size_opcode,
> cur->id, 0);
> - arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, leaf,
> - iova, size, granule);
> + arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, single_ril,
> + leaf, iova, size, granule);
> break;
> case INV_TYPE_S2_VMID:
> cmd = arm_smmu_make_cmd_tlbi(cur->size_opcode,
> 0, cur->id);
> - arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, leaf,
> - iova, size, granule);
> + arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, single_ril,
> + leaf, iova, size, granule);
> break;
> case INV_TYPE_S2_VMID_S1_CLEAR:
> /* CMDQ_OP_TLBI_S12_VMALL already flushed S1 entries */
> @@ -2684,6 +2724,9 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
> unsigned int granule, bool leaf)
> {
> struct arm_smmu_invs *invs;
> + bool single_ril =
> + smmu_domain->stage == ARM_SMMU_DOMAIN_SVA &&
> + (smmu_domain->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL);
>
> /*
> * An invalidation request must follow some IOPTE change and then load
> @@ -2723,10 +2766,12 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
> unsigned long flags;
>
> read_lock_irqsave(&invs->rwlock, flags);
> - __arm_smmu_domain_inv_range(invs, iova, size, granule, leaf);
> + __arm_smmu_domain_inv_range(invs, iova, size, granule,
> + single_ril, leaf);
> read_unlock_irqrestore(&invs->rwlock, flags);
> } else {
> - __arm_smmu_domain_inv_range(invs, iova, size, granule, leaf);
> + __arm_smmu_domain_inv_range(invs, iova, size, granule,
> + single_ril, leaf);
> }
>
> rcu_read_unlock();
> @@ -5009,12 +5054,20 @@ static void arm_smmu_device_iidr_probe(struct arm_smmu_device *smmu)
> /* Arm errata 2268618, 2812531 */
> smmu->features &= ~ARM_SMMU_FEAT_NESTING;
> }
> + /* Arm errata 3777127 */
> + smmu->options |= ARM_SMMU_OPT_FULL_CONT_RIL;
> break;
> case IIDR_PRODUCTID_ARM_MMU_L1:
> - case IIDR_PRODUCTID_ARM_MMU_S3:
> - /* Arm errata 3878312/3995052 */
> + /* Arm errata 3878312 */
> smmu->features &= ~ARM_SMMU_FEAT_BTM;
> break;
> + case IIDR_PRODUCTID_ARM_MMU_S3:
> + /* Arm errata 3995052 */
> + smmu->features &= ~ARM_SMMU_FEAT_BTM;
> + /* Arm errata 3673557 */
> + if (variant < 1 || (variant == 1 && revision < 1))
> + smmu->options |= ARM_SMMU_OPT_FULL_CONT_RIL;
> + break;
> }
> break;
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 50f8321e979cef..065d76eb148119 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -934,6 +934,8 @@ struct arm_smmu_device {
> #define ARM_SMMU_OPT_MSIPOLL (1 << 2)
> #define ARM_SMMU_OPT_CMDQ_FORCE_SYNC (1 << 3)
> #define ARM_SMMU_OPT_TEGRA241_CMDQV (1 << 4)
> +/* RANGE_INV is mandatory and one RIL must fully span an invalidated CONT */
> +#define ARM_SMMU_OPT_FULL_CONT_RIL (1 << 5)
> u32 options;
>
> struct arm_smmu_cmdq cmdq;
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 1/9] iommu/arm-smmu-v3: Handle ARM erratum for CONT under invalidation with SVA
2026-09-07 14:21 ` Mostafa Saleh
@ 2026-09-07 15:11 ` Jason Gunthorpe
2026-09-07 15:17 ` Mostafa Saleh
0 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-07 15:11 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 02:21:44PM +0000, Mostafa Saleh wrote:
> > +static bool arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> > + struct arm_smmu_cmdq_batch *cmds,
> > + struct arm_smmu_cmd *cmd,
> > + unsigned long iova, size_t size,
> > + u8 tgsz_lg2)
> > +{
> > + u64 cur_tg = iova >> tgsz_lg2;
>
> Nit: tg suffix is a bit confusing, I guess pfn is more accurate, but
> no strong opinion.
I don't want to use pfn in any of this code because convention has PFN
as PAGE_SIZE, while this is a "translation granule", ie the base
granule size of the iommu domain. Which are not always the same thing..
Jason
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 1/9] iommu/arm-smmu-v3: Handle ARM erratum for CONT under invalidation with SVA
2026-09-07 15:11 ` Jason Gunthorpe
@ 2026-09-07 15:17 ` Mostafa Saleh
2026-09-07 15:18 ` Jason Gunthorpe
0 siblings, 1 reply; 25+ messages in thread
From: Mostafa Saleh @ 2026-09-07 15:17 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 12:11:45PM -0300, Jason Gunthorpe wrote:
> On Mon, Sep 07, 2026 at 02:21:44PM +0000, Mostafa Saleh wrote:
> > > +static bool arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> > > + struct arm_smmu_cmdq_batch *cmds,
> > > + struct arm_smmu_cmd *cmd,
> > > + unsigned long iova, size_t size,
> > > + u8 tgsz_lg2)
> > > +{
> > > + u64 cur_tg = iova >> tgsz_lg2;
> >
> > Nit: tg suffix is a bit confusing, I guess pfn is more accurate, but
> > no strong opinion.
>
> I don't want to use pfn in any of this code because convention has PFN
> as PAGE_SIZE, while this is a "translation granule", ie the base
> granule size of the iommu domain. Which are not always the same thing..
But this is an address not a granule size, not sure what that could
be called then :)
Thanks,
Mostafa
>
> Jason
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 1/9] iommu/arm-smmu-v3: Handle ARM erratum for CONT under invalidation with SVA
2026-09-07 15:17 ` Mostafa Saleh
@ 2026-09-07 15:18 ` Jason Gunthorpe
0 siblings, 0 replies; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-07 15:18 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 03:17:04PM +0000, Mostafa Saleh wrote:
> On Mon, Sep 07, 2026 at 12:11:45PM -0300, Jason Gunthorpe wrote:
> > On Mon, Sep 07, 2026 at 02:21:44PM +0000, Mostafa Saleh wrote:
> > > > +static bool arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> > > > + struct arm_smmu_cmdq_batch *cmds,
> > > > + struct arm_smmu_cmd *cmd,
> > > > + unsigned long iova, size_t size,
> > > > + u8 tgsz_lg2)
> > > > +{
> > > > + u64 cur_tg = iova >> tgsz_lg2;
> > >
> > > Nit: tg suffix is a bit confusing, I guess pfn is more accurate, but
> > > no strong opinion.
> >
> > I don't want to use pfn in any of this code because convention has PFN
> > as PAGE_SIZE, while this is a "translation granule", ie the base
> > granule size of the iommu domain. Which are not always the same thing..
>
> But this is an address not a granule size, not sure what that could
> be called then :)
It is the iova in granule sized units. PFN is a mm sort of thing that
refers to PAGE_SIZE units.
Jison
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v5 2/9] iommu/arm-smmu-v3: Pass the parameters for the invalidation in a struct
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
2026-09-01 17:49 ` [PATCH v5 1/9] iommu/arm-smmu-v3: Handle ARM erratum for CONT under invalidation with SVA Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-01 17:49 ` [PATCH v5 3/9] iommu/arm-smmu-v3: Move pgsize out of arm_smmu_inv Jason Gunthorpe
` (6 subsequent siblings)
8 siblings, 0 replies; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
These parameters go to a lot of different functions and the next
patches will add more. Put them into a struct to keep things tidy.
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 90 ++++++++++-----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 9 +++
2 files changed, 54 insertions(+), 45 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index d6896e25b6632a..03723bb00de412 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2363,8 +2363,8 @@ static irqreturn_t arm_smmu_combined_irq_handler(int irq, void *dev)
return IRQ_WAKE_THREAD;
}
-static struct arm_smmu_cmd
-arm_smmu_atc_inv_to_cmd(u32 sid, int ssid, unsigned long iova, size_t size)
+static struct arm_smmu_cmd arm_smmu_atc_inv_to_cmd(u32 sid, int ssid,
+ struct arm_smmu_tlbi *tlbi)
{
size_t log2_span;
size_t span_mask;
@@ -2386,8 +2386,8 @@ arm_smmu_atc_inv_to_cmd(u32 sid, int ssid, unsigned long iova, size_t size)
* This has the unpleasant side-effect of invalidating all PASID-tagged
* ATC entries within the address range.
*/
- page_start = iova >> inval_grain_shift;
- page_end = (iova + size - 1) >> inval_grain_shift;
+ page_start = tlbi->iova >> inval_grain_shift;
+ page_end = (tlbi->iova + tlbi->size - 1) >> inval_grain_shift;
/*
* In an ATS Invalidate Request, the address must be aligned on the
@@ -2462,20 +2462,23 @@ static void arm_smmu_tlb_inv_context(void *cookie)
static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
struct arm_smmu_cmdq_batch *cmds,
- struct arm_smmu_cmd *cmd, bool leaf,
- unsigned long iova, size_t size,
- size_t granule, size_t pgsize)
+ struct arm_smmu_cmd *cmd,
+ struct arm_smmu_tlbi *tlbi,
+ size_t pgsize)
{
- unsigned long end = iova + size, num_pages = 0, tg = pgsize;
+ size_t inv_range = tlbi->iopte_size;
+ unsigned long iova = tlbi->iova;
+ unsigned long end = iova + tlbi->size;
+ unsigned long num_pages = 0;
+ unsigned int tg = pgsize;
u64 orig_data0 = cmd->data[0];
- size_t inv_range = granule;
u8 ttl = 0, tg_enc = 0;
- if (WARN_ON_ONCE(!size))
+ if (WARN_ON_ONCE(!tlbi->size))
return;
if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
- num_pages = size >> tg;
+ num_pages = tlbi->size >> tg;
/* Convert page size of 12,14,16 (log2) to 1,2,3 */
tg_enc = (tg - 10) / 2;
@@ -2488,8 +2491,8 @@ static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
* want to use a range command, so avoid the SVA corner case
* where both scale and num could be 0 as well.
*/
- if (leaf)
- ttl = 4 - ((ilog2(granule) - 3) / (tg - 3));
+ if (tlbi->leaf_only)
+ ttl = 4 - ((ilog2(tlbi->iopte_size) - 3) / (tg - 3));
else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) == 1)
num_pages++;
}
@@ -2528,7 +2531,7 @@ static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
* command and something would be very broken if iova had them
* set.
*/
- cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, leaf) |
+ cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) |
(iova & ~GENMASK_U64(11, 0));
@@ -2568,13 +2571,13 @@ static bool arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
return true;
}
-static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t size,
- size_t granule)
+static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu,
+ struct arm_smmu_tlbi *tlbi)
{
size_t max_tlbi_ops;
/* 0 size means invalidate all */
- if (!size || size == SIZE_MAX)
+ if (!tlbi->size || tlbi->size == SIZE_MAX)
return true;
if (smmu->features & ARM_SMMU_FEAT_RANGE_INV)
@@ -2587,32 +2590,31 @@ static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t size,
* invalidation feature, where there can be too many per-granule TLBIs,
* resulting in a soft lockup.
*/
- max_tlbi_ops = 1 << (ilog2(granule) - 3);
- return size >= max_tlbi_ops * granule;
+ max_tlbi_ops = 1 << (ilog2(tlbi->iopte_size) - 3);
+ return tlbi->size >= max_tlbi_ops * tlbi->iopte_size;
}
/* Used by non INV_TYPE_ATS* invalidations */
static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
struct arm_smmu_cmdq_batch *cmds,
struct arm_smmu_cmd *cmd,
- bool single_ril, bool leaf,
- unsigned long iova, size_t size,
- unsigned int granule)
+ struct arm_smmu_tlbi *tlbi)
{
struct arm_smmu_cmd nsize_cmd;
- if (arm_smmu_inv_size_too_big(inv->smmu, size, granule))
+ if (arm_smmu_inv_size_too_big(inv->smmu, tlbi))
goto full_inv;
- if (single_ril && size > granule) {
- if (!arm_smmu_cmdq_batch_add_ril(inv->smmu, cmds, cmd, iova,
- size, inv->pgsize))
+ if (tlbi->has_cont && tlbi->size > tlbi->iopte_size &&
+ (inv->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
+ if (!arm_smmu_cmdq_batch_add_ril(inv->smmu, cmds, cmd,
+ tlbi->iova, tlbi->size,
+ inv->pgsize))
goto full_inv;
return;
}
- arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, leaf, iova, size,
- granule, inv->pgsize);
+ arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi, inv->pgsize);
return;
full_inv:
@@ -2637,10 +2639,8 @@ static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
return false;
}
-static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,
- unsigned long iova, size_t size,
- unsigned int granule, bool single_ril,
- bool leaf)
+static void __arm_smmu_domain_inv_range(struct arm_smmu_tlbi *tlbi,
+ struct arm_smmu_invs *invs)
{
struct arm_smmu_cmdq_batch cmds = {};
struct arm_smmu_inv *cur;
@@ -2670,18 +2670,16 @@ static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,
case INV_TYPE_S1_ASID:
cmd = arm_smmu_make_cmd_tlbi(cur->size_opcode,
cur->id, 0);
- arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, single_ril,
- leaf, iova, size, granule);
+ arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, tlbi);
break;
case INV_TYPE_S2_VMID:
cmd = arm_smmu_make_cmd_tlbi(cur->size_opcode,
0, cur->id);
- arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, single_ril,
- leaf, iova, size, granule);
+ arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, tlbi);
break;
case INV_TYPE_S2_VMID_S1_CLEAR:
/* CMDQ_OP_TLBI_S12_VMALL already flushed S1 entries */
- if (arm_smmu_inv_size_too_big(cur->smmu, size, granule))
+ if (arm_smmu_inv_size_too_big(cur->smmu, tlbi))
break;
arm_smmu_cmdq_batch_add_cmd(
smmu, &cmds,
@@ -2692,7 +2690,7 @@ static void __arm_smmu_domain_inv_range(struct arm_smmu_invs *invs,
arm_smmu_cmdq_batch_add_cmd(
smmu, &cmds,
arm_smmu_atc_inv_to_cmd(cur->id, cur->ssid,
- iova, size));
+ tlbi));
break;
case INV_TYPE_ATS_FULL:
arm_smmu_cmdq_batch_add_cmd(
@@ -2723,10 +2721,14 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
unsigned long iova, size_t size,
unsigned int granule, bool leaf)
{
+ struct arm_smmu_tlbi tlbi = {
+ .iova = iova,
+ .size = size,
+ .iopte_size = granule,
+ .has_cont = smmu_domain->stage == ARM_SMMU_DOMAIN_SVA,
+ .leaf_only = leaf,
+ };
struct arm_smmu_invs *invs;
- bool single_ril =
- smmu_domain->stage == ARM_SMMU_DOMAIN_SVA &&
- (smmu_domain->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL);
/*
* An invalidation request must follow some IOPTE change and then load
@@ -2766,12 +2768,10 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
unsigned long flags;
read_lock_irqsave(&invs->rwlock, flags);
- __arm_smmu_domain_inv_range(invs, iova, size, granule,
- single_ril, leaf);
+ __arm_smmu_domain_inv_range(&tlbi, invs);
read_unlock_irqrestore(&invs->rwlock, flags);
} else {
- __arm_smmu_domain_inv_range(invs, iova, size, granule,
- single_ril, leaf);
+ __arm_smmu_domain_inv_range(&tlbi, invs);
}
rcu_read_unlock();
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 065d76eb148119..d3e81056f24afc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -805,6 +805,15 @@ static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs)
return new_invs;
}
+struct arm_smmu_tlbi {
+ unsigned long iova;
+ size_t size;
+ /* page or block size of the leaf iopte */
+ unsigned int iopte_size;
+ bool has_cont;
+ bool leaf_only;
+};
+
struct arm_smmu_evtq {
struct arm_smmu_queue q;
struct iopf_queue *iopf;
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v5 3/9] iommu/arm-smmu-v3: Move pgsize out of arm_smmu_inv
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
2026-09-01 17:49 ` [PATCH v5 1/9] iommu/arm-smmu-v3: Handle ARM erratum for CONT under invalidation with SVA Jason Gunthorpe
2026-09-01 17:49 ` [PATCH v5 2/9] iommu/arm-smmu-v3: Pass the parameters for the invalidation in a struct Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-07 14:22 ` Mostafa Saleh
2026-09-01 17:49 ` [PATCH v5 4/9] iommu/arm-smmu-v3: Optimize range invalidation for latency Jason Gunthorpe
` (5 subsequent siblings)
8 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
pgsize is a constant property of the domain, it is the base translation
granule of the page table (4k, 16k, 64k) in log2.
Store it to the struct arm_smmu_domain based on how the page table was
created.
Pass it around in the tlbi.
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 1 +
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 29 ++++++++-----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 ++++---
3 files changed, 19 insertions(+), 21 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index a0c9078646fb55..7acb70d060ec4e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -345,6 +345,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
* ARM_SMMU_FEAT_RANGE_INV is present
*/
smmu_domain->domain.pgsize_bitmap = PAGE_SIZE;
+ smmu_domain->tgsz_lg2 = PAGE_SHIFT;
smmu_domain->stage = ARM_SMMU_DOMAIN_SVA;
smmu_domain->smmu = smmu;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 03723bb00de412..6413eb4021ce22 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2463,14 +2463,13 @@ static void arm_smmu_tlb_inv_context(void *cookie)
static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
struct arm_smmu_cmdq_batch *cmds,
struct arm_smmu_cmd *cmd,
- struct arm_smmu_tlbi *tlbi,
- size_t pgsize)
+ struct arm_smmu_tlbi *tlbi)
{
size_t inv_range = tlbi->iopte_size;
unsigned long iova = tlbi->iova;
unsigned long end = iova + tlbi->size;
unsigned long num_pages = 0;
- unsigned int tg = pgsize;
+ u8 tg = tlbi->tgsz_lg2;
u64 orig_data0 = cmd->data[0];
u8 ttl = 0, tg_enc = 0;
@@ -2609,12 +2608,12 @@ static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
(inv->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
if (!arm_smmu_cmdq_batch_add_ril(inv->smmu, cmds, cmd,
tlbi->iova, tlbi->size,
- inv->pgsize))
+ tlbi->tgsz_lg2))
goto full_inv;
return;
}
- arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi, inv->pgsize);
+ arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi);
return;
full_inv:
@@ -2722,6 +2721,7 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
unsigned int granule, bool leaf)
{
struct arm_smmu_tlbi tlbi = {
+ .tgsz_lg2 = smmu_domain->tgsz_lg2,
.iova = iova,
.size = size,
.iopte_size = granule,
@@ -2978,6 +2978,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
return -ENOMEM;
smmu_domain->domain.pgsize_bitmap = pgtbl_cfg.pgsize_bitmap;
+ smmu_domain->tgsz_lg2 = __ffs(pgtbl_cfg.pgsize_bitmap);
smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1;
smmu_domain->domain.geometry.force_aperture = true;
if (enable_dirty && smmu_domain->stage == ARM_SMMU_DOMAIN_S1)
@@ -3217,15 +3218,13 @@ static void arm_smmu_disable_iopf(struct arm_smmu_master *master,
static struct arm_smmu_inv *
arm_smmu_master_build_inv(struct arm_smmu_master *master,
- enum arm_smmu_inv_type type, u32 id, ioasid_t ssid,
- size_t pgsize)
+ enum arm_smmu_inv_type type, u32 id, ioasid_t ssid)
{
struct arm_smmu_invs *build_invs = master->build_invs;
struct arm_smmu_inv *cur, inv = {
.smmu = master->smmu,
.type = type,
.id = id,
- .pgsize = pgsize,
};
if (WARN_ON(build_invs->num_invs >= build_invs->max_invs))
@@ -3277,28 +3276,24 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
ioasid_t ssid, struct arm_smmu_domain *smmu_domain)
{
const bool nesting = smmu_domain->nest_parent;
- size_t pgsize = 0, i;
+ size_t i;
iommu_group_mutex_assert(master->dev);
master->build_invs->num_invs = 0;
- /* Range-based invalidation requires the leaf pgsize for calculation */
- if (master->smmu->features & ARM_SMMU_FEAT_RANGE_INV)
- pgsize = __ffs(smmu_domain->domain.pgsize_bitmap);
-
switch (smmu_domain->stage) {
case ARM_SMMU_DOMAIN_SVA:
case ARM_SMMU_DOMAIN_S1:
if (!arm_smmu_master_build_inv(master, INV_TYPE_S1_ASID,
smmu_domain->cd.asid,
- IOMMU_NO_PASID, pgsize))
+ IOMMU_NO_PASID))
return NULL;
break;
case ARM_SMMU_DOMAIN_S2:
if (!arm_smmu_master_build_inv(master, INV_TYPE_S2_VMID,
smmu_domain->s2_cfg.vmid,
- IOMMU_NO_PASID, pgsize))
+ IOMMU_NO_PASID))
return NULL;
break;
default:
@@ -3310,7 +3305,7 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
if (nesting) {
if (!arm_smmu_master_build_inv(
master, INV_TYPE_S2_VMID_S1_CLEAR,
- smmu_domain->s2_cfg.vmid, IOMMU_NO_PASID, 0))
+ smmu_domain->s2_cfg.vmid, IOMMU_NO_PASID))
return NULL;
}
@@ -3321,7 +3316,7 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
*/
if (!arm_smmu_master_build_inv(
master, nesting ? INV_TYPE_ATS_FULL : INV_TYPE_ATS,
- master->streams[i].id, ssid, 0))
+ master->streams[i].id, ssid))
return NULL;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index d3e81056f24afc..3e7f94339ab60e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -736,10 +736,9 @@ struct arm_smmu_inv {
u8 size_opcode;
u8 nsize_opcode;
u32 id; /* ASID or VMID or SID */
- union {
- size_t pgsize; /* ARM_SMMU_FEAT_RANGE_INV */
- u32 ssid; /* INV_TYPE_ATS */
- };
+
+ /* Only used by INV_TYPE_ATS */
+ u32 ssid;
int users; /* users=0 to mark as a trash to be purged */
};
@@ -810,6 +809,8 @@ struct arm_smmu_tlbi {
size_t size;
/* page or block size of the leaf iopte */
unsigned int iopte_size;
+ /* Base Translation Granule of the page table */
+ u8 tgsz_lg2;
bool has_cont;
bool leaf_only;
};
@@ -1060,6 +1061,7 @@ struct arm_smmu_domain {
spinlock_t devices_lock;
bool enforce_cache_coherency : 1;
bool nest_parent : 1;
+ u8 tgsz_lg2;
struct mmu_notifier mmu_notifier;
};
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v5 3/9] iommu/arm-smmu-v3: Move pgsize out of arm_smmu_inv
2026-09-01 17:49 ` [PATCH v5 3/9] iommu/arm-smmu-v3: Move pgsize out of arm_smmu_inv Jason Gunthorpe
@ 2026-09-07 14:22 ` Mostafa Saleh
0 siblings, 0 replies; 25+ messages in thread
From: Mostafa Saleh @ 2026-09-07 14:22 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Tue, Sep 01, 2026 at 02:49:52PM -0300, Jason Gunthorpe wrote:
> pgsize is a constant property of the domain, it is the base translation
> granule of the page table (4k, 16k, 64k) in log2.
>
> Store it to the struct arm_smmu_domain based on how the page table was
> created.
>
> Pass it around in the tlbi.
>
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Thanks,
Mostafa
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 1 +
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 29 ++++++++-----------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 10 ++++---
> 3 files changed, 19 insertions(+), 21 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index a0c9078646fb55..7acb70d060ec4e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -345,6 +345,7 @@ struct iommu_domain *arm_smmu_sva_domain_alloc(struct device *dev,
> * ARM_SMMU_FEAT_RANGE_INV is present
> */
> smmu_domain->domain.pgsize_bitmap = PAGE_SIZE;
> + smmu_domain->tgsz_lg2 = PAGE_SHIFT;
> smmu_domain->stage = ARM_SMMU_DOMAIN_SVA;
> smmu_domain->smmu = smmu;
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 03723bb00de412..6413eb4021ce22 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2463,14 +2463,13 @@ static void arm_smmu_tlb_inv_context(void *cookie)
> static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
> struct arm_smmu_cmdq_batch *cmds,
> struct arm_smmu_cmd *cmd,
> - struct arm_smmu_tlbi *tlbi,
> - size_t pgsize)
> + struct arm_smmu_tlbi *tlbi)
> {
> size_t inv_range = tlbi->iopte_size;
> unsigned long iova = tlbi->iova;
> unsigned long end = iova + tlbi->size;
> unsigned long num_pages = 0;
> - unsigned int tg = pgsize;
> + u8 tg = tlbi->tgsz_lg2;
> u64 orig_data0 = cmd->data[0];
> u8 ttl = 0, tg_enc = 0;
>
> @@ -2609,12 +2608,12 @@ static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
> (inv->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
> if (!arm_smmu_cmdq_batch_add_ril(inv->smmu, cmds, cmd,
> tlbi->iova, tlbi->size,
> - inv->pgsize))
> + tlbi->tgsz_lg2))
> goto full_inv;
> return;
> }
>
> - arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi, inv->pgsize);
> + arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi);
> return;
>
> full_inv:
> @@ -2722,6 +2721,7 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
> unsigned int granule, bool leaf)
> {
> struct arm_smmu_tlbi tlbi = {
> + .tgsz_lg2 = smmu_domain->tgsz_lg2,
> .iova = iova,
> .size = size,
> .iopte_size = granule,
> @@ -2978,6 +2978,7 @@ static int arm_smmu_domain_finalise(struct arm_smmu_domain *smmu_domain,
> return -ENOMEM;
>
> smmu_domain->domain.pgsize_bitmap = pgtbl_cfg.pgsize_bitmap;
> + smmu_domain->tgsz_lg2 = __ffs(pgtbl_cfg.pgsize_bitmap);
> smmu_domain->domain.geometry.aperture_end = (1UL << pgtbl_cfg.ias) - 1;
> smmu_domain->domain.geometry.force_aperture = true;
> if (enable_dirty && smmu_domain->stage == ARM_SMMU_DOMAIN_S1)
> @@ -3217,15 +3218,13 @@ static void arm_smmu_disable_iopf(struct arm_smmu_master *master,
>
> static struct arm_smmu_inv *
> arm_smmu_master_build_inv(struct arm_smmu_master *master,
> - enum arm_smmu_inv_type type, u32 id, ioasid_t ssid,
> - size_t pgsize)
> + enum arm_smmu_inv_type type, u32 id, ioasid_t ssid)
> {
> struct arm_smmu_invs *build_invs = master->build_invs;
> struct arm_smmu_inv *cur, inv = {
> .smmu = master->smmu,
> .type = type,
> .id = id,
> - .pgsize = pgsize,
> };
>
> if (WARN_ON(build_invs->num_invs >= build_invs->max_invs))
> @@ -3277,28 +3276,24 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
> ioasid_t ssid, struct arm_smmu_domain *smmu_domain)
> {
> const bool nesting = smmu_domain->nest_parent;
> - size_t pgsize = 0, i;
> + size_t i;
>
> iommu_group_mutex_assert(master->dev);
>
> master->build_invs->num_invs = 0;
>
> - /* Range-based invalidation requires the leaf pgsize for calculation */
> - if (master->smmu->features & ARM_SMMU_FEAT_RANGE_INV)
> - pgsize = __ffs(smmu_domain->domain.pgsize_bitmap);
> -
> switch (smmu_domain->stage) {
> case ARM_SMMU_DOMAIN_SVA:
> case ARM_SMMU_DOMAIN_S1:
> if (!arm_smmu_master_build_inv(master, INV_TYPE_S1_ASID,
> smmu_domain->cd.asid,
> - IOMMU_NO_PASID, pgsize))
> + IOMMU_NO_PASID))
> return NULL;
> break;
> case ARM_SMMU_DOMAIN_S2:
> if (!arm_smmu_master_build_inv(master, INV_TYPE_S2_VMID,
> smmu_domain->s2_cfg.vmid,
> - IOMMU_NO_PASID, pgsize))
> + IOMMU_NO_PASID))
> return NULL;
> break;
> default:
> @@ -3310,7 +3305,7 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
> if (nesting) {
> if (!arm_smmu_master_build_inv(
> master, INV_TYPE_S2_VMID_S1_CLEAR,
> - smmu_domain->s2_cfg.vmid, IOMMU_NO_PASID, 0))
> + smmu_domain->s2_cfg.vmid, IOMMU_NO_PASID))
> return NULL;
> }
>
> @@ -3321,7 +3316,7 @@ arm_smmu_master_build_invs(struct arm_smmu_master *master, bool ats_enabled,
> */
> if (!arm_smmu_master_build_inv(
> master, nesting ? INV_TYPE_ATS_FULL : INV_TYPE_ATS,
> - master->streams[i].id, ssid, 0))
> + master->streams[i].id, ssid))
> return NULL;
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index d3e81056f24afc..3e7f94339ab60e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -736,10 +736,9 @@ struct arm_smmu_inv {
> u8 size_opcode;
> u8 nsize_opcode;
> u32 id; /* ASID or VMID or SID */
> - union {
> - size_t pgsize; /* ARM_SMMU_FEAT_RANGE_INV */
> - u32 ssid; /* INV_TYPE_ATS */
> - };
> +
> + /* Only used by INV_TYPE_ATS */
> + u32 ssid;
>
> int users; /* users=0 to mark as a trash to be purged */
> };
> @@ -810,6 +809,8 @@ struct arm_smmu_tlbi {
> size_t size;
> /* page or block size of the leaf iopte */
> unsigned int iopte_size;
> + /* Base Translation Granule of the page table */
> + u8 tgsz_lg2;
> bool has_cont;
> bool leaf_only;
> };
> @@ -1060,6 +1061,7 @@ struct arm_smmu_domain {
> spinlock_t devices_lock;
> bool enforce_cache_coherency : 1;
> bool nest_parent : 1;
> + u8 tgsz_lg2;
>
> struct mmu_notifier mmu_notifier;
> };
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v5 4/9] iommu/arm-smmu-v3: Optimize range invalidation for latency
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
` (2 preceding siblings ...)
2026-09-01 17:49 ` [PATCH v5 3/9] iommu/arm-smmu-v3: Move pgsize out of arm_smmu_inv Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-07 14:26 ` Mostafa Saleh
2026-09-01 17:49 ` [PATCH v5 5/9] iommu/arm-smmu-v3: Keep track in the arm_smmu_invs if RIL is used Jason Gunthorpe
` (4 subsequent siblings)
8 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
The server IOMMU drivers focus on invalidation latency by default,
over-invalidating if necessary, to round the invalidation range up to a
single command. I think this represents a trade off for DMA non-FQ and SVA
where stalling the operation is overall worse than re-loading the IOTLB.
For instance AMD and VT-d both round the range up to the largest aligned
power of two and invalidate that. This causes over-invalidation but that
is preferred on real HW over trying to issue a number of smaller
range invalidations.
SMMUv3 on the other hand will try to do up to 512 single invalidations,
otherwise falls back to full invalidation, or it will try to issue an
string of RIL invalidations for every 5 bits of IOVA range to perfectly
cover it.
While the RIL optimization is pretty good we can still do better and
reduce it to only 2 invalidations at maximum using the idea Robin came up
with to overlap two RILs in the middle. This preserves the exact coverage
of the current RIL and still caps the number of RILs at 2.
This works because the RIL can start at any IOVA, so we can place a RIL
forwards from the start and backwards from the end, overlapping in the
middle if the scale isn't precise enough.
In the cases where this produces 2 RILS it continues to produce RILs that
don't overlap, but the exact split point is different than the original
algorithm due to the new calculation method. In cases where the original
algorithm would produce 3 or more RILs this will produce 2 RILS with an
overlap.
The SVA under invalidation errata work around is maintained by "rounding up" to
generate a single RIL that over-covers the entire range.
Since the normal path is now the only one with a loop, split them into two
functions and fold a simplified version of arm_smmu_inv_size_too_big()
directly into the normal flow in a way that directly limits the number of
single invalidation commands generated, again focusing on controlling
latency.
The end result is any gather is converted into either:
- One invalidate all
- One or two range invalidate ops
- At most 512 single invalidation ops
arm_smmu_domain_inv() is modified to always pass in the tgsz because the
new logic relies on it being valid.
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 354 ++++++++++++--------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 +-
2 files changed, 210 insertions(+), 147 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6413eb4021ce22..9b79d7a5389812 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2460,166 +2460,219 @@ static void arm_smmu_tlb_inv_context(void *cookie)
arm_smmu_domain_inv(smmu_domain);
}
-static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
+/*
+ * Check address alignment for TTL hint per SMMUv3 H.a Section 4.4.1.
+ * Address bits below the alignment must be zero, otherwise UNPREDICTABLE.
+ */
+static bool arm_smmu_ttl_addr_aligned(u64 address, unsigned int tg,
+ unsigned int ttl)
+{
+ unsigned int pgsz_lg2 = (tg - 3) * (3 - ttl) + tg;
+
+ return !(address & GENMASK_U64(pgsz_lg2 - 1, 0));
+}
+
+struct arm_smmu_ril_range {
+ u64 start_tg;
+ /* Normal integer, not encoded. 0 means 0.*/
+ u64 num;
+ unsigned int scale;
+};
+
+/*
+ * Initialize the smallest RIL covering num_tg and ending at last_tg.
+ */
+static struct arm_smmu_ril_range arm_smmu_ril_init_end(u64 last_tg, u64 num_tg)
+{
+ struct arm_smmu_ril_range ril = {};
+
+ if (!num_tg)
+ return ril;
+
+ ril.scale = fls64((num_tg - 1) / 32);
+ ril.num = DIV_ROUND_UP_ULL(num_tg, 1ULL << ril.scale);
+ ril.start_tg = last_tg - ((ril.num << ril.scale) - 1);
+ return ril;
+}
+
+static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
+ struct arm_smmu_cmdq_batch *cmds,
+ struct arm_smmu_cmd *ref_cmd,
+ bool leaf_only,
+ const struct arm_smmu_ril_range *ril,
+ u8 ttl, u8 tg_enc)
+{
+ struct arm_smmu_cmd cmd;
+ unsigned int tgsz_lg2 = tg_enc * 2 + 10;
+ u64 iova = ril->start_tg << tgsz_lg2;
+ unsigned int num = ril->num - 1;
+
+ /* 16K granule TTL=1 is reserved (Section 4.4.1) */
+ if (WARN_ON(tgsz_lg2 == 14 && ttl == 1))
+ ttl = 0;
+
+ /* Verify address alignment for the TTL hint */
+ if (ttl && !arm_smmu_ttl_addr_aligned(iova, tgsz_lg2, ttl))
+ ttl = 0;
+
+ /*
+ * SMMUv3 H.a Section 4.4.1: TG!=0, NUM==0, SCALE==0, TTL==0 is Reserved
+ * and causes CERROR_ILL. Single tg uses NUM=0, SCALE=0 with a TTL hint
+ * to target only the exact leaf entry.
+ *
+ * For a single tg invalidation a 0 TTL can come from places like the
+ * SVA path that don't have enough information to get a TTL, or as a
+ * side of effect of the splitting.
+ *
+ * A single-TG invalidation cannot reach this point if it is part of a
+ * CONT group, so it is safe to transform it into a single invalidation.
+ * The ARM_SMMU_OPT_FULL_CONT_RIL errata does not apply.
+ */
+ if (!num && !ril->scale && !ttl)
+ tg_enc = 0;
+
+ cmd.data[0] = ref_cmd->data[0] | FIELD_PREP(CMDQ_TLBI_0_NUM, num) |
+ FIELD_PREP(CMDQ_TLBI_0_SCALE, ril->scale);
+ cmd.data[1] = ref_cmd->data[1] |
+ FIELD_PREP(CMDQ_TLBI_1_LEAF, leaf_only) |
+ FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
+ FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova;
+ arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, &cmd);
+}
+
+/*
+ * Issue up to two range TLBI commands covering [iova, iova+size). Returns true
+ * if successful, false if the range is too large to fit into RIL commands.
+ *
+ * Normally the first RIL is the largest representable span which does not
+ * exceed the requested range. If necessary, the second RIL is the smallest
+ * representable range covering the remainder and is anchored at the end. Any
+ * excess coverage from the second RIL overlaps the first instead of exceeding
+ * the requested range.
+ *
+ * If a SVA is being invalidated and the SMMU has the ARM_SMMU_OPT_FULL_CONT_RIL
+ * errata this produces only a single RIL and overinvalidates to ensure any
+ * potential CONT is covered with a single RIL.
+ */
+static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
struct arm_smmu_cmdq_batch *cmds,
struct arm_smmu_cmd *cmd,
struct arm_smmu_tlbi *tlbi)
{
- size_t inv_range = tlbi->iopte_size;
- unsigned long iova = tlbi->iova;
- unsigned long end = iova + tlbi->size;
- unsigned long num_pages = 0;
- u8 tg = tlbi->tgsz_lg2;
- u64 orig_data0 = cmd->data[0];
- u8 ttl = 0, tg_enc = 0;
+ u8 tgsz_lg2 = tlbi->tgsz_lg2;
+ struct arm_smmu_ril_range first = { .start_tg = tlbi->iova >>
+ tgsz_lg2 };
+ u64 last_tg = (tlbi->iova + tlbi->size - 1) >> tgsz_lg2;
+ u64 num_tg = last_tg - first.start_tg + 1;
+ u8 tg_enc = (tgsz_lg2 - 10) / 2;
+ struct arm_smmu_ril_range trail;
+ u8 ttl = 0;
- if (WARN_ON_ONCE(!tlbi->size))
- return;
-
- if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
- num_pages = tlbi->size >> tg;
-
- /* Convert page size of 12,14,16 (log2) to 1,2,3 */
- tg_enc = (tg - 10) / 2;
-
- /*
- * Determine what level the granule is at. For non-leaf, both
- * io-pgtable and SVA pass a nominal last-level granule because
- * they don't know what level(s) actually apply, so ignore that
- * and leave TTL=0. However for various errata reasons we still
- * want to use a range command, so avoid the SVA corner case
- * where both scale and num could be 0 as well.
- */
- if (tlbi->leaf_only)
- ttl = 4 - ((ilog2(tlbi->iopte_size) - 3) / (tg - 3));
- else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) == 1)
- num_pages++;
- }
-
- while (iova < end) {
- if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
- /*
- * On each iteration of the loop, the range is 5 bits
- * worth of the aligned size remaining.
- * The range in pages is:
- *
- * range = (num_pages & (0x1f << __ffs(num_pages)))
- */
- unsigned long scale, num;
-
- /* Determine the power of 2 multiple number of pages */
- scale = __ffs(num_pages);
-
- /* Determine how many chunks of 2^scale size we have */
- num = (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX;
-
- /* Keep the pre-DS 5-bit truncation when scale > 31 */
- cmd->data[0] = orig_data0 |
- FIELD_PREP(CMDQ_TLBI_0_NUM, num - 1) |
- FIELD_PREP(CMDQ_TLBI_0_SCALE, scale & 0x1f);
-
- /* range is num * 2^scale * pgsize */
- inv_range = num << (scale + tg);
-
- /* Clear out the lower order bits for the next iteration */
- num_pages -= num << scale;
- }
-
- /*
- * IPA has fewer bits than VA, but they are reserved in the
- * command and something would be very broken if iova had them
- * set.
- */
- cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
- FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
- FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) |
- (iova & ~GENMASK_U64(11, 0));
-
- arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
- iova += inv_range;
- }
-}
-
-/*
- * Generate a RIL for ARM_SMMU_OPT_FULL_CONT_RIL by ensuring the entire SVA
- * requested range is covered with a single RIL command. The scale is adjusted
- * so that the RIL may extend past the end of the requested range. This ensures
- * that any CONT the MM is invalidating is covered by a single RIL. TTL and LEAF
- * are always 0 because this is only used by SVA.
- */
-static bool arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
- struct arm_smmu_cmdq_batch *cmds,
- struct arm_smmu_cmd *cmd,
- unsigned long iova, size_t size,
- u8 tgsz_lg2)
-{
- u64 cur_tg = iova >> tgsz_lg2;
- u64 num_tg = ((iova + size - 1) >> tgsz_lg2) - cur_tg + 1;
- unsigned int scale = fls64((num_tg - 1) / 32);
-
- if (scale > 31)
- return false;
-
- cmd->data[0] |=
- FIELD_PREP(CMDQ_TLBI_0_NUM,
- DIV_ROUND_UP_ULL(num_tg, 1ULL << scale) - 1) |
- FIELD_PREP(CMDQ_TLBI_0_SCALE, scale);
- cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_TG, (tgsz_lg2 - 10) / 2) |
- (cur_tg << tgsz_lg2);
- arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
- return true;
-}
-
-static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu,
- struct arm_smmu_tlbi *tlbi)
-{
- size_t max_tlbi_ops;
-
- /* 0 size means invalidate all */
- if (!tlbi->size || tlbi->size == SIZE_MAX)
- return true;
-
- if (smmu->features & ARM_SMMU_FEAT_RANGE_INV)
+ if (!tlbi->size)
return false;
/*
- * Borrowed from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h,
- * this is used as a threshold to replace "size_opcode" commands with a
- * single "nsize_opcode" command, when SMMU doesn't implement the range
- * invalidation feature, where there can be too many per-granule TLBIs,
- * resulting in a soft lockup.
+ * Determine what level the granule is at. For non-leaf, both
+ * io-pgtable and SVA pass a nominal last-level granule because they
+ * don't know what level(s) actually apply, so leave TTL=0.
*/
- max_tlbi_ops = 1 << (ilog2(tlbi->iopte_size) - 3);
- return tlbi->size >= max_tlbi_ops * tlbi->iopte_size;
+ if (tlbi->leaf_only)
+ ttl = 4 - ((ilog2(tlbi->iopte_size) - 3) / (tgsz_lg2 - 3));
+
+ /*
+ * The spec defines the invalidated range as:
+ * Range = ((NUM+1) * 2^SCALE) * Translation_Granule_Size
+ * NUM is 5 bits, so (NUM+1) covers 1..32 granules. Find the smallest
+ * SCALE at which a single command could cover num_tg.
+ *
+ * Unlike other IOMMUs the spec has no alignment requirement on the
+ * address beyond alignment to tg (so long as TTL=0).
+ */
+ first.scale = fls64((num_tg - 1) / 32);
+ if (first.scale > 31) {
+ /* Range too large for a single command do full invalidation */
+ return false;
+ }
+
+ if (tlbi->has_cont && (smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
+ /*
+ * Produce a single invalidation by rounding up and disabling
+ * the trailer.
+ */
+ first.num = DIV_ROUND_UP_ULL(num_tg, 1ULL << first.scale);
+ trail.num = 0;
+ } else {
+ /*
+ * Produce two invalidations by rounding down and adding a
+ * second trailing RIL anchored at the end.
+ */
+ first.num = num_tg >> first.scale;
+ trail = arm_smmu_ril_init_end(
+ last_tg, num_tg - ((u64)first.num << first.scale));
+ }
+ arm_smmu_cmdq_batch_add_ril(smmu, cmds, cmd, tlbi->leaf_only, &first,
+ ttl, tg_enc);
+
+ if (trail.num)
+ arm_smmu_cmdq_batch_add_ril(smmu, cmds, cmd, tlbi->leaf_only,
+ &trail, ttl, tg_enc);
+ return true;
}
-/* Used by non INV_TYPE_ATS* invalidations */
-static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
+/*
+ * One TLBI command per IOTLB entry, assuming the entries are all at least
+ * iopte_granule sized. Returns false if too many commands would be needed which
+ * indicates too high a latency. The threshold is similar to MAX_DVM_OPS in
+ * arch/arm64/include/asm/tlbflush.h for the 4k PAGE_SIZE.
+ */
+static bool arm_smmu_cmdq_batch_add_single(struct arm_smmu_device *smmu,
+ struct arm_smmu_cmdq_batch *cmds,
+ struct arm_smmu_cmd *cmd,
+ struct arm_smmu_tlbi *tlbi)
+{
+ unsigned long num_ops = tlbi->size / tlbi->iopte_size;
+ unsigned long iova = tlbi->iova;
+ unsigned long i;
+
+ if (!num_ops || num_ops > 512)
+ return false;
+
+ for (i = 0; i < num_ops; i++) {
+ cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
+ (iova & ~GENMASK_U64(11, 0));
+ arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
+ iova += tlbi->iopte_size;
+ }
+ return true;
+}
+
+static void arm_smmu_inv_all_cmd(struct arm_smmu_inv *inv,
+ struct arm_smmu_cmdq_batch *cmds,
+ struct arm_smmu_cmd *cmd)
+{
+ u64p_replace_bits(&cmd->data[0], inv->nsize_opcode, CMDQ_0_OP);
+ arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, cmd);
+}
+
+/*
+ * Used by non INV_TYPE_ATS* invalidations. Returns true if it fell back to
+ * full invalidation using nsize_opcode.
+ */
+static bool arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
struct arm_smmu_cmdq_batch *cmds,
struct arm_smmu_cmd *cmd,
struct arm_smmu_tlbi *tlbi)
{
- struct arm_smmu_cmd nsize_cmd;
-
- if (arm_smmu_inv_size_too_big(inv->smmu, tlbi))
- goto full_inv;
-
- if (tlbi->has_cont && tlbi->size > tlbi->iopte_size &&
- (inv->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
- if (!arm_smmu_cmdq_batch_add_ril(inv->smmu, cmds, cmd,
- tlbi->iova, tlbi->size,
- tlbi->tgsz_lg2))
- goto full_inv;
- return;
+ if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
+ if (arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi))
+ return false;
+ } else {
+ if (arm_smmu_cmdq_batch_add_single(inv->smmu, cmds, cmd, tlbi))
+ return false;
}
- arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi);
- return;
-
-full_inv:
- nsize_cmd = *cmd;
- u64p_replace_bits(&nsize_cmd.data[0], inv->nsize_opcode, CMDQ_0_OP);
- arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, &nsize_cmd);
+ arm_smmu_inv_all_cmd(inv, cmds, cmd);
+ return true;
}
static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
@@ -2641,6 +2694,7 @@ static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
static void __arm_smmu_domain_inv_range(struct arm_smmu_tlbi *tlbi,
struct arm_smmu_invs *invs)
{
+ struct arm_smmu_inv *used_s12_vmall = NULL;
struct arm_smmu_cmdq_batch cmds = {};
struct arm_smmu_inv *cur;
struct arm_smmu_inv *end;
@@ -2674,11 +2728,19 @@ static void __arm_smmu_domain_inv_range(struct arm_smmu_tlbi *tlbi,
case INV_TYPE_S2_VMID:
cmd = arm_smmu_make_cmd_tlbi(cur->size_opcode,
0, cur->id);
- arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, tlbi);
+ if (arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, tlbi))
+ used_s12_vmall = cur + 1;
break;
case INV_TYPE_S2_VMID_S1_CLEAR:
- /* CMDQ_OP_TLBI_S12_VMALL already flushed S1 entries */
- if (arm_smmu_inv_size_too_big(cur->smmu, tlbi))
+ /*
+ * S2_VMID used CMDQ_OP_TLBI_S12_VMALL which already
+ * flushed S1 entries. These two types always come in
+ * pairs and arm_smmu_inv_cmp() ensures that they are
+ * consecutive in the list for the same SMMU. There may
+ * be several pairings so check this is paired with the
+ * one that did the full invalidation.
+ */
+ if (used_s12_vmall == cur)
break;
arm_smmu_cmdq_batch_add_cmd(
smmu, &cmds,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 3e7f94339ab60e..8b075e05d900aa 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -1173,7 +1173,8 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
static inline void arm_smmu_domain_inv(struct arm_smmu_domain *smmu_domain)
{
- arm_smmu_domain_inv_range(smmu_domain, 0, 0, 0, false);
+ arm_smmu_domain_inv_range(smmu_domain, 0, 0, 1 << smmu_domain->tgsz_lg2,
+ false);
}
void __arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu,
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v5 4/9] iommu/arm-smmu-v3: Optimize range invalidation for latency
2026-09-01 17:49 ` [PATCH v5 4/9] iommu/arm-smmu-v3: Optimize range invalidation for latency Jason Gunthorpe
@ 2026-09-07 14:26 ` Mostafa Saleh
2026-09-07 15:23 ` Jason Gunthorpe
0 siblings, 1 reply; 25+ messages in thread
From: Mostafa Saleh @ 2026-09-07 14:26 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Tue, Sep 01, 2026 at 02:49:53PM -0300, Jason Gunthorpe wrote:
> The server IOMMU drivers focus on invalidation latency by default,
> over-invalidating if necessary, to round the invalidation range up to a
> single command. I think this represents a trade off for DMA non-FQ and SVA
> where stalling the operation is overall worse than re-loading the IOTLB.
>
> For instance AMD and VT-d both round the range up to the largest aligned
> power of two and invalidate that. This causes over-invalidation but that
> is preferred on real HW over trying to issue a number of smaller
> range invalidations.
>
> SMMUv3 on the other hand will try to do up to 512 single invalidations,
> otherwise falls back to full invalidation, or it will try to issue an
> string of RIL invalidations for every 5 bits of IOVA range to perfectly
> cover it.
>
> While the RIL optimization is pretty good we can still do better and
> reduce it to only 2 invalidations at maximum using the idea Robin came up
> with to overlap two RILs in the middle. This preserves the exact coverage
> of the current RIL and still caps the number of RILs at 2.
>
> This works because the RIL can start at any IOVA, so we can place a RIL
> forwards from the start and backwards from the end, overlapping in the
> middle if the scale isn't precise enough.
>
> In the cases where this produces 2 RILS it continues to produce RILs that
> don't overlap, but the exact split point is different than the original
> algorithm due to the new calculation method. In cases where the original
> algorithm would produce 3 or more RILs this will produce 2 RILS with an
> overlap.
>
> The SVA under invalidation errata work around is maintained by "rounding up" to
> generate a single RIL that over-covers the entire range.
>
> Since the normal path is now the only one with a loop, split them into two
> functions and fold a simplified version of arm_smmu_inv_size_too_big()
> directly into the normal flow in a way that directly limits the number of
> single invalidation commands generated, again focusing on controlling
> latency.
>
> The end result is any gather is converted into either:
> - One invalidate all
> - One or two range invalidate ops
> - At most 512 single invalidation ops
>
> arm_smmu_domain_inv() is modified to always pass in the tgsz because the
> new logic relies on it being valid.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 354 ++++++++++++--------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 +-
> 2 files changed, 210 insertions(+), 147 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 6413eb4021ce22..9b79d7a5389812 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2460,166 +2460,219 @@ static void arm_smmu_tlb_inv_context(void *cookie)
> arm_smmu_domain_inv(smmu_domain);
> }
>
> -static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
> +/*
> + * Check address alignment for TTL hint per SMMUv3 H.a Section 4.4.1.
> + * Address bits below the alignment must be zero, otherwise UNPREDICTABLE.
> + */
> +static bool arm_smmu_ttl_addr_aligned(u64 address, unsigned int tg,
> + unsigned int ttl)
> +{
> + unsigned int pgsz_lg2 = (tg - 3) * (3 - ttl) + tg;
> +
> + return !(address & GENMASK_U64(pgsz_lg2 - 1, 0));
> +}
> +
> +struct arm_smmu_ril_range {
> + u64 start_tg;
> + /* Normal integer, not encoded. 0 means 0.*/
> + u64 num;
> + unsigned int scale;
> +};
> +
> +/*
> + * Initialize the smallest RIL covering num_tg and ending at last_tg.
> + */
> +static struct arm_smmu_ril_range arm_smmu_ril_init_end(u64 last_tg, u64 num_tg)
> +{
> + struct arm_smmu_ril_range ril = {};
> +
> + if (!num_tg)
> + return ril;
> +
> + ril.scale = fls64((num_tg - 1) / 32);
> + ril.num = DIV_ROUND_UP_ULL(num_tg, 1ULL << ril.scale);
> + ril.start_tg = last_tg - ((ril.num << ril.scale) - 1);
> + return ril;
> +}
> +
> +static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> + struct arm_smmu_cmdq_batch *cmds,
> + struct arm_smmu_cmd *ref_cmd,
> + bool leaf_only,
> + const struct arm_smmu_ril_range *ril,
> + u8 ttl, u8 tg_enc)
> +{
> + struct arm_smmu_cmd cmd;
> + unsigned int tgsz_lg2 = tg_enc * 2 + 10;
> + u64 iova = ril->start_tg << tgsz_lg2;
> + unsigned int num = ril->num - 1;
> +
> + /* 16K granule TTL=1 is reserved (Section 4.4.1) */
> + if (WARN_ON(tgsz_lg2 == 14 && ttl == 1))
> + ttl = 0;
> +
> + /* Verify address alignment for the TTL hint */
> + if (ttl && !arm_smmu_ttl_addr_aligned(iova, tgsz_lg2, ttl))
> + ttl = 0;
Is that a separate fix? Can this be a separate patch in that case.
Also, in what cases can this happen? io-pgtable-arm should do the
right thing and SVA will use ttl=0
> +
> + /*
> + * SMMUv3 H.a Section 4.4.1: TG!=0, NUM==0, SCALE==0, TTL==0 is Reserved
> + * and causes CERROR_ILL. Single tg uses NUM=0, SCALE=0 with a TTL hint
> + * to target only the exact leaf entry.
> + *
> + * For a single tg invalidation a 0 TTL can come from places like the
> + * SVA path that don't have enough information to get a TTL, or as a
> + * side of effect of the splitting.
> + *
> + * A single-TG invalidation cannot reach this point if it is part of a
> + * CONT group, so it is safe to transform it into a single invalidation.
> + * The ARM_SMMU_OPT_FULL_CONT_RIL errata does not apply.
> + */
> + if (!num && !ril->scale && !ttl)
> + tg_enc = 0;
> +
> + cmd.data[0] = ref_cmd->data[0] | FIELD_PREP(CMDQ_TLBI_0_NUM, num) |
> + FIELD_PREP(CMDQ_TLBI_0_SCALE, ril->scale);
> + cmd.data[1] = ref_cmd->data[1] |
> + FIELD_PREP(CMDQ_TLBI_1_LEAF, leaf_only) |
> + FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
> + FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova;
> + arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, &cmd);
> +}
> +
> +/*
> + * Issue up to two range TLBI commands covering [iova, iova+size). Returns true
> + * if successful, false if the range is too large to fit into RIL commands.
> + *
> + * Normally the first RIL is the largest representable span which does not
> + * exceed the requested range. If necessary, the second RIL is the smallest
> + * representable range covering the remainder and is anchored at the end. Any
> + * excess coverage from the second RIL overlaps the first instead of exceeding
> + * the requested range.
> + *
> + * If a SVA is being invalidated and the SMMU has the ARM_SMMU_OPT_FULL_CONT_RIL
> + * errata this produces only a single RIL and overinvalidates to ensure any
> + * potential CONT is covered with a single RIL.
> + */
> +static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
> struct arm_smmu_cmdq_batch *cmds,
> struct arm_smmu_cmd *cmd,
> struct arm_smmu_tlbi *tlbi)
> {
> - size_t inv_range = tlbi->iopte_size;
> - unsigned long iova = tlbi->iova;
> - unsigned long end = iova + tlbi->size;
> - unsigned long num_pages = 0;
> - u8 tg = tlbi->tgsz_lg2;
> - u64 orig_data0 = cmd->data[0];
> - u8 ttl = 0, tg_enc = 0;
> + u8 tgsz_lg2 = tlbi->tgsz_lg2;
> + struct arm_smmu_ril_range first = { .start_tg = tlbi->iova >>
> + tgsz_lg2 };
> + u64 last_tg = (tlbi->iova + tlbi->size - 1) >> tgsz_lg2;
> + u64 num_tg = last_tg - first.start_tg + 1;
> + u8 tg_enc = (tgsz_lg2 - 10) / 2;
> + struct arm_smmu_ril_range trail;
> + u8 ttl = 0;
>
> - if (WARN_ON_ONCE(!tlbi->size))
> - return;
> -
> - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
> - num_pages = tlbi->size >> tg;
> -
> - /* Convert page size of 12,14,16 (log2) to 1,2,3 */
> - tg_enc = (tg - 10) / 2;
> -
> - /*
> - * Determine what level the granule is at. For non-leaf, both
> - * io-pgtable and SVA pass a nominal last-level granule because
> - * they don't know what level(s) actually apply, so ignore that
> - * and leave TTL=0. However for various errata reasons we still
> - * want to use a range command, so avoid the SVA corner case
> - * where both scale and num could be 0 as well.
> - */
> - if (tlbi->leaf_only)
> - ttl = 4 - ((ilog2(tlbi->iopte_size) - 3) / (tg - 3));
> - else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) == 1)
> - num_pages++;
> - }
> -
> - while (iova < end) {
> - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
> - /*
> - * On each iteration of the loop, the range is 5 bits
> - * worth of the aligned size remaining.
> - * The range in pages is:
> - *
> - * range = (num_pages & (0x1f << __ffs(num_pages)))
> - */
> - unsigned long scale, num;
> -
> - /* Determine the power of 2 multiple number of pages */
> - scale = __ffs(num_pages);
> -
> - /* Determine how many chunks of 2^scale size we have */
> - num = (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX;
> -
> - /* Keep the pre-DS 5-bit truncation when scale > 31 */
> - cmd->data[0] = orig_data0 |
> - FIELD_PREP(CMDQ_TLBI_0_NUM, num - 1) |
> - FIELD_PREP(CMDQ_TLBI_0_SCALE, scale & 0x1f);
That just deletes the DS code which was recently merged.
I see that is added again in the last patch, but it would make more
sense to do it in the same patch I guess.
Thanks,
Mostafa
> -
> - /* range is num * 2^scale * pgsize */
> - inv_range = num << (scale + tg);
> -
> - /* Clear out the lower order bits for the next iteration */
> - num_pages -= num << scale;
> - }
> -
> - /*
> - * IPA has fewer bits than VA, but they are reserved in the
> - * command and something would be very broken if iova had them
> - * set.
> - */
> - cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
> - FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
> - FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) |
> - (iova & ~GENMASK_U64(11, 0));
> -
> - arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
> - iova += inv_range;
> - }
> -}
> -
> -/*
> - * Generate a RIL for ARM_SMMU_OPT_FULL_CONT_RIL by ensuring the entire SVA
> - * requested range is covered with a single RIL command. The scale is adjusted
> - * so that the RIL may extend past the end of the requested range. This ensures
> - * that any CONT the MM is invalidating is covered by a single RIL. TTL and LEAF
> - * are always 0 because this is only used by SVA.
> - */
> -static bool arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> - struct arm_smmu_cmdq_batch *cmds,
> - struct arm_smmu_cmd *cmd,
> - unsigned long iova, size_t size,
> - u8 tgsz_lg2)
> -{
> - u64 cur_tg = iova >> tgsz_lg2;
> - u64 num_tg = ((iova + size - 1) >> tgsz_lg2) - cur_tg + 1;
> - unsigned int scale = fls64((num_tg - 1) / 32);
> -
> - if (scale > 31)
> - return false;
> -
> - cmd->data[0] |=
> - FIELD_PREP(CMDQ_TLBI_0_NUM,
> - DIV_ROUND_UP_ULL(num_tg, 1ULL << scale) - 1) |
> - FIELD_PREP(CMDQ_TLBI_0_SCALE, scale);
> - cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_TG, (tgsz_lg2 - 10) / 2) |
> - (cur_tg << tgsz_lg2);
> - arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
> - return true;
> -}
> -
> -static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu,
> - struct arm_smmu_tlbi *tlbi)
> -{
> - size_t max_tlbi_ops;
> -
> - /* 0 size means invalidate all */
> - if (!tlbi->size || tlbi->size == SIZE_MAX)
> - return true;
> -
> - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV)
> + if (!tlbi->size)
> return false;
>
> /*
> - * Borrowed from the MAX_TLBI_OPS in arch/arm64/include/asm/tlbflush.h,
> - * this is used as a threshold to replace "size_opcode" commands with a
> - * single "nsize_opcode" command, when SMMU doesn't implement the range
> - * invalidation feature, where there can be too many per-granule TLBIs,
> - * resulting in a soft lockup.
> + * Determine what level the granule is at. For non-leaf, both
> + * io-pgtable and SVA pass a nominal last-level granule because they
> + * don't know what level(s) actually apply, so leave TTL=0.
> */
> - max_tlbi_ops = 1 << (ilog2(tlbi->iopte_size) - 3);
> - return tlbi->size >= max_tlbi_ops * tlbi->iopte_size;
> + if (tlbi->leaf_only)
> + ttl = 4 - ((ilog2(tlbi->iopte_size) - 3) / (tgsz_lg2 - 3));
> +
> + /*
> + * The spec defines the invalidated range as:
> + * Range = ((NUM+1) * 2^SCALE) * Translation_Granule_Size
> + * NUM is 5 bits, so (NUM+1) covers 1..32 granules. Find the smallest
> + * SCALE at which a single command could cover num_tg.
> + *
> + * Unlike other IOMMUs the spec has no alignment requirement on the
> + * address beyond alignment to tg (so long as TTL=0).
> + */
> + first.scale = fls64((num_tg - 1) / 32);
> + if (first.scale > 31) {
> + /* Range too large for a single command do full invalidation */
> + return false;
> + }
> +
> + if (tlbi->has_cont && (smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
> + /*
> + * Produce a single invalidation by rounding up and disabling
> + * the trailer.
> + */
> + first.num = DIV_ROUND_UP_ULL(num_tg, 1ULL << first.scale);
> + trail.num = 0;
> + } else {
> + /*
> + * Produce two invalidations by rounding down and adding a
> + * second trailing RIL anchored at the end.
> + */
> + first.num = num_tg >> first.scale;
> + trail = arm_smmu_ril_init_end(
> + last_tg, num_tg - ((u64)first.num << first.scale));
> + }
> + arm_smmu_cmdq_batch_add_ril(smmu, cmds, cmd, tlbi->leaf_only, &first,
> + ttl, tg_enc);
> +
> + if (trail.num)
> + arm_smmu_cmdq_batch_add_ril(smmu, cmds, cmd, tlbi->leaf_only,
> + &trail, ttl, tg_enc);
> + return true;
> }
>
> -/* Used by non INV_TYPE_ATS* invalidations */
> -static void arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
> +/*
> + * One TLBI command per IOTLB entry, assuming the entries are all at least
> + * iopte_granule sized. Returns false if too many commands would be needed which
> + * indicates too high a latency. The threshold is similar to MAX_DVM_OPS in
> + * arch/arm64/include/asm/tlbflush.h for the 4k PAGE_SIZE.
> + */
> +static bool arm_smmu_cmdq_batch_add_single(struct arm_smmu_device *smmu,
> + struct arm_smmu_cmdq_batch *cmds,
> + struct arm_smmu_cmd *cmd,
> + struct arm_smmu_tlbi *tlbi)
> +{
> + unsigned long num_ops = tlbi->size / tlbi->iopte_size;
> + unsigned long iova = tlbi->iova;
> + unsigned long i;
> +
> + if (!num_ops || num_ops > 512)
> + return false;
I am guessing "!num_ops" is impossible as both params are passed from
io-pgtable-arm so size must be >= iopte_size. However, in case size
was less than the iopte_size, wouldn’t we need to round up in that case?
> +
> + for (i = 0; i < num_ops; i++) {
> + cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
> + (iova & ~GENMASK_U64(11, 0));
> + arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
> + iova += tlbi->iopte_size;
> + }
> + return true;
> +}
> +
> +static void arm_smmu_inv_all_cmd(struct arm_smmu_inv *inv,
> + struct arm_smmu_cmdq_batch *cmds,
> + struct arm_smmu_cmd *cmd)
> +{
> + u64p_replace_bits(&cmd->data[0], inv->nsize_opcode, CMDQ_0_OP);
> + arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, cmd);
> +}
> +
> +/*
> + * Used by non INV_TYPE_ATS* invalidations. Returns true if it fell back to
> + * full invalidation using nsize_opcode.
> + */
> +static bool arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
> struct arm_smmu_cmdq_batch *cmds,
> struct arm_smmu_cmd *cmd,
> struct arm_smmu_tlbi *tlbi)
> {
> - struct arm_smmu_cmd nsize_cmd;
> -
> - if (arm_smmu_inv_size_too_big(inv->smmu, tlbi))
> - goto full_inv;
> -
> - if (tlbi->has_cont && tlbi->size > tlbi->iopte_size &&
> - (inv->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
> - if (!arm_smmu_cmdq_batch_add_ril(inv->smmu, cmds, cmd,
> - tlbi->iova, tlbi->size,
> - tlbi->tgsz_lg2))
> - goto full_inv;
> - return;
> + if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
> + if (arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi))
> + return false;
> + } else {
> + if (arm_smmu_cmdq_batch_add_single(inv->smmu, cmds, cmd, tlbi))
> + return false;
> }
>
> - arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi);
> - return;
> -
> -full_inv:
> - nsize_cmd = *cmd;
> - u64p_replace_bits(&nsize_cmd.data[0], inv->nsize_opcode, CMDQ_0_OP);
> - arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, &nsize_cmd);
> + arm_smmu_inv_all_cmd(inv, cmds, cmd);
> + return true;
> }
>
> static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
> @@ -2641,6 +2694,7 @@ static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
> static void __arm_smmu_domain_inv_range(struct arm_smmu_tlbi *tlbi,
> struct arm_smmu_invs *invs)
> {
> + struct arm_smmu_inv *used_s12_vmall = NULL;
> struct arm_smmu_cmdq_batch cmds = {};
> struct arm_smmu_inv *cur;
> struct arm_smmu_inv *end;
> @@ -2674,11 +2728,19 @@ static void __arm_smmu_domain_inv_range(struct arm_smmu_tlbi *tlbi,
> case INV_TYPE_S2_VMID:
> cmd = arm_smmu_make_cmd_tlbi(cur->size_opcode,
> 0, cur->id);
> - arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, tlbi);
> + if (arm_smmu_inv_to_cmdq_batch(cur, &cmds, &cmd, tlbi))
> + used_s12_vmall = cur + 1;
> break;
> case INV_TYPE_S2_VMID_S1_CLEAR:
> - /* CMDQ_OP_TLBI_S12_VMALL already flushed S1 entries */
> - if (arm_smmu_inv_size_too_big(cur->smmu, tlbi))
> + /*
> + * S2_VMID used CMDQ_OP_TLBI_S12_VMALL which already
> + * flushed S1 entries. These two types always come in
> + * pairs and arm_smmu_inv_cmp() ensures that they are
> + * consecutive in the list for the same SMMU. There may
> + * be several pairings so check this is paired with the
> + * one that did the full invalidation.
> + */
> + if (used_s12_vmall == cur)
> break;
> arm_smmu_cmdq_batch_add_cmd(
> smmu, &cmds,
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 3e7f94339ab60e..8b075e05d900aa 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -1173,7 +1173,8 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
>
> static inline void arm_smmu_domain_inv(struct arm_smmu_domain *smmu_domain)
> {
> - arm_smmu_domain_inv_range(smmu_domain, 0, 0, 0, false);
> + arm_smmu_domain_inv_range(smmu_domain, 0, 0, 1 << smmu_domain->tgsz_lg2,
> + false);
> }
>
> void __arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu,
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 4/9] iommu/arm-smmu-v3: Optimize range invalidation for latency
2026-09-07 14:26 ` Mostafa Saleh
@ 2026-09-07 15:23 ` Jason Gunthorpe
2026-09-07 19:34 ` Jason Gunthorpe
0 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-07 15:23 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 02:26:16PM +0000, Mostafa Saleh wrote:
> > +static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> > + struct arm_smmu_cmdq_batch *cmds,
> > + struct arm_smmu_cmd *ref_cmd,
> > + bool leaf_only,
> > + const struct arm_smmu_ril_range *ril,
> > + u8 ttl, u8 tg_enc)
> > +{
> > + struct arm_smmu_cmd cmd;
> > + unsigned int tgsz_lg2 = tg_enc * 2 + 10;
> > + u64 iova = ril->start_tg << tgsz_lg2;
> > + unsigned int num = ril->num - 1;
> > +
> > + /* 16K granule TTL=1 is reserved (Section 4.4.1) */
> > + if (WARN_ON(tgsz_lg2 == 14 && ttl == 1))
> > + ttl = 0;
> > +
> > + /* Verify address alignment for the TTL hint */
> > + if (ttl && !arm_smmu_ttl_addr_aligned(iova, tgsz_lg2, ttl))
> > + ttl = 0;
>
> Is that a separate fix? Can this be a separate patch in that case.
I don't know if it is because:
> Also, in what cases can this happen? io-pgtable-arm should do the
> right thing and SVA will use ttl=0
This is more getting things setup right for iommupt in the next series
which creates every imaginable kind of gather.
> > - /* Determine how many chunks of 2^scale size we have */
> > - num = (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX;
> > -
> > - /* Keep the pre-DS 5-bit truncation when scale > 31 */
> > - cmd->data[0] = orig_data0 |
> > - FIELD_PREP(CMDQ_TLBI_0_NUM, num - 1) |
> > - FIELD_PREP(CMDQ_TLBI_0_SCALE, scale & 0x1f);
>
> That just deletes the DS code which was recently merged.
> I see that is added again in the last patch, but it would make more
> sense to do it in the same patch I guess.
I missed it when rebasing, I'll adjust it
Thanks,
Jason
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 4/9] iommu/arm-smmu-v3: Optimize range invalidation for latency
2026-09-07 15:23 ` Jason Gunthorpe
@ 2026-09-07 19:34 ` Jason Gunthorpe
0 siblings, 0 replies; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-07 19:34 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 12:23:05PM -0300, Jason Gunthorpe wrote:
> > > - /* Determine how many chunks of 2^scale size we have */
> > > - num = (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX;
> > > -
> > > - /* Keep the pre-DS 5-bit truncation when scale > 31 */
> > > - cmd->data[0] = orig_data0 |
> > > - FIELD_PREP(CMDQ_TLBI_0_NUM, num - 1) |
> > > - FIELD_PREP(CMDQ_TLBI_0_SCALE, scale & 0x1f);
> >
> > That just deletes the DS code which was recently merged.
> > I see that is added again in the last patch, but it would make more
> > sense to do it in the same patch I guess.
>
> I missed it when rebasing, I'll adjust it
Actually it is correct like this. The earlier commit 773f2b9e3670
("iommu/arm-smmu-v3: Support IDR5.DS and widen the TLBI SCALE field")
did not actually support a wider scale it just widened the field and
added this protective truncate.
So this commit still does not support a wider scale and we don't need
the protective truncate because this version directly limits scale's
value.
The later patch truely is the one to add proper DS support and it
doesn't bring back the "& 0x1f", it adjusts the new code that limited
scale.
Jason
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v5 5/9] iommu/arm-smmu-v3: Keep track in the arm_smmu_invs if RIL is used
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
` (3 preceding siblings ...)
2026-09-01 17:49 ` [PATCH v5 4/9] iommu/arm-smmu-v3: Optimize range invalidation for latency Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-01 17:49 ` [PATCH v5 6/9] iommu/arm-smmu-v3: Precompute the invalidation commands Jason Gunthorpe
` (3 subsequent siblings)
8 siblings, 0 replies; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
Summarize if any of the inv entries will use RIL and if any need
ARM_SMMU_OPT_FULL_CONT_RIL. The next patch will use this to avoid RIL
pre-calculations unless RIL is being used by the invalidation.
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Reviewed-by: Mostafa Saleh <smostafa@google.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c | 38 +++++++++++--------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 19 ++++++++--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 4 ++
3 files changed, 41 insertions(+), 20 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
index add671363c828c..6f8917ff33a02b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c
@@ -24,7 +24,9 @@ struct arm_smmu_test_writer {
static struct arm_smmu_ste bypass_ste;
static struct arm_smmu_ste abort_ste;
static struct arm_smmu_device smmu = {
- .features = ARM_SMMU_FEAT_STALLS | ARM_SMMU_FEAT_ATTR_TYPES_OVR
+ .features = ARM_SMMU_FEAT_STALLS | ARM_SMMU_FEAT_ATTR_TYPES_OVR |
+ ARM_SMMU_FEAT_RANGE_INV,
+ .options = ARM_SMMU_OPT_FULL_CONT_RIL,
};
static struct mm_struct sva_mm = {
.pgd = (void *)0xdaedbeefdeadbeefULL,
@@ -645,6 +647,8 @@ static void arm_smmu_v3_invs_test_verify(struct kunit *test,
{
KUNIT_EXPECT_EQ(test, invs->num_invs, num_invs);
KUNIT_EXPECT_EQ(test, invs->num_trashes, num_trashes);
+ KUNIT_EXPECT_TRUE(test, invs->has_range_inv);
+ KUNIT_EXPECT_TRUE(test, invs->has_full_cont_ril);
while (num_invs--) {
KUNIT_EXPECT_EQ(test, invs->inv[num_invs].id, ids[num_invs]);
KUNIT_EXPECT_EQ(test, READ_ONCE(invs->inv[num_invs].users),
@@ -655,37 +659,37 @@ static void arm_smmu_v3_invs_test_verify(struct kunit *test,
static struct arm_smmu_invs invs1 = {
.num_invs = 3,
- .inv = { { .type = INV_TYPE_S2_VMID, .id = 1, },
- { .type = INV_TYPE_S2_VMID_S1_CLEAR, .id = 1, },
- { .type = INV_TYPE_ATS, .id = 3, }, },
+ .inv = { { .smmu = &smmu, .type = INV_TYPE_S2_VMID, .id = 1, },
+ { .smmu = &smmu, .type = INV_TYPE_S2_VMID_S1_CLEAR, .id = 1, },
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 3, }, },
};
static struct arm_smmu_invs invs2 = {
.num_invs = 3,
- .inv = { { .type = INV_TYPE_S2_VMID, .id = 1, }, /* duplicated */
- { .type = INV_TYPE_ATS, .id = 4, },
- { .type = INV_TYPE_ATS, .id = 5, }, },
+ .inv = { { .smmu = &smmu, .type = INV_TYPE_S2_VMID, .id = 1, }, /* duplicated */
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 4, },
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 5, }, },
};
static struct arm_smmu_invs invs3 = {
.num_invs = 3,
- .inv = { { .type = INV_TYPE_S2_VMID, .id = 1, }, /* duplicated */
- { .type = INV_TYPE_ATS, .id = 5, }, /* recover a trash */
- { .type = INV_TYPE_ATS, .id = 6, }, },
+ .inv = { { .smmu = &smmu, .type = INV_TYPE_S2_VMID, .id = 1, }, /* duplicated */
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 5, }, /* recover a trash */
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 6, }, },
};
static struct arm_smmu_invs invs4 = {
.num_invs = 3,
- .inv = { { .type = INV_TYPE_ATS, .id = 10, .ssid = 1 },
- { .type = INV_TYPE_ATS, .id = 10, .ssid = 3 },
- { .type = INV_TYPE_ATS, .id = 12, .ssid = 1 }, },
+ .inv = { { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 10, .ssid = 1 },
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 10, .ssid = 3 },
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 12, .ssid = 1 }, },
};
static struct arm_smmu_invs invs5 = {
.num_invs = 3,
- .inv = { { .type = INV_TYPE_ATS, .id = 10, .ssid = 2 },
- { .type = INV_TYPE_ATS, .id = 10, .ssid = 3 }, /* duplicate */
- { .type = INV_TYPE_ATS, .id = 12, .ssid = 2 }, },
+ .inv = { { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 10, .ssid = 2 },
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 10, .ssid = 3 }, /* duplicate */
+ { .smmu = &smmu, .type = INV_TYPE_ATS, .id = 12, .ssid = 2 }, },
};
static void arm_smmu_v3_invs_test(struct kunit *test)
@@ -705,6 +709,8 @@ static void arm_smmu_v3_invs_test(struct kunit *test)
/* New array */
test_a = arm_smmu_invs_alloc(0);
KUNIT_EXPECT_EQ(test, test_a->num_invs, 0);
+ KUNIT_EXPECT_FALSE(test, test_a->has_range_inv);
+ KUNIT_EXPECT_FALSE(test, test_a->has_full_cont_ril);
/* Test1: merge invs1 (new array) */
test_b = arm_smmu_invs_merge(test_a, &invs1);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 9b79d7a5389812..68dd7b69392737 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1053,6 +1053,19 @@ static inline int arm_smmu_invs_iter_next_cmp(struct arm_smmu_invs *invs_l,
return arm_smmu_inv_cmp(cur_l, &invs_r->inv[next_r]);
}
+static void arm_smmu_invs_update_caps(struct arm_smmu_invs *invs,
+ const struct arm_smmu_inv *inv)
+{
+ if (arm_smmu_inv_is_ats(inv))
+ invs->has_ats = true;
+
+ if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
+ invs->has_range_inv = true;
+ if (inv->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)
+ invs->has_full_cont_ril = true;
+ }
+}
+
/**
* arm_smmu_invs_for_each_cmp - Iterate over two sorted arrays computing for
* arm_smmu_invs_merge() or arm_smmu_invs_unref()
@@ -1123,8 +1136,7 @@ struct arm_smmu_invs *arm_smmu_invs_merge(struct arm_smmu_invs *invs,
*/
if (new != new_invs->inv)
WARN_ON_ONCE(arm_smmu_inv_cmp(new - 1, new) == 1);
- if (arm_smmu_inv_is_ats(new))
- new_invs->has_ats = true;
+ arm_smmu_invs_update_caps(new_invs, new);
new++;
}
@@ -1234,8 +1246,7 @@ struct arm_smmu_invs *arm_smmu_invs_purge(struct arm_smmu_invs *invs)
arm_smmu_invs_for_each_entry(invs, i, inv) {
new_invs->inv[num_invs] = *inv;
- if (arm_smmu_inv_is_ats(inv))
- new_invs->has_ats = true;
+ arm_smmu_invs_update_caps(new_invs, inv);
num_invs++;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 8b075e05d900aa..4b9f04825eefa2 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -758,6 +758,8 @@ static inline bool arm_smmu_inv_is_ats(const struct arm_smmu_inv *inv)
* Must not be greater than @num_invs
* @rwlock: optional rwlock to fence ATS operations
* @has_ats: flag if the array contains an INV_TYPE_ATS or INV_TYPE_ATS_FULL
+ * @has_range_inv: flag if any entry's SMMU supports range invalidation
+ * @has_full_cont_ril: flag if any entry's SMMU requires the CONT RIL workaround
* @rcu: rcu head for kfree_rcu()
* @inv: flexible invalidation array
*
@@ -787,6 +789,8 @@ struct arm_smmu_invs {
size_t num_trashes;
rwlock_t rwlock;
bool has_ats;
+ bool has_range_inv;
+ bool has_full_cont_ril;
struct rcu_head rcu;
struct arm_smmu_inv inv[] __counted_by(max_invs);
};
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v5 6/9] iommu/arm-smmu-v3: Precompute the invalidation commands
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
` (4 preceding siblings ...)
2026-09-01 17:49 ` [PATCH v5 5/9] iommu/arm-smmu-v3: Keep track in the arm_smmu_invs if RIL is used Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-07 14:27 ` Mostafa Saleh
2026-09-01 17:49 ` [PATCH v5 7/9] iommu/arm-smmu-v3: Populate the tlbi at the top of the call chain Jason Gunthorpe
` (2 subsequent siblings)
8 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
Store the required cmd data in the tlbi and just copy it out when
processing each item in the invs list. The cmd form only depends on
if the instance supports RIL or not, otherwise it is always the same.
This avoids redundant calculations for each invs entry.
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 141 +++++++++++---------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 12 +-
2 files changed, 91 insertions(+), 62 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 68dd7b69392737..883dfc584ed6c6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2506,14 +2506,12 @@ static struct arm_smmu_ril_range arm_smmu_ril_init_end(u64 last_tg, u64 num_tg)
return ril;
}
-static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
- struct arm_smmu_cmdq_batch *cmds,
- struct arm_smmu_cmd *ref_cmd,
- bool leaf_only,
+static void arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi,
const struct arm_smmu_ril_range *ril,
u8 ttl, u8 tg_enc)
{
- struct arm_smmu_cmd cmd;
+ struct arm_smmu_cmd *cmd =
+ &tlbi->range.cmds[tlbi->range.num_cmds++];
unsigned int tgsz_lg2 = tg_enc * 2 + 10;
u64 iova = ril->start_tg << tgsz_lg2;
unsigned int num = ril->num - 1;
@@ -2542,18 +2540,16 @@ static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
if (!num && !ril->scale && !ttl)
tg_enc = 0;
- cmd.data[0] = ref_cmd->data[0] | FIELD_PREP(CMDQ_TLBI_0_NUM, num) |
- FIELD_PREP(CMDQ_TLBI_0_SCALE, ril->scale);
- cmd.data[1] = ref_cmd->data[1] |
- FIELD_PREP(CMDQ_TLBI_1_LEAF, leaf_only) |
- FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
- FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova;
- arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, &cmd);
+ cmd->data[0] = FIELD_PREP(CMDQ_TLBI_0_NUM, num) |
+ FIELD_PREP(CMDQ_TLBI_0_SCALE, ril->scale);
+ cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
+ FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
+ FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova;
}
/*
- * Issue up to two range TLBI commands covering [iova, iova+size). Returns true
- * if successful, false if the range is too large to fit into RIL commands.
+ * Generate up to two range TLBI command payloads covering [iova, iova+size).
+ * Sets use_full_inv if the range is too large to represent.
*
* Normally the first RIL is the largest representable span which does not
* exceed the requested range. If necessary, the second RIL is the smallest
@@ -2561,14 +2557,12 @@ static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
* excess coverage from the second RIL overlaps the first instead of exceeding
* the requested range.
*
- * If a SVA is being invalidated and the SMMU has the ARM_SMMU_OPT_FULL_CONT_RIL
- * errata this produces only a single RIL and overinvalidates to ensure any
- * potential CONT is covered with a single RIL.
+ * For SVA on an invs containing an SMMU with ARM_SMMU_OPT_FULL_CONT_RIL,
+ * produce only a single RIL and overinvalidate so any potential CONT is
+ * covered by one command.
*/
-static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
- struct arm_smmu_cmdq_batch *cmds,
- struct arm_smmu_cmd *cmd,
- struct arm_smmu_tlbi *tlbi)
+static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
+ bool single_ril)
{
u8 tgsz_lg2 = tlbi->tgsz_lg2;
struct arm_smmu_ril_range first = { .start_tg = tlbi->iova >>
@@ -2579,9 +2573,6 @@ static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
struct arm_smmu_ril_range trail;
u8 ttl = 0;
- if (!tlbi->size)
- return false;
-
/*
* Determine what level the granule is at. For non-leaf, both
* io-pgtable and SVA pass a nominal last-level granule because they
@@ -2602,10 +2593,11 @@ static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
first.scale = fls64((num_tg - 1) / 32);
if (first.scale > 31) {
/* Range too large for a single command do full invalidation */
- return false;
+ tlbi->range.use_full_inv = true;
+ return;
}
- if (tlbi->has_cont && (smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
+ if (single_ril) {
/*
* Produce a single invalidation by rounding up and disabling
* the trailer.
@@ -2621,40 +2613,27 @@ static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
trail = arm_smmu_ril_init_end(
last_tg, num_tg - ((u64)first.num << first.scale));
}
- arm_smmu_cmdq_batch_add_ril(smmu, cmds, cmd, tlbi->leaf_only, &first,
- ttl, tg_enc);
+ arm_smmu_tlbi_add_range_cmd(tlbi, &first, ttl, tg_enc);
if (trail.num)
- arm_smmu_cmdq_batch_add_ril(smmu, cmds, cmd, tlbi->leaf_only,
- &trail, ttl, tg_enc);
- return true;
+ arm_smmu_tlbi_add_range_cmd(tlbi, &trail, ttl, tg_enc);
}
/*
* One TLBI command per IOTLB entry, assuming the entries are all at least
- * iopte_granule sized. Returns false if too many commands would be needed which
- * indicates too high a latency. The threshold is similar to MAX_DVM_OPS in
- * arch/arm64/include/asm/tlbflush.h for the 4k PAGE_SIZE.
+ * iopte_granule sized. Sets use_full_inv if too many commands would be needed
+ * which indicates too high a latency. The threshold is similar to MAX_DVM_OPS
+ * in arch/arm64/include/asm/tlbflush.h for the 4k PAGE_SIZE.
*/
-static bool arm_smmu_cmdq_batch_add_single(struct arm_smmu_device *smmu,
- struct arm_smmu_cmdq_batch *cmds,
- struct arm_smmu_cmd *cmd,
- struct arm_smmu_tlbi *tlbi)
+static void arm_smmu_tlbi_calc_single(struct arm_smmu_tlbi *tlbi)
{
unsigned long num_ops = tlbi->size / tlbi->iopte_size;
- unsigned long iova = tlbi->iova;
- unsigned long i;
- if (!num_ops || num_ops > 512)
- return false;
-
- for (i = 0; i < num_ops; i++) {
- cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
- (iova & ~GENMASK_U64(11, 0));
- arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
- iova += tlbi->iopte_size;
+ if (!num_ops || num_ops > 512) {
+ tlbi->single.use_full_inv = true;
+ return;
}
- return true;
+ tlbi->single.num = num_ops;
}
static void arm_smmu_inv_all_cmd(struct arm_smmu_inv *inv,
@@ -2674,16 +2653,37 @@ static bool arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
struct arm_smmu_cmd *cmd,
struct arm_smmu_tlbi *tlbi)
{
+ u64 iova = tlbi->iova;
+ unsigned int i;
+
if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
- if (arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi))
- return false;
- } else {
- if (arm_smmu_cmdq_batch_add_single(inv->smmu, cmds, cmd, tlbi))
- return false;
+ if (tlbi->range.use_full_inv) {
+ arm_smmu_inv_all_cmd(inv, cmds, cmd);
+ return true;
+ }
+ for (i = 0; i < tlbi->range.num_cmds; i++) {
+ struct arm_smmu_cmd range_cmd = tlbi->range.cmds[i];
+
+ range_cmd.data[0] |= cmd->data[0];
+ range_cmd.data[1] |= cmd->data[1];
+ arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds,
+ &range_cmd);
+ }
+ return false;
}
- arm_smmu_inv_all_cmd(inv, cmds, cmd);
- return true;
+ if (tlbi->single.use_full_inv) {
+ arm_smmu_inv_all_cmd(inv, cmds, cmd);
+ return true;
+ }
+
+ for (i = 0; i < tlbi->single.num; i++) {
+ cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
+ (iova & ~GENMASK_U64(11, 0));
+ iova += tlbi->iopte_size;
+ arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, cmd);
+ }
+ return false;
}
static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
@@ -2702,8 +2702,8 @@ static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
return false;
}
-static void __arm_smmu_domain_inv_range(struct arm_smmu_tlbi *tlbi,
- struct arm_smmu_invs *invs)
+static void arm_smmu_domain_tlbi_inv(struct arm_smmu_tlbi *tlbi,
+ struct arm_smmu_invs *invs)
{
struct arm_smmu_inv *used_s12_vmall = NULL;
struct arm_smmu_cmdq_batch cmds = {};
@@ -2798,11 +2798,17 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
.iova = iova,
.size = size,
.iopte_size = granule,
- .has_cont = smmu_domain->stage == ARM_SMMU_DOMAIN_SVA,
.leaf_only = leaf,
};
struct arm_smmu_invs *invs;
+ if (!size || size == SIZE_MAX) {
+ tlbi.single.use_full_inv = true;
+ tlbi.range.use_full_inv = true;
+ } else {
+ arm_smmu_tlbi_calc_single(&tlbi);
+ }
+
/*
* An invalidation request must follow some IOPTE change and then load
* an invalidation array. In the meantime, a domain attachment mutates
@@ -2833,6 +2839,19 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
rcu_read_lock();
invs = rcu_dereference(smmu_domain->invs);
+ /*
+ * Only precalculate RIL if it will be used, invs generation ensures
+ * this matches the instances used for invalidation.
+ */
+ if (invs->has_range_inv) {
+ if (!tlbi.range.use_full_inv) {
+ arm_smmu_tlbi_calc_range(
+ &tlbi,
+ smmu_domain->stage == ARM_SMMU_DOMAIN_SVA &&
+ invs->has_full_cont_ril);
+ }
+ }
+
/*
* Avoid locking unless ATS is being used. No ATC invalidation can be
* going on after a domain is detached.
@@ -2841,10 +2860,10 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
unsigned long flags;
read_lock_irqsave(&invs->rwlock, flags);
- __arm_smmu_domain_inv_range(&tlbi, invs);
+ arm_smmu_domain_tlbi_inv(&tlbi, invs);
read_unlock_irqrestore(&invs->rwlock, flags);
} else {
- __arm_smmu_domain_inv_range(&tlbi, invs);
+ arm_smmu_domain_tlbi_inv(&tlbi, invs);
}
rcu_read_unlock();
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 4b9f04825eefa2..33ef99775aea87 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -815,8 +815,18 @@ struct arm_smmu_tlbi {
unsigned int iopte_size;
/* Base Translation Granule of the page table */
u8 tgsz_lg2;
- bool has_cont;
bool leaf_only;
+
+ struct {
+ bool use_full_inv;
+ u16 num;
+ } single;
+
+ struct {
+ bool use_full_inv;
+ u8 num_cmds;
+ struct arm_smmu_cmd cmds[2];
+ } range;
};
struct arm_smmu_evtq {
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v5 6/9] iommu/arm-smmu-v3: Precompute the invalidation commands
2026-09-01 17:49 ` [PATCH v5 6/9] iommu/arm-smmu-v3: Precompute the invalidation commands Jason Gunthorpe
@ 2026-09-07 14:27 ` Mostafa Saleh
2026-09-07 15:19 ` Jason Gunthorpe
0 siblings, 1 reply; 25+ messages in thread
From: Mostafa Saleh @ 2026-09-07 14:27 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Tue, Sep 01, 2026 at 02:49:55PM -0300, Jason Gunthorpe wrote:
> Store the required cmd data in the tlbi and just copy it out when
> processing each item in the invs list. The cmd form only depends on
> if the instance supports RIL or not, otherwise it is always the same.
>
> This avoids redundant calculations for each invs entry.
As I mentioned on v2, I don’t see a value for this without smmu
sharing over the same domain, as this just adds extra complexity IMHO,
but that's up to Robin and Will.
Thanks,
Mostafa
>
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 141 +++++++++++---------
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 12 +-
> 2 files changed, 91 insertions(+), 62 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index 68dd7b69392737..883dfc584ed6c6 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2506,14 +2506,12 @@ static struct arm_smmu_ril_range arm_smmu_ril_init_end(u64 last_tg, u64 num_tg)
> return ril;
> }
>
> -static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> - struct arm_smmu_cmdq_batch *cmds,
> - struct arm_smmu_cmd *ref_cmd,
> - bool leaf_only,
> +static void arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi,
> const struct arm_smmu_ril_range *ril,
> u8 ttl, u8 tg_enc)
> {
> - struct arm_smmu_cmd cmd;
> + struct arm_smmu_cmd *cmd =
> + &tlbi->range.cmds[tlbi->range.num_cmds++];
> unsigned int tgsz_lg2 = tg_enc * 2 + 10;
> u64 iova = ril->start_tg << tgsz_lg2;
> unsigned int num = ril->num - 1;
> @@ -2542,18 +2540,16 @@ static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> if (!num && !ril->scale && !ttl)
> tg_enc = 0;
>
> - cmd.data[0] = ref_cmd->data[0] | FIELD_PREP(CMDQ_TLBI_0_NUM, num) |
> - FIELD_PREP(CMDQ_TLBI_0_SCALE, ril->scale);
> - cmd.data[1] = ref_cmd->data[1] |
> - FIELD_PREP(CMDQ_TLBI_1_LEAF, leaf_only) |
> - FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
> - FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova;
> - arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, &cmd);
> + cmd->data[0] = FIELD_PREP(CMDQ_TLBI_0_NUM, num) |
> + FIELD_PREP(CMDQ_TLBI_0_SCALE, ril->scale);
> + cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
> + FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
> + FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova;
> }
>
> /*
> - * Issue up to two range TLBI commands covering [iova, iova+size). Returns true
> - * if successful, false if the range is too large to fit into RIL commands.
> + * Generate up to two range TLBI command payloads covering [iova, iova+size).
> + * Sets use_full_inv if the range is too large to represent.
> *
> * Normally the first RIL is the largest representable span which does not
> * exceed the requested range. If necessary, the second RIL is the smallest
> @@ -2561,14 +2557,12 @@ static void arm_smmu_cmdq_batch_add_ril(struct arm_smmu_device *smmu,
> * excess coverage from the second RIL overlaps the first instead of exceeding
> * the requested range.
> *
> - * If a SVA is being invalidated and the SMMU has the ARM_SMMU_OPT_FULL_CONT_RIL
> - * errata this produces only a single RIL and overinvalidates to ensure any
> - * potential CONT is covered with a single RIL.
> + * For SVA on an invs containing an SMMU with ARM_SMMU_OPT_FULL_CONT_RIL,
> + * produce only a single RIL and overinvalidate so any potential CONT is
> + * covered by one command.
> */
> -static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
> - struct arm_smmu_cmdq_batch *cmds,
> - struct arm_smmu_cmd *cmd,
> - struct arm_smmu_tlbi *tlbi)
> +static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
> + bool single_ril)
> {
> u8 tgsz_lg2 = tlbi->tgsz_lg2;
> struct arm_smmu_ril_range first = { .start_tg = tlbi->iova >>
> @@ -2579,9 +2573,6 @@ static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
> struct arm_smmu_ril_range trail;
> u8 ttl = 0;
>
> - if (!tlbi->size)
> - return false;
> -
> /*
> * Determine what level the granule is at. For non-leaf, both
> * io-pgtable and SVA pass a nominal last-level granule because they
> @@ -2602,10 +2593,11 @@ static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
> first.scale = fls64((num_tg - 1) / 32);
> if (first.scale > 31) {
> /* Range too large for a single command do full invalidation */
> - return false;
> + tlbi->range.use_full_inv = true;
> + return;
> }
>
> - if (tlbi->has_cont && (smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)) {
> + if (single_ril) {
> /*
> * Produce a single invalidation by rounding up and disabling
> * the trailer.
> @@ -2621,40 +2613,27 @@ static bool arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu,
> trail = arm_smmu_ril_init_end(
> last_tg, num_tg - ((u64)first.num << first.scale));
> }
> - arm_smmu_cmdq_batch_add_ril(smmu, cmds, cmd, tlbi->leaf_only, &first,
> - ttl, tg_enc);
> + arm_smmu_tlbi_add_range_cmd(tlbi, &first, ttl, tg_enc);
>
> if (trail.num)
> - arm_smmu_cmdq_batch_add_ril(smmu, cmds, cmd, tlbi->leaf_only,
> - &trail, ttl, tg_enc);
> - return true;
> + arm_smmu_tlbi_add_range_cmd(tlbi, &trail, ttl, tg_enc);
> }
>
> /*
> * One TLBI command per IOTLB entry, assuming the entries are all at least
> - * iopte_granule sized. Returns false if too many commands would be needed which
> - * indicates too high a latency. The threshold is similar to MAX_DVM_OPS in
> - * arch/arm64/include/asm/tlbflush.h for the 4k PAGE_SIZE.
> + * iopte_granule sized. Sets use_full_inv if too many commands would be needed
> + * which indicates too high a latency. The threshold is similar to MAX_DVM_OPS
> + * in arch/arm64/include/asm/tlbflush.h for the 4k PAGE_SIZE.
> */
> -static bool arm_smmu_cmdq_batch_add_single(struct arm_smmu_device *smmu,
> - struct arm_smmu_cmdq_batch *cmds,
> - struct arm_smmu_cmd *cmd,
> - struct arm_smmu_tlbi *tlbi)
> +static void arm_smmu_tlbi_calc_single(struct arm_smmu_tlbi *tlbi)
> {
> unsigned long num_ops = tlbi->size / tlbi->iopte_size;
> - unsigned long iova = tlbi->iova;
> - unsigned long i;
>
> - if (!num_ops || num_ops > 512)
> - return false;
> -
> - for (i = 0; i < num_ops; i++) {
> - cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
> - (iova & ~GENMASK_U64(11, 0));
> - arm_smmu_cmdq_batch_add_cmd_p(smmu, cmds, cmd);
> - iova += tlbi->iopte_size;
> + if (!num_ops || num_ops > 512) {
> + tlbi->single.use_full_inv = true;
> + return;
> }
> - return true;
> + tlbi->single.num = num_ops;
> }
>
> static void arm_smmu_inv_all_cmd(struct arm_smmu_inv *inv,
> @@ -2674,16 +2653,37 @@ static bool arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
> struct arm_smmu_cmd *cmd,
> struct arm_smmu_tlbi *tlbi)
> {
> + u64 iova = tlbi->iova;
> + unsigned int i;
> +
> if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
> - if (arm_smmu_cmdq_batch_add_range(inv->smmu, cmds, cmd, tlbi))
> - return false;
> - } else {
> - if (arm_smmu_cmdq_batch_add_single(inv->smmu, cmds, cmd, tlbi))
> - return false;
> + if (tlbi->range.use_full_inv) {
> + arm_smmu_inv_all_cmd(inv, cmds, cmd);
> + return true;
> + }
> + for (i = 0; i < tlbi->range.num_cmds; i++) {
> + struct arm_smmu_cmd range_cmd = tlbi->range.cmds[i];
> +
> + range_cmd.data[0] |= cmd->data[0];
> + range_cmd.data[1] |= cmd->data[1];
> + arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds,
> + &range_cmd);
> + }
> + return false;
> }
>
> - arm_smmu_inv_all_cmd(inv, cmds, cmd);
> - return true;
> + if (tlbi->single.use_full_inv) {
> + arm_smmu_inv_all_cmd(inv, cmds, cmd);
> + return true;
> + }
> +
> + for (i = 0; i < tlbi->single.num; i++) {
> + cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
> + (iova & ~GENMASK_U64(11, 0));
> + iova += tlbi->iopte_size;
> + arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, cmd);
> + }
> + return false;
> }
>
> static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
> @@ -2702,8 +2702,8 @@ static inline bool arm_smmu_invs_end_batch(struct arm_smmu_inv *cur,
> return false;
> }
>
> -static void __arm_smmu_domain_inv_range(struct arm_smmu_tlbi *tlbi,
> - struct arm_smmu_invs *invs)
> +static void arm_smmu_domain_tlbi_inv(struct arm_smmu_tlbi *tlbi,
> + struct arm_smmu_invs *invs)
> {
> struct arm_smmu_inv *used_s12_vmall = NULL;
> struct arm_smmu_cmdq_batch cmds = {};
> @@ -2798,11 +2798,17 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
> .iova = iova,
> .size = size,
> .iopte_size = granule,
> - .has_cont = smmu_domain->stage == ARM_SMMU_DOMAIN_SVA,
> .leaf_only = leaf,
> };
> struct arm_smmu_invs *invs;
>
> + if (!size || size == SIZE_MAX) {
> + tlbi.single.use_full_inv = true;
> + tlbi.range.use_full_inv = true;
> + } else {
> + arm_smmu_tlbi_calc_single(&tlbi);
> + }
> +
> /*
> * An invalidation request must follow some IOPTE change and then load
> * an invalidation array. In the meantime, a domain attachment mutates
> @@ -2833,6 +2839,19 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
> rcu_read_lock();
> invs = rcu_dereference(smmu_domain->invs);
>
> + /*
> + * Only precalculate RIL if it will be used, invs generation ensures
> + * this matches the instances used for invalidation.
> + */
> + if (invs->has_range_inv) {
> + if (!tlbi.range.use_full_inv) {
> + arm_smmu_tlbi_calc_range(
> + &tlbi,
> + smmu_domain->stage == ARM_SMMU_DOMAIN_SVA &&
> + invs->has_full_cont_ril);
> + }
> + }
> +
> /*
> * Avoid locking unless ATS is being used. No ATC invalidation can be
> * going on after a domain is detached.
> @@ -2841,10 +2860,10 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
> unsigned long flags;
>
> read_lock_irqsave(&invs->rwlock, flags);
> - __arm_smmu_domain_inv_range(&tlbi, invs);
> + arm_smmu_domain_tlbi_inv(&tlbi, invs);
> read_unlock_irqrestore(&invs->rwlock, flags);
> } else {
> - __arm_smmu_domain_inv_range(&tlbi, invs);
> + arm_smmu_domain_tlbi_inv(&tlbi, invs);
> }
>
> rcu_read_unlock();
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 4b9f04825eefa2..33ef99775aea87 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -815,8 +815,18 @@ struct arm_smmu_tlbi {
> unsigned int iopte_size;
> /* Base Translation Granule of the page table */
> u8 tgsz_lg2;
> - bool has_cont;
> bool leaf_only;
> +
> + struct {
> + bool use_full_inv;
> + u16 num;
> + } single;
> +
> + struct {
> + bool use_full_inv;
> + u8 num_cmds;
> + struct arm_smmu_cmd cmds[2];
> + } range;
> };
>
> struct arm_smmu_evtq {
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 6/9] iommu/arm-smmu-v3: Precompute the invalidation commands
2026-09-07 14:27 ` Mostafa Saleh
@ 2026-09-07 15:19 ` Jason Gunthorpe
0 siblings, 0 replies; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-07 15:19 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 02:27:52PM +0000, Mostafa Saleh wrote:
> On Tue, Sep 01, 2026 at 02:49:55PM -0300, Jason Gunthorpe wrote:
> > Store the required cmd data in the tlbi and just copy it out when
> > processing each item in the invs list. The cmd form only depends on
> > if the instance supports RIL or not, otherwise it is always the same.
> >
> > This avoids redundant calculations for each invs entry.
>
> As I mentioned on v2, I don’t see a value for this without smmu
> sharing over the same domain, as this just adds extra complexity IMHO,
> but that's up to Robin and Will.
Nicolin already has a series that does exactly this, waiting for stuff
already on the list to get merged before posting it. I'd rather have
things organized in the way we know we need them..
Otherwise we need to go around and change this stuff again..
Jason
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v5 7/9] iommu/arm-smmu-v3: Populate the tlbi at the top of the call chain
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
` (5 preceding siblings ...)
2026-09-01 17:49 ` [PATCH v5 6/9] iommu/arm-smmu-v3: Precompute the invalidation commands Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-01 17:49 ` [PATCH v5 8/9] iommu/arm-smmu-v3: Change how the tlbi describes the invalidation Jason Gunthorpe
2026-09-01 17:49 ` [PATCH v5 9/9] iommu/arm-smmu-v3: Support the DS expansion of RIL's SCALE Jason Gunthorpe
8 siblings, 0 replies; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
Each of these have their own unique situation, populate the tlbi right
at the top and pass it into arm_smmu_domain_inv_range(). They will
diverge further when the iommupt invalidation scheme is introduced.
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 21 ++++----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 54 ++++++++++---------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 15 ++++--
3 files changed, 51 insertions(+), 39 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 7acb70d060ec4e..47eba08bd1161f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -140,16 +140,19 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
{
struct arm_smmu_domain *smmu_domain =
container_of(mn, struct arm_smmu_domain, mmu_notifier);
- size_t size;
+ struct arm_smmu_tlbi tlbi = {
+ .tgsz_lg2 = smmu_domain->tgsz_lg2,
+ .iova = start,
+ /*
+ * The mm_types defines vm_end as the first byte after the end
+ * address, different from IOMMU subsystem using the last
+ * address of an address range.
+ */
+ .size = end - start,
+ .iopte_size = PAGE_SIZE,
+ };
- /*
- * The mm_types defines vm_end as the first byte after the end address,
- * different from IOMMU subsystem using the last address of an address
- * range. So do a simple translation here by calculating size correctly.
- */
- size = end - start;
-
- arm_smmu_domain_inv_range(smmu_domain, start, size, PAGE_SIZE, false);
+ arm_smmu_domain_tlbi(&tlbi, smmu_domain);
}
static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 883dfc584ed6c6..b9efba3404c68b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2789,25 +2789,13 @@ static void arm_smmu_domain_tlbi_inv(struct arm_smmu_tlbi *tlbi,
}
}
-void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
- unsigned long iova, size_t size,
- unsigned int granule, bool leaf)
+void arm_smmu_domain_tlbi(struct arm_smmu_tlbi *tlbi,
+ struct arm_smmu_domain *smmu_domain)
{
- struct arm_smmu_tlbi tlbi = {
- .tgsz_lg2 = smmu_domain->tgsz_lg2,
- .iova = iova,
- .size = size,
- .iopte_size = granule,
- .leaf_only = leaf,
- };
struct arm_smmu_invs *invs;
- if (!size || size == SIZE_MAX) {
- tlbi.single.use_full_inv = true;
- tlbi.range.use_full_inv = true;
- } else {
- arm_smmu_tlbi_calc_single(&tlbi);
- }
+ if (!tlbi->single.use_full_inv)
+ arm_smmu_tlbi_calc_single(tlbi);
/*
* An invalidation request must follow some IOPTE change and then load
@@ -2826,7 +2814,7 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
*
* [CPU0] | [CPU1]
* change IOPTE on new domain: |
- * arm_smmu_domain_inv_range() { | arm_smmu_install_new_domain_invs()
+ * arm_smmu_domain_tlbi() { | arm_smmu_install_new_domain_invs()
* smp_mb(); // ensures IOPTE | arm_smmu_install_ste_for_dev {
* // seen by SMMU | dma_wmb(); // ensures invs update
* // load the updated invs | // before updating STE
@@ -2844,9 +2832,9 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
* this matches the instances used for invalidation.
*/
if (invs->has_range_inv) {
- if (!tlbi.range.use_full_inv) {
+ if (!tlbi->range.use_full_inv) {
arm_smmu_tlbi_calc_range(
- &tlbi,
+ tlbi,
smmu_domain->stage == ARM_SMMU_DOMAIN_SVA &&
invs->has_full_cont_ril);
}
@@ -2860,10 +2848,10 @@ void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
unsigned long flags;
read_lock_irqsave(&invs->rwlock, flags);
- arm_smmu_domain_tlbi_inv(&tlbi, invs);
+ arm_smmu_domain_tlbi_inv(tlbi, invs);
read_unlock_irqrestore(&invs->rwlock, flags);
} else {
- arm_smmu_domain_tlbi_inv(&tlbi, invs);
+ arm_smmu_domain_tlbi_inv(tlbi, invs);
}
rcu_read_unlock();
@@ -2879,12 +2867,23 @@ static void arm_smmu_tlb_inv_page_nosync(struct iommu_iotlb_gather *gather,
iommu_iotlb_gather_add_page(domain, gather, iova, granule);
}
+/*
+ * Called by io-pgtable-arm.c for each single table level it wants to remove.
+ * size is the size of the table level and granule is the tg in bytes. This must
+ * clear the walk cache and any leaves within the range.
+ */
static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size,
size_t granule, void *cookie)
{
struct arm_smmu_domain *smmu_domain = cookie;
+ struct arm_smmu_tlbi tlbi = {
+ .tgsz_lg2 = smmu_domain->tgsz_lg2,
+ .iova = iova,
+ .size = size,
+ .iopte_size = 1 << smmu_domain->tgsz_lg2,
+ };
- arm_smmu_domain_inv_range(smmu_domain, iova, size, granule, false);
+ arm_smmu_domain_tlbi(&tlbi, smmu_domain);
}
static const struct iommu_flush_ops arm_smmu_flush_ops = {
@@ -4157,13 +4156,18 @@ static void arm_smmu_iotlb_sync(struct iommu_domain *domain,
struct iommu_iotlb_gather *gather)
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ struct arm_smmu_tlbi tlbi = {
+ .tgsz_lg2 = smmu_domain->tgsz_lg2,
+ .iova = gather->start,
+ .size = gather->end - gather->start + 1,
+ .iopte_size = gather->pgsize,
+ .leaf_only = true,
+ };
if (!gather->pgsize)
return;
- arm_smmu_domain_inv_range(smmu_domain, gather->start,
- gather->end - gather->start + 1,
- gather->pgsize, true);
+ arm_smmu_domain_tlbi(&tlbi, smmu_domain);
}
static phys_addr_t
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 33ef99775aea87..09befe7b284fed 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -1181,14 +1181,19 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
struct arm_smmu_cd *cd, struct iommu_domain *old);
-void arm_smmu_domain_inv_range(struct arm_smmu_domain *smmu_domain,
- unsigned long iova, size_t size,
- unsigned int granule, bool leaf);
+void arm_smmu_domain_tlbi(struct arm_smmu_tlbi *tlbi,
+ struct arm_smmu_domain *smmu_domain);
static inline void arm_smmu_domain_inv(struct arm_smmu_domain *smmu_domain)
{
- arm_smmu_domain_inv_range(smmu_domain, 0, 0, 1 << smmu_domain->tgsz_lg2,
- false);
+ /* Prefilled for invalidate all */
+ struct arm_smmu_tlbi tlbi = {
+ .tgsz_lg2 = smmu_domain->tgsz_lg2,
+ .single.use_full_inv = true,
+ .range.use_full_inv = true,
+ };
+
+ arm_smmu_domain_tlbi(&tlbi, smmu_domain);
}
void __arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu,
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v5 8/9] iommu/arm-smmu-v3: Change how the tlbi describes the invalidation
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
` (6 preceding siblings ...)
2026-09-01 17:49 ` [PATCH v5 7/9] iommu/arm-smmu-v3: Populate the tlbi at the top of the call chain Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-07 14:42 ` Mostafa Saleh
2026-09-01 17:49 ` [PATCH v5 9/9] iommu/arm-smmu-v3: Support the DS expansion of RIL's SCALE Jason Gunthorpe
8 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
The RIL logic has long had a FIXME that there is not enough
information to properly compute the RIL. There is also subtly not
enough information to properly compute the single stride either.
Change tlbi to use the information format that iommupt is going to
use for ARM. This prepares the invalidation code to support iommupt
and fixes two small limitations with the current code.
iommupt is designed to accumulate all invalidation into a single
gather, then the iommu driver should issue a small number of commands
to execute the gather to control invalidation latency. This is in
contrast to io-pgtable-arm.c which generates many gather flushes and
direct walk cache flushes as it progresses.
To accommodate this the gather will accumulate "damage" in bitmaps,
one for leaf changes and one for table changes. This is enough
information for SMMUv3 to compute the proper stride for single
invalidation and to generate ideal hints for range invalidation.
Change the inner workings of the tlbi process to directly use this
new-style gather description with the idea that the iommupt
conversion will just direct assign the gather fields to the tlbi.
Rework the three places creating the tlbi to express their needs in
terms of the new bitmaps.
1) Simple iotlb invalidation always gets a single range of leaf
levels, so it can set a single leaf bit
2) Walk invalidation is expected to clear the table and all the leaves it
could contain. Set a single table bit and all the leaf bits.
There is a weakness in the existing io-pgtable where it double
invalidates the leaves, once through the arm_smmu_tlb_inv_walk() then
again through the gather. Since these are separate operations they are
not capped by the invalidation count limits and a single unmap may end
up doing thousands of tlbi commands. Eventually converting to iommupt's
gather only approach will correct this.
3) SVA invalidation has no idea what the MM did, so it will set all
the bits in the bitmaps.
This corrects another weakness where the RIL invalidation logic
was generating hints assuming the #2 rules which isn't correct
for SVA.
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 29 ++-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 176 +++++++++++++-----
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 24 ++-
3 files changed, 177 insertions(+), 52 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 47eba08bd1161f..07ba4e7910ab84 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -140,17 +140,34 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
{
struct arm_smmu_domain *smmu_domain =
container_of(mn, struct arm_smmu_domain, mmu_notifier);
+ u8 tgsz_lg2 = smmu_domain->tgsz_lg2;
struct arm_smmu_tlbi tlbi = {
.tgsz_lg2 = smmu_domain->tgsz_lg2,
- .iova = start,
+ .start = start,
+ .last = end - 1,
/*
- * The mm_types defines vm_end as the first byte after the end
- * address, different from IOMMU subsystem using the last
- * address of an address range.
+ * No information comes from the mm, assume the worst case that
+ * it changed every table level. The way this is hooked into the
+ * mm is tricky, the range won't be expanded to include an
+ * entire table level if one was removed like the iommu gather
+ * does. Thus even if this is a 4k invalidation it may be
+ * including any table level too.
*/
- .size = end - start,
- .iopte_size = PAGE_SIZE,
+ .table_levels_bitmap = 0xfe,
};
+ u8 pmd_lg2sz = (tgsz_lg2 - 3) * 1 + tgsz_lg2;
+
+ /*
+ * If the size is small then we can infer the invalidation is PTE only
+ * and set the PTE level only. Otherwise it could be some other
+ * combination so just set them all. This allows RIL to use TTL=3 in
+ * cases of PTE only changes. The mm must not try to partially
+ * invalidate pmd/etc.
+ */
+ if (end - start < BIT_U64(pmd_lg2sz))
+ tlbi.leaf_levels_bitmap = 1;
+ else
+ tlbi.leaf_levels_bitmap = 0xff;
arm_smmu_domain_tlbi(&tlbi, smmu_domain);
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index b9efba3404c68b..b99229a5bbacca 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2397,8 +2397,8 @@ static struct arm_smmu_cmd arm_smmu_atc_inv_to_cmd(u32 sid, int ssid,
* This has the unpleasant side-effect of invalidating all PASID-tagged
* ATC entries within the address range.
*/
- page_start = tlbi->iova >> inval_grain_shift;
- page_end = (tlbi->iova + tlbi->size - 1) >> inval_grain_shift;
+ page_start = tlbi->start >> inval_grain_shift;
+ page_end = tlbi->last >> inval_grain_shift;
/*
* In an ATS Invalidate Request, the address must be aligned on the
@@ -2512,16 +2512,11 @@ static void arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi,
{
struct arm_smmu_cmd *cmd =
&tlbi->range.cmds[tlbi->range.num_cmds++];
- unsigned int tgsz_lg2 = tg_enc * 2 + 10;
- u64 iova = ril->start_tg << tgsz_lg2;
+ u64 iova = ril->start_tg << tlbi->tgsz_lg2;
unsigned int num = ril->num - 1;
- /* 16K granule TTL=1 is reserved (Section 4.4.1) */
- if (WARN_ON(tgsz_lg2 == 14 && ttl == 1))
- ttl = 0;
-
/* Verify address alignment for the TTL hint */
- if (ttl && !arm_smmu_ttl_addr_aligned(iova, tgsz_lg2, ttl))
+ if (ttl && !arm_smmu_ttl_addr_aligned(iova, tlbi->tgsz_lg2, ttl))
ttl = 0;
/*
@@ -2529,9 +2524,8 @@ static void arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi,
* and causes CERROR_ILL. Single tg uses NUM=0, SCALE=0 with a TTL hint
* to target only the exact leaf entry.
*
- * For a single tg invalidation a 0 TTL can come from places like the
- * SVA path that don't have enough information to get a TTL, or as a
- * side of effect of the splitting.
+ * For a single tg invalidation a 0 TTL can come as a side of effect of
+ * the splitting.
*
* A single-TG invalidation cannot reach this point if it is part of a
* CONT group, so it is safe to transform it into a single invalidation.
@@ -2542,14 +2536,83 @@ static void arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi,
cmd->data[0] = FIELD_PREP(CMDQ_TLBI_0_NUM, num) |
FIELD_PREP(CMDQ_TLBI_0_SCALE, ril->scale);
- cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
+ cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF,
+ !tlbi->table_levels_bitmap) |
FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova;
}
+static int arm_smmu_bitmap_to_level(u8 bitmap)
+{
+ return 3 - (int)__ffs(bitmap);
+}
+
/*
- * Generate up to two range TLBI command payloads covering [iova, iova+size).
- * Sets use_full_inv if the range is too large to represent.
+ * Compute the TTL hint from leaf/table level bitmaps. 0 ttl means no hint
+ * invalidate all levels.
+ */
+static unsigned int arm_smmu_compute_ttl(u8 leaf_bitmap, u8 table_bitmap,
+ u8 tgsz_lg2)
+{
+ int ttl;
+
+ if (leaf_bitmap) {
+ /* If TTL is used then only leaves at the TTL are invalidated */
+ if (!is_power_of_2(leaf_bitmap))
+ return 0;
+
+ ttl = arm_smmu_bitmap_to_level(leaf_bitmap);
+ if (table_bitmap) {
+ int table_ttl = arm_smmu_bitmap_to_level(table_bitmap) + 1;
+
+ /*
+ * A RIL invalidation with !leaf_only clears out all
+ * table levels above the leaf level ttl only.
+ */
+ if (table_ttl > ttl)
+ return 0;
+ }
+ } else if (table_bitmap) {
+ /*
+ * Table-only invalidation. Spec says:
+ * For operations with Leaf=0, invalidation of cached Table
+ * descriptors for the address and scope additionally occurs at
+ * levels between the start of the walk and the level before
+ * the last level given by TTL.
+ * Choose a TTL hint that covers the only target table
+ * descriptor levels.
+ */
+ ttl = arm_smmu_bitmap_to_level(table_bitmap) + 1;
+
+ /*
+ * 16K granule, ARM TTL=1 is reserved (SMMUv3 H.a Section
+ * 4.4.1.1) if DS=0, avoid it always for table invalidations
+ * since we don't know what instance this will be applied to
+ * yet.
+ */
+ if (tgsz_lg2 == 14 && ttl == 1)
+ return 0;
+ } else {
+ /* Both bitmaps zero is not allowed */
+ WARN_ON(true);
+ return 0;
+ }
+
+ /*
+ * Assumes the page table is formed properly and does not trigger the
+ * 16k TTL=1 condition for leaf-only unless DS is enabled.
+ *
+ * ARM level -1 never has a leaf so something has gone wrong. ARM Level
+ * 0 cannot be hinted because ttl=0 means no-hint.
+ */
+ if (WARN_ON(ttl < 0))
+ return 0;
+ return ttl;
+}
+
+/*
+ * Generate up to two range TLBI command payloads covering [start, last]. Sets
+ * use_full_inv if the range is too large to represent.
*
* Normally the first RIL is the largest representable span which does not
* exceed the requested range. If necessary, the second RIL is the smallest
@@ -2565,21 +2628,14 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
bool single_ril)
{
u8 tgsz_lg2 = tlbi->tgsz_lg2;
- struct arm_smmu_ril_range first = { .start_tg = tlbi->iova >>
+ unsigned int ttl = arm_smmu_compute_ttl(
+ tlbi->leaf_levels_bitmap, tlbi->table_levels_bitmap, tgsz_lg2);
+ struct arm_smmu_ril_range first = { .start_tg = tlbi->start >>
tgsz_lg2 };
- u64 last_tg = (tlbi->iova + tlbi->size - 1) >> tgsz_lg2;
+ u64 last_tg = tlbi->last >> tgsz_lg2;
u64 num_tg = last_tg - first.start_tg + 1;
u8 tg_enc = (tgsz_lg2 - 10) / 2;
struct arm_smmu_ril_range trail;
- u8 ttl = 0;
-
- /*
- * Determine what level the granule is at. For non-leaf, both
- * io-pgtable and SVA pass a nominal last-level granule because they
- * don't know what level(s) actually apply, so leave TTL=0.
- */
- if (tlbi->leaf_only)
- ttl = 4 - ((ilog2(tlbi->iopte_size) - 3) / (tgsz_lg2 - 3));
/*
* The spec defines the invalidated range as:
@@ -2620,20 +2676,44 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
}
/*
- * One TLBI command per IOTLB entry, assuming the entries are all at least
- * iopte_granule sized. Sets use_full_inv if too many commands would be needed
- * which indicates too high a latency. The threshold is similar to MAX_DVM_OPS
- * in arch/arm64/include/asm/tlbflush.h for the 4k PAGE_SIZE.
+ * Compute the stride for non-RIL single-page invalidation. Returns the log2
+ * stride of the lowest affected level. Single invalidation removes all IOPTEs
+ * that contain the IOVA invalidated, and we can reliably assume that the
+ * architected page size and table sizes (not contiguous!) are reflected in the
+ * IOTLB. Thus if there is a 2M leaf entry we only need to issue a single IOTLB
+ * invalidation within that 2M IOVA.
+ */
+static u8 arm_smmu_tlbi_calc_stride(struct arm_smmu_tlbi *tlbi)
+{
+ u8 combined = tlbi->table_levels_bitmap | tlbi->leaf_levels_bitmap;
+ u8 tg_szlg2 = tlbi->tgsz_lg2;
+
+ if (WARN_ON(!combined))
+ return U8_MAX;
+ return (tg_szlg2 - 3) * __ffs(combined) + tg_szlg2;
+}
+
+/*
+ * One TLBI command per stride-sized entry. Sets use_full_inv if too many
+ * commands would be needed. The threshold is similar to MAX_DVM_OPS in
+ * arch/arm64/include/asm/tlbflush.h.
*/
static void arm_smmu_tlbi_calc_single(struct arm_smmu_tlbi *tlbi)
{
- unsigned long num_ops = tlbi->size / tlbi->iopte_size;
+ u8 stride_lg2 = arm_smmu_tlbi_calc_stride(tlbi);
+ unsigned long num_ops;
+ if (stride_lg2 == U8_MAX) {
+ tlbi->single.use_full_inv = true;
+ return;
+ }
+ num_ops = (tlbi->last - tlbi->start + 1) >> stride_lg2;
if (!num_ops || num_ops > 512) {
tlbi->single.use_full_inv = true;
return;
}
tlbi->single.num = num_ops;
+ tlbi->single.stride_lg2 = stride_lg2;
}
static void arm_smmu_inv_all_cmd(struct arm_smmu_inv *inv,
@@ -2653,7 +2733,7 @@ static bool arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
struct arm_smmu_cmd *cmd,
struct arm_smmu_tlbi *tlbi)
{
- u64 iova = tlbi->iova;
+ u64 iova = tlbi->start;
unsigned int i;
if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
@@ -2678,9 +2758,10 @@ static bool arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
}
for (i = 0; i < tlbi->single.num; i++) {
- cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
+ cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF,
+ !tlbi->table_levels_bitmap) |
(iova & ~GENMASK_U64(11, 0));
- iova += tlbi->iopte_size;
+ iova += BIT_U64(tlbi->single.stride_lg2);
arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, cmd);
}
return false;
@@ -2876,12 +2957,16 @@ static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size,
size_t granule, void *cookie)
{
struct arm_smmu_domain *smmu_domain = cookie;
+ u8 tgsz_lg2 = smmu_domain->tgsz_lg2;
struct arm_smmu_tlbi tlbi = {
.tgsz_lg2 = smmu_domain->tgsz_lg2,
- .iova = iova,
- .size = size,
- .iopte_size = 1 << smmu_domain->tgsz_lg2,
+ .start = iova,
+ .last = iova + size - 1,
};
+ u8 table_levels = BIT((ilog2(size) - tgsz_lg2) / (tgsz_lg2 - 3));
+
+ tlbi.table_levels_bitmap = table_levels;
+ tlbi.leaf_levels_bitmap = table_levels - 1;
arm_smmu_domain_tlbi(&tlbi, smmu_domain);
}
@@ -4152,21 +4237,28 @@ static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain)
arm_smmu_tlb_inv_context(smmu_domain);
}
+/*
+ * Called by io-pgtable-arm.c for each run of same pgsize leaf only
+ * invalidation. If it has to change to a different leaf level then it flushes
+ * the gather and starts a fresh one. Thus this always targets only a single
+ * leaf level.
+ */
static void arm_smmu_iotlb_sync(struct iommu_domain *domain,
struct iommu_iotlb_gather *gather)
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
+ unsigned int tg = smmu_domain->tgsz_lg2;
struct arm_smmu_tlbi tlbi = {
.tgsz_lg2 = smmu_domain->tgsz_lg2,
- .iova = gather->start,
- .size = gather->end - gather->start + 1,
- .iopte_size = gather->pgsize,
- .leaf_only = true,
+ .start = gather->start,
+ .last = gather->end,
};
- if (!gather->pgsize)
+ if (WARN_ON(gather->pgsize < BIT(tg)))
return;
+ tlbi.leaf_levels_bitmap = BIT((ilog2(gather->pgsize) - tg) / (tg - 3));
+
arm_smmu_domain_tlbi(&tlbi, smmu_domain);
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 09befe7b284fed..6582203eb7425f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -809,17 +809,31 @@ static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs)
}
struct arm_smmu_tlbi {
- unsigned long iova;
- size_t size;
- /* page or block size of the leaf iopte */
- unsigned int iopte_size;
+ unsigned long start;
+ unsigned long last;
/* Base Translation Granule of the page table */
u8 tgsz_lg2;
bool leaf_only;
+ /*
+ * Level bitmaps use iommupt numbering: bit 0 is the leaf-only level
+ * (ARM level 3), bit 1 is the next level up (ARM level 2), etc. These
+ * match the iommu_iotlb_gather.pt fields. Each set bit indicates a
+ * change at that level. The contiguous hint has no effect on
+ * invalidation processing because HW can ignore the hint.
+ *
+ * The pair selects the invalidation scope:
+ * table!=0, leaf==0 : walk cache only
+ * table==0, leaf!=0 : leaves only
+ * table!=0, leaf!=0 : walk cache + all leaves
+ * table==0, leaf==0 : illegal
+ */
+ u8 leaf_levels_bitmap;
+ u8 table_levels_bitmap;
struct {
bool use_full_inv;
u16 num;
+ u8 stride_lg2;
} single;
struct {
@@ -1188,6 +1202,8 @@ static inline void arm_smmu_domain_inv(struct arm_smmu_domain *smmu_domain)
{
/* Prefilled for invalidate all */
struct arm_smmu_tlbi tlbi = {
+ .start = 0,
+ .last = ULONG_MAX,
.tgsz_lg2 = smmu_domain->tgsz_lg2,
.single.use_full_inv = true,
.range.use_full_inv = true,
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v5 8/9] iommu/arm-smmu-v3: Change how the tlbi describes the invalidation
2026-09-01 17:49 ` [PATCH v5 8/9] iommu/arm-smmu-v3: Change how the tlbi describes the invalidation Jason Gunthorpe
@ 2026-09-07 14:42 ` Mostafa Saleh
2026-09-07 15:34 ` Jason Gunthorpe
0 siblings, 1 reply; 25+ messages in thread
From: Mostafa Saleh @ 2026-09-07 14:42 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Tue, Sep 01, 2026 at 02:49:57PM -0300, Jason Gunthorpe wrote:
> The RIL logic has long had a FIXME that there is not enough
> information to properly compute the RIL. There is also subtly not
> enough information to properly compute the single stride either.
>
> Change tlbi to use the information format that iommupt is going to
> use for ARM. This prepares the invalidation code to support iommupt
> and fixes two small limitations with the current code.
>
> iommupt is designed to accumulate all invalidation into a single
> gather, then the iommu driver should issue a small number of commands
> to execute the gather to control invalidation latency. This is in
> contrast to io-pgtable-arm.c which generates many gather flushes and
> direct walk cache flushes as it progresses.
>
> To accommodate this the gather will accumulate "damage" in bitmaps,
> one for leaf changes and one for table changes. This is enough
> information for SMMUv3 to compute the proper stride for single
> invalidation and to generate ideal hints for range invalidation.
I am not sure I understand that, in what situation the leaf_bitmap
would be used instead of a single page size?
Would iommupt combine different page sizes in a single invalidation?
And then I see in this patch it has:
if (!is_power_of_2(leaf_bitmap))
return 0;
Would that actually be better for performance than using 2 sets of
RILs, one for each page size?
As I'd imagine the HW will spend more effort on the TTL=0 case
otherwise it wouldn't require it.
>
> Change the inner workings of the tlbi process to directly use this
> new-style gather description with the idea that the iommupt
> conversion will just direct assign the gather fields to the tlbi.
>
> Rework the three places creating the tlbi to express their needs in
> terms of the new bitmaps.
>
> 1) Simple iotlb invalidation always gets a single range of leaf
> levels, so it can set a single leaf bit
>
> 2) Walk invalidation is expected to clear the table and all the leaves it
> could contain. Set a single table bit and all the leaf bits.
>
> There is a weakness in the existing io-pgtable where it double
> invalidates the leaves, once through the arm_smmu_tlb_inv_walk() then
> again through the gather. Since these are separate operations they are
> not capped by the invalidation count limits and a single unmap may end
> up doing thousands of tlbi commands. Eventually converting to iommupt's
> gather only approach will correct this.
>
> 3) SVA invalidation has no idea what the MM did, so it will set all
> the bits in the bitmaps.
>
> This corrects another weakness where the RIL invalidation logic
> was generating hints assuming the #2 rules which isn't correct
> for SVA.
>
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 29 ++-
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 176 +++++++++++++-----
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 24 ++-
> 3 files changed, 177 insertions(+), 52 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> index 47eba08bd1161f..07ba4e7910ab84 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
> @@ -140,17 +140,34 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> {
> struct arm_smmu_domain *smmu_domain =
> container_of(mn, struct arm_smmu_domain, mmu_notifier);
> + u8 tgsz_lg2 = smmu_domain->tgsz_lg2;
> struct arm_smmu_tlbi tlbi = {
> .tgsz_lg2 = smmu_domain->tgsz_lg2,
> - .iova = start,
> + .start = start,
> + .last = end - 1,
> /*
> - * The mm_types defines vm_end as the first byte after the end
> - * address, different from IOMMU subsystem using the last
> - * address of an address range.
> + * No information comes from the mm, assume the worst case that
> + * it changed every table level. The way this is hooked into the
> + * mm is tricky, the range won't be expanded to include an
> + * entire table level if one was removed like the iommu gather
> + * does. Thus even if this is a 4k invalidation it may be
> + * including any table level too.
> */
> - .size = end - start,
> - .iopte_size = PAGE_SIZE,
> + .table_levels_bitmap = 0xfe,
What does that mean, won't arm have a max of 4 levels?
> };
> + u8 pmd_lg2sz = (tgsz_lg2 - 3) * 1 + tgsz_lg2;
> +
> + /*
> + * If the size is small then we can infer the invalidation is PTE only
> + * and set the PTE level only. Otherwise it could be some other
> + * combination so just set them all. This allows RIL to use TTL=3 in
> + * cases of PTE only changes. The mm must not try to partially
> + * invalidate pmd/etc.
> + */
How does that work with splitting blocks? I imagine that might be
ossible with userspace.
> + if (end - start < BIT_U64(pmd_lg2sz))
> + tlbi.leaf_levels_bitmap = 1;
> + else
> + tlbi.leaf_levels_bitmap = 0xff;
>
> arm_smmu_domain_tlbi(&tlbi, smmu_domain);
> }
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index b9efba3404c68b..b99229a5bbacca 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -2397,8 +2397,8 @@ static struct arm_smmu_cmd arm_smmu_atc_inv_to_cmd(u32 sid, int ssid,
> * This has the unpleasant side-effect of invalidating all PASID-tagged
> * ATC entries within the address range.
> */
> - page_start = tlbi->iova >> inval_grain_shift;
> - page_end = (tlbi->iova + tlbi->size - 1) >> inval_grain_shift;
> + page_start = tlbi->start >> inval_grain_shift;
> + page_end = tlbi->last >> inval_grain_shift;
>
> /*
> * In an ATS Invalidate Request, the address must be aligned on the
> @@ -2512,16 +2512,11 @@ static void arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi,
> {
> struct arm_smmu_cmd *cmd =
> &tlbi->range.cmds[tlbi->range.num_cmds++];
> - unsigned int tgsz_lg2 = tg_enc * 2 + 10;
> - u64 iova = ril->start_tg << tgsz_lg2;
> + u64 iova = ril->start_tg << tlbi->tgsz_lg2;
> unsigned int num = ril->num - 1;
>
> - /* 16K granule TTL=1 is reserved (Section 4.4.1) */
> - if (WARN_ON(tgsz_lg2 == 14 && ttl == 1))
> - ttl = 0;
> -
> /* Verify address alignment for the TTL hint */
> - if (ttl && !arm_smmu_ttl_addr_aligned(iova, tgsz_lg2, ttl))
> + if (ttl && !arm_smmu_ttl_addr_aligned(iova, tlbi->tgsz_lg2, ttl))
> ttl = 0;
>
> /*
> @@ -2529,9 +2524,8 @@ static void arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi,
> * and causes CERROR_ILL. Single tg uses NUM=0, SCALE=0 with a TTL hint
> * to target only the exact leaf entry.
> *
> - * For a single tg invalidation a 0 TTL can come from places like the
> - * SVA path that don't have enough information to get a TTL, or as a
> - * side of effect of the splitting.
> + * For a single tg invalidation a 0 TTL can come as a side of effect of
> + * the splitting.
> *
> * A single-TG invalidation cannot reach this point if it is part of a
> * CONT group, so it is safe to transform it into a single invalidation.
> @@ -2542,14 +2536,83 @@ static void arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi,
>
> cmd->data[0] = FIELD_PREP(CMDQ_TLBI_0_NUM, num) |
> FIELD_PREP(CMDQ_TLBI_0_SCALE, ril->scale);
> - cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
> + cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF,
> + !tlbi->table_levels_bitmap) |
> FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) |
> FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova;
> }
>
> +static int arm_smmu_bitmap_to_level(u8 bitmap)
> +{
> + return 3 - (int)__ffs(bitmap);
> +}
> +
> /*
> - * Generate up to two range TLBI command payloads covering [iova, iova+size).
> - * Sets use_full_inv if the range is too large to represent.
> + * Compute the TTL hint from leaf/table level bitmaps. 0 ttl means no hint
> + * invalidate all levels.
> + */
> +static unsigned int arm_smmu_compute_ttl(u8 leaf_bitmap, u8 table_bitmap,
> + u8 tgsz_lg2)
> +{
> + int ttl;
> +
> + if (leaf_bitmap) {
> + /* If TTL is used then only leaves at the TTL are invalidated */
> + if (!is_power_of_2(leaf_bitmap))
> + return 0;
> +
> + ttl = arm_smmu_bitmap_to_level(leaf_bitmap);
> + if (table_bitmap) {
> + int table_ttl = arm_smmu_bitmap_to_level(table_bitmap) + 1;
> +
> + /*
> + * A RIL invalidation with !leaf_only clears out all
> + * table levels above the leaf level ttl only.
> + */
> + if (table_ttl > ttl)
> + return 0;
> + }
> + } else if (table_bitmap) {
> + /*
> + * Table-only invalidation. Spec says:
> + * For operations with Leaf=0, invalidation of cached Table
> + * descriptors for the address and scope additionally occurs at
> + * levels between the start of the walk and the level before
> + * the last level given by TTL.
> + * Choose a TTL hint that covers the only target table
> + * descriptor levels.
> + */
> + ttl = arm_smmu_bitmap_to_level(table_bitmap) + 1;
> +
> + /*
> + * 16K granule, ARM TTL=1 is reserved (SMMUv3 H.a Section
> + * 4.4.1.1) if DS=0, avoid it always for table invalidations
> + * since we don't know what instance this will be applied to
> + * yet.
> + */
> + if (tgsz_lg2 == 14 && ttl == 1)
> + return 0;
> + } else {
> + /* Both bitmaps zero is not allowed */
> + WARN_ON(true);
> + return 0;
> + }
> +
> + /*
> + * Assumes the page table is formed properly and does not trigger the
> + * 16k TTL=1 condition for leaf-only unless DS is enabled.
> + *
> + * ARM level -1 never has a leaf so something has gone wrong. ARM Level
> + * 0 cannot be hinted because ttl=0 means no-hint.
> + */
> + if (WARN_ON(ttl < 0))
> + return 0;
> + return ttl;
> +}
> +
> +/*
> + * Generate up to two range TLBI command payloads covering [start, last]. Sets
> + * use_full_inv if the range is too large to represent.
> *
> * Normally the first RIL is the largest representable span which does not
> * exceed the requested range. If necessary, the second RIL is the smallest
> @@ -2565,21 +2628,14 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
> bool single_ril)
> {
> u8 tgsz_lg2 = tlbi->tgsz_lg2;
> - struct arm_smmu_ril_range first = { .start_tg = tlbi->iova >>
> + unsigned int ttl = arm_smmu_compute_ttl(
> + tlbi->leaf_levels_bitmap, tlbi->table_levels_bitmap, tgsz_lg2);
> + struct arm_smmu_ril_range first = { .start_tg = tlbi->start >>
> tgsz_lg2 };
> - u64 last_tg = (tlbi->iova + tlbi->size - 1) >> tgsz_lg2;
> + u64 last_tg = tlbi->last >> tgsz_lg2;
> u64 num_tg = last_tg - first.start_tg + 1;
> u8 tg_enc = (tgsz_lg2 - 10) / 2;
> struct arm_smmu_ril_range trail;
> - u8 ttl = 0;
> -
> - /*
> - * Determine what level the granule is at. For non-leaf, both
> - * io-pgtable and SVA pass a nominal last-level granule because they
> - * don't know what level(s) actually apply, so leave TTL=0.
> - */
> - if (tlbi->leaf_only)
> - ttl = 4 - ((ilog2(tlbi->iopte_size) - 3) / (tgsz_lg2 - 3));
>
> /*
> * The spec defines the invalidated range as:
> @@ -2620,20 +2676,44 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
> }
>
> /*
> - * One TLBI command per IOTLB entry, assuming the entries are all at least
> - * iopte_granule sized. Sets use_full_inv if too many commands would be needed
> - * which indicates too high a latency. The threshold is similar to MAX_DVM_OPS
> - * in arch/arm64/include/asm/tlbflush.h for the 4k PAGE_SIZE.
> + * Compute the stride for non-RIL single-page invalidation. Returns the log2
> + * stride of the lowest affected level. Single invalidation removes all IOPTEs
> + * that contain the IOVA invalidated, and we can reliably assume that the
> + * architected page size and table sizes (not contiguous!) are reflected in the
> + * IOTLB. Thus if there is a 2M leaf entry we only need to issue a single IOTLB
> + * invalidation within that 2M IOVA.
> + */
> +static u8 arm_smmu_tlbi_calc_stride(struct arm_smmu_tlbi *tlbi)
> +{
> + u8 combined = tlbi->table_levels_bitmap | tlbi->leaf_levels_bitmap;
> + u8 tg_szlg2 = tlbi->tgsz_lg2;
> +
> + if (WARN_ON(!combined))
> + return U8_MAX;
When can that happen?
> + return (tg_szlg2 - 3) * __ffs(combined) + tg_szlg2;
> +}
> +
> +/*
> + * One TLBI command per stride-sized entry. Sets use_full_inv if too many
> + * commands would be needed. The threshold is similar to MAX_DVM_OPS in
> + * arch/arm64/include/asm/tlbflush.h.
> */
> static void arm_smmu_tlbi_calc_single(struct arm_smmu_tlbi *tlbi)
> {
> - unsigned long num_ops = tlbi->size / tlbi->iopte_size;
> + u8 stride_lg2 = arm_smmu_tlbi_calc_stride(tlbi);
> + unsigned long num_ops;
>
> + if (stride_lg2 == U8_MAX) {
> + tlbi->single.use_full_inv = true;
> + return;
> + }
> + num_ops = (tlbi->last - tlbi->start + 1) >> stride_lg2;
> if (!num_ops || num_ops > 512) {
> tlbi->single.use_full_inv = true;
> return;
> }
> tlbi->single.num = num_ops;
> + tlbi->single.stride_lg2 = stride_lg2;
> }
>
> static void arm_smmu_inv_all_cmd(struct arm_smmu_inv *inv,
> @@ -2653,7 +2733,7 @@ static bool arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
> struct arm_smmu_cmd *cmd,
> struct arm_smmu_tlbi *tlbi)
> {
> - u64 iova = tlbi->iova;
> + u64 iova = tlbi->start;
> unsigned int i;
>
> if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
> @@ -2678,9 +2758,10 @@ static bool arm_smmu_inv_to_cmdq_batch(struct arm_smmu_inv *inv,
> }
>
> for (i = 0; i < tlbi->single.num; i++) {
> - cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) |
> + cmd->data[1] = FIELD_PREP(CMDQ_TLBI_1_LEAF,
> + !tlbi->table_levels_bitmap) |
> (iova & ~GENMASK_U64(11, 0));
> - iova += tlbi->iopte_size;
> + iova += BIT_U64(tlbi->single.stride_lg2);
> arm_smmu_cmdq_batch_add_cmd_p(inv->smmu, cmds, cmd);
> }
> return false;
> @@ -2876,12 +2957,16 @@ static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size,
> size_t granule, void *cookie)
> {
> struct arm_smmu_domain *smmu_domain = cookie;
> + u8 tgsz_lg2 = smmu_domain->tgsz_lg2;
> struct arm_smmu_tlbi tlbi = {
> .tgsz_lg2 = smmu_domain->tgsz_lg2,
> - .iova = iova,
> - .size = size,
> - .iopte_size = 1 << smmu_domain->tgsz_lg2,
> + .start = iova,
> + .last = iova + size - 1,
> };
> + u8 table_levels = BIT((ilog2(size) - tgsz_lg2) / (tgsz_lg2 - 3));
> +
> + tlbi.table_levels_bitmap = table_levels;
> + tlbi.leaf_levels_bitmap = table_levels - 1;
>
> arm_smmu_domain_tlbi(&tlbi, smmu_domain);
> }
> @@ -4152,21 +4237,28 @@ static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain)
> arm_smmu_tlb_inv_context(smmu_domain);
> }
>
> +/*
> + * Called by io-pgtable-arm.c for each run of same pgsize leaf only
I believe that it is called from dma-iommu.c, io-pgtable-arm.c will
call the tlb_add_page which builds the gather though.
> + * invalidation. If it has to change to a different leaf level then it flushes
> + * the gather and starts a fresh one. Thus this always targets only a single
> + * leaf level.
> + */
> static void arm_smmu_iotlb_sync(struct iommu_domain *domain,
> struct iommu_iotlb_gather *gather)
> {
> struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> + unsigned int tg = smmu_domain->tgsz_lg2;
> struct arm_smmu_tlbi tlbi = {
> .tgsz_lg2 = smmu_domain->tgsz_lg2,
> - .iova = gather->start,
> - .size = gather->end - gather->start + 1,
> - .iopte_size = gather->pgsize,
> - .leaf_only = true,
> + .start = gather->start,
> + .last = gather->end,
> };
>
> - if (!gather->pgsize)
> + if (WARN_ON(gather->pgsize < BIT(tg)))
> return;
>
> + tlbi.leaf_levels_bitmap = BIT((ilog2(gather->pgsize) - tg) / (tg - 3));
Having some page table macros would be helpful (and in other places in
this patch)
Thanks,
Mostafa
> +
> arm_smmu_domain_tlbi(&tlbi, smmu_domain);
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 09befe7b284fed..6582203eb7425f 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -809,17 +809,31 @@ static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs)
> }
>
> struct arm_smmu_tlbi {
> - unsigned long iova;
> - size_t size;
> - /* page or block size of the leaf iopte */
> - unsigned int iopte_size;
> + unsigned long start;
> + unsigned long last;
> /* Base Translation Granule of the page table */
> u8 tgsz_lg2;
> bool leaf_only;
> + /*
> + * Level bitmaps use iommupt numbering: bit 0 is the leaf-only level
> + * (ARM level 3), bit 1 is the next level up (ARM level 2), etc. These
> + * match the iommu_iotlb_gather.pt fields. Each set bit indicates a
> + * change at that level. The contiguous hint has no effect on
> + * invalidation processing because HW can ignore the hint.
> + *
> + * The pair selects the invalidation scope:
> + * table!=0, leaf==0 : walk cache only
> + * table==0, leaf!=0 : leaves only
> + * table!=0, leaf!=0 : walk cache + all leaves
> + * table==0, leaf==0 : illegal
> + */
> + u8 leaf_levels_bitmap;
> + u8 table_levels_bitmap;
>
> struct {
> bool use_full_inv;
> u16 num;
> + u8 stride_lg2;
> } single;
>
> struct {
> @@ -1188,6 +1202,8 @@ static inline void arm_smmu_domain_inv(struct arm_smmu_domain *smmu_domain)
> {
> /* Prefilled for invalidate all */
> struct arm_smmu_tlbi tlbi = {
> + .start = 0,
> + .last = ULONG_MAX,
> .tgsz_lg2 = smmu_domain->tgsz_lg2,
> .single.use_full_inv = true,
> .range.use_full_inv = true,
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 8/9] iommu/arm-smmu-v3: Change how the tlbi describes the invalidation
2026-09-07 14:42 ` Mostafa Saleh
@ 2026-09-07 15:34 ` Jason Gunthorpe
2026-09-09 15:57 ` Jason Gunthorpe
0 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-07 15:34 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 02:42:57PM +0000, Mostafa Saleh wrote:
> On Tue, Sep 01, 2026 at 02:49:57PM -0300, Jason Gunthorpe wrote:
> > The RIL logic has long had a FIXME that there is not enough
> > information to properly compute the RIL. There is also subtly not
> > enough information to properly compute the single stride either.
> >
> > Change tlbi to use the information format that iommupt is going to
> > use for ARM. This prepares the invalidation code to support iommupt
> > and fixes two small limitations with the current code.
> >
> > iommupt is designed to accumulate all invalidation into a single
> > gather, then the iommu driver should issue a small number of commands
> > to execute the gather to control invalidation latency. This is in
> > contrast to io-pgtable-arm.c which generates many gather flushes and
> > direct walk cache flushes as it progresses.
> >
> > To accommodate this the gather will accumulate "damage" in bitmaps,
> > one for leaf changes and one for table changes. This is enough
> > information for SMMUv3 to compute the proper stride for single
> > invalidation and to generate ideal hints for range invalidation.
>
> I am not sure I understand that, in what situation the leaf_bitmap
> would be used instead of a single page size?=
> Would iommupt combine different page sizes in a single invalidation?
Yes
> And then I see in this patch it has:
> if (!is_power_of_2(leaf_bitmap))
> return 0;
Right, ARM doesn't support mixed leaves in a RIL so we can't use TTL
if iommupt has constructed something like that.
> Would that actually be better for performance than using 2 sets of
> RILs, one for each page size?
> As I'd imagine the HW will spend more effort on the TTL=0 case
> otherwise it wouldn't require it.
I have no idea, it is a hint. Since SW has no knowledge I think it
should just issue as few commands as possible. There is no way to know
what will work better on any particular HW.
> > @@ -140,17 +140,34 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
> > {
> > struct arm_smmu_domain *smmu_domain =
> > container_of(mn, struct arm_smmu_domain, mmu_notifier);
> > + u8 tgsz_lg2 = smmu_domain->tgsz_lg2;
> > struct arm_smmu_tlbi tlbi = {
> > .tgsz_lg2 = smmu_domain->tgsz_lg2,
> > - .iova = start,
> > + .start = start,
> > + .last = end - 1,
> > /*
> > - * The mm_types defines vm_end as the first byte after the end
> > - * address, different from IOMMU subsystem using the last
> > - * address of an address range.
> > + * No information comes from the mm, assume the worst case that
> > + * it changed every table level. The way this is hooked into the
> > + * mm is tricky, the range won't be expanded to include an
> > + * entire table level if one was removed like the iommu gather
> > + * does. Thus even if this is a 4k invalidation it may be
> > + * including any table level too.
> > */
> > - .size = end - start,
> > - .iopte_size = PAGE_SIZE,
> > + .table_levels_bitmap = 0xfe,
>
> What does that mean, won't arm have a max of 4 levels?
It is really ~1, the extra leading 1s don't matter. Just can't have
the leaf bit set.
> > + /*
> > + * If the size is small then we can infer the invalidation is PTE only
> > + * and set the PTE level only. Otherwise it could be some other
> > + * combination so just set them all. This allows RIL to use TTL=3 in
> > + * cases of PTE only changes. The mm must not try to partially
> > + * invalidate pmd/etc.
> > + */
>
> How does that work with splitting blocks? I imagine that might be
> ossible with userspace.
If mm splits anything then the invalidation will not be PAGE_SIZE
big. A split requires invalidating the original larger size.
> > +static u8 arm_smmu_tlbi_calc_stride(struct arm_smmu_tlbi *tlbi)
> > +{
> > + u8 combined = tlbi->table_levels_bitmap | tlbi->leaf_levels_bitmap;
> > + u8 tg_szlg2 = tlbi->tgsz_lg2;
> > +
> > + if (WARN_ON(!combined))
> > + return U8_MAX;
>
> When can that happen?
It can't, thats why it is a WARN_ON :)
> > @@ -4152,21 +4237,28 @@ static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain)
> > arm_smmu_tlb_inv_context(smmu_domain);
> > }
> >
> > +/*
> > + * Called by io-pgtable-arm.c for each run of same pgsize leaf only
>
> I believe that it is called from dma-iommu.c, io-pgtable-arm.c will
> call the tlb_add_page which builds the gather though.
Sort of, for the purposes of this comment the important flush is
initiated by io-pgtable-arm.c under tlb_add_page() when it calls
arm_smmu_tlb_inv_page_nosync(), which calls
iommu_iotlb_gather_add_page(), which calls iommu_iotlb_sync()
That's done in a way that guarentees the same-pgsize property:
if ((gather->pgsize && gather->pgsize != size) ||
Yes it is also called from dma-iommu.c, but only for the "trailing"
gather and that doesn't do anything to change what is in the gather..
I'll add a few more words here
> > + * invalidation. If it has to change to a different leaf level then it flushes
> > + * the gather and starts a fresh one. Thus this always targets only a single
> > + * leaf level.
> > + */
> > static void arm_smmu_iotlb_sync(struct iommu_domain *domain,
> > struct iommu_iotlb_gather *gather)
> > {
> > struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
> > + unsigned int tg = smmu_domain->tgsz_lg2;
> > struct arm_smmu_tlbi tlbi = {
> > .tgsz_lg2 = smmu_domain->tgsz_lg2,
> > - .iova = gather->start,
> > - .size = gather->end - gather->start + 1,
> > - .iopte_size = gather->pgsize,
> > - .leaf_only = true,
> > + .start = gather->start,
> > + .last = gather->end,
> > };
> >
> > - if (!gather->pgsize)
> > + if (WARN_ON(gather->pgsize < BIT(tg)))
> > return;
> >
> > + tlbi.leaf_levels_bitmap = BIT((ilog2(gather->pgsize) - tg) / (tg - 3));
>
> Having some page table macros would be helpful (and in other places in
> this patch)
At least this one gets deleted in the next series, so I left it like
this deliberately. Was there something else you saw that had
duplication?
Thanks,
Jason
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 8/9] iommu/arm-smmu-v3: Change how the tlbi describes the invalidation
2026-09-07 15:34 ` Jason Gunthorpe
@ 2026-09-09 15:57 ` Jason Gunthorpe
0 siblings, 0 replies; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-09 15:57 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 12:34:29PM -0300, Jason Gunthorpe wrote:
> > > + tlbi.leaf_levels_bitmap = BIT((ilog2(gather->pgsize) - tg) / (tg - 3));
> >
> > Having some page table macros would be helpful (and in other places in
> > this patch)
>
> At least this one gets deleted in the next series, so I left it like
> this deliberately. Was there something else you saw that had
> duplication?
I ended up with this:
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
index 07ba4e7910ab84..fc9622cdbac808 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
@@ -155,7 +155,7 @@ static void arm_smmu_mm_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
*/
.table_levels_bitmap = 0xfe,
};
- u8 pmd_lg2sz = (tgsz_lg2 - 3) * 1 + tgsz_lg2;
+ u8 pmd_lg2sz = arm_smmu_pt_level_to_lg2sz(tgsz_lg2, 1);
/*
* If the size is small then we can infer the invalidation is PTE only
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 2a159be751891e..8dd5d30c569771 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2485,7 +2485,7 @@ static void arm_smmu_tlb_inv_context(void *cookie)
static bool arm_smmu_ttl_addr_aligned(u64 address, unsigned int tg,
unsigned int ttl)
{
- unsigned int pgsz_lg2 = (tg - 3) * (3 - ttl) + tg;
+ unsigned int pgsz_lg2 = arm_smmu_pt_level_to_lg2sz(tg, 3 - ttl);
return !(address & GENMASK_U64(pgsz_lg2 - 1, 0));
}
@@ -2497,6 +2497,16 @@ struct arm_smmu_ril_range {
unsigned int scale;
};
+static unsigned int arm_smmu_ril_calc_scale(u64 num_tg)
+{
+ return fls64((num_tg - 1) / (CMDQ_TLBI_RANGE_NUM_MAX + 1));
+}
+
+static u64 arm_smmu_ril_calc_num(u64 num_tg, unsigned int scale)
+{
+ return DIV_ROUND_UP_ULL(num_tg, 1ULL << scale);
+}
+
/*
* Initialize the smallest RIL covering num_tg and ending at last_tg.
*/
@@ -2507,8 +2517,8 @@ static struct arm_smmu_ril_range arm_smmu_ril_init_end(u64 last_tg, u64 num_tg)
if (!num_tg)
return ril;
- ril.scale = fls64((num_tg - 1) / 32);
- ril.num = DIV_ROUND_UP_ULL(num_tg, 1ULL << ril.scale);
+ ril.scale = arm_smmu_ril_calc_scale(num_tg);
+ ril.num = arm_smmu_ril_calc_num(num_tg, ril.scale);
ril.start_tg = last_tg - ((ril.num << ril.scale) - 1);
return ril;
}
@@ -2654,7 +2664,7 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
* Unlike other IOMMUs the spec has no alignment requirement on the
* address beyond alignment to tg (so long as TTL=0).
*/
- first.scale = fls64((num_tg - 1) / 32);
+ first.scale = arm_smmu_ril_calc_scale(num_tg);
if (first.scale > scale_max) {
/* Range too large for a single command do full invalidation */
tlbi->range.use_full_inv = true;
@@ -2666,7 +2676,7 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
* Produce a single invalidation by rounding up and disabling
* the trailer.
*/
- first.num = DIV_ROUND_UP_ULL(num_tg, 1ULL << first.scale);
+ first.num = arm_smmu_ril_calc_num(num_tg, first.scale);
trail.num = 0;
} else {
/*
@@ -2694,11 +2704,10 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
static u8 arm_smmu_tlbi_calc_stride(struct arm_smmu_tlbi *tlbi)
{
u8 combined = tlbi->table_levels_bitmap | tlbi->leaf_levels_bitmap;
- u8 tg_szlg2 = tlbi->tgsz_lg2;
if (WARN_ON(!combined))
return U8_MAX;
- return (tg_szlg2 - 3) * __ffs(combined) + tg_szlg2;
+ return arm_smmu_pt_level_to_lg2sz(tlbi->tgsz_lg2, __ffs(combined));
}
/*
@@ -2972,7 +2981,8 @@ static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size,
.start = iova,
.last = iova + size - 1,
};
- u8 table_levels = BIT((ilog2(size) - tgsz_lg2) / (tgsz_lg2 - 3));
+ u8 table_levels =
+ BIT(arm_smmu_pt_lg2sz_to_level(tgsz_lg2, ilog2(size)));
tlbi.table_levels_bitmap = table_levels;
tlbi.leaf_levels_bitmap = table_levels - 1;
@@ -4247,10 +4257,12 @@ static void arm_smmu_flush_iotlb_all(struct iommu_domain *domain)
}
/*
- * Called by io-pgtable-arm.c for each run of same pgsize leaf only
- * invalidation. If it has to change to a different leaf level then it flushes
- * the gather and starts a fresh one. Thus this always targets only a single
- * leaf level.
+ * io-pgtable-arm.c calls this function either under
+ * arm_smmu_tlb_inv_page_nosync() or via the normal iommu code to flush the
+ * gather. Due to how iommu_iotlb_gather_add_page() works the gather will end up
+ * with a single uniform pgsize leaf. If it has to change to a different leaf
+ * level then it flushes the gather and starts a fresh one. Thus this always
+ * targets only a single leaf level.
*/
static void arm_smmu_iotlb_sync(struct iommu_domain *domain,
struct iommu_iotlb_gather *gather)
@@ -4266,7 +4278,8 @@ static void arm_smmu_iotlb_sync(struct iommu_domain *domain,
if (WARN_ON(gather->pgsize < BIT(tg)))
return;
- tlbi.leaf_levels_bitmap = BIT((ilog2(gather->pgsize) - tg) / (tg - 3));
+ tlbi.leaf_levels_bitmap =
+ BIT(arm_smmu_pt_lg2sz_to_level(tg, ilog2(gather->pgsize)));
arm_smmu_domain_tlbi(&tlbi, smmu_domain);
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 8b251a188810c9..70d732a96be27f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -810,12 +810,24 @@ static inline struct arm_smmu_invs *arm_smmu_invs_alloc(size_t num_invs)
return new_invs;
}
+/* Generic page-table level 0 is the leaf-only level. */
+static inline unsigned int arm_smmu_pt_level_to_lg2sz(unsigned int tgsz_lg2,
+ unsigned int level)
+{
+ return tgsz_lg2 + (tgsz_lg2 - ilog2(sizeof(u64))) * level;
+}
+
+static inline unsigned int arm_smmu_pt_lg2sz_to_level(unsigned int tgsz_lg2,
+ unsigned int lg2sz)
+{
+ return (lg2sz - tgsz_lg2) / (tgsz_lg2 - ilog2(sizeof(u64)));
+}
+
struct arm_smmu_tlbi {
unsigned long start;
unsigned long last;
/* Base Translation Granule of the page table */
u8 tgsz_lg2;
- bool leaf_only;
/*
* Level bitmaps use iommupt numbering: bit 0 is the leaf-only level
* (ARM level 3), bit 1 is the next level up (ARM level 2), etc. These
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v5 9/9] iommu/arm-smmu-v3: Support the DS expansion of RIL's SCALE
2026-09-01 17:49 [PATCH v5 0/9] Organize the SMMUv3 invalidation flow so iommupt can use it Jason Gunthorpe
` (7 preceding siblings ...)
2026-09-01 17:49 ` [PATCH v5 8/9] iommu/arm-smmu-v3: Change how the tlbi describes the invalidation Jason Gunthorpe
@ 2026-09-01 17:49 ` Jason Gunthorpe
2026-09-07 14:44 ` Mostafa Saleh
8 siblings, 1 reply; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-01 17:49 UTC (permalink / raw)
To: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon
Cc: David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, Mostafa Saleh, stable, Vijayanand Jitta
If DS is supported then SCALE can go up to 39. Compute a scale max that is
compatible for the entire invs list.
Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
Tested-by: Nicolin Chen <nicolinc@nvidia.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 ++++++++++++---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
2 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index b99229a5bbacca..2a159be751891e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1060,9 +1060,16 @@ static void arm_smmu_invs_update_caps(struct arm_smmu_invs *invs,
invs->has_ats = true;
if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
+ unsigned int scale_max;
+
invs->has_range_inv = true;
if (inv->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)
invs->has_full_cont_ril = true;
+
+ scale_max = (inv->smmu->features & ARM_SMMU_FEAT_DS) ? 39 : 31;
+ if (!invs->range_inv_scale_max ||
+ scale_max < invs->range_inv_scale_max)
+ invs->range_inv_scale_max = scale_max;
}
}
@@ -2625,7 +2632,8 @@ static unsigned int arm_smmu_compute_ttl(u8 leaf_bitmap, u8 table_bitmap,
* covered by one command.
*/
static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
- bool single_ril)
+ bool single_ril,
+ unsigned int scale_max)
{
u8 tgsz_lg2 = tlbi->tgsz_lg2;
unsigned int ttl = arm_smmu_compute_ttl(
@@ -2647,7 +2655,7 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
* address beyond alignment to tg (so long as TTL=0).
*/
first.scale = fls64((num_tg - 1) / 32);
- if (first.scale > 31) {
+ if (first.scale > scale_max) {
/* Range too large for a single command do full invalidation */
tlbi->range.use_full_inv = true;
return;
@@ -2917,7 +2925,8 @@ void arm_smmu_domain_tlbi(struct arm_smmu_tlbi *tlbi,
arm_smmu_tlbi_calc_range(
tlbi,
smmu_domain->stage == ARM_SMMU_DOMAIN_SVA &&
- invs->has_full_cont_ril);
+ invs->has_full_cont_ril,
+ invs->range_inv_scale_max);
}
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 6582203eb7425f..8b251a188810c9 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -760,6 +760,7 @@ static inline bool arm_smmu_inv_is_ats(const struct arm_smmu_inv *inv)
* @has_ats: flag if the array contains an INV_TYPE_ATS or INV_TYPE_ATS_FULL
* @has_range_inv: flag if any entry's SMMU supports range invalidation
* @has_full_cont_ril: flag if any entry's SMMU requires the CONT RIL workaround
+ * @range_inv_scale_max: max SCALE usable by all range-capable SMMUs
* @rcu: rcu head for kfree_rcu()
* @inv: flexible invalidation array
*
@@ -791,6 +792,7 @@ struct arm_smmu_invs {
bool has_ats;
bool has_range_inv;
bool has_full_cont_ril;
+ u8 range_inv_scale_max;
struct rcu_head rcu;
struct arm_smmu_inv inv[] __counted_by(max_invs);
};
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v5 9/9] iommu/arm-smmu-v3: Support the DS expansion of RIL's SCALE
2026-09-01 17:49 ` [PATCH v5 9/9] iommu/arm-smmu-v3: Support the DS expansion of RIL's SCALE Jason Gunthorpe
@ 2026-09-07 14:44 ` Mostafa Saleh
2026-09-07 19:39 ` Jason Gunthorpe
0 siblings, 1 reply; 25+ messages in thread
From: Mostafa Saleh @ 2026-09-07 14:44 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Tue, Sep 01, 2026 at 02:49:58PM -0300, Jason Gunthorpe wrote:
> If DS is supported then SCALE can go up to 39. Compute a scale max that is
> compatible for the entire invs list.
I see in some places scale is still assumed to 32 as in:
- arm_smmu_ril_init_end
- arm_smmu_tlbi_calc_range
Would those need to be updated also?
Thanks,
Mostafa
>
> Reviewed-by: Nicolin Chen <nicolinc@nvidia.com>
> Tested-by: Nicolin Chen <nicolinc@nvidia.com>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 15 ++++++++++++---
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 2 ++
> 2 files changed, 14 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index b99229a5bbacca..2a159be751891e 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -1060,9 +1060,16 @@ static void arm_smmu_invs_update_caps(struct arm_smmu_invs *invs,
> invs->has_ats = true;
>
> if (inv->smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
> + unsigned int scale_max;
> +
> invs->has_range_inv = true;
> if (inv->smmu->options & ARM_SMMU_OPT_FULL_CONT_RIL)
> invs->has_full_cont_ril = true;
> +
> + scale_max = (inv->smmu->features & ARM_SMMU_FEAT_DS) ? 39 : 31;
> + if (!invs->range_inv_scale_max ||
> + scale_max < invs->range_inv_scale_max)
> + invs->range_inv_scale_max = scale_max;
> }
> }
>
> @@ -2625,7 +2632,8 @@ static unsigned int arm_smmu_compute_ttl(u8 leaf_bitmap, u8 table_bitmap,
> * covered by one command.
> */
> static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
> - bool single_ril)
> + bool single_ril,
> + unsigned int scale_max)
> {
> u8 tgsz_lg2 = tlbi->tgsz_lg2;
> unsigned int ttl = arm_smmu_compute_ttl(
> @@ -2647,7 +2655,7 @@ static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi,
> * address beyond alignment to tg (so long as TTL=0).
> */
> first.scale = fls64((num_tg - 1) / 32);
> - if (first.scale > 31) {
> + if (first.scale > scale_max) {
> /* Range too large for a single command do full invalidation */
> tlbi->range.use_full_inv = true;
> return;
> @@ -2917,7 +2925,8 @@ void arm_smmu_domain_tlbi(struct arm_smmu_tlbi *tlbi,
> arm_smmu_tlbi_calc_range(
> tlbi,
> smmu_domain->stage == ARM_SMMU_DOMAIN_SVA &&
> - invs->has_full_cont_ril);
> + invs->has_full_cont_ril,
> + invs->range_inv_scale_max);
> }
> }
>
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> index 6582203eb7425f..8b251a188810c9 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
> @@ -760,6 +760,7 @@ static inline bool arm_smmu_inv_is_ats(const struct arm_smmu_inv *inv)
> * @has_ats: flag if the array contains an INV_TYPE_ATS or INV_TYPE_ATS_FULL
> * @has_range_inv: flag if any entry's SMMU supports range invalidation
> * @has_full_cont_ril: flag if any entry's SMMU requires the CONT RIL workaround
> + * @range_inv_scale_max: max SCALE usable by all range-capable SMMUs
> * @rcu: rcu head for kfree_rcu()
> * @inv: flexible invalidation array
> *
> @@ -791,6 +792,7 @@ struct arm_smmu_invs {
> bool has_ats;
> bool has_range_inv;
> bool has_full_cont_ril;
> + u8 range_inv_scale_max;
> struct rcu_head rcu;
> struct arm_smmu_inv inv[] __counted_by(max_invs);
> };
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 25+ messages in thread* Re: [PATCH v5 9/9] iommu/arm-smmu-v3: Support the DS expansion of RIL's SCALE
2026-09-07 14:44 ` Mostafa Saleh
@ 2026-09-07 19:39 ` Jason Gunthorpe
0 siblings, 0 replies; 25+ messages in thread
From: Jason Gunthorpe @ 2026-09-07 19:39 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Catalin Marinas, Jonathan Corbet, iommu, Joerg Roedel (AMD),
Jean-Philippe Brucker, linux-arm-kernel, linux-doc, Mark Rutland,
Randy Dunlap, Robin Murphy, Shuah Khan, Will Deacon,
David Matlack, Jean-Philippe Brucker, Jonathan Cameron,
Nicolin Chen, Pasha Tatashin, patches, Pranjal Shrivastava,
Samiullah Khawaja, stable, Vijayanand Jitta
On Mon, Sep 07, 2026 at 02:44:22PM +0000, Mostafa Saleh wrote:
> On Tue, Sep 01, 2026 at 02:49:58PM -0300, Jason Gunthorpe wrote:
> > If DS is supported then SCALE can go up to 39. Compute a scale max that is
> > compatible for the entire invs list.
>
> I see in some places scale is still assumed to 32 as in:
> - arm_smmu_ril_init_end
> - arm_smmu_tlbi_calc_range
Neither of those are scale, they are the range of NUM:
first.scale = fls64((num_tg - 1) / 32);
ie what scale do you need to fit in at most 32 NUM.
I will add a constant "MAX_RIL_NUM", it will be clearer.
Jason
^ permalink raw reply [flat|nested] 25+ messages in thread