[PATCH 0/7] Support non-leaf and range invalidation features in RISC-V

[PATCH 0/7] Support non-leaf and range invalidation features in RISC-V

[Jason Gunthorpe]

This is part of the patch pile to get SMMUv3 moved to iommupt. From
that perspective it introduces the PT_FEAT_DETAILED_GATHER which will
be used by both RISC-V and SMMUv3 to generate optimized invalidation
commands.

I don't have any RISC-V anything so this needs to be tested by someone who
does.

Improve the invalidation flow in RISC-V:
 - For single address invalidation, compute a stride based on what
   IOPTEs were changed by having the gather track level changes in
   bitmaps. This allows things like unmapping hugetlb-backed mappings
   to avoid jumping to global invalidation.

 - Support the NL bit to avoid global invalidation when table
   structure is changed

 - Support the S range invalidation feature to convert any gather
   into a single invalidation command using an aligned power-of-2
   size like AMD and VT-d.

This is also a requirement for the kunit iotlb series, which was used
to test the math used to generate the invalidation commands from the
gather in this series.

It replaces a similar series Fangyu sent prior to the iommupt
conversion:

https://lore.kernel.org/linux-iommu/20260208144213.94856-1-fangyu.yu@linux.alibaba.com/

The full patch pile is here:

https://github.com/jgunthorpe/linux/commits/iommu_pt_arm64/

Jason Gunthorpe (7):
  iommu: Split the kdoc comment for struct iommu_iotlb_gather
  iommupt: Add struct iommupt_pending_gather
  iommupt: Add PT_FEAT_DETAILED_GATHER
  iommu/riscv: Enable PT_FEAT_DETAILED_GATHER and pass gather to
    iotlb_inval
  iommu/riscv: Compute best stride for single invalidation
  iommu/riscv: Add RISCV_IOMMU_CAPABILITIES_NL
  iommu/riscv: Add NAPOT range invalidation support

 drivers/iommu/generic_pt/fmt/iommu_riscv64.c |   4 +-
 drivers/iommu/generic_pt/iommu_pt.h          |  98 ++++++++----
 drivers/iommu/riscv/iommu-bits.h             |  24 +++
 drivers/iommu/riscv/iommu.c                  | 160 ++++++++++++++-----
 include/linux/generic_pt/common.h            |   5 +
 include/linux/iommu.h                        |  42 ++++-
 6 files changed, 259 insertions(+), 74 deletions(-)


base-commit: ac81c0f2e6a779e444c5b6fd43756ef811656098
-- 
2.43.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 1/7] iommu: Split the kdoc comment for struct iommu_iotlb_gather

[Jason Gunthorpe]

Use in-line member documentation and add some small clarifications to
the members. This is preparation to add more members.

- Note that pgsize is only used by arm-smmuv3

- Note that freelist is only used by iommupt

- Reword queued to emphasize the flush-all behavior

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 include/linux/iommu.h | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index d8fcdb61e44c42..47a559616706a1 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -345,12 +345,6 @@ struct iommu_pages_list {
 /**
  * struct iommu_iotlb_gather - Range information for a pending IOTLB flush
  *
- * @start: IOVA representing the start of the range to be flushed
- * @end: IOVA representing the end of the range to be flushed (inclusive)
- * @pgsize: The interval at which to perform the flush
- * @freelist: Removed pages to free after sync
- * @queued: Indicates that the flush will be queued
- *
  * This structure is intended to be updated by multiple calls to the
  * ->unmap() function in struct iommu_ops before eventually being passed
  * into ->iotlb_sync(). Drivers can add pages to @freelist to be freed after
@@ -359,10 +353,24 @@ struct iommu_pages_list {
  * later instead of ->iotlb_sync(), so drivers may optimise accordingly.
  */
 struct iommu_iotlb_gather {
+	/** @start: IOVA representing the start of the range to be flushed */
 	unsigned long		start;
+	/**
+	 * @end: IOVA representing the end of the range to be
+	 *       flushed (inclusive)
+	 */
 	unsigned long		end;
+	/**
+	 * @pgsize: The interval at which to perform the flush, only used
+	 *          by arm-smmu-v3
+	 */
 	size_t			pgsize;
+	/**
+	 * @freelist: Removed pages to free after sync, only used by
+	 *            iommupt
+	 */
 	struct iommu_pages_list	freelist;
+	/** @queued: True if the gather will be completed with a flush all */
 	bool			queued;
 };
 
-- 
2.43.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 2/7] iommupt: Add struct iommupt_pending_gather

[Jason Gunthorpe]

Add a struct to keep track of all the things that are pending to be
merged into the gather. The way gather merging works, the pending
range is checked against the current gather, and the current gather
can be flushed before the pending things are added.

Thus, if new things have to be recorded in the gather they need to be
kept in the pending struct until after the gather is optionally
flushed.

The next patch adds new items to the gather and the pending struct.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/generic_pt/iommu_pt.h | 75 ++++++++++++++++++-----------
 1 file changed, 47 insertions(+), 28 deletions(-)

diff --git a/drivers/iommu/generic_pt/iommu_pt.h b/drivers/iommu/generic_pt/iommu_pt.h
index 19b6daf88f2ab1..5ec135cf43e2d6 100644
--- a/drivers/iommu/generic_pt/iommu_pt.h
+++ b/drivers/iommu/generic_pt/iommu_pt.h
@@ -40,15 +40,27 @@ static void flush_writes_item(const struct pt_state *pts)
 			PT_ITEM_WORD_SIZE);
 }
 
-static void gather_range_pages(struct iommu_iotlb_gather *iotlb_gather,
-			       struct pt_iommu *iommu_table, pt_vaddr_t iova,
-			       pt_vaddr_t len,
-			       struct iommu_pages_list *free_list)
+struct iommupt_pending_gather {
+	struct iommu_iotlb_gather *iotlb_gather;
+	struct iommu_pages_list free_list;
+};
+
+static void gather_add_table(struct iommupt_pending_gather *pending,
+			     const struct pt_state *pts,
+			     struct pt_table_p *table)
 {
+	iommu_pages_list_add(&pending->free_list, table);
+}
+
+static void gather_range_pending(struct iommupt_pending_gather *pending,
+				 struct pt_iommu *iommu_table, pt_vaddr_t iova,
+				 pt_vaddr_t len)
+{
+	struct iommu_iotlb_gather *iotlb_gather = pending->iotlb_gather;
 	struct pt_common *common = common_from_iommu(iommu_table);
 
 	if (pt_feature(common, PT_FEAT_DMA_INCOHERENT))
-		iommu_pages_stop_incoherent_list(free_list,
+		iommu_pages_stop_incoherent_list(&pending->free_list,
 						 iommu_table->iommu_device);
 
 	/*
@@ -72,7 +84,8 @@ static void gather_range_pages(struct iommu_iotlb_gather *iotlb_gather,
 		iommu_iotlb_gather_add_range(iotlb_gather, iova, len);
 	}
 
-	iommu_pages_list_splice(free_list, &iotlb_gather->freelist);
+	iommu_pages_list_splice(&pending->free_list, &iotlb_gather->freelist);
+	INIT_LIST_HEAD(&pending->free_list.pages);
 }
 
 #define DOMAIN_NS(op) CONCATENATE(CONCATENATE(pt_iommu_, PTPFX), op)
@@ -341,7 +354,7 @@ static int __maybe_unused NS(set_dirty)(struct pt_iommu *iommu_table,
 }
 
 struct pt_iommu_collect_args {
-	struct iommu_pages_list free_list;
+	struct iommupt_pending_gather pending;
 	/* Fail if any OAs are within the range */
 	u8 check_mapped : 1;
 };
@@ -358,7 +371,8 @@ static int __collect_tables(struct pt_range *range, void *arg,
 
 	for_each_pt_level_entry(&pts) {
 		if (pts.type == PT_ENTRY_TABLE) {
-			iommu_pages_list_add(&collect->free_list, pts.table_lower);
+			gather_add_table(&collect->pending, &pts,
+					 pts.table_lower);
 			ret = pt_descend(&pts, arg, __collect_tables);
 			if (ret)
 				return ret;
@@ -493,15 +507,18 @@ static int clear_contig(const struct pt_state *start_pts,
 	struct pt_range range = *start_pts->range;
 	struct pt_state pts =
 		pt_init(&range, start_pts->level, start_pts->table);
-	struct pt_iommu_collect_args collect = { .check_mapped = true };
+	struct pt_iommu_collect_args collect = {
+		.check_mapped = true,
+		.pending.iotlb_gather = iotlb_gather,
+		.pending.free_list = IOMMU_PAGES_LIST_INIT(
+			collect.pending.free_list),
+	};
 	int ret;
 
 	pts.index = start_pts->index;
 	pts.end_index = start_pts->index + step;
 	for (; _pt_iter_load(&pts); pt_next_entry(&pts)) {
 		if (pts.type == PT_ENTRY_TABLE) {
-			collect.free_list =
-				IOMMU_PAGES_LIST_INIT(collect.free_list);
 			ret = pt_walk_descend_all(&pts, __collect_tables,
 						  &collect);
 			if (ret)
@@ -514,12 +531,11 @@ static int clear_contig(const struct pt_state *start_pts,
 			pt_clear_entries(&pts, ilog2(1));
 			flush_writes_item(&pts);
 
-			iommu_pages_list_add(&collect.free_list,
-					     pt_table_ptr(&pts));
-			gather_range_pages(
-				iotlb_gather, iommu_table, range.va,
-				log2_to_int(pt_table_item_lg2sz(&pts)),
-				&collect.free_list);
+			gather_add_table(&collect.pending, &pts,
+					 pts.table_lower);
+			gather_range_pending(
+				&collect.pending, iommu_table, range.va,
+				log2_to_int(pt_table_item_lg2sz(&pts)));
 		} else if (pts.type != PT_ENTRY_EMPTY) {
 			return -EADDRINUSE;
 		}
@@ -962,7 +978,7 @@ static int NS(map_range)(struct pt_iommu *iommu_table, dma_addr_t iova,
 }
 
 struct pt_unmap_args {
-	struct iommu_pages_list free_list;
+	struct iommupt_pending_gather pending;
 	pt_vaddr_t unmapped;
 };
 
@@ -1025,8 +1041,8 @@ static __maybe_unused int __unmap_range(struct pt_range *range, void *arg,
 			 * succeed in clearing the lower table levels.
 			 */
 			if (fully_covered) {
-				iommu_pages_list_add(&unmap->free_list,
-						     pts.table_lower);
+				gather_add_table(&unmap->pending, &pts,
+						 pts.table_lower);
 				pt_clear_entries(&pts, ilog2(1));
 				if (pts.index < flush_start_index)
 					flush_start_index = pts.index;
@@ -1065,8 +1081,11 @@ static size_t NS(unmap_range)(struct pt_iommu *iommu_table, dma_addr_t iova,
 			      dma_addr_t len,
 			      struct iommu_iotlb_gather *iotlb_gather)
 {
-	struct pt_unmap_args unmap = { .free_list = IOMMU_PAGES_LIST_INIT(
-					       unmap.free_list) };
+	struct pt_unmap_args unmap = {
+		.pending.iotlb_gather = iotlb_gather,
+		.pending.free_list = IOMMU_PAGES_LIST_INIT(
+			unmap.pending.free_list),
+	};
 	struct pt_range range;
 	int ret;
 
@@ -1076,8 +1095,7 @@ static size_t NS(unmap_range)(struct pt_iommu *iommu_table, dma_addr_t iova,
 
 	pt_walk_range(&range, __unmap_range, &unmap);
 
-	gather_range_pages(iotlb_gather, iommu_table, iova, unmap.unmapped,
-			   &unmap.free_list);
+	gather_range_pending(&unmap.pending, iommu_table, iova, unmap.unmapped);
 
 	return unmap.unmapped;
 }
@@ -1111,10 +1129,11 @@ static void NS(deinit)(struct pt_iommu *iommu_table)
 	struct pt_common *common = common_from_iommu(iommu_table);
 	struct pt_range range = pt_all_range(common);
 	struct pt_iommu_collect_args collect = {
-		.free_list = IOMMU_PAGES_LIST_INIT(collect.free_list),
+		.pending.free_list = IOMMU_PAGES_LIST_INIT(
+			collect.pending.free_list),
 	};
 
-	iommu_pages_list_add(&collect.free_list, range.top_table);
+	iommu_pages_list_add(&collect.pending.free_list, range.top_table);
 	pt_walk_range(&range, __collect_tables, &collect);
 
 	/*
@@ -1122,9 +1141,9 @@ static void NS(deinit)(struct pt_iommu *iommu_table)
 	 * and invalidated any caching referring to this memory.
 	 */
 	if (pt_feature(common, PT_FEAT_DMA_INCOHERENT))
-		iommu_pages_stop_incoherent_list(&collect.free_list,
+		iommu_pages_stop_incoherent_list(&collect.pending.free_list,
 						 iommu_table->iommu_device);
-	iommu_put_pages_list(&collect.free_list);
+	iommu_put_pages_list(&collect.pending.free_list);
 }
 
 static const struct pt_iommu_ops NS(ops) = {
-- 
2.43.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 3/7] iommupt: Add PT_FEAT_DETAILED_GATHER

[Jason Gunthorpe]

Generating the ARM SMMUv3 and RISC-V invalidation commands optimally
requires some additional details from iommupt:

- leaf_levels_bitmap is used to compute the ARM Range Invalidation
  Table Top Level hint

- leaf_levels_bitmap is also used to compute the stride when
  generating single invalidations to invalidate once per leaf

- table_levels_bitmap also computes the ARM TTL for future cases when
  there are no leaves

Put these under a feature since only two drivers need to calculate
them.

This is also useful for the coming kunit iotlb invalidation test to
know more about what invalidation is happening.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/generic_pt/iommu_pt.h | 23 ++++++++++++++++++++++
 include/linux/generic_pt/common.h   |  5 +++++
 include/linux/iommu.h               | 30 ++++++++++++++++++++++++-----
 3 files changed, 53 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/generic_pt/iommu_pt.h b/drivers/iommu/generic_pt/iommu_pt.h
index 5ec135cf43e2d6..61c6d79c712cfa 100644
--- a/drivers/iommu/generic_pt/iommu_pt.h
+++ b/drivers/iommu/generic_pt/iommu_pt.h
@@ -43,6 +43,8 @@ static void flush_writes_item(const struct pt_state *pts)
 struct iommupt_pending_gather {
 	struct iommu_iotlb_gather *iotlb_gather;
 	struct iommu_pages_list free_list;
+	u8 leaf_levels_bitmap;
+	u8 table_levels_bitmap;
 };
 
 static void gather_add_table(struct iommupt_pending_gather *pending,
@@ -50,6 +52,17 @@ static void gather_add_table(struct iommupt_pending_gather *pending,
 			     struct pt_table_p *table)
 {
 	iommu_pages_list_add(&pending->free_list, table);
+	if (pts_feature(pts, PT_FEAT_DETAILED_GATHER))
+		pending->table_levels_bitmap |= BIT(pts->level);
+}
+
+static void gather_add_leaf(struct iommupt_pending_gather *pending,
+			    const struct pt_state *pts)
+{
+	if (!pts_feature(pts, PT_FEAT_DETAILED_GATHER))
+		return;
+
+	pending->leaf_levels_bitmap |= BIT(pts->level);
 }
 
 static void gather_range_pending(struct iommupt_pending_gather *pending,
@@ -86,6 +99,15 @@ static void gather_range_pending(struct iommupt_pending_gather *pending,
 
 	iommu_pages_list_splice(&pending->free_list, &iotlb_gather->freelist);
 	INIT_LIST_HEAD(&pending->free_list.pages);
+
+	if (pt_feature(common, PT_FEAT_DETAILED_GATHER)) {
+		iotlb_gather->pt.leaf_levels_bitmap |=
+			pending->leaf_levels_bitmap;
+		iotlb_gather->pt.table_levels_bitmap |=
+			pending->table_levels_bitmap;
+		pending->leaf_levels_bitmap = 0;
+		pending->table_levels_bitmap = 0;
+	}
 }
 
 #define DOMAIN_NS(op) CONCATENATE(CONCATENATE(pt_iommu_, PTPFX), op)
@@ -1059,6 +1081,7 @@ static __maybe_unused int __unmap_range(struct pt_range *range, void *arg,
 			 */
 			num_contig_lg2 = pt_entry_num_contig_lg2(&pts);
 			pt_clear_entries(&pts, num_contig_lg2);
+			gather_add_leaf(&unmap->pending, &pts);
 			num_oas += log2_to_int(num_contig_lg2);
 			if (pts.index < flush_start_index)
 				flush_start_index = pts.index;
diff --git a/include/linux/generic_pt/common.h b/include/linux/generic_pt/common.h
index fc5d0b5edadc08..230347f00353bc 100644
--- a/include/linux/generic_pt/common.h
+++ b/include/linux/generic_pt/common.h
@@ -134,6 +134,11 @@ enum pt_features {
 	 * significant amount of page table.
 	 */
 	PT_FEAT_FLUSH_RANGE_NO_GAPS,
+	/**
+	 * @PT_FEAT_DETAILED_GATHER: Fill in the struct iommu_iotlb_gather pt
+	 * sub structure with information about which levels were changed.
+	 */
+	PT_FEAT_DETAILED_GATHER,
 	/* private: */
 	PT_FEAT_FMT_START,
 };
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 47a559616706a1..0abe234fd12c33 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -360,11 +360,31 @@ struct iommu_iotlb_gather {
 	 *       flushed (inclusive)
 	 */
 	unsigned long		end;
-	/**
-	 * @pgsize: The interval at which to perform the flush, only used
-	 *          by arm-smmu-v3
-	 */
-	size_t			pgsize;
+
+	union {
+		/**
+		 * @pgsize: The interval at which to perform the flush, only
+		 *          used by arm-smmu-v3
+		 */
+		size_t pgsize;
+		struct {
+			/**
+			 * @pt.leaf_levels_bitmap: Bitmap of generic_pt
+			 * levels where leaf entries were unmapped. Bit 0
+			 * means the leaf only level. If 0 no leafs
+			 * were unmapped.
+			 */
+			u8 leaf_levels_bitmap;
+			/**
+			 * @pt.table_levels_bitmap: Bitmap of generic_pt levels
+			 * of table entries that were removed. Bit 0 is never
+			 * set, bit 1 means a table of all leafs was removed.
+			 * When freelist is empty this must be 0.
+			 */
+			u8 table_levels_bitmap;
+		} pt;
+	};
+
 	/**
 	 * @freelist: Removed pages to free after sync, only used by
 	 *            iommupt
-- 
2.43.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 4/7] iommu/riscv: Enable PT_FEAT_DETAILED_GATHER and pass gather to iotlb_inval

[Jason Gunthorpe]

RISC-V can use the information from PT_FEAT_DETAILED_GATHER to
compute the best stride to generate the single TLB invalidations.

Pass the gather down to the lower functions and create a full-range
gather for the flush-all callback.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/generic_pt/fmt/iommu_riscv64.c |  4 +-
 drivers/iommu/riscv/iommu.c                  | 44 +++++++++++++-------
 2 files changed, 31 insertions(+), 17 deletions(-)

diff --git a/drivers/iommu/generic_pt/fmt/iommu_riscv64.c b/drivers/iommu/generic_pt/fmt/iommu_riscv64.c
index cbf60fffa9bf7d..b18fc4d109f538 100644
--- a/drivers/iommu/generic_pt/fmt/iommu_riscv64.c
+++ b/drivers/iommu/generic_pt/fmt/iommu_riscv64.c
@@ -6,6 +6,8 @@
 #define PT_FMT_VARIANT 64
 #define PT_SUPPORTED_FEATURES                                  \
 	(BIT(PT_FEAT_SIGN_EXTEND) | BIT(PT_FEAT_FLUSH_RANGE) | \
-	 BIT(PT_FEAT_RISCV_SVNAPOT_64K))
+	 BIT(PT_FEAT_RISCV_SVNAPOT_64K) |                      \
+	 BIT(PT_FEAT_DETAILED_GATHER))
+#define PT_FORCE_ENABLED_FEATURES BIT(PT_FEAT_DETAILED_GATHER)
 
 #include "iommu_template.h"
diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
index a31f50bbad3535..556d5dc9c58681 100644
--- a/drivers/iommu/riscv/iommu.c
+++ b/drivers/iommu/riscv/iommu.c
@@ -929,12 +929,28 @@ static void riscv_iommu_bond_unlink(struct riscv_iommu_domain *domain,
 #define RISCV_IOMMU_IOTLB_INVAL_LIMIT	(2 << 20)
 
 static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
-				    unsigned long start, unsigned long end)
+				    struct iommu_iotlb_gather *gather)
 {
+	unsigned long start;
+	unsigned long end;
 	struct riscv_iommu_bond *bond;
 	struct riscv_iommu_device *iommu, *prev;
 	struct riscv_iommu_command cmd;
 
+	/*
+	 * When non-leaf page table entries were changed, the base spec
+	 * requires a full PSCID invalidation (AV=0) since there is no
+	 * way to do targeted non-leaf invalidation without the NL
+	 * extension. Force global invalidation to preserve correctness.
+	 */
+	if (gather->pt.table_levels_bitmap) {
+		start = 0;
+		end = ULONG_MAX;
+	} else {
+		start = gather->start;
+		end = gather->end;
+	}
+
 	/*
 	 * For each IOMMU linked with this protection domain (via bonds->dev),
 	 * an IOTLB invaliation command will be submitted and executed.
@@ -1145,8 +1161,14 @@ static void riscv_iommu_iodir_update(struct riscv_iommu_device *iommu,
 static void riscv_iommu_iotlb_flush_all(struct iommu_domain *iommu_domain)
 {
 	struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
+	struct iommu_iotlb_gather gather = {
+		.start = 0,
+		.end = ULONG_MAX,
+		.pt.leaf_levels_bitmap = 0xFF,
+		.pt.table_levels_bitmap = 0xFE,
+	};
 
-	riscv_iommu_iotlb_inval(domain, 0, ULONG_MAX);
+	riscv_iommu_iotlb_inval(domain, &gather);
 }
 
 static void riscv_iommu_iotlb_sync(struct iommu_domain *iommu_domain,
@@ -1154,19 +1176,8 @@ static void riscv_iommu_iotlb_sync(struct iommu_domain *iommu_domain,
 {
 	struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
 
-	if (iommu_pages_list_empty(&gather->freelist)) {
-		riscv_iommu_iotlb_inval(domain, gather->start, gather->end);
-	} else {
-		/*
-		 * In 1.0 spec version, the smallest scope we can use to
-		 * invalidate all levels of page table (i.e. leaf and non-leaf)
-		 * is an invalidate-all-PSCID IOTINVAL.VMA with AV=0.
-		 * This will be updated with hardware support for
-		 * capability.NL (non-leaf) IOTINVAL command.
-		 */
-		riscv_iommu_iotlb_inval(domain, 0, ULONG_MAX);
-		iommu_put_pages_list(&gather->freelist);
-	}
+	riscv_iommu_iotlb_inval(domain, gather);
+	iommu_put_pages_list(&gather->freelist);
 }
 
 static void riscv_iommu_free_paging_domain(struct iommu_domain *iommu_domain)
@@ -1267,7 +1278,8 @@ static struct iommu_domain *riscv_iommu_alloc_paging_domain(struct device *dev)
 	 */
 	cfg.common.features = BIT(PT_FEAT_SIGN_EXTEND) |
 			      BIT(PT_FEAT_FLUSH_RANGE) |
-			      BIT(PT_FEAT_RISCV_SVNAPOT_64K);
+			      BIT(PT_FEAT_RISCV_SVNAPOT_64K) |
+			      BIT(PT_FEAT_DETAILED_GATHER);
 	domain->riscvpt.iommu.nid = dev_to_node(iommu->dev);
 	domain->domain.ops = &riscv_iommu_paging_domain_ops;
 
-- 
2.43.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 4/7] iommu/riscv: Enable PT_FEAT_DETAILED_GATHER and pass gather to iotlb_inval

[Tomasz Jeznach]

On Fri, Apr 10, 2026 at 8:57 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> RISC-V can use the information from PT_FEAT_DETAILED_GATHER to
> compute the best stride to generate the single TLB invalidations.
>
> Pass the gather down to the lower functions and create a full-range
> gather for the flush-all callback.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/iommu/generic_pt/fmt/iommu_riscv64.c |  4 +-
>  drivers/iommu/riscv/iommu.c                  | 44 +++++++++++++-------
>  2 files changed, 31 insertions(+), 17 deletions(-)
>
> diff --git a/drivers/iommu/generic_pt/fmt/iommu_riscv64.c b/drivers/iommu/generic_pt/fmt/iommu_riscv64.c
> index cbf60fffa9bf7d..b18fc4d109f538 100644
> --- a/drivers/iommu/generic_pt/fmt/iommu_riscv64.c
> +++ b/drivers/iommu/generic_pt/fmt/iommu_riscv64.c
> @@ -6,6 +6,8 @@
>  #define PT_FMT_VARIANT 64
>  #define PT_SUPPORTED_FEATURES                                  \
>         (BIT(PT_FEAT_SIGN_EXTEND) | BIT(PT_FEAT_FLUSH_RANGE) | \
> -        BIT(PT_FEAT_RISCV_SVNAPOT_64K))
> +        BIT(PT_FEAT_RISCV_SVNAPOT_64K) |                      \
> +        BIT(PT_FEAT_DETAILED_GATHER))
> +#define PT_FORCE_ENABLED_FEATURES BIT(PT_FEAT_DETAILED_GATHER)
>
>  #include "iommu_template.h"
> diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> index a31f50bbad3535..556d5dc9c58681 100644
> --- a/drivers/iommu/riscv/iommu.c
> +++ b/drivers/iommu/riscv/iommu.c
> @@ -929,12 +929,28 @@ static void riscv_iommu_bond_unlink(struct riscv_iommu_domain *domain,
>  #define RISCV_IOMMU_IOTLB_INVAL_LIMIT  (2 << 20)
>
>  static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
> -                                   unsigned long start, unsigned long end)
> +                                   struct iommu_iotlb_gather *gather)
>  {
> +       unsigned long start;
> +       unsigned long end;
>         struct riscv_iommu_bond *bond;
>         struct riscv_iommu_device *iommu, *prev;
>         struct riscv_iommu_command cmd;
>
> +       /*
> +        * When non-leaf page table entries were changed, the base spec
> +        * requires a full PSCID invalidation (AV=0) since there is no
> +        * way to do targeted non-leaf invalidation without the NL
> +        * extension. Force global invalidation to preserve correctness.
> +        */
> +       if (gather->pt.table_levels_bitmap) {
> +               start = 0;
> +               end = ULONG_MAX;
> +       } else {
> +               start = gather->start;
> +               end = gather->end;
> +       }
> +
>         /*
>          * For each IOMMU linked with this protection domain (via bonds->dev),
>          * an IOTLB invaliation command will be submitted and executed.
> @@ -1145,8 +1161,14 @@ static void riscv_iommu_iodir_update(struct riscv_iommu_device *iommu,
>  static void riscv_iommu_iotlb_flush_all(struct iommu_domain *iommu_domain)
>  {
>         struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
> +       struct iommu_iotlb_gather gather = {
> +               .start = 0,
> +               .end = ULONG_MAX,
> +               .pt.leaf_levels_bitmap = 0xFF,
> +               .pt.table_levels_bitmap = 0xFE,
> +       };
>
> -       riscv_iommu_iotlb_inval(domain, 0, ULONG_MAX);
> +       riscv_iommu_iotlb_inval(domain, &gather);
>  }
>
>  static void riscv_iommu_iotlb_sync(struct iommu_domain *iommu_domain,
> @@ -1154,19 +1176,8 @@ static void riscv_iommu_iotlb_sync(struct iommu_domain *iommu_domain,
>  {
>         struct riscv_iommu_domain *domain = iommu_domain_to_riscv(iommu_domain);
>
> -       if (iommu_pages_list_empty(&gather->freelist)) {
> -               riscv_iommu_iotlb_inval(domain, gather->start, gather->end);
> -       } else {
> -               /*
> -                * In 1.0 spec version, the smallest scope we can use to
> -                * invalidate all levels of page table (i.e. leaf and non-leaf)
> -                * is an invalidate-all-PSCID IOTINVAL.VMA with AV=0.
> -                * This will be updated with hardware support for
> -                * capability.NL (non-leaf) IOTINVAL command.
> -                */
> -               riscv_iommu_iotlb_inval(domain, 0, ULONG_MAX);
> -               iommu_put_pages_list(&gather->freelist);
> -       }
> +       riscv_iommu_iotlb_inval(domain, gather);
> +       iommu_put_pages_list(&gather->freelist);
>  }
>
>  static void riscv_iommu_free_paging_domain(struct iommu_domain *iommu_domain)
> @@ -1267,7 +1278,8 @@ static struct iommu_domain *riscv_iommu_alloc_paging_domain(struct device *dev)
>          */
>         cfg.common.features = BIT(PT_FEAT_SIGN_EXTEND) |
>                               BIT(PT_FEAT_FLUSH_RANGE) |
> -                             BIT(PT_FEAT_RISCV_SVNAPOT_64K);
> +                             BIT(PT_FEAT_RISCV_SVNAPOT_64K) |
> +                             BIT(PT_FEAT_DETAILED_GATHER);
>         domain->riscvpt.iommu.nid = dev_to_node(iommu->dev);
>         domain->domain.ops = &riscv_iommu_paging_domain_ops;
>
> --
> 2.43.0
>

Reviewed-by: Tomasz Jeznach <tjeznach@rivosinc.com>

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 5/7] iommu/riscv: Compute best stride for single invalidation

[Jason Gunthorpe]

Replace the per-page IOTLB invalidation loop with stride-based
invalidation that uses the level bitmaps from iommu_iotlb_gather.

Pre-calculate the invalidation information before running over the
bonds loop as it is the same for every entry.

The lowest set bit in the PT_FEAT_DETAILED_GATHER bitmaps indicates
the stride. This design ignores the SVNAPOT contiguous pages on the
assumption that they still have to be individually invalidated like
ARM requires, though it is not clear from the spec.

Replace the 2M cutoff for global invalidation with a 512 command
limit. This is the same for a 4k stride and now scales with the
stride size.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/riscv/iommu.c | 103 +++++++++++++++++++++++++-----------
 1 file changed, 71 insertions(+), 32 deletions(-)

diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
index 556d5dc9c58681..fd9c5294dbc082 100644
--- a/drivers/iommu/riscv/iommu.c
+++ b/drivers/iommu/riscv/iommu.c
@@ -920,22 +920,58 @@ static void riscv_iommu_bond_unlink(struct riscv_iommu_domain *domain,
 	}
 }
 
-/*
- * Send IOTLB.INVAL for whole address space for ranges larger than 2MB.
- * This limit will be replaced with range invalidations, if supported by
- * the hardware, when RISC-V IOMMU architecture specification update for
- * range invalidations update will be available.
- */
-#define RISCV_IOMMU_IOTLB_INVAL_LIMIT	(2 << 20)
+struct riscv_iommu_tlbi {
+	u64 start;
+	u64 last;
+	bool non_leaf;
+	struct {
+		bool use_global;
+		u8 stride_lg2;
+		unsigned int num;
+	} single;
+};
 
-static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
-				    struct iommu_iotlb_gather *gather)
+static void riscv_iommu_tlbi_calc(struct riscv_iommu_tlbi *tlbi,
+				  struct iommu_iotlb_gather *gather)
+{
+	u8 combined = gather->pt.leaf_levels_bitmap |
+		      gather->pt.table_levels_bitmap;
+	u64 num;
+
+	tlbi->non_leaf = gather->pt.table_levels_bitmap != 0;
+	tlbi->start = gather->start;
+	tlbi->last = gather->end;
+
+	/* No level information available */
+	if (!combined) {
+		tlbi->single.use_global = true;
+		return;
+	}
+
+	/*
+	 * Calculate stride from the lowest changed level. RISC-V uses 4KiB
+	 * granule with 9 bits per level.
+	 */
+	tlbi->single.stride_lg2 = 9 * __ffs(combined) + 12;
+	num = (tlbi->last - tlbi->start + 1) >> tlbi->single.stride_lg2;
+	if (!num || num > 512) {
+		tlbi->single.use_global = true;
+	} else {
+		tlbi->single.num = num;
+		tlbi->single.use_global = false;
+	}
+}
+
+static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
+					  int pscid,
+					  struct riscv_iommu_tlbi *tlbi)
 {
-	unsigned long start;
-	unsigned long end;
-	struct riscv_iommu_bond *bond;
-	struct riscv_iommu_device *iommu, *prev;
 	struct riscv_iommu_command cmd;
+	unsigned long iova;
+	unsigned int i;
+
+	riscv_iommu_cmd_inval_vma(&cmd);
+	riscv_iommu_cmd_inval_set_pscid(&cmd, pscid);
 
 	/*
 	 * When non-leaf page table entries were changed, the base spec
@@ -943,13 +979,28 @@ static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
 	 * way to do targeted non-leaf invalidation without the NL
 	 * extension. Force global invalidation to preserve correctness.
 	 */
-	if (gather->pt.table_levels_bitmap) {
-		start = 0;
-		end = ULONG_MAX;
-	} else {
-		start = gather->start;
-		end = gather->end;
+	if (tlbi->single.use_global || tlbi->non_leaf)
+		goto global;
+
+	iova = tlbi->start;
+	for (i = 0; i < tlbi->single.num; i++) {
+		riscv_iommu_cmd_inval_set_addr(&cmd, iova);
+		riscv_iommu_cmd_send(iommu, &cmd);
+		iova += 1ULL << tlbi->single.stride_lg2;
 	}
+	return;
+global:
+	riscv_iommu_cmd_send(iommu, &cmd);
+}
+
+static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
+				    struct iommu_iotlb_gather *gather)
+{
+	struct riscv_iommu_device *iommu, *prev;
+	struct riscv_iommu_bond *bond;
+	struct riscv_iommu_tlbi tlbi;
+
+	riscv_iommu_tlbi_calc(&tlbi, gather);
 
 	/*
 	 * For each IOMMU linked with this protection domain (via bonds->dev),
@@ -990,19 +1041,7 @@ static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
 		if (iommu == prev)
 			continue;
 
-		riscv_iommu_cmd_inval_vma(&cmd);
-		riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid);
-		if (end - start < RISCV_IOMMU_IOTLB_INVAL_LIMIT - 1) {
-			unsigned long iova = start;
-
-			do {
-				riscv_iommu_cmd_inval_set_addr(&cmd, iova);
-				riscv_iommu_cmd_send(iommu, &cmd);
-			} while (!check_add_overflow(iova, PAGE_SIZE, &iova) &&
-				 iova < end);
-		} else {
-			riscv_iommu_cmd_send(iommu, &cmd);
-		}
+		riscv_iommu_iotlb_inval_iommu(iommu, domain->pscid, &tlbi);
 		prev = iommu;
 	}
 
-- 
2.43.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 5/7] iommu/riscv: Compute best stride for single invalidation

[Tomasz Jeznach]

On Fri, Apr 10, 2026 at 8:57 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Replace the per-page IOTLB invalidation loop with stride-based
> invalidation that uses the level bitmaps from iommu_iotlb_gather.
>
> Pre-calculate the invalidation information before running over the
> bonds loop as it is the same for every entry.
>
> The lowest set bit in the PT_FEAT_DETAILED_GATHER bitmaps indicates
> the stride. This design ignores the SVNAPOT contiguous pages on the
> assumption that they still have to be individually invalidated like
> ARM requires, though it is not clear from the spec.
>
> Replace the 2M cutoff for global invalidation with a 512 command
> limit. This is the same for a 4k stride and now scales with the
> stride size.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/iommu/riscv/iommu.c | 103 +++++++++++++++++++++++++-----------
>  1 file changed, 71 insertions(+), 32 deletions(-)
>
> diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> index 556d5dc9c58681..fd9c5294dbc082 100644
> --- a/drivers/iommu/riscv/iommu.c
> +++ b/drivers/iommu/riscv/iommu.c
> @@ -920,22 +920,58 @@ static void riscv_iommu_bond_unlink(struct riscv_iommu_domain *domain,
>         }
>  }
>
> -/*
> - * Send IOTLB.INVAL for whole address space for ranges larger than 2MB.
> - * This limit will be replaced with range invalidations, if supported by
> - * the hardware, when RISC-V IOMMU architecture specification update for
> - * range invalidations update will be available.
> - */
> -#define RISCV_IOMMU_IOTLB_INVAL_LIMIT  (2 << 20)
> +struct riscv_iommu_tlbi {
> +       u64 start;
> +       u64 last;
> +       bool non_leaf;
> +       struct {
> +               bool use_global;
> +               u8 stride_lg2;
> +               unsigned int num;
> +       } single;
> +};
>
> -static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
> -                                   struct iommu_iotlb_gather *gather)
> +static void riscv_iommu_tlbi_calc(struct riscv_iommu_tlbi *tlbi,
> +                                 struct iommu_iotlb_gather *gather)
> +{
> +       u8 combined = gather->pt.leaf_levels_bitmap |
> +                     gather->pt.table_levels_bitmap;
> +       u64 num;
> +
> +       tlbi->non_leaf = gather->pt.table_levels_bitmap != 0;
> +       tlbi->start = gather->start;
> +       tlbi->last = gather->end;
> +
> +       /* No level information available */
> +       if (!combined) {
> +               tlbi->single.use_global = true;
> +               return;
> +       }
> +
> +       /*
> +        * Calculate stride from the lowest changed level. RISC-V uses 4KiB
> +        * granule with 9 bits per level.
> +        */
> +       tlbi->single.stride_lg2 = 9 * __ffs(combined) + 12;
> +       num = (tlbi->last - tlbi->start + 1) >> tlbi->single.stride_lg2;
> +       if (!num || num > 512) {
> +               tlbi->single.use_global = true;
> +       } else {
> +               tlbi->single.num = num;
> +               tlbi->single.use_global = false;
> +       }
> +}
> +
> +static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
> +                                         int pscid,
> +                                         struct riscv_iommu_tlbi *tlbi)
>  {
> -       unsigned long start;
> -       unsigned long end;
> -       struct riscv_iommu_bond *bond;
> -       struct riscv_iommu_device *iommu, *prev;
>         struct riscv_iommu_command cmd;
> +       unsigned long iova;
> +       unsigned int i;
> +
> +       riscv_iommu_cmd_inval_vma(&cmd);
> +       riscv_iommu_cmd_inval_set_pscid(&cmd, pscid);
>
>         /*
>          * When non-leaf page table entries were changed, the base spec
> @@ -943,13 +979,28 @@ static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
>          * way to do targeted non-leaf invalidation without the NL
>          * extension. Force global invalidation to preserve correctness.
>          */
> -       if (gather->pt.table_levels_bitmap) {
> -               start = 0;
> -               end = ULONG_MAX;
> -       } else {
> -               start = gather->start;
> -               end = gather->end;
> +       if (tlbi->single.use_global || tlbi->non_leaf)
> +               goto global;
> +
> +       iova = tlbi->start;
> +       for (i = 0; i < tlbi->single.num; i++) {
> +               riscv_iommu_cmd_inval_set_addr(&cmd, iova);
> +               riscv_iommu_cmd_send(iommu, &cmd);
> +               iova += 1ULL << tlbi->single.stride_lg2;
>         }
> +       return;
> +global:
> +       riscv_iommu_cmd_send(iommu, &cmd);
> +}
> +
> +static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
> +                                   struct iommu_iotlb_gather *gather)
> +{
> +       struct riscv_iommu_device *iommu, *prev;
> +       struct riscv_iommu_bond *bond;
> +       struct riscv_iommu_tlbi tlbi;
> +
> +       riscv_iommu_tlbi_calc(&tlbi, gather);
>
>         /*
>          * For each IOMMU linked with this protection domain (via bonds->dev),
> @@ -990,19 +1041,7 @@ static void riscv_iommu_iotlb_inval(struct riscv_iommu_domain *domain,
>                 if (iommu == prev)
>                         continue;
>
> -               riscv_iommu_cmd_inval_vma(&cmd);
> -               riscv_iommu_cmd_inval_set_pscid(&cmd, domain->pscid);
> -               if (end - start < RISCV_IOMMU_IOTLB_INVAL_LIMIT - 1) {
> -                       unsigned long iova = start;
> -
> -                       do {
> -                               riscv_iommu_cmd_inval_set_addr(&cmd, iova);
> -                               riscv_iommu_cmd_send(iommu, &cmd);
> -                       } while (!check_add_overflow(iova, PAGE_SIZE, &iova) &&
> -                                iova < end);
> -               } else {
> -                       riscv_iommu_cmd_send(iommu, &cmd);
> -               }
> +               riscv_iommu_iotlb_inval_iommu(iommu, domain->pscid, &tlbi);
>                 prev = iommu;
>         }
>
> --
> 2.43.0
>

Reviewed-by: Tomasz Jeznach <tjeznach@rivosinc.com>

Thank you
- Tomasz

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 6/7] iommu/riscv: Add RISCV_IOMMU_CAPABILITIES_NL

[Jason Gunthorpe]

Non-leaf invalidation allows the single invalidate command to also
clear the walk cache. If NL is available, set the NL bit if the
gather indicates tables have been changed. The stride is already
calculated properly.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/riscv/iommu-bits.h |  7 +++++++
 drivers/iommu/riscv/iommu.c      | 12 +++++++-----
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
index 29a0040b1c32ea..f01b49ac815586 100644
--- a/drivers/iommu/riscv/iommu-bits.h
+++ b/drivers/iommu/riscv/iommu-bits.h
@@ -63,6 +63,7 @@
 #define RISCV_IOMMU_CAPABILITIES_PD8		BIT_ULL(38)
 #define RISCV_IOMMU_CAPABILITIES_PD17		BIT_ULL(39)
 #define RISCV_IOMMU_CAPABILITIES_PD20		BIT_ULL(40)
+#define RISCV_IOMMU_CAPABILITIES_NL		BIT_ULL(42)
 
 /**
  * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
@@ -473,6 +474,7 @@ struct riscv_iommu_command {
 #define RISCV_IOMMU_CMD_IOTINVAL_PSCV		BIT_ULL(32)
 #define RISCV_IOMMU_CMD_IOTINVAL_GV		BIT_ULL(33)
 #define RISCV_IOMMU_CMD_IOTINVAL_GSCID		GENMASK_ULL(59, 44)
+#define RISCV_IOMMU_CMD_IOTINVAL_NL		BIT_ULL(34)
 /* dword1[61:10] is the 4K-aligned page address */
 #define RISCV_IOMMU_CMD_IOTINVAL_ADDR		GENMASK_ULL(61, 10)
 
@@ -724,6 +726,11 @@ static inline void riscv_iommu_cmd_inval_set_addr(struct riscv_iommu_command *cm
 	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV;
 }
 
+static inline void riscv_iommu_cmd_inval_set_nl(struct riscv_iommu_command *cmd)
+{
+	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_NL;
+}
+
 static inline void riscv_iommu_cmd_inval_set_pscid(struct riscv_iommu_command *cmd,
 						   int pscid)
 {
diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
index fd9c5294dbc082..ea14630430451a 100644
--- a/drivers/iommu/riscv/iommu.c
+++ b/drivers/iommu/riscv/iommu.c
@@ -966,6 +966,8 @@ static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
 					  int pscid,
 					  struct riscv_iommu_tlbi *tlbi)
 {
+	bool use_nl = tlbi->non_leaf &&
+		      (iommu->caps & RISCV_IOMMU_CAPABILITIES_NL);
 	struct riscv_iommu_command cmd;
 	unsigned long iova;
 	unsigned int i;
@@ -974,17 +976,17 @@ static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
 	riscv_iommu_cmd_inval_set_pscid(&cmd, pscid);
 
 	/*
-	 * When non-leaf page table entries were changed, the base spec
-	 * requires a full PSCID invalidation (AV=0) since there is no
-	 * way to do targeted non-leaf invalidation without the NL
-	 * extension. Force global invalidation to preserve correctness.
+	 * If non-leaf entries were changed and the IOMMU doesn't
+	 * support NL, we must fall back to global invalidation (AV=0).
 	 */
-	if (tlbi->single.use_global || tlbi->non_leaf)
+	if (tlbi->single.use_global || (tlbi->non_leaf && !use_nl))
 		goto global;
 
 	iova = tlbi->start;
 	for (i = 0; i < tlbi->single.num; i++) {
 		riscv_iommu_cmd_inval_set_addr(&cmd, iova);
+		if (use_nl)
+			riscv_iommu_cmd_inval_set_nl(&cmd);
 		riscv_iommu_cmd_send(iommu, &cmd);
 		iova += 1ULL << tlbi->single.stride_lg2;
 	}
-- 
2.43.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 6/7] iommu/riscv: Add RISCV_IOMMU_CAPABILITIES_NL

[Tomasz Jeznach]

On Fri, Apr 10, 2026 at 8:57 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Non-leaf invalidation allows the single invalidate command to also
> clear the walk cache. If NL is available, set the NL bit if the
> gather indicates tables have been changed. The stride is already
> calculated properly.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/iommu/riscv/iommu-bits.h |  7 +++++++
>  drivers/iommu/riscv/iommu.c      | 12 +++++++-----
>  2 files changed, 14 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
> index 29a0040b1c32ea..f01b49ac815586 100644
> --- a/drivers/iommu/riscv/iommu-bits.h
> +++ b/drivers/iommu/riscv/iommu-bits.h
> @@ -63,6 +63,7 @@
>  #define RISCV_IOMMU_CAPABILITIES_PD8           BIT_ULL(38)
>  #define RISCV_IOMMU_CAPABILITIES_PD17          BIT_ULL(39)
>  #define RISCV_IOMMU_CAPABILITIES_PD20          BIT_ULL(40)
> +#define RISCV_IOMMU_CAPABILITIES_NL            BIT_ULL(42)
>
>  /**
>   * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
> @@ -473,6 +474,7 @@ struct riscv_iommu_command {
>  #define RISCV_IOMMU_CMD_IOTINVAL_PSCV          BIT_ULL(32)
>  #define RISCV_IOMMU_CMD_IOTINVAL_GV            BIT_ULL(33)
>  #define RISCV_IOMMU_CMD_IOTINVAL_GSCID         GENMASK_ULL(59, 44)
> +#define RISCV_IOMMU_CMD_IOTINVAL_NL            BIT_ULL(34)
>  /* dword1[61:10] is the 4K-aligned page address */
>  #define RISCV_IOMMU_CMD_IOTINVAL_ADDR          GENMASK_ULL(61, 10)
>
> @@ -724,6 +726,11 @@ static inline void riscv_iommu_cmd_inval_set_addr(struct riscv_iommu_command *cm
>         cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV;
>  }
>
> +static inline void riscv_iommu_cmd_inval_set_nl(struct riscv_iommu_command *cmd)
> +{
> +       cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_NL;
> +}
> +
>  static inline void riscv_iommu_cmd_inval_set_pscid(struct riscv_iommu_command *cmd,
>                                                    int pscid)
>  {
> diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> index fd9c5294dbc082..ea14630430451a 100644
> --- a/drivers/iommu/riscv/iommu.c
> +++ b/drivers/iommu/riscv/iommu.c
> @@ -966,6 +966,8 @@ static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
>                                           int pscid,
>                                           struct riscv_iommu_tlbi *tlbi)
>  {
> +       bool use_nl = tlbi->non_leaf &&
> +                     (iommu->caps & RISCV_IOMMU_CAPABILITIES_NL);
>         struct riscv_iommu_command cmd;
>         unsigned long iova;
>         unsigned int i;
> @@ -974,17 +976,17 @@ static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
>         riscv_iommu_cmd_inval_set_pscid(&cmd, pscid);
>
>         /*
> -        * When non-leaf page table entries were changed, the base spec
> -        * requires a full PSCID invalidation (AV=0) since there is no
> -        * way to do targeted non-leaf invalidation without the NL
> -        * extension. Force global invalidation to preserve correctness.
> +        * If non-leaf entries were changed and the IOMMU doesn't
> +        * support NL, we must fall back to global invalidation (AV=0).
>          */
> -       if (tlbi->single.use_global || tlbi->non_leaf)
> +       if (tlbi->single.use_global || (tlbi->non_leaf && !use_nl))
>                 goto global;
>
>         iova = tlbi->start;
>         for (i = 0; i < tlbi->single.num; i++) {
>                 riscv_iommu_cmd_inval_set_addr(&cmd, iova);
> +               if (use_nl)
> +                       riscv_iommu_cmd_inval_set_nl(&cmd);
>                 riscv_iommu_cmd_send(iommu, &cmd);
>                 iova += 1ULL << tlbi->single.stride_lg2;
>         }
> --
> 2.43.0
>

Reviewed-by: Tomasz Jeznach <tjeznach@rivosinc.com>

Thank you,
- Tomasz

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

[PATCH 7/7] iommu/riscv: Add NAPOT range invalidation support

[Jason Gunthorpe]

Use the RISC-V IOMMU Address Range Invalidation extension
(capabilities.S, spec section 9.3) to invalidate an IOVA range with
a single IOTINVAL.VMA command using NAPOT-encoded addressing.

One iommu_iotlb_gather maps to one NAPOT invalidation command. The
smallest power-of-two aligned range covering the gather is used since
over-invalidation is always safe.

S and NL seem to be orthogonal in the spec, so if NL is not
supported then global invalidation is probably always going to happen
as wiping a large range without a table change is not common.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/iommu/riscv/iommu-bits.h | 17 +++++++++++++
 drivers/iommu/riscv/iommu.c      | 43 +++++++++++++++++++++++++++-----
 2 files changed, 54 insertions(+), 6 deletions(-)

diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
index f01b49ac815586..32b3ad3ac9ae59 100644
--- a/drivers/iommu/riscv/iommu-bits.h
+++ b/drivers/iommu/riscv/iommu-bits.h
@@ -64,6 +64,7 @@
 #define RISCV_IOMMU_CAPABILITIES_PD17		BIT_ULL(39)
 #define RISCV_IOMMU_CAPABILITIES_PD20		BIT_ULL(40)
 #define RISCV_IOMMU_CAPABILITIES_NL		BIT_ULL(42)
+#define RISCV_IOMMU_CAPABILITIES_S		BIT_ULL(43)
 
 /**
  * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
@@ -475,6 +476,7 @@ struct riscv_iommu_command {
 #define RISCV_IOMMU_CMD_IOTINVAL_GV		BIT_ULL(33)
 #define RISCV_IOMMU_CMD_IOTINVAL_GSCID		GENMASK_ULL(59, 44)
 #define RISCV_IOMMU_CMD_IOTINVAL_NL		BIT_ULL(34)
+#define RISCV_IOMMU_CMD_IOTINVAL_S		BIT_ULL(9)
 /* dword1[61:10] is the 4K-aligned page address */
 #define RISCV_IOMMU_CMD_IOTINVAL_ADDR		GENMASK_ULL(61, 10)
 
@@ -731,6 +733,21 @@ static inline void riscv_iommu_cmd_inval_set_nl(struct riscv_iommu_command *cmd)
 	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_NL;
 }
 
+/*
+ * Set NAPOT-encoded address for range invalidation (S=1).
+ * sz_lg2: log2 of total range in bytes, must be >= 13 (8KiB, 2 pages).
+ * addr must be naturally aligned to 2^sz_lg2.
+ */
+static inline void riscv_iommu_cmd_inval_set_napot(
+	struct riscv_iommu_command *cmd, u64 addr, unsigned int sz_lg2)
+{
+	u64 pfn = addr >> 12;
+
+	pfn |= BIT_U64(sz_lg2 - 13) - 1;
+	cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, pfn);
+	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV | RISCV_IOMMU_CMD_IOTINVAL_S;
+}
+
 static inline void riscv_iommu_cmd_inval_set_pscid(struct riscv_iommu_command *cmd,
 						   int pscid)
 {
diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
index ea14630430451a..41e26e267a7fd6 100644
--- a/drivers/iommu/riscv/iommu.c
+++ b/drivers/iommu/riscv/iommu.c
@@ -929,6 +929,10 @@ struct riscv_iommu_tlbi {
 		u8 stride_lg2;
 		unsigned int num;
 	} single;
+	struct {
+		u8 sz_lg2;
+		u64 addr;
+	} range;
 };
 
 static void riscv_iommu_tlbi_calc(struct riscv_iommu_tlbi *tlbi,
@@ -945,9 +949,23 @@ static void riscv_iommu_tlbi_calc(struct riscv_iommu_tlbi *tlbi,
 	/* No level information available */
 	if (!combined) {
 		tlbi->single.use_global = true;
+		tlbi->range.sz_lg2 = 0;
 		return;
 	}
 
+	/*
+	 * Calculate the smallest NAPOT range containing [start, last].
+	 * NAPOT encoding requires a power-of-two sized, naturally aligned
+	 * range. Over-invalidation is always safe.
+	 */
+	tlbi->range.sz_lg2 = fls64(tlbi->start ^ tlbi->last);
+	if (unlikely(tlbi->range.sz_lg2 >= 64)) {
+		tlbi->single.use_global = true;
+		tlbi->range.sz_lg2 = 0;
+		return;
+	}
+	tlbi->range.addr = tlbi->start & ~(BIT_U64(tlbi->range.sz_lg2) - 1);
+
 	/*
 	 * Calculate stride from the lowest changed level. RISC-V uses 4KiB
 	 * granule with 9 bits per level.
@@ -969,7 +987,6 @@ static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
 	bool use_nl = tlbi->non_leaf &&
 		      (iommu->caps & RISCV_IOMMU_CAPABILITIES_NL);
 	struct riscv_iommu_command cmd;
-	unsigned long iova;
 	unsigned int i;
 
 	riscv_iommu_cmd_inval_vma(&cmd);
@@ -979,16 +996,30 @@ static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
 	 * If non-leaf entries were changed and the IOMMU doesn't
 	 * support NL, we must fall back to global invalidation (AV=0).
 	 */
-	if (tlbi->single.use_global || (tlbi->non_leaf && !use_nl))
+	if (tlbi->non_leaf && !use_nl)
 		goto global;
 
-	iova = tlbi->start;
-	for (i = 0; i < tlbi->single.num; i++) {
-		riscv_iommu_cmd_inval_set_addr(&cmd, iova);
+	if (iommu->caps & RISCV_IOMMU_CAPABILITIES_S &&
+	    tlbi->range.sz_lg2 >= 13) {
+		riscv_iommu_cmd_inval_set_napot(&cmd, tlbi->range.addr,
+						tlbi->range.sz_lg2);
 		if (use_nl)
 			riscv_iommu_cmd_inval_set_nl(&cmd);
 		riscv_iommu_cmd_send(iommu, &cmd);
-		iova += 1ULL << tlbi->single.stride_lg2;
+	} else {
+		unsigned long iova;
+
+		if (tlbi->single.use_global)
+			goto global;
+
+		iova = tlbi->start;
+		for (i = 0; i < tlbi->single.num; i++) {
+			riscv_iommu_cmd_inval_set_addr(&cmd, iova);
+			if (use_nl)
+				riscv_iommu_cmd_inval_set_nl(&cmd);
+			riscv_iommu_cmd_send(iommu, &cmd);
+			iova += 1ULL << tlbi->single.stride_lg2;
+		}
 	}
 	return;
 global:
-- 
2.43.0


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 7/7] iommu/riscv: Add NAPOT range invalidation support

[Tomasz Jeznach]

On Fri, Apr 10, 2026 at 8:57 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> Use the RISC-V IOMMU Address Range Invalidation extension
> (capabilities.S, spec section 9.3) to invalidate an IOVA range with
> a single IOTINVAL.VMA command using NAPOT-encoded addressing.
>
> One iommu_iotlb_gather maps to one NAPOT invalidation command. The
> smallest power-of-two aligned range covering the gather is used since
> over-invalidation is always safe.
>
> S and NL seem to be orthogonal in the spec, so if NL is not
> supported then global invalidation is probably always going to happen
> as wiping a large range without a table change is not common.
>
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/iommu/riscv/iommu-bits.h | 17 +++++++++++++
>  drivers/iommu/riscv/iommu.c      | 43 +++++++++++++++++++++++++++-----
>  2 files changed, 54 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
> index f01b49ac815586..32b3ad3ac9ae59 100644
> --- a/drivers/iommu/riscv/iommu-bits.h
> +++ b/drivers/iommu/riscv/iommu-bits.h
> @@ -64,6 +64,7 @@
>  #define RISCV_IOMMU_CAPABILITIES_PD17          BIT_ULL(39)
>  #define RISCV_IOMMU_CAPABILITIES_PD20          BIT_ULL(40)
>  #define RISCV_IOMMU_CAPABILITIES_NL            BIT_ULL(42)
> +#define RISCV_IOMMU_CAPABILITIES_S             BIT_ULL(43)
>
>  /**
>   * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
> @@ -475,6 +476,7 @@ struct riscv_iommu_command {
>  #define RISCV_IOMMU_CMD_IOTINVAL_GV            BIT_ULL(33)
>  #define RISCV_IOMMU_CMD_IOTINVAL_GSCID         GENMASK_ULL(59, 44)
>  #define RISCV_IOMMU_CMD_IOTINVAL_NL            BIT_ULL(34)
> +#define RISCV_IOMMU_CMD_IOTINVAL_S             BIT_ULL(9)
>  /* dword1[61:10] is the 4K-aligned page address */
>  #define RISCV_IOMMU_CMD_IOTINVAL_ADDR          GENMASK_ULL(61, 10)
>
> @@ -731,6 +733,21 @@ static inline void riscv_iommu_cmd_inval_set_nl(struct riscv_iommu_command *cmd)
>         cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_NL;
>  }
>
> +/*
> + * Set NAPOT-encoded address for range invalidation (S=1).
> + * sz_lg2: log2 of total range in bytes, must be >= 13 (8KiB, 2 pages).
> + * addr must be naturally aligned to 2^sz_lg2.
> + */
> +static inline void riscv_iommu_cmd_inval_set_napot(
> +       struct riscv_iommu_command *cmd, u64 addr, unsigned int sz_lg2)
> +{
> +       u64 pfn = addr >> 12;
> +
> +       pfn |= BIT_U64(sz_lg2 - 13) - 1;
> +       cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, pfn);
> +       cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV | RISCV_IOMMU_CMD_IOTINVAL_S;
> +}
> +
>  static inline void riscv_iommu_cmd_inval_set_pscid(struct riscv_iommu_command *cmd,
>                                                    int pscid)
>  {
> diff --git a/drivers/iommu/riscv/iommu.c b/drivers/iommu/riscv/iommu.c
> index ea14630430451a..41e26e267a7fd6 100644
> --- a/drivers/iommu/riscv/iommu.c
> +++ b/drivers/iommu/riscv/iommu.c
> @@ -929,6 +929,10 @@ struct riscv_iommu_tlbi {
>                 u8 stride_lg2;
>                 unsigned int num;
>         } single;
> +       struct {
> +               u8 sz_lg2;
> +               u64 addr;
> +       } range;
>  };
>
>  static void riscv_iommu_tlbi_calc(struct riscv_iommu_tlbi *tlbi,
> @@ -945,9 +949,23 @@ static void riscv_iommu_tlbi_calc(struct riscv_iommu_tlbi *tlbi,
>         /* No level information available */
>         if (!combined) {
>                 tlbi->single.use_global = true;
> +               tlbi->range.sz_lg2 = 0;
>                 return;
>         }
>
> +       /*
> +        * Calculate the smallest NAPOT range containing [start, last].
> +        * NAPOT encoding requires a power-of-two sized, naturally aligned
> +        * range. Over-invalidation is always safe.
> +        */
> +       tlbi->range.sz_lg2 = fls64(tlbi->start ^ tlbi->last);
> +       if (unlikely(tlbi->range.sz_lg2 >= 64)) {
> +               tlbi->single.use_global = true;
> +               tlbi->range.sz_lg2 = 0;
> +               return;
> +       }
> +       tlbi->range.addr = tlbi->start & ~(BIT_U64(tlbi->range.sz_lg2) - 1);
> +
>         /*
>          * Calculate stride from the lowest changed level. RISC-V uses 4KiB
>          * granule with 9 bits per level.
> @@ -969,7 +987,6 @@ static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
>         bool use_nl = tlbi->non_leaf &&
>                       (iommu->caps & RISCV_IOMMU_CAPABILITIES_NL);
>         struct riscv_iommu_command cmd;
> -       unsigned long iova;
>         unsigned int i;
>
>         riscv_iommu_cmd_inval_vma(&cmd);
> @@ -979,16 +996,30 @@ static void riscv_iommu_iotlb_inval_iommu(struct riscv_iommu_device *iommu,
>          * If non-leaf entries were changed and the IOMMU doesn't
>          * support NL, we must fall back to global invalidation (AV=0).
>          */
> -       if (tlbi->single.use_global || (tlbi->non_leaf && !use_nl))
> +       if (tlbi->non_leaf && !use_nl)
>                 goto global;
>
> -       iova = tlbi->start;
> -       for (i = 0; i < tlbi->single.num; i++) {
> -               riscv_iommu_cmd_inval_set_addr(&cmd, iova);
> +       if (iommu->caps & RISCV_IOMMU_CAPABILITIES_S &&
> +           tlbi->range.sz_lg2 >= 13) {
> +               riscv_iommu_cmd_inval_set_napot(&cmd, tlbi->range.addr,
> +                                               tlbi->range.sz_lg2);
>                 if (use_nl)
>                         riscv_iommu_cmd_inval_set_nl(&cmd);
>                 riscv_iommu_cmd_send(iommu, &cmd);
> -               iova += 1ULL << tlbi->single.stride_lg2;
> +       } else {
> +               unsigned long iova;
> +
> +               if (tlbi->single.use_global)
> +                       goto global;
> +
> +               iova = tlbi->start;
> +               for (i = 0; i < tlbi->single.num; i++) {
> +                       riscv_iommu_cmd_inval_set_addr(&cmd, iova);
> +                       if (use_nl)
> +                               riscv_iommu_cmd_inval_set_nl(&cmd);
> +                       riscv_iommu_cmd_send(iommu, &cmd);
> +                       iova += 1ULL << tlbi->single.stride_lg2;
> +               }
>         }
>         return;
>  global:
> --
> 2.43.0
>

Reviewed-by: Tomasz Jeznach <tjeznach@rivosinc.com>

Thank you,
- Tomasz

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 7/7] iommu/riscv: Add NAPOT range invalidation support

[Andrew Jones]

On Fri, Apr 10, 2026 at 12:57:08PM -0300, Jason Gunthorpe wrote:
> Use the RISC-V IOMMU Address Range Invalidation extension
> (capabilities.S, spec section 9.3) to invalidate an IOVA range with
> a single IOTINVAL.VMA command using NAPOT-encoded addressing.
> 
> One iommu_iotlb_gather maps to one NAPOT invalidation command. The
> smallest power-of-two aligned range covering the gather is used since
> over-invalidation is always safe.
> 
> S and NL seem to be orthogonal in the spec, so if NL is not
> supported then global invalidation is probably always going to happen
> as wiping a large range without a table change is not common.
> 
> Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> ---
>  drivers/iommu/riscv/iommu-bits.h | 17 +++++++++++++
>  drivers/iommu/riscv/iommu.c      | 43 +++++++++++++++++++++++++++-----
>  2 files changed, 54 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
> index f01b49ac815586..32b3ad3ac9ae59 100644
> --- a/drivers/iommu/riscv/iommu-bits.h
> +++ b/drivers/iommu/riscv/iommu-bits.h
> @@ -64,6 +64,7 @@
>  #define RISCV_IOMMU_CAPABILITIES_PD17		BIT_ULL(39)
>  #define RISCV_IOMMU_CAPABILITIES_PD20		BIT_ULL(40)
>  #define RISCV_IOMMU_CAPABILITIES_NL		BIT_ULL(42)
> +#define RISCV_IOMMU_CAPABILITIES_S		BIT_ULL(43)
>  
>  /**
>   * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
> @@ -475,6 +476,7 @@ struct riscv_iommu_command {
>  #define RISCV_IOMMU_CMD_IOTINVAL_GV		BIT_ULL(33)
>  #define RISCV_IOMMU_CMD_IOTINVAL_GSCID		GENMASK_ULL(59, 44)
>  #define RISCV_IOMMU_CMD_IOTINVAL_NL		BIT_ULL(34)
> +#define RISCV_IOMMU_CMD_IOTINVAL_S		BIT_ULL(9)

We should add a comment here that this is actually bit 73, i.e.

#define RISCV_IOMMU_CMD_IOTINVAL_S           BIT_ULL(9)	/* bit 73 (dword1 bit 9) */

because...

>  /* dword1[61:10] is the 4K-aligned page address */
>  #define RISCV_IOMMU_CMD_IOTINVAL_ADDR		GENMASK_ULL(61, 10)
>  
> @@ -731,6 +733,21 @@ static inline void riscv_iommu_cmd_inval_set_nl(struct riscv_iommu_command *cmd)
>  	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_NL;
>  }
>  
> +/*
> + * Set NAPOT-encoded address for range invalidation (S=1).
> + * sz_lg2: log2 of total range in bytes, must be >= 13 (8KiB, 2 pages).
> + * addr must be naturally aligned to 2^sz_lg2.
> + */
> +static inline void riscv_iommu_cmd_inval_set_napot(
> +	struct riscv_iommu_command *cmd, u64 addr, unsigned int sz_lg2)
> +{
> +	u64 pfn = addr >> 12;
> +
> +	pfn |= BIT_U64(sz_lg2 - 13) - 1;
> +	cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, pfn);
> +	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV | RISCV_IOMMU_CMD_IOTINVAL_S;

...here we're setting the wrong dword. This should be

	cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, pfn) |
		      RISCV_IOMMU_CMD_IOTINVAL_S;
	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV;

Thanks,
drew

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 7/7] iommu/riscv: Add NAPOT range invalidation support

[Tomasz Jeznach]

On Tue, May 5, 2026 at 3:47 PM Andrew Jones
<andrew.jones@oss.qualcomm.com> wrote:
>
> On Fri, Apr 10, 2026 at 12:57:08PM -0300, Jason Gunthorpe wrote:
> > Use the RISC-V IOMMU Address Range Invalidation extension
> > (capabilities.S, spec section 9.3) to invalidate an IOVA range with
> > a single IOTINVAL.VMA command using NAPOT-encoded addressing.
> >
> > One iommu_iotlb_gather maps to one NAPOT invalidation command. The
> > smallest power-of-two aligned range covering the gather is used since
> > over-invalidation is always safe.
> >
> > S and NL seem to be orthogonal in the spec, so if NL is not
> > supported then global invalidation is probably always going to happen
> > as wiping a large range without a table change is not common.
> >
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> >  drivers/iommu/riscv/iommu-bits.h | 17 +++++++++++++
> >  drivers/iommu/riscv/iommu.c      | 43 +++++++++++++++++++++++++++-----
> >  2 files changed, 54 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
> > index f01b49ac815586..32b3ad3ac9ae59 100644
> > --- a/drivers/iommu/riscv/iommu-bits.h
> > +++ b/drivers/iommu/riscv/iommu-bits.h
> > @@ -64,6 +64,7 @@
> >  #define RISCV_IOMMU_CAPABILITIES_PD17                BIT_ULL(39)
> >  #define RISCV_IOMMU_CAPABILITIES_PD20                BIT_ULL(40)
> >  #define RISCV_IOMMU_CAPABILITIES_NL          BIT_ULL(42)
> > +#define RISCV_IOMMU_CAPABILITIES_S           BIT_ULL(43)
> >
> >  /**
> >   * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
> > @@ -475,6 +476,7 @@ struct riscv_iommu_command {
> >  #define RISCV_IOMMU_CMD_IOTINVAL_GV          BIT_ULL(33)
> >  #define RISCV_IOMMU_CMD_IOTINVAL_GSCID               GENMASK_ULL(59, 44)
> >  #define RISCV_IOMMU_CMD_IOTINVAL_NL          BIT_ULL(34)
> > +#define RISCV_IOMMU_CMD_IOTINVAL_S           BIT_ULL(9)
>
> We should add a comment here that this is actually bit 73, i.e.
>
> #define RISCV_IOMMU_CMD_IOTINVAL_S           BIT_ULL(9) /* bit 73 (dword1 bit 9) */
>
> because...
>
> >  /* dword1[61:10] is the 4K-aligned page address */
> >  #define RISCV_IOMMU_CMD_IOTINVAL_ADDR                GENMASK_ULL(61, 10)
> >
> > @@ -731,6 +733,21 @@ static inline void riscv_iommu_cmd_inval_set_nl(struct riscv_iommu_command *cmd)
> >       cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_NL;
> >  }
> >
> > +/*
> > + * Set NAPOT-encoded address for range invalidation (S=1).
> > + * sz_lg2: log2 of total range in bytes, must be >= 13 (8KiB, 2 pages).
> > + * addr must be naturally aligned to 2^sz_lg2.
> > + */
> > +static inline void riscv_iommu_cmd_inval_set_napot(
> > +     struct riscv_iommu_command *cmd, u64 addr, unsigned int sz_lg2)
> > +{
> > +     u64 pfn = addr >> 12;
> > +
> > +     pfn |= BIT_U64(sz_lg2 - 13) - 1;
> > +     cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, pfn);
> > +     cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV | RISCV_IOMMU_CMD_IOTINVAL_S;
>
> ...here we're setting the wrong dword. This should be
>
>         cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, pfn) |
>                       RISCV_IOMMU_CMD_IOTINVAL_S;
>         cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV;
>
> Thanks,
> drew

Yes, good catch. Similar to the RFC commit [1] ...

- Tomasz

[1] https://github.com/tjeznach/linux/commit/ef51d4be750d2d7c449747ab53fd5c12d5a4a175

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 7/7] iommu/riscv: Add NAPOT range invalidation support

[Jason Gunthorpe]

On Tue, May 05, 2026 at 05:47:10PM -0500, Andrew Jones wrote:
> On Fri, Apr 10, 2026 at 12:57:08PM -0300, Jason Gunthorpe wrote:
> > Use the RISC-V IOMMU Address Range Invalidation extension
> > (capabilities.S, spec section 9.3) to invalidate an IOVA range with
> > a single IOTINVAL.VMA command using NAPOT-encoded addressing.
> > 
> > One iommu_iotlb_gather maps to one NAPOT invalidation command. The
> > smallest power-of-two aligned range covering the gather is used since
> > over-invalidation is always safe.
> > 
> > S and NL seem to be orthogonal in the spec, so if NL is not
> > supported then global invalidation is probably always going to happen
> > as wiping a large range without a table change is not common.
> > 
> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
> > ---
> >  drivers/iommu/riscv/iommu-bits.h | 17 +++++++++++++
> >  drivers/iommu/riscv/iommu.c      | 43 +++++++++++++++++++++++++++-----
> >  2 files changed, 54 insertions(+), 6 deletions(-)
> > 
> > diff --git a/drivers/iommu/riscv/iommu-bits.h b/drivers/iommu/riscv/iommu-bits.h
> > index f01b49ac815586..32b3ad3ac9ae59 100644
> > --- a/drivers/iommu/riscv/iommu-bits.h
> > +++ b/drivers/iommu/riscv/iommu-bits.h
> > @@ -64,6 +64,7 @@
> >  #define RISCV_IOMMU_CAPABILITIES_PD17		BIT_ULL(39)
> >  #define RISCV_IOMMU_CAPABILITIES_PD20		BIT_ULL(40)
> >  #define RISCV_IOMMU_CAPABILITIES_NL		BIT_ULL(42)
> > +#define RISCV_IOMMU_CAPABILITIES_S		BIT_ULL(43)
> >  
> >  /**
> >   * enum riscv_iommu_igs_settings - Interrupt Generation Support Settings
> > @@ -475,6 +476,7 @@ struct riscv_iommu_command {
> >  #define RISCV_IOMMU_CMD_IOTINVAL_GV		BIT_ULL(33)
> >  #define RISCV_IOMMU_CMD_IOTINVAL_GSCID		GENMASK_ULL(59, 44)
> >  #define RISCV_IOMMU_CMD_IOTINVAL_NL		BIT_ULL(34)
> > +#define RISCV_IOMMU_CMD_IOTINVAL_S		BIT_ULL(9)
> 
> We should add a comment here that this is actually bit 73, i.e.
> 
> #define RISCV_IOMMU_CMD_IOTINVAL_S           BIT_ULL(9)	/* bit 73 (dword1 bit 9) */
> 
> because...

!! That's a big mistake, the common thing in other drivers is
something like CMD0 CMD1 to designate the word.

> > +static inline void riscv_iommu_cmd_inval_set_napot(
> > +	struct riscv_iommu_command *cmd, u64 addr, unsigned int sz_lg2)
> > +{
> > +	u64 pfn = addr >> 12;
> > +
> > +	pfn |= BIT_U64(sz_lg2 - 13) - 1;
> > +	cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, pfn);
> > +	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV | RISCV_IOMMU_CMD_IOTINVAL_S;
> 
> ...here we're setting the wrong dword. This should be
> 
> 	cmd->dword1 = FIELD_PREP(RISCV_IOMMU_CMD_IOTINVAL_ADDR, pfn) |
> 		      RISCV_IOMMU_CMD_IOTINVAL_S;
> 	cmd->dword0 |= RISCV_IOMMU_CMD_IOTINVAL_AV;

Right, I will fix it up

Thanks,
Jason

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

Re: [PATCH 0/7] Support non-leaf and range invalidation features in RISC-V

[Tomasz Jeznach]

Thanks Jason for this series.
I've verified github/iommu_pt_arm64 on RISC-V, looks good.

- Tomasz

On Fri, Apr 10, 2026 at 8:57 AM Jason Gunthorpe <jgg@nvidia.com> wrote:
>
> This is part of the patch pile to get SMMUv3 moved to iommupt. From
> that perspective it introduces the PT_FEAT_DETAILED_GATHER which will
> be used by both RISC-V and SMMUv3 to generate optimized invalidation
> commands.
>
> I don't have any RISC-V anything so this needs to be tested by someone who
> does.
>
> Improve the invalidation flow in RISC-V:
>  - For single address invalidation, compute a stride based on what
>    IOPTEs were changed by having the gather track level changes in
>    bitmaps. This allows things like unmapping hugetlb-backed mappings
>    to avoid jumping to global invalidation.
>
>  - Support the NL bit to avoid global invalidation when table
>    structure is changed
>
>  - Support the S range invalidation feature to convert any gather
>    into a single invalidation command using an aligned power-of-2
>    size like AMD and VT-d.
>
> This is also a requirement for the kunit iotlb series, which was used
> to test the math used to generate the invalidation commands from the
> gather in this series.
>
> It replaces a similar series Fangyu sent prior to the iommupt
> conversion:
>
> https://lore.kernel.org/linux-iommu/20260208144213.94856-1-fangyu.yu@linux.alibaba.com/
>
> The full patch pile is here:
>
> https://github.com/jgunthorpe/linux/commits/iommu_pt_arm64/
>
> Jason Gunthorpe (7):
>   iommu: Split the kdoc comment for struct iommu_iotlb_gather
>   iommupt: Add struct iommupt_pending_gather
>   iommupt: Add PT_FEAT_DETAILED_GATHER
>   iommu/riscv: Enable PT_FEAT_DETAILED_GATHER and pass gather to
>     iotlb_inval
>   iommu/riscv: Compute best stride for single invalidation
>   iommu/riscv: Add RISCV_IOMMU_CAPABILITIES_NL
>   iommu/riscv: Add NAPOT range invalidation support
>
>  drivers/iommu/generic_pt/fmt/iommu_riscv64.c |   4 +-
>  drivers/iommu/generic_pt/iommu_pt.h          |  98 ++++++++----
>  drivers/iommu/riscv/iommu-bits.h             |  24 +++
>  drivers/iommu/riscv/iommu.c                  | 160 ++++++++++++++-----
>  include/linux/generic_pt/common.h            |   5 +
>  include/linux/iommu.h                        |  42 ++++-
>  6 files changed, 259 insertions(+), 74 deletions(-)
>
>
> base-commit: ac81c0f2e6a779e444c5b6fd43756ef811656098
> --
> 2.43.0
>

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv