* [PATCH v3 01/29] KVM: arm64: Add a new function to donate memory with prot
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 02/29] KVM: arm64: Donate MMIO to the hypervisor Mostafa Saleh
` (27 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Soon, IOMMU drivers running in the hypervisor might interact with
non-coherent devices, so it needs a mechanism to map memory as
non cacheable.
Add ___pkvm_host_donate_hyp() which accepts a new argument for prot,
so the driver can add KVM_PGTABLE_PROT_NORMAL_NC.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 1 +
arch/arm64/kvm/hyp/nvhe/mem_protect.c | 11 +++++++++--
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
index 5f9d56754e39..52d7ee91e18c 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
@@ -36,6 +36,7 @@ int __pkvm_prot_finalize(void);
int __pkvm_host_share_hyp(u64 pfn);
int __pkvm_host_unshare_hyp(u64 pfn);
int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages);
+int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot prot);
int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages);
int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages);
int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages);
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index 8957734d6183..861e448183fd 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -769,13 +769,15 @@ int __pkvm_host_unshare_hyp(u64 pfn)
return ret;
}
-int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages)
+int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot prot)
{
u64 phys = hyp_pfn_to_phys(pfn);
u64 size = PAGE_SIZE * nr_pages;
void *virt = __hyp_va(phys);
int ret;
+ WARN_ON(prot & KVM_PGTABLE_PROT_X);
+
host_lock_component();
hyp_lock_component();
@@ -787,7 +789,7 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages)
goto unlock;
__hyp_set_page_state_range(phys, size, PKVM_PAGE_OWNED);
- WARN_ON(pkvm_create_mappings_locked(virt, virt + size, PAGE_HYP));
+ WARN_ON(pkvm_create_mappings_locked(virt, virt + size, prot));
WARN_ON(host_stage2_set_owner_locked(phys, size, PKVM_ID_HYP));
unlock:
@@ -797,6 +799,11 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages)
return ret;
}
+int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages)
+{
+ return ___pkvm_host_donate_hyp(pfn, nr_pages, PAGE_HYP);
+}
+
int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages)
{
u64 phys = hyp_pfn_to_phys(pfn);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 02/29] KVM: arm64: Donate MMIO to the hypervisor
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 01/29] KVM: arm64: Add a new function to donate memory with prot Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 03/29] KVM: arm64: pkvm: Add pkvm_time_get() Mostafa Saleh
` (26 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Add a function to donate MMIO to the hypervisor so IOMMU hypervisor
drivers can use that to protect the MMIO of IOMMU.
The initial attempt to implement this was to have a new flag to
"___pkvm_host_donate_hyp" to accept MMIO. However that had many problems,
it was quite intrusive for host/hyp to check/set page state to make it
aware of MMIO and to encode the state in the page table in that case.
Which is called in paths that can be sensitive to performance (FFA, VMs..)
As donating MMIO is very rare, and we don’t need to encode the full state,
it’s reasonable to have a separate function to do this.
It will init the host s2 page table with an invalid leaf with the owner ID
to prevent the host from mapping the page on faults.
Also, prevent kvm_pgtable_stage2_unmap() from removing owner ID from
stage-2 PTEs, as this can be triggered from recycle logic under memory
pressure. There is no code relying on this, as all ownership changes is
done via kvm_pgtable_stage2_set_owner()
For error path in IOMMU drivers, add a function to donate MMIO back
from hyp to host.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 +
arch/arm64/kvm/hyp/nvhe/mem_protect.c | 64 +++++++++++++++++++
arch/arm64/kvm/hyp/pgtable.c | 9 +--
3 files changed, 68 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
index 52d7ee91e18c..98e173da0f9b 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h
@@ -37,6 +37,8 @@ int __pkvm_host_share_hyp(u64 pfn);
int __pkvm_host_unshare_hyp(u64 pfn);
int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages);
int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot prot);
+int __pkvm_host_donate_hyp_mmio(u64 pfn);
+int __pkvm_hyp_donate_host_mmio(u64 pfn);
int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages);
int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages);
int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages);
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index 861e448183fd..c9a15ef6b18d 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -799,6 +799,70 @@ int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot prot)
return ret;
}
+int __pkvm_host_donate_hyp_mmio(u64 pfn)
+{
+ u64 phys = hyp_pfn_to_phys(pfn);
+ void *virt = __hyp_va(phys);
+ int ret;
+ kvm_pte_t pte;
+
+ host_lock_component();
+ hyp_lock_component();
+
+ ret = kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, NULL);
+ if (ret)
+ goto unlock;
+
+ if (pte && !kvm_pte_valid(pte)) {
+ ret = -EPERM;
+ goto unlock;
+ }
+
+ ret = kvm_pgtable_get_leaf(&pkvm_pgtable, (u64)virt, &pte, NULL);
+ if (ret)
+ goto unlock;
+ if (pte) {
+ ret = -EBUSY;
+ goto unlock;
+ }
+
+ ret = pkvm_create_mappings_locked(virt, virt + PAGE_SIZE, PAGE_HYP_DEVICE);
+ if (ret)
+ goto unlock;
+ /*
+ * We set HYP as the owner of the MMIO pages in the host stage-2, for:
+ * - host aborts: host_stage2_adjust_range() would fail for invalid non zero PTEs.
+ * - recycle under memory pressure: host_stage2_unmap_dev_all() would call
+ * kvm_pgtable_stage2_unmap() which will not clear non zero invalid ptes (counted).
+ * - other MMIO donation: Would fail as we check that the PTE is valid or empty.
+ */
+ WARN_ON(host_stage2_try(kvm_pgtable_stage2_set_owner, &host_mmu.pgt, phys,
+ PAGE_SIZE, &host_s2_pool, PKVM_ID_HYP));
+unlock:
+ hyp_unlock_component();
+ host_unlock_component();
+
+ return ret;
+}
+
+int __pkvm_hyp_donate_host_mmio(u64 pfn)
+{
+ u64 phys = hyp_pfn_to_phys(pfn);
+ u64 virt = (u64)__hyp_va(phys);
+ size_t size = PAGE_SIZE;
+
+ host_lock_component();
+ hyp_lock_component();
+
+ WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, virt, size) != size);
+ WARN_ON(host_stage2_try(kvm_pgtable_stage2_set_owner, &host_mmu.pgt, phys,
+ PAGE_SIZE, &host_s2_pool, PKVM_ID_HOST));
+ hyp_unlock_component();
+ host_unlock_component();
+
+ return 0;
+}
+
int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages)
{
return ___pkvm_host_donate_hyp(pfn, nr_pages, PAGE_HYP);
diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index c351b4abd5db..ba06b0c21d5a 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -1095,13 +1095,8 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx,
kvm_pte_t *childp = NULL;
bool need_flush = false;
- if (!kvm_pte_valid(ctx->old)) {
- if (stage2_pte_is_counted(ctx->old)) {
- kvm_clear_pte(ctx->ptep);
- mm_ops->put_page(ctx->ptep);
- }
- return 0;
- }
+ if (!kvm_pte_valid(ctx->old))
+ return stage2_pte_is_counted(ctx->old) ? -EPERM : 0;
if (kvm_pte_table(ctx->old, ctx->level)) {
childp = kvm_pte_follow(ctx->old, mm_ops);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 03/29] KVM: arm64: pkvm: Add pkvm_time_get()
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 01/29] KVM: arm64: Add a new function to donate memory with prot Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 02/29] KVM: arm64: Donate MMIO to the hypervisor Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 04/29] iommu/io-pgtable-arm: Split the page table driver Mostafa Saleh
` (25 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Add a function to return time in us.
This can be used from IOMMU drivers while waiting for conditions as
for SMMUv3 TLB invalidation waiting for sync.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 2 ++
arch/arm64/kvm/hyp/nvhe/setup.c | 4 ++++
arch/arm64/kvm/hyp/nvhe/timer-sr.c | 33 ++++++++++++++++++++++++++
3 files changed, 39 insertions(+)
diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
index ce31d3b73603..6c19691720cd 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h
@@ -87,4 +87,6 @@ bool kvm_handle_pvm_restricted(struct kvm_vcpu *vcpu, u64 *exit_code);
void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu);
int kvm_check_pvm_sysreg_table(void);
+int pkvm_timer_init(void);
+u64 pkvm_time_get(void);
#endif /* __ARM64_KVM_NVHE_PKVM_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
index a48d3f5a5afb..ee6435473204 100644
--- a/arch/arm64/kvm/hyp/nvhe/setup.c
+++ b/arch/arm64/kvm/hyp/nvhe/setup.c
@@ -304,6 +304,10 @@ void __noreturn __pkvm_init_finalise(void)
};
pkvm_pgtable.mm_ops = &pkvm_pgtable_mm_ops;
+ ret = pkvm_timer_init();
+ if (ret)
+ goto out;
+
ret = fix_host_ownership();
if (ret)
goto out;
diff --git a/arch/arm64/kvm/hyp/nvhe/timer-sr.c b/arch/arm64/kvm/hyp/nvhe/timer-sr.c
index ff176f4ce7de..e166cd5a56b8 100644
--- a/arch/arm64/kvm/hyp/nvhe/timer-sr.c
+++ b/arch/arm64/kvm/hyp/nvhe/timer-sr.c
@@ -11,6 +11,10 @@
#include <asm/kvm_hyp.h>
#include <asm/kvm_mmu.h>
+#include <nvhe/pkvm.h>
+
+static u32 timer_freq;
+
void __kvm_timer_set_cntvoff(u64 cntvoff)
{
write_sysreg(cntvoff, cntvoff_el2);
@@ -68,3 +72,32 @@ void __timer_enable_traps(struct kvm_vcpu *vcpu)
sysreg_clear_set(cnthctl_el2, clr, set);
}
+
+static u64 pkvm_ticks_get(void)
+{
+ return __arch_counter_get_cntvct();
+}
+
+#define SEC_TO_US 1000000
+
+int pkvm_timer_init(void)
+{
+ timer_freq = read_sysreg(cntfrq_el0);
+ /*
+ * TODO: The highest privileged level is supposed to initialize this
+ * register. But on some systems (which?), this information is only
+ * contained in the device-tree, so we'll need to find it out some other
+ * way.
+ */
+ if (!timer_freq || timer_freq < SEC_TO_US)
+ return -ENODEV;
+ return 0;
+}
+
+#define pkvm_time_ticks_to_us(ticks) ((u64)(ticks) * SEC_TO_US / timer_freq)
+
+/* Return time in us. */
+u64 pkvm_time_get(void)
+{
+ return pkvm_time_ticks_to_us(pkvm_ticks_get());
+}
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 04/29] iommu/io-pgtable-arm: Split the page table driver
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (2 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 03/29] KVM: arm64: pkvm: Add pkvm_time_get() Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 05/29] iommu/io-pgtable-arm: Split initialization Mostafa Saleh
` (24 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
To allow the KVM IOMMU driver to populate page tables using the
io-pgtable-arm code, move the shared bits into io-pgtable-arm-common.c.
Here we move the bulk of the common code, and a subsequent patch handles
the bits that require more care. phys_to_virt() and virt_to_phys() do
need special handling here because the hypervisor will have its own
version. It will also implement its own version of
__arm_lpae_alloc_pages(), __arm_lpae_free_pages() and
__arm_lpae_sync_pte() since the hypervisor needs some assistance for
allocating pages.
Some other minor changes, as there is a special hook for splitting
blocks, as the hypervisor can't print string on WARN, also, in
the future the hypervsior might implement that logic (or bring it
back for itself) as an optimization.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/io-pgtable-arm-common.c | 399 +++++++++++++++++
drivers/iommu/io-pgtable-arm.c | 590 +-------------------------
drivers/iommu/io-pgtable-arm.h | 255 +++++++++--
4 files changed, 643 insertions(+), 603 deletions(-)
create mode 100644 drivers/iommu/io-pgtable-arm-common.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 355294fa9033..a075eefca480 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
-obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
+obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o io-pgtable-arm-common.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_DART) += io-pgtable-dart.o
obj-$(CONFIG_IOMMU_IOVA) += iova.o
obj-$(CONFIG_OF_IOMMU) += of_iommu.o
diff --git a/drivers/iommu/io-pgtable-arm-common.c b/drivers/iommu/io-pgtable-arm-common.c
new file mode 100644
index 000000000000..6d5ae1b340b2
--- /dev/null
+++ b/drivers/iommu/io-pgtable-arm-common.c
@@ -0,0 +1,399 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * CPU-agnostic ARM page table allocator.
+ *
+ * Copyright (C) 2022 ARM Limited
+ *
+ * Author: Will Deacon <will.deacon@arm.com>
+ */
+#include <linux/sizes.h>
+#include <linux/types.h>
+
+#include "io-pgtable-arm.h"
+
+/*
+ * Convert an index returned by ARM_LPAE_PGD_IDX(), which can point into
+ * a concatenated PGD, into the maximum number of entries that can be
+ * mapped in the same table page.
+ */
+static inline int arm_lpae_max_entries(int i, struct arm_lpae_io_pgtable *data)
+{
+ int ptes_per_table = ARM_LPAE_PTES_PER_TABLE(data);
+
+ return ptes_per_table - (i & (ptes_per_table - 1));
+}
+
+static void __arm_lpae_clear_pte(arm_lpae_iopte *ptep, struct io_pgtable_cfg *cfg, int num_entries)
+{
+ for (int i = 0; i < num_entries; i++)
+ ptep[i] = 0;
+
+ if (!cfg->coherent_walk && num_entries)
+ __arm_lpae_sync_pte(ptep, num_entries, cfg);
+}
+
+static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,
+ struct iommu_iotlb_gather *gather,
+ unsigned long iova, size_t size, size_t pgcount,
+ int lvl, arm_lpae_iopte *ptep);
+
+static void __arm_lpae_init_pte(struct arm_lpae_io_pgtable *data,
+ phys_addr_t paddr, arm_lpae_iopte prot,
+ int lvl, int num_entries, arm_lpae_iopte *ptep)
+{
+ arm_lpae_iopte pte = prot;
+ struct io_pgtable_cfg *cfg = &data->iop.cfg;
+ size_t sz = ARM_LPAE_BLOCK_SIZE(lvl, data);
+ int i;
+
+ if (data->iop.fmt != ARM_MALI_LPAE && lvl == ARM_LPAE_MAX_LEVELS - 1)
+ pte |= ARM_LPAE_PTE_TYPE_PAGE;
+ else
+ pte |= ARM_LPAE_PTE_TYPE_BLOCK;
+
+ for (i = 0; i < num_entries; i++)
+ ptep[i] = pte | paddr_to_iopte(paddr + i * sz, data);
+
+ if (!cfg->coherent_walk)
+ __arm_lpae_sync_pte(ptep, num_entries, cfg);
+}
+
+static int arm_lpae_init_pte(struct arm_lpae_io_pgtable *data,
+ unsigned long iova, phys_addr_t paddr,
+ arm_lpae_iopte prot, int lvl, int num_entries,
+ arm_lpae_iopte *ptep)
+{
+ int i;
+
+ for (i = 0; i < num_entries; i++)
+ if (iopte_leaf(ptep[i], lvl, data->iop.fmt)) {
+ /* We require an unmap first */
+ WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
+ return -EEXIST;
+ } else if (iopte_type(ptep[i]) == ARM_LPAE_PTE_TYPE_TABLE) {
+ /*
+ * We need to unmap and free the old table before
+ * overwriting it with a block entry.
+ */
+ arm_lpae_iopte *tblp;
+ size_t sz = ARM_LPAE_BLOCK_SIZE(lvl, data);
+
+ tblp = ptep - ARM_LPAE_LVL_IDX(iova, lvl, data);
+ if (__arm_lpae_unmap(data, NULL, iova + i * sz, sz, 1,
+ lvl, tblp) != sz) {
+ WARN_ON(1);
+ return -EINVAL;
+ }
+ }
+
+ __arm_lpae_init_pte(data, paddr, prot, lvl, num_entries, ptep);
+ return 0;
+}
+
+static arm_lpae_iopte arm_lpae_install_table(arm_lpae_iopte *table,
+ arm_lpae_iopte *ptep,
+ arm_lpae_iopte curr,
+ struct arm_lpae_io_pgtable *data)
+{
+ arm_lpae_iopte old, new;
+ struct io_pgtable_cfg *cfg = &data->iop.cfg;
+
+ new = paddr_to_iopte(__arm_lpae_virt_to_phys(table), data) |
+ ARM_LPAE_PTE_TYPE_TABLE;
+ if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_NS)
+ new |= ARM_LPAE_PTE_NSTABLE;
+
+ /*
+ * Ensure the table itself is visible before its PTE can be.
+ * Whilst we could get away with cmpxchg64_release below, this
+ * doesn't have any ordering semantics when !CONFIG_SMP.
+ */
+ dma_wmb();
+
+ old = cmpxchg64_relaxed(ptep, curr, new);
+
+ if (cfg->coherent_walk || (old & ARM_LPAE_PTE_SW_SYNC))
+ return old;
+
+ /* Even if it's not ours, there's no point waiting; just kick it */
+ __arm_lpae_sync_pte(ptep, 1, cfg);
+ if (old == curr)
+ WRITE_ONCE(*ptep, new | ARM_LPAE_PTE_SW_SYNC);
+
+ return old;
+}
+
+static int __arm_lpae_map(struct arm_lpae_io_pgtable *data, unsigned long iova,
+ phys_addr_t paddr, size_t size, size_t pgcount,
+ arm_lpae_iopte prot, int lvl, arm_lpae_iopte *ptep,
+ gfp_t gfp, size_t *mapped)
+{
+ arm_lpae_iopte *cptep, pte;
+ size_t block_size = ARM_LPAE_BLOCK_SIZE(lvl, data);
+ size_t tblsz = ARM_LPAE_GRANULE(data);
+ struct io_pgtable_cfg *cfg = &data->iop.cfg;
+ int ret = 0, num_entries, max_entries, map_idx_start;
+
+ /* Find our entry at the current level */
+ map_idx_start = ARM_LPAE_LVL_IDX(iova, lvl, data);
+ ptep += map_idx_start;
+
+ /* If we can install a leaf entry at this level, then do so */
+ if (size == block_size) {
+ max_entries = arm_lpae_max_entries(map_idx_start, data);
+ num_entries = min_t(int, pgcount, max_entries);
+ ret = arm_lpae_init_pte(data, iova, paddr, prot, lvl, num_entries, ptep);
+ if (!ret)
+ *mapped += num_entries * size;
+
+ return ret;
+ }
+
+ /* We can't allocate tables at the final level */
+ if (WARN_ON(lvl >= ARM_LPAE_MAX_LEVELS - 1))
+ return -EINVAL;
+
+ /* Grab a pointer to the next level */
+ pte = READ_ONCE(*ptep);
+ if (!pte) {
+ cptep = __arm_lpae_alloc_pages(tblsz, gfp, cfg, data->iop.cookie);
+ if (!cptep)
+ return -ENOMEM;
+
+ pte = arm_lpae_install_table(cptep, ptep, 0, data);
+ if (pte)
+ __arm_lpae_free_pages(cptep, tblsz, cfg, data->iop.cookie);
+ } else if (!cfg->coherent_walk && !(pte & ARM_LPAE_PTE_SW_SYNC)) {
+ __arm_lpae_sync_pte(ptep, 1, cfg);
+ }
+
+ if (pte && !iopte_leaf(pte, lvl, data->iop.fmt)) {
+ cptep = iopte_deref(pte, data);
+ } else if (pte) {
+ /* We require an unmap first */
+ WARN_ON(!(cfg->quirks & IO_PGTABLE_QUIRK_NO_WARN));
+ return -EEXIST;
+ }
+
+ /* Rinse, repeat */
+ return __arm_lpae_map(data, iova, paddr, size, pgcount, prot, lvl + 1,
+ cptep, gfp, mapped);
+}
+
+static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
+ int prot)
+{
+ arm_lpae_iopte pte;
+
+ if (data->iop.fmt == ARM_64_LPAE_S1 ||
+ data->iop.fmt == ARM_32_LPAE_S1) {
+ pte = ARM_LPAE_PTE_nG;
+ if (!(prot & IOMMU_WRITE) && (prot & IOMMU_READ))
+ pte |= ARM_LPAE_PTE_AP_RDONLY;
+ else if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_HD)
+ pte |= ARM_LPAE_PTE_DBM;
+ if (!(prot & IOMMU_PRIV))
+ pte |= ARM_LPAE_PTE_AP_UNPRIV;
+ } else {
+ pte = ARM_LPAE_PTE_HAP_FAULT;
+ if (prot & IOMMU_READ)
+ pte |= ARM_LPAE_PTE_HAP_READ;
+ if (prot & IOMMU_WRITE)
+ pte |= ARM_LPAE_PTE_HAP_WRITE;
+ }
+
+ /*
+ * Note that this logic is structured to accommodate Mali LPAE
+ * having stage-1-like attributes but stage-2-like permissions.
+ */
+ if (data->iop.fmt == ARM_64_LPAE_S2 ||
+ data->iop.fmt == ARM_32_LPAE_S2) {
+ if (prot & IOMMU_MMIO) {
+ pte |= ARM_LPAE_PTE_MEMATTR_DEV;
+ } else if (prot & IOMMU_CACHE) {
+ if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_S2FWB)
+ pte |= ARM_LPAE_PTE_MEMATTR_FWB_WB;
+ else
+ pte |= ARM_LPAE_PTE_MEMATTR_OIWB;
+ } else {
+ pte |= ARM_LPAE_PTE_MEMATTR_NC;
+ }
+ } else {
+ if (prot & IOMMU_MMIO)
+ pte |= (ARM_LPAE_MAIR_ATTR_IDX_DEV
+ << ARM_LPAE_PTE_ATTRINDX_SHIFT);
+ else if (prot & IOMMU_CACHE)
+ pte |= (ARM_LPAE_MAIR_ATTR_IDX_CACHE
+ << ARM_LPAE_PTE_ATTRINDX_SHIFT);
+ }
+
+ /*
+ * Also Mali has its own notions of shareability wherein its Inner
+ * domain covers the cores within the GPU, and its Outer domain is
+ * "outside the GPU" (i.e. either the Inner or System domain in CPU
+ * terms, depending on coherency).
+ */
+ if (prot & IOMMU_CACHE && data->iop.fmt != ARM_MALI_LPAE)
+ pte |= ARM_LPAE_PTE_SH_IS;
+ else
+ pte |= ARM_LPAE_PTE_SH_OS;
+
+ if (prot & IOMMU_NOEXEC)
+ pte |= ARM_LPAE_PTE_XN;
+
+ if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_NS)
+ pte |= ARM_LPAE_PTE_NS;
+
+ if (data->iop.fmt != ARM_MALI_LPAE)
+ pte |= ARM_LPAE_PTE_AF;
+
+ return pte;
+}
+
+int arm_lpae_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int iommu_prot, gfp_t gfp, size_t *mapped)
+{
+ struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
+ struct io_pgtable_cfg *cfg = &data->iop.cfg;
+ arm_lpae_iopte *ptep = data->pgd;
+ int ret, lvl = data->start_level;
+ arm_lpae_iopte prot;
+ long iaext = (s64)iova >> cfg->ias;
+
+ if (WARN_ON(!pgsize || (pgsize & cfg->pgsize_bitmap) != pgsize))
+ return -EINVAL;
+
+ if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1)
+ iaext = ~iaext;
+ if (WARN_ON(iaext || paddr >> cfg->oas))
+ return -ERANGE;
+
+ if (!(iommu_prot & (IOMMU_READ | IOMMU_WRITE)))
+ return -EINVAL;
+
+ prot = arm_lpae_prot_to_pte(data, iommu_prot);
+ ret = __arm_lpae_map(data, iova, paddr, pgsize, pgcount, prot, lvl,
+ ptep, gfp, mapped);
+ /*
+ * Synchronise all PTE updates for the new mapping before there's
+ * a chance for anything to kick off a table walk for the new iova.
+ */
+ wmb();
+
+ return ret;
+}
+
+void __arm_lpae_free_pgtable(struct arm_lpae_io_pgtable *data, int lvl,
+ arm_lpae_iopte *ptep)
+{
+ arm_lpae_iopte *start, *end;
+ unsigned long table_size;
+
+ if (lvl == data->start_level)
+ table_size = ARM_LPAE_PGD_SIZE(data);
+ else
+ table_size = ARM_LPAE_GRANULE(data);
+
+ start = ptep;
+
+ /* Only leaf entries at the last level */
+ if (lvl == ARM_LPAE_MAX_LEVELS - 1)
+ end = ptep;
+ else
+ end = (void *)ptep + table_size;
+
+ while (ptep != end) {
+ arm_lpae_iopte pte = *ptep++;
+
+ if (!pte || iopte_leaf(pte, lvl, data->iop.fmt))
+ continue;
+
+ __arm_lpae_free_pgtable(data, lvl + 1, iopte_deref(pte, data));
+ }
+
+ __arm_lpae_free_pages(start, table_size, &data->iop.cfg, data->iop.cookie);
+}
+
+static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,
+ struct iommu_iotlb_gather *gather,
+ unsigned long iova, size_t size, size_t pgcount,
+ int lvl, arm_lpae_iopte *ptep)
+{
+ arm_lpae_iopte pte;
+ struct io_pgtable *iop = &data->iop;
+ int i = 0, num_entries, max_entries, unmap_idx_start;
+
+ /* Something went horribly wrong and we ran out of page table */
+ if (WARN_ON(lvl == ARM_LPAE_MAX_LEVELS))
+ return 0;
+
+ unmap_idx_start = ARM_LPAE_LVL_IDX(iova, lvl, data);
+ ptep += unmap_idx_start;
+ pte = READ_ONCE(*ptep);
+ if (!pte) {
+ WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
+ return -ENOENT;
+ }
+
+ /* If the size matches this level, we're in the right place */
+ if (size == ARM_LPAE_BLOCK_SIZE(lvl, data)) {
+ max_entries = arm_lpae_max_entries(unmap_idx_start, data);
+ num_entries = min_t(int, pgcount, max_entries);
+
+ /* Find and handle non-leaf entries */
+ for (i = 0; i < num_entries; i++) {
+ pte = READ_ONCE(ptep[i]);
+ if (!pte) {
+ WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
+ break;
+ }
+
+ if (!iopte_leaf(pte, lvl, iop->fmt)) {
+ __arm_lpae_clear_pte(&ptep[i], &iop->cfg, 1);
+
+ /* Also flush any partial walks */
+ io_pgtable_tlb_flush_walk(iop, iova + i * size, size,
+ ARM_LPAE_GRANULE(data));
+ __arm_lpae_free_pgtable(data, lvl + 1, iopte_deref(pte, data));
+ }
+ }
+
+ /* Clear the remaining entries */
+ __arm_lpae_clear_pte(ptep, &iop->cfg, i);
+
+ if (gather && !iommu_iotlb_gather_queued(gather))
+ for (int j = 0; j < i; j++)
+ io_pgtable_tlb_add_page(iop, gather, iova + j * size, size);
+
+ return i * size;
+ } else if (iopte_leaf(pte, lvl, iop->fmt)) {
+ arm_lpae_split_blk();
+ return 0;
+ }
+
+ /* Keep on walkin' */
+ ptep = iopte_deref(pte, data);
+ return __arm_lpae_unmap(data, gather, iova, size, pgcount, lvl + 1, ptep);
+}
+
+size_t arm_lpae_unmap_pages(struct io_pgtable_ops *ops, unsigned long iova,
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *gather)
+{
+ struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
+ struct io_pgtable_cfg *cfg = &data->iop.cfg;
+ arm_lpae_iopte *ptep = data->pgd;
+ long iaext = (s64)iova >> cfg->ias;
+
+ if (WARN_ON(!pgsize || (pgsize & cfg->pgsize_bitmap) != pgsize || !pgcount))
+ return 0;
+
+ if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1)
+ iaext = ~iaext;
+ if (WARN_ON(iaext))
+ return 0;
+
+ return __arm_lpae_unmap(data, gather, iova, pgsize, pgcount,
+ data->start_level, ptep);
+}
diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c
index 96425e92f313..ca4467ad3c40 100644
--- a/drivers/iommu/io-pgtable-arm.c
+++ b/drivers/iommu/io-pgtable-arm.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* CPU-agnostic ARM page table allocator.
+ * Host-specific functions. The rest is in io-pgtable-arm-common.c.
*
* Copyright (C) 2014 ARM Limited
*
@@ -24,204 +25,15 @@
#include "io-pgtable-arm.h"
#include "iommu-pages.h"
-#define ARM_LPAE_MAX_ADDR_BITS 52
-#define ARM_LPAE_S2_MAX_CONCAT_PAGES 16
-#define ARM_LPAE_MAX_LEVELS 4
-
-/* Struct accessors */
-#define io_pgtable_to_data(x) \
- container_of((x), struct arm_lpae_io_pgtable, iop)
-
-#define io_pgtable_ops_to_data(x) \
- io_pgtable_to_data(io_pgtable_ops_to_pgtable(x))
-
-/*
- * Calculate the right shift amount to get to the portion describing level l
- * in a virtual address mapped by the pagetable in d.
- */
-#define ARM_LPAE_LVL_SHIFT(l,d) \
- (((ARM_LPAE_MAX_LEVELS - (l)) * (d)->bits_per_level) + \
- ilog2(sizeof(arm_lpae_iopte)))
-
-#define ARM_LPAE_GRANULE(d) \
- (sizeof(arm_lpae_iopte) << (d)->bits_per_level)
-#define ARM_LPAE_PGD_SIZE(d) \
- (sizeof(arm_lpae_iopte) << (d)->pgd_bits)
-
-#define ARM_LPAE_PTES_PER_TABLE(d) \
- (ARM_LPAE_GRANULE(d) >> ilog2(sizeof(arm_lpae_iopte)))
-
-/*
- * Calculate the index at level l used to map virtual address a using the
- * pagetable in d.
- */
-#define ARM_LPAE_PGD_IDX(l,d) \
- ((l) == (d)->start_level ? (d)->pgd_bits - (d)->bits_per_level : 0)
-
-#define ARM_LPAE_LVL_IDX(a,l,d) \
- (((u64)(a) >> ARM_LPAE_LVL_SHIFT(l,d)) & \
- ((1 << ((d)->bits_per_level + ARM_LPAE_PGD_IDX(l,d))) - 1))
-
-/* Calculate the block/page mapping size at level l for pagetable in d. */
-#define ARM_LPAE_BLOCK_SIZE(l,d) (1ULL << ARM_LPAE_LVL_SHIFT(l,d))
-
-/* Page table bits */
-#define ARM_LPAE_PTE_TYPE_SHIFT 0
-#define ARM_LPAE_PTE_TYPE_MASK 0x3
-
-#define ARM_LPAE_PTE_TYPE_BLOCK 1
-#define ARM_LPAE_PTE_TYPE_TABLE 3
-#define ARM_LPAE_PTE_TYPE_PAGE 3
-
-#define ARM_LPAE_PTE_ADDR_MASK GENMASK_ULL(47,12)
-
-#define ARM_LPAE_PTE_NSTABLE (((arm_lpae_iopte)1) << 63)
-#define ARM_LPAE_PTE_XN (((arm_lpae_iopte)3) << 53)
-#define ARM_LPAE_PTE_DBM (((arm_lpae_iopte)1) << 51)
-#define ARM_LPAE_PTE_AF (((arm_lpae_iopte)1) << 10)
-#define ARM_LPAE_PTE_SH_NS (((arm_lpae_iopte)0) << 8)
-#define ARM_LPAE_PTE_SH_OS (((arm_lpae_iopte)2) << 8)
-#define ARM_LPAE_PTE_SH_IS (((arm_lpae_iopte)3) << 8)
-#define ARM_LPAE_PTE_NS (((arm_lpae_iopte)1) << 5)
-#define ARM_LPAE_PTE_VALID (((arm_lpae_iopte)1) << 0)
-
-#define ARM_LPAE_PTE_ATTR_LO_MASK (((arm_lpae_iopte)0x3ff) << 2)
-/* Ignore the contiguous bit for block splitting */
-#define ARM_LPAE_PTE_ATTR_HI_MASK (ARM_LPAE_PTE_XN | ARM_LPAE_PTE_DBM)
-#define ARM_LPAE_PTE_ATTR_MASK (ARM_LPAE_PTE_ATTR_LO_MASK | \
- ARM_LPAE_PTE_ATTR_HI_MASK)
-/* Software bit for solving coherency races */
-#define ARM_LPAE_PTE_SW_SYNC (((arm_lpae_iopte)1) << 55)
-
-/* Stage-1 PTE */
-#define ARM_LPAE_PTE_AP_UNPRIV (((arm_lpae_iopte)1) << 6)
-#define ARM_LPAE_PTE_AP_RDONLY_BIT 7
-#define ARM_LPAE_PTE_AP_RDONLY (((arm_lpae_iopte)1) << \
- ARM_LPAE_PTE_AP_RDONLY_BIT)
-#define ARM_LPAE_PTE_AP_WR_CLEAN_MASK (ARM_LPAE_PTE_AP_RDONLY | \
- ARM_LPAE_PTE_DBM)
-#define ARM_LPAE_PTE_ATTRINDX_SHIFT 2
-#define ARM_LPAE_PTE_nG (((arm_lpae_iopte)1) << 11)
-
-/* Stage-2 PTE */
-#define ARM_LPAE_PTE_HAP_FAULT (((arm_lpae_iopte)0) << 6)
-#define ARM_LPAE_PTE_HAP_READ (((arm_lpae_iopte)1) << 6)
-#define ARM_LPAE_PTE_HAP_WRITE (((arm_lpae_iopte)2) << 6)
-/*
- * For !FWB these code to:
- * 1111 = Normal outer write back cachable / Inner Write Back Cachable
- * Permit S1 to override
- * 0101 = Normal Non-cachable / Inner Non-cachable
- * 0001 = Device / Device-nGnRE
- * For S2FWB these code:
- * 0110 Force Normal Write Back
- * 0101 Normal* is forced Normal-NC, Device unchanged
- * 0001 Force Device-nGnRE
- */
-#define ARM_LPAE_PTE_MEMATTR_FWB_WB (((arm_lpae_iopte)0x6) << 2)
-#define ARM_LPAE_PTE_MEMATTR_OIWB (((arm_lpae_iopte)0xf) << 2)
-#define ARM_LPAE_PTE_MEMATTR_NC (((arm_lpae_iopte)0x5) << 2)
-#define ARM_LPAE_PTE_MEMATTR_DEV (((arm_lpae_iopte)0x1) << 2)
-
-/* Register bits */
-#define ARM_LPAE_VTCR_SL0_MASK 0x3
-
-#define ARM_LPAE_TCR_T0SZ_SHIFT 0
-
-#define ARM_LPAE_VTCR_PS_SHIFT 16
-#define ARM_LPAE_VTCR_PS_MASK 0x7
-
-#define ARM_LPAE_MAIR_ATTR_SHIFT(n) ((n) << 3)
-#define ARM_LPAE_MAIR_ATTR_MASK 0xff
-#define ARM_LPAE_MAIR_ATTR_DEVICE 0x04
-#define ARM_LPAE_MAIR_ATTR_NC 0x44
-#define ARM_LPAE_MAIR_ATTR_INC_OWBRWA 0xf4
-#define ARM_LPAE_MAIR_ATTR_WBRWA 0xff
-#define ARM_LPAE_MAIR_ATTR_IDX_NC 0
-#define ARM_LPAE_MAIR_ATTR_IDX_CACHE 1
-#define ARM_LPAE_MAIR_ATTR_IDX_DEV 2
-#define ARM_LPAE_MAIR_ATTR_IDX_INC_OCACHE 3
-
-#define ARM_MALI_LPAE_TTBR_ADRMODE_TABLE (3u << 0)
-#define ARM_MALI_LPAE_TTBR_READ_INNER BIT(2)
-#define ARM_MALI_LPAE_TTBR_SHARE_OUTER BIT(4)
-
-#define ARM_MALI_LPAE_MEMATTR_IMP_DEF 0x88ULL
-#define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL
-
-/* IOPTE accessors */
-#define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d))
-
-#define iopte_type(pte) \
- (((pte) >> ARM_LPAE_PTE_TYPE_SHIFT) & ARM_LPAE_PTE_TYPE_MASK)
-
-#define iopte_prot(pte) ((pte) & ARM_LPAE_PTE_ATTR_MASK)
-
#define iopte_writeable_dirty(pte) \
(((pte) & ARM_LPAE_PTE_AP_WR_CLEAN_MASK) == ARM_LPAE_PTE_DBM)
#define iopte_set_writeable_clean(ptep) \
set_bit(ARM_LPAE_PTE_AP_RDONLY_BIT, (unsigned long *)(ptep))
-struct arm_lpae_io_pgtable {
- struct io_pgtable iop;
-
- int pgd_bits;
- int start_level;
- int bits_per_level;
-
- void *pgd;
-};
-
-typedef u64 arm_lpae_iopte;
-
-static inline bool iopte_leaf(arm_lpae_iopte pte, int lvl,
- enum io_pgtable_fmt fmt)
-{
- if (lvl == (ARM_LPAE_MAX_LEVELS - 1) && fmt != ARM_MALI_LPAE)
- return iopte_type(pte) == ARM_LPAE_PTE_TYPE_PAGE;
-
- return iopte_type(pte) == ARM_LPAE_PTE_TYPE_BLOCK;
-}
-
-static inline bool iopte_table(arm_lpae_iopte pte, int lvl)
-{
- if (lvl == (ARM_LPAE_MAX_LEVELS - 1))
- return false;
- return iopte_type(pte) == ARM_LPAE_PTE_TYPE_TABLE;
-}
-
-static arm_lpae_iopte paddr_to_iopte(phys_addr_t paddr,
- struct arm_lpae_io_pgtable *data)
+void arm_lpae_split_blk(void)
{
- arm_lpae_iopte pte = paddr;
-
- /* Of the bits which overlap, either 51:48 or 15:12 are always RES0 */
- return (pte | (pte >> (48 - 12))) & ARM_LPAE_PTE_ADDR_MASK;
-}
-
-static phys_addr_t iopte_to_paddr(arm_lpae_iopte pte,
- struct arm_lpae_io_pgtable *data)
-{
- u64 paddr = pte & ARM_LPAE_PTE_ADDR_MASK;
-
- if (ARM_LPAE_GRANULE(data) < SZ_64K)
- return paddr;
-
- /* Rotate the packed high-order bits back to the top */
- return (paddr | (paddr << (48 - 12))) & (ARM_LPAE_PTE_ADDR_MASK << 4);
-}
-
-/*
- * Convert an index returned by ARM_LPAE_PGD_IDX(), which can point into
- * a concatenated PGD, into the maximum number of entries that can be
- * mapped in the same table page.
- */
-static inline int arm_lpae_max_entries(int i, struct arm_lpae_io_pgtable *data)
-{
- int ptes_per_table = ARM_LPAE_PTES_PER_TABLE(data);
-
- return ptes_per_table - (i & (ptes_per_table - 1));
+ WARN_ONCE(true, "Unmap of a partial large IOPTE is not allowed");
}
/*
@@ -257,9 +69,9 @@ static dma_addr_t __arm_lpae_dma_addr(void *pages)
return (dma_addr_t)virt_to_phys(pages);
}
-static void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp,
- struct io_pgtable_cfg *cfg,
- void *cookie)
+void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp,
+ struct io_pgtable_cfg *cfg,
+ void *cookie)
{
struct device *dev = cfg->iommu_dev;
size_t alloc_size;
@@ -308,9 +120,9 @@ static void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp,
return NULL;
}
-static void __arm_lpae_free_pages(void *pages, size_t size,
- struct io_pgtable_cfg *cfg,
- void *cookie)
+void __arm_lpae_free_pages(void *pages, size_t size,
+ struct io_pgtable_cfg *cfg,
+ void *cookie)
{
if (!cfg->coherent_walk)
dma_unmap_single(cfg->iommu_dev, __arm_lpae_dma_addr(pages),
@@ -322,393 +134,19 @@ static void __arm_lpae_free_pages(void *pages, size_t size,
iommu_free_pages(pages);
}
-static void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries,
- struct io_pgtable_cfg *cfg)
+void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries,
+ struct io_pgtable_cfg *cfg)
{
dma_sync_single_for_device(cfg->iommu_dev, __arm_lpae_dma_addr(ptep),
sizeof(*ptep) * num_entries, DMA_TO_DEVICE);
}
-static void __arm_lpae_clear_pte(arm_lpae_iopte *ptep, struct io_pgtable_cfg *cfg, int num_entries)
-{
- for (int i = 0; i < num_entries; i++)
- ptep[i] = 0;
-
- if (!cfg->coherent_walk && num_entries)
- __arm_lpae_sync_pte(ptep, num_entries, cfg);
-}
-
-static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,
- struct iommu_iotlb_gather *gather,
- unsigned long iova, size_t size, size_t pgcount,
- int lvl, arm_lpae_iopte *ptep);
-
-static void __arm_lpae_init_pte(struct arm_lpae_io_pgtable *data,
- phys_addr_t paddr, arm_lpae_iopte prot,
- int lvl, int num_entries, arm_lpae_iopte *ptep)
-{
- arm_lpae_iopte pte = prot;
- struct io_pgtable_cfg *cfg = &data->iop.cfg;
- size_t sz = ARM_LPAE_BLOCK_SIZE(lvl, data);
- int i;
-
- if (data->iop.fmt != ARM_MALI_LPAE && lvl == ARM_LPAE_MAX_LEVELS - 1)
- pte |= ARM_LPAE_PTE_TYPE_PAGE;
- else
- pte |= ARM_LPAE_PTE_TYPE_BLOCK;
-
- for (i = 0; i < num_entries; i++)
- ptep[i] = pte | paddr_to_iopte(paddr + i * sz, data);
-
- if (!cfg->coherent_walk)
- __arm_lpae_sync_pte(ptep, num_entries, cfg);
-}
-
-static int arm_lpae_init_pte(struct arm_lpae_io_pgtable *data,
- unsigned long iova, phys_addr_t paddr,
- arm_lpae_iopte prot, int lvl, int num_entries,
- arm_lpae_iopte *ptep)
-{
- int i;
-
- for (i = 0; i < num_entries; i++)
- if (iopte_leaf(ptep[i], lvl, data->iop.fmt)) {
- /* We require an unmap first */
- WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
- return -EEXIST;
- } else if (iopte_type(ptep[i]) == ARM_LPAE_PTE_TYPE_TABLE) {
- /*
- * We need to unmap and free the old table before
- * overwriting it with a block entry.
- */
- arm_lpae_iopte *tblp;
- size_t sz = ARM_LPAE_BLOCK_SIZE(lvl, data);
-
- tblp = ptep - ARM_LPAE_LVL_IDX(iova, lvl, data);
- if (__arm_lpae_unmap(data, NULL, iova + i * sz, sz, 1,
- lvl, tblp) != sz) {
- WARN_ON(1);
- return -EINVAL;
- }
- }
-
- __arm_lpae_init_pte(data, paddr, prot, lvl, num_entries, ptep);
- return 0;
-}
-
-static arm_lpae_iopte arm_lpae_install_table(arm_lpae_iopte *table,
- arm_lpae_iopte *ptep,
- arm_lpae_iopte curr,
- struct arm_lpae_io_pgtable *data)
-{
- arm_lpae_iopte old, new;
- struct io_pgtable_cfg *cfg = &data->iop.cfg;
-
- new = paddr_to_iopte(__pa(table), data) | ARM_LPAE_PTE_TYPE_TABLE;
- if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_NS)
- new |= ARM_LPAE_PTE_NSTABLE;
-
- /*
- * Ensure the table itself is visible before its PTE can be.
- * Whilst we could get away with cmpxchg64_release below, this
- * doesn't have any ordering semantics when !CONFIG_SMP.
- */
- dma_wmb();
-
- old = cmpxchg64_relaxed(ptep, curr, new);
-
- if (cfg->coherent_walk || (old & ARM_LPAE_PTE_SW_SYNC))
- return old;
-
- /* Even if it's not ours, there's no point waiting; just kick it */
- __arm_lpae_sync_pte(ptep, 1, cfg);
- if (old == curr)
- WRITE_ONCE(*ptep, new | ARM_LPAE_PTE_SW_SYNC);
-
- return old;
-}
-
-static int __arm_lpae_map(struct arm_lpae_io_pgtable *data, unsigned long iova,
- phys_addr_t paddr, size_t size, size_t pgcount,
- arm_lpae_iopte prot, int lvl, arm_lpae_iopte *ptep,
- gfp_t gfp, size_t *mapped)
-{
- arm_lpae_iopte *cptep, pte;
- size_t block_size = ARM_LPAE_BLOCK_SIZE(lvl, data);
- size_t tblsz = ARM_LPAE_GRANULE(data);
- struct io_pgtable_cfg *cfg = &data->iop.cfg;
- int ret = 0, num_entries, max_entries, map_idx_start;
-
- /* Find our entry at the current level */
- map_idx_start = ARM_LPAE_LVL_IDX(iova, lvl, data);
- ptep += map_idx_start;
-
- /* If we can install a leaf entry at this level, then do so */
- if (size == block_size) {
- max_entries = arm_lpae_max_entries(map_idx_start, data);
- num_entries = min_t(int, pgcount, max_entries);
- ret = arm_lpae_init_pte(data, iova, paddr, prot, lvl, num_entries, ptep);
- if (!ret)
- *mapped += num_entries * size;
-
- return ret;
- }
-
- /* We can't allocate tables at the final level */
- if (WARN_ON(lvl >= ARM_LPAE_MAX_LEVELS - 1))
- return -EINVAL;
-
- /* Grab a pointer to the next level */
- pte = READ_ONCE(*ptep);
- if (!pte) {
- cptep = __arm_lpae_alloc_pages(tblsz, gfp, cfg, data->iop.cookie);
- if (!cptep)
- return -ENOMEM;
-
- pte = arm_lpae_install_table(cptep, ptep, 0, data);
- if (pte)
- __arm_lpae_free_pages(cptep, tblsz, cfg, data->iop.cookie);
- } else if (!cfg->coherent_walk && !(pte & ARM_LPAE_PTE_SW_SYNC)) {
- __arm_lpae_sync_pte(ptep, 1, cfg);
- }
-
- if (pte && !iopte_leaf(pte, lvl, data->iop.fmt)) {
- cptep = iopte_deref(pte, data);
- } else if (pte) {
- /* We require an unmap first */
- WARN_ON(!(cfg->quirks & IO_PGTABLE_QUIRK_NO_WARN));
- return -EEXIST;
- }
-
- /* Rinse, repeat */
- return __arm_lpae_map(data, iova, paddr, size, pgcount, prot, lvl + 1,
- cptep, gfp, mapped);
-}
-
-static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
- int prot)
-{
- arm_lpae_iopte pte;
-
- if (data->iop.fmt == ARM_64_LPAE_S1 ||
- data->iop.fmt == ARM_32_LPAE_S1) {
- pte = ARM_LPAE_PTE_nG;
- if (!(prot & IOMMU_WRITE) && (prot & IOMMU_READ))
- pte |= ARM_LPAE_PTE_AP_RDONLY;
- else if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_HD)
- pte |= ARM_LPAE_PTE_DBM;
- if (!(prot & IOMMU_PRIV))
- pte |= ARM_LPAE_PTE_AP_UNPRIV;
- } else {
- pte = ARM_LPAE_PTE_HAP_FAULT;
- if (prot & IOMMU_READ)
- pte |= ARM_LPAE_PTE_HAP_READ;
- if (prot & IOMMU_WRITE)
- pte |= ARM_LPAE_PTE_HAP_WRITE;
- }
-
- /*
- * Note that this logic is structured to accommodate Mali LPAE
- * having stage-1-like attributes but stage-2-like permissions.
- */
- if (data->iop.fmt == ARM_64_LPAE_S2 ||
- data->iop.fmt == ARM_32_LPAE_S2) {
- if (prot & IOMMU_MMIO) {
- pte |= ARM_LPAE_PTE_MEMATTR_DEV;
- } else if (prot & IOMMU_CACHE) {
- if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_S2FWB)
- pte |= ARM_LPAE_PTE_MEMATTR_FWB_WB;
- else
- pte |= ARM_LPAE_PTE_MEMATTR_OIWB;
- } else {
- pte |= ARM_LPAE_PTE_MEMATTR_NC;
- }
- } else {
- if (prot & IOMMU_MMIO)
- pte |= (ARM_LPAE_MAIR_ATTR_IDX_DEV
- << ARM_LPAE_PTE_ATTRINDX_SHIFT);
- else if (prot & IOMMU_CACHE)
- pte |= (ARM_LPAE_MAIR_ATTR_IDX_CACHE
- << ARM_LPAE_PTE_ATTRINDX_SHIFT);
- }
-
- /*
- * Also Mali has its own notions of shareability wherein its Inner
- * domain covers the cores within the GPU, and its Outer domain is
- * "outside the GPU" (i.e. either the Inner or System domain in CPU
- * terms, depending on coherency).
- */
- if (prot & IOMMU_CACHE && data->iop.fmt != ARM_MALI_LPAE)
- pte |= ARM_LPAE_PTE_SH_IS;
- else
- pte |= ARM_LPAE_PTE_SH_OS;
-
- if (prot & IOMMU_NOEXEC)
- pte |= ARM_LPAE_PTE_XN;
-
- if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_NS)
- pte |= ARM_LPAE_PTE_NS;
-
- if (data->iop.fmt != ARM_MALI_LPAE)
- pte |= ARM_LPAE_PTE_AF;
-
- return pte;
-}
-
-static int arm_lpae_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
- phys_addr_t paddr, size_t pgsize, size_t pgcount,
- int iommu_prot, gfp_t gfp, size_t *mapped)
-{
- struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
- struct io_pgtable_cfg *cfg = &data->iop.cfg;
- arm_lpae_iopte *ptep = data->pgd;
- int ret, lvl = data->start_level;
- arm_lpae_iopte prot;
- long iaext = (s64)iova >> cfg->ias;
-
- if (WARN_ON(!pgsize || (pgsize & cfg->pgsize_bitmap) != pgsize))
- return -EINVAL;
-
- if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1)
- iaext = ~iaext;
- if (WARN_ON(iaext || paddr >> cfg->oas))
- return -ERANGE;
-
- if (!(iommu_prot & (IOMMU_READ | IOMMU_WRITE)))
- return -EINVAL;
-
- prot = arm_lpae_prot_to_pte(data, iommu_prot);
- ret = __arm_lpae_map(data, iova, paddr, pgsize, pgcount, prot, lvl,
- ptep, gfp, mapped);
- /*
- * Synchronise all PTE updates for the new mapping before there's
- * a chance for anything to kick off a table walk for the new iova.
- */
- wmb();
-
- return ret;
-}
-
-static void __arm_lpae_free_pgtable(struct arm_lpae_io_pgtable *data, int lvl,
- arm_lpae_iopte *ptep)
-{
- arm_lpae_iopte *start, *end;
- unsigned long table_size;
-
- if (lvl == data->start_level)
- table_size = ARM_LPAE_PGD_SIZE(data);
- else
- table_size = ARM_LPAE_GRANULE(data);
-
- start = ptep;
-
- /* Only leaf entries at the last level */
- if (lvl == ARM_LPAE_MAX_LEVELS - 1)
- end = ptep;
- else
- end = (void *)ptep + table_size;
-
- while (ptep != end) {
- arm_lpae_iopte pte = *ptep++;
-
- if (!pte || iopte_leaf(pte, lvl, data->iop.fmt))
- continue;
-
- __arm_lpae_free_pgtable(data, lvl + 1, iopte_deref(pte, data));
- }
-
- __arm_lpae_free_pages(start, table_size, &data->iop.cfg, data->iop.cookie);
-}
-
static void arm_lpae_free_pgtable(struct io_pgtable *iop)
{
- struct arm_lpae_io_pgtable *data = io_pgtable_to_data(iop);
-
- __arm_lpae_free_pgtable(data, data->start_level, data->pgd);
- kfree(data);
-}
-
-static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,
- struct iommu_iotlb_gather *gather,
- unsigned long iova, size_t size, size_t pgcount,
- int lvl, arm_lpae_iopte *ptep)
-{
- arm_lpae_iopte pte;
- struct io_pgtable *iop = &data->iop;
- int i = 0, num_entries, max_entries, unmap_idx_start;
-
- /* Something went horribly wrong and we ran out of page table */
- if (WARN_ON(lvl == ARM_LPAE_MAX_LEVELS))
- return 0;
-
- unmap_idx_start = ARM_LPAE_LVL_IDX(iova, lvl, data);
- ptep += unmap_idx_start;
- pte = READ_ONCE(*ptep);
- if (!pte) {
- WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
- return -ENOENT;
- }
-
- /* If the size matches this level, we're in the right place */
- if (size == ARM_LPAE_BLOCK_SIZE(lvl, data)) {
- max_entries = arm_lpae_max_entries(unmap_idx_start, data);
- num_entries = min_t(int, pgcount, max_entries);
-
- /* Find and handle non-leaf entries */
- for (i = 0; i < num_entries; i++) {
- pte = READ_ONCE(ptep[i]);
- if (!pte) {
- WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
- break;
- }
-
- if (!iopte_leaf(pte, lvl, iop->fmt)) {
- __arm_lpae_clear_pte(&ptep[i], &iop->cfg, 1);
-
- /* Also flush any partial walks */
- io_pgtable_tlb_flush_walk(iop, iova + i * size, size,
- ARM_LPAE_GRANULE(data));
- __arm_lpae_free_pgtable(data, lvl + 1, iopte_deref(pte, data));
- }
- }
-
- /* Clear the remaining entries */
- __arm_lpae_clear_pte(ptep, &iop->cfg, i);
-
- if (gather && !iommu_iotlb_gather_queued(gather))
- for (int j = 0; j < i; j++)
- io_pgtable_tlb_add_page(iop, gather, iova + j * size, size);
-
- return i * size;
- } else if (iopte_leaf(pte, lvl, iop->fmt)) {
- WARN_ONCE(true, "Unmap of a partial large IOPTE is not allowed");
- return 0;
- }
-
- /* Keep on walkin' */
- ptep = iopte_deref(pte, data);
- return __arm_lpae_unmap(data, gather, iova, size, pgcount, lvl + 1, ptep);
-}
-
-static size_t arm_lpae_unmap_pages(struct io_pgtable_ops *ops, unsigned long iova,
- size_t pgsize, size_t pgcount,
- struct iommu_iotlb_gather *gather)
-{
- struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
- struct io_pgtable_cfg *cfg = &data->iop.cfg;
- arm_lpae_iopte *ptep = data->pgd;
- long iaext = (s64)iova >> cfg->ias;
-
- if (WARN_ON(!pgsize || (pgsize & cfg->pgsize_bitmap) != pgsize || !pgcount))
- return 0;
-
- if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1)
- iaext = ~iaext;
- if (WARN_ON(iaext))
- return 0;
+ struct arm_lpae_io_pgtable *data = io_pgtable_to_data(iop);
- return __arm_lpae_unmap(data, gather, iova, pgsize, pgcount,
- data->start_level, ptep);
+ __arm_lpae_free_pgtable(data, data->start_level, data->pgd);
+ kfree(data);
}
struct io_pgtable_walk_data {
diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h
index ba7cfdf7afa0..c3a3b4fd44c3 100644
--- a/drivers/iommu/io-pgtable-arm.h
+++ b/drivers/iommu/io-pgtable-arm.h
@@ -2,29 +2,232 @@
#ifndef IO_PGTABLE_ARM_H_
#define IO_PGTABLE_ARM_H_
-#define ARM_LPAE_TCR_TG0_4K 0
-#define ARM_LPAE_TCR_TG0_64K 1
-#define ARM_LPAE_TCR_TG0_16K 2
-
-#define ARM_LPAE_TCR_TG1_16K 1
-#define ARM_LPAE_TCR_TG1_4K 2
-#define ARM_LPAE_TCR_TG1_64K 3
-
-#define ARM_LPAE_TCR_SH_NS 0
-#define ARM_LPAE_TCR_SH_OS 2
-#define ARM_LPAE_TCR_SH_IS 3
-
-#define ARM_LPAE_TCR_RGN_NC 0
-#define ARM_LPAE_TCR_RGN_WBWA 1
-#define ARM_LPAE_TCR_RGN_WT 2
-#define ARM_LPAE_TCR_RGN_WB 3
-
-#define ARM_LPAE_TCR_PS_32_BIT 0x0ULL
-#define ARM_LPAE_TCR_PS_36_BIT 0x1ULL
-#define ARM_LPAE_TCR_PS_40_BIT 0x2ULL
-#define ARM_LPAE_TCR_PS_42_BIT 0x3ULL
-#define ARM_LPAE_TCR_PS_44_BIT 0x4ULL
-#define ARM_LPAE_TCR_PS_48_BIT 0x5ULL
-#define ARM_LPAE_TCR_PS_52_BIT 0x6ULL
-
-#endif /* IO_PGTABLE_ARM_H_ */
+#include <linux/io-pgtable.h>
+
+typedef u64 arm_lpae_iopte;
+
+struct arm_lpae_io_pgtable {
+ struct io_pgtable iop;
+
+ int pgd_bits;
+ int start_level;
+ int bits_per_level;
+
+ void *pgd;
+};
+
+#define ARM_LPAE_MAX_ADDR_BITS 52
+#define ARM_LPAE_S2_MAX_CONCAT_PAGES 16
+#define ARM_LPAE_MAX_LEVELS 4
+
+/* Struct accessors */
+#define io_pgtable_to_data(x) \
+ container_of((x), struct arm_lpae_io_pgtable, iop)
+
+#define io_pgtable_ops_to_data(x) \
+ io_pgtable_to_data(io_pgtable_ops_to_pgtable(x))
+
+/*
+ * Calculate the right shift amount to get to the portion describing level l
+ * in a virtual address mapped by the pagetable in d.
+ */
+#define ARM_LPAE_LVL_SHIFT(l,d) \
+ (((ARM_LPAE_MAX_LEVELS - (l)) * (d)->bits_per_level) + \
+ ilog2(sizeof(arm_lpae_iopte)))
+
+#define ARM_LPAE_GRANULE(d) \
+ (sizeof(arm_lpae_iopte) << (d)->bits_per_level)
+#define ARM_LPAE_PGD_SIZE(d) \
+ (sizeof(arm_lpae_iopte) << (d)->pgd_bits)
+
+#define ARM_LPAE_PTES_PER_TABLE(d) \
+ (ARM_LPAE_GRANULE(d) >> ilog2(sizeof(arm_lpae_iopte)))
+
+/*
+ * Calculate the index at level l used to map virtual address a using the
+ * pagetable in d.
+ */
+#define ARM_LPAE_PGD_IDX(l,d) \
+ ((l) == (d)->start_level ? (d)->pgd_bits - (d)->bits_per_level : 0)
+
+#define ARM_LPAE_LVL_IDX(a,l,d) \
+ (((u64)(a) >> ARM_LPAE_LVL_SHIFT(l,d)) & \
+ ((1 << ((d)->bits_per_level + ARM_LPAE_PGD_IDX(l,d))) - 1))
+
+/* Calculate the block/page mapping size at level l for pagetable in d. */
+#define ARM_LPAE_BLOCK_SIZE(l,d) (1ULL << ARM_LPAE_LVL_SHIFT(l,d))
+
+/* Page table bits */
+#define ARM_LPAE_PTE_TYPE_SHIFT 0
+#define ARM_LPAE_PTE_TYPE_MASK 0x3
+
+#define ARM_LPAE_PTE_TYPE_BLOCK 1
+#define ARM_LPAE_PTE_TYPE_TABLE 3
+#define ARM_LPAE_PTE_TYPE_PAGE 3
+
+#define ARM_LPAE_PTE_ADDR_MASK GENMASK_ULL(47,12)
+
+#define ARM_LPAE_PTE_NSTABLE (((arm_lpae_iopte)1) << 63)
+#define ARM_LPAE_PTE_XN (((arm_lpae_iopte)3) << 53)
+#define ARM_LPAE_PTE_DBM (((arm_lpae_iopte)1) << 51)
+#define ARM_LPAE_PTE_AF (((arm_lpae_iopte)1) << 10)
+#define ARM_LPAE_PTE_SH_NS (((arm_lpae_iopte)0) << 8)
+#define ARM_LPAE_PTE_SH_OS (((arm_lpae_iopte)2) << 8)
+#define ARM_LPAE_PTE_SH_IS (((arm_lpae_iopte)3) << 8)
+#define ARM_LPAE_PTE_NS (((arm_lpae_iopte)1) << 5)
+#define ARM_LPAE_PTE_VALID (((arm_lpae_iopte)1) << 0)
+
+#define ARM_LPAE_PTE_ATTR_LO_MASK (((arm_lpae_iopte)0x3ff) << 2)
+/* Ignore the contiguous bit for block splitting */
+#define ARM_LPAE_PTE_ATTR_HI_MASK (ARM_LPAE_PTE_XN | ARM_LPAE_PTE_DBM)
+#define ARM_LPAE_PTE_ATTR_MASK (ARM_LPAE_PTE_ATTR_LO_MASK | \
+ ARM_LPAE_PTE_ATTR_HI_MASK)
+/* Software bit for solving coherency races */
+#define ARM_LPAE_PTE_SW_SYNC (((arm_lpae_iopte)1) << 55)
+
+/* Stage-1 PTE */
+#define ARM_LPAE_PTE_AP_UNPRIV (((arm_lpae_iopte)1) << 6)
+#define ARM_LPAE_PTE_AP_RDONLY_BIT 7
+#define ARM_LPAE_PTE_AP_RDONLY (((arm_lpae_iopte)1) << \
+ ARM_LPAE_PTE_AP_RDONLY_BIT)
+#define ARM_LPAE_PTE_AP_WR_CLEAN_MASK (ARM_LPAE_PTE_AP_RDONLY | \
+ ARM_LPAE_PTE_DBM)
+#define ARM_LPAE_PTE_ATTRINDX_SHIFT 2
+#define ARM_LPAE_PTE_nG (((arm_lpae_iopte)1) << 11)
+
+/* Stage-2 PTE */
+#define ARM_LPAE_PTE_HAP_FAULT (((arm_lpae_iopte)0) << 6)
+#define ARM_LPAE_PTE_HAP_READ (((arm_lpae_iopte)1) << 6)
+#define ARM_LPAE_PTE_HAP_WRITE (((arm_lpae_iopte)2) << 6)
+/*
+ * For !FWB these code to:
+ * 1111 = Normal outer write back cachable / Inner Write Back Cachable
+ * Permit S1 to override
+ * 0101 = Normal Non-cachable / Inner Non-cachable
+ * 0001 = Device / Device-nGnRE
+ * For S2FWB these code:
+ * 0110 Force Normal Write Back
+ * 0101 Normal* is forced Normal-NC, Device unchanged
+ * 0001 Force Device-nGnRE
+ */
+#define ARM_LPAE_PTE_MEMATTR_FWB_WB (((arm_lpae_iopte)0x6) << 2)
+#define ARM_LPAE_PTE_MEMATTR_OIWB (((arm_lpae_iopte)0xf) << 2)
+#define ARM_LPAE_PTE_MEMATTR_NC (((arm_lpae_iopte)0x5) << 2)
+#define ARM_LPAE_PTE_MEMATTR_DEV (((arm_lpae_iopte)0x1) << 2)
+
+/* Register bits */
+#define ARM_LPAE_VTCR_SL0_MASK 0x3
+
+#define ARM_LPAE_TCR_T0SZ_SHIFT 0
+
+#define ARM_LPAE_VTCR_PS_SHIFT 16
+#define ARM_LPAE_VTCR_PS_MASK 0x7
+
+#define ARM_LPAE_MAIR_ATTR_SHIFT(n) ((n) << 3)
+#define ARM_LPAE_MAIR_ATTR_MASK 0xff
+#define ARM_LPAE_MAIR_ATTR_DEVICE 0x04
+#define ARM_LPAE_MAIR_ATTR_NC 0x44
+#define ARM_LPAE_MAIR_ATTR_INC_OWBRWA 0xf4
+#define ARM_LPAE_MAIR_ATTR_WBRWA 0xff
+#define ARM_LPAE_MAIR_ATTR_IDX_NC 0
+#define ARM_LPAE_MAIR_ATTR_IDX_CACHE 1
+#define ARM_LPAE_MAIR_ATTR_IDX_DEV 2
+#define ARM_LPAE_MAIR_ATTR_IDX_INC_OCACHE 3
+
+#define ARM_MALI_LPAE_TTBR_ADRMODE_TABLE (3u << 0)
+#define ARM_MALI_LPAE_TTBR_READ_INNER BIT(2)
+#define ARM_MALI_LPAE_TTBR_SHARE_OUTER BIT(4)
+
+#define ARM_MALI_LPAE_MEMATTR_IMP_DEF 0x88ULL
+#define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL
+
+#define ARM_LPAE_TCR_TG0_4K 0
+#define ARM_LPAE_TCR_TG0_64K 1
+#define ARM_LPAE_TCR_TG0_16K 2
+
+#define ARM_LPAE_TCR_TG1_16K 1
+#define ARM_LPAE_TCR_TG1_4K 2
+#define ARM_LPAE_TCR_TG1_64K 3
+
+#define ARM_LPAE_TCR_SH_NS 0
+#define ARM_LPAE_TCR_SH_OS 2
+#define ARM_LPAE_TCR_SH_IS 3
+
+#define ARM_LPAE_TCR_RGN_NC 0
+#define ARM_LPAE_TCR_RGN_WBWA 1
+#define ARM_LPAE_TCR_RGN_WT 2
+#define ARM_LPAE_TCR_RGN_WB 3
+
+#define ARM_LPAE_TCR_PS_32_BIT 0x0ULL
+#define ARM_LPAE_TCR_PS_36_BIT 0x1ULL
+#define ARM_LPAE_TCR_PS_40_BIT 0x2ULL
+#define ARM_LPAE_TCR_PS_42_BIT 0x3ULL
+#define ARM_LPAE_TCR_PS_44_BIT 0x4ULL
+#define ARM_LPAE_TCR_PS_48_BIT 0x5ULL
+#define ARM_LPAE_TCR_PS_52_BIT 0x6ULL
+
+/* IOPTE accessors */
+#define iopte_type(pte) \
+ (((pte) >> ARM_LPAE_PTE_TYPE_SHIFT) & ARM_LPAE_PTE_TYPE_MASK)
+
+#define iopte_prot(pte) ((pte) & ARM_LPAE_PTE_ATTR_MASK)
+
+static inline bool iopte_leaf(arm_lpae_iopte pte, int lvl,
+ enum io_pgtable_fmt fmt)
+{
+ if (lvl == (ARM_LPAE_MAX_LEVELS - 1) && fmt != ARM_MALI_LPAE)
+ return iopte_type(pte) == ARM_LPAE_PTE_TYPE_PAGE;
+
+ return iopte_type(pte) == ARM_LPAE_PTE_TYPE_BLOCK;
+}
+
+static inline bool iopte_table(arm_lpae_iopte pte, int lvl)
+{
+ if (lvl == (ARM_LPAE_MAX_LEVELS - 1))
+ return false;
+ return iopte_type(pte) == ARM_LPAE_PTE_TYPE_TABLE;
+}
+
+#define __arm_lpae_virt_to_phys __pa
+#define __arm_lpae_phys_to_virt __va
+
+static inline phys_addr_t iopte_to_paddr(arm_lpae_iopte pte,
+ struct arm_lpae_io_pgtable *data)
+{
+ u64 paddr = pte & ARM_LPAE_PTE_ADDR_MASK;
+
+ if (ARM_LPAE_GRANULE(data) < SZ_64K)
+ return paddr;
+
+ /* Rotate the packed high-order bits back to the top */
+ return (paddr | (paddr << (48 - 12))) & (ARM_LPAE_PTE_ADDR_MASK << 4);
+}
+
+#define iopte_deref(pte, d) __arm_lpae_phys_to_virt(iopte_to_paddr(pte, d))
+
+static inline arm_lpae_iopte paddr_to_iopte(phys_addr_t paddr,
+ struct arm_lpae_io_pgtable *data)
+{
+ arm_lpae_iopte pte = paddr;
+
+ /* Of the bits which overlap, either 51:48 or 15:12 are always RES0 */
+ return (pte | (pte >> (48 - 12))) & ARM_LPAE_PTE_ADDR_MASK;
+}
+
+/* Generic functions */
+int arm_lpae_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int iommu_prot, gfp_t gfp, size_t *mapped);
+size_t arm_lpae_unmap_pages(struct io_pgtable_ops *ops, unsigned long iova,
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *gather);
+
+/* Host/hyp-specific functions */
+void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, struct io_pgtable_cfg *cfg, void *cookie);
+void __arm_lpae_free_pages(void *pages, size_t size, struct io_pgtable_cfg *cfg, void *cookie);
+void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries,
+ struct io_pgtable_cfg *cfg);
+void __arm_lpae_free_pgtable(struct arm_lpae_io_pgtable *data, int lvl,
+ arm_lpae_iopte *ptep);
+void arm_lpae_split_blk(void);
+#endif /* IO_PGTABLE_H_ */
\ No newline at end of file
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 05/29] iommu/io-pgtable-arm: Split initialization
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (3 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 04/29] iommu/io-pgtable-arm: Split the page table driver Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 06/29] iommu/arm-smmu-v3: Move some definitions to a new common file Mostafa Saleh
` (23 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Extract the configuration part from io-pgtable-arm.c, move it to
io-pgtable-arm-common.c.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
drivers/iommu/io-pgtable-arm-common.c | 295 ++++++++++++++++++++++++++
drivers/iommu/io-pgtable-arm.c | 280 +-----------------------
drivers/iommu/io-pgtable-arm.h | 8 +
3 files changed, 312 insertions(+), 271 deletions(-)
diff --git a/drivers/iommu/io-pgtable-arm-common.c b/drivers/iommu/io-pgtable-arm-common.c
index 6d5ae1b340b2..40bd5a6bad88 100644
--- a/drivers/iommu/io-pgtable-arm-common.c
+++ b/drivers/iommu/io-pgtable-arm-common.c
@@ -11,6 +11,301 @@
#include "io-pgtable-arm.h"
+/*
+ * Check if concatenated PGDs are mandatory according to Arm DDI0487 (K.a)
+ * 1) R_DXBSH: For 16KB, and 48-bit input size, use level 1 instead of 0.
+ * 2) R_SRKBC: After de-ciphering the table for PA size and valid initial lookup
+ * a) 40 bits PA size with 4K: use level 1 instead of level 0 (2 tables for ias = oas)
+ * b) 40 bits PA size with 16K: use level 2 instead of level 1 (16 tables for ias = oas)
+ * c) 42 bits PA size with 4K: use level 1 instead of level 0 (8 tables for ias = oas)
+ * d) 48 bits PA size with 16K: use level 1 instead of level 0 (2 tables for ias = oas)
+ */
+static inline bool arm_lpae_concat_mandatory(struct io_pgtable_cfg *cfg,
+ struct arm_lpae_io_pgtable *data)
+{
+ unsigned int ias = cfg->ias;
+ unsigned int oas = cfg->oas;
+
+ /* Covers 1 and 2.d */
+ if ((ARM_LPAE_GRANULE(data) == SZ_16K) && (data->start_level == 0))
+ return (oas == 48) || (ias == 48);
+
+ /* Covers 2.a and 2.c */
+ if ((ARM_LPAE_GRANULE(data) == SZ_4K) && (data->start_level == 0))
+ return (oas == 40) || (oas == 42);
+
+ /* Case 2.b */
+ return (ARM_LPAE_GRANULE(data) == SZ_16K) &&
+ (data->start_level == 1) && (oas == 40);
+}
+
+static void arm_lpae_restrict_pgsizes(struct io_pgtable_cfg *cfg)
+{
+ unsigned long granule, page_sizes;
+ unsigned int max_addr_bits = 48;
+
+ /*
+ * We need to restrict the supported page sizes to match the
+ * translation regime for a particular granule. Aim to match
+ * the CPU page size if possible, otherwise prefer smaller sizes.
+ * While we're at it, restrict the block sizes to match the
+ * chosen granule.
+ */
+ if (cfg->pgsize_bitmap & PAGE_SIZE)
+ granule = PAGE_SIZE;
+ else if (cfg->pgsize_bitmap & ~PAGE_MASK)
+ granule = 1UL << __fls(cfg->pgsize_bitmap & ~PAGE_MASK);
+ else if (cfg->pgsize_bitmap & PAGE_MASK)
+ granule = 1UL << __ffs(cfg->pgsize_bitmap & PAGE_MASK);
+ else
+ granule = 0;
+
+ switch (granule) {
+ case SZ_4K:
+ page_sizes = (SZ_4K | SZ_2M | SZ_1G);
+ break;
+ case SZ_16K:
+ page_sizes = (SZ_16K | SZ_32M);
+ break;
+ case SZ_64K:
+ max_addr_bits = 52;
+ page_sizes = (SZ_64K | SZ_512M);
+ if (cfg->oas > 48)
+ page_sizes |= 1ULL << 42; /* 4TB */
+ break;
+ default:
+ page_sizes = 0;
+ }
+
+ cfg->pgsize_bitmap &= page_sizes;
+ cfg->ias = min(cfg->ias, max_addr_bits);
+ cfg->oas = min(cfg->oas, max_addr_bits);
+}
+
+int arm_lpae_init_pgtable(struct io_pgtable_cfg *cfg,
+ struct arm_lpae_io_pgtable *data)
+{
+ int levels, va_bits, pg_shift;
+
+ arm_lpae_restrict_pgsizes(cfg);
+
+ if (!(cfg->pgsize_bitmap & (SZ_4K | SZ_16K | SZ_64K)))
+ return -EINVAL;
+
+ if (cfg->ias > ARM_LPAE_MAX_ADDR_BITS)
+ return E2BIG;
+
+ if (cfg->oas > ARM_LPAE_MAX_ADDR_BITS)
+ return -E2BIG;
+
+ pg_shift = __ffs(cfg->pgsize_bitmap);
+ data->bits_per_level = pg_shift - ilog2(sizeof(arm_lpae_iopte));
+
+ va_bits = cfg->ias - pg_shift;
+ levels = DIV_ROUND_UP(va_bits, data->bits_per_level);
+ data->start_level = ARM_LPAE_MAX_LEVELS - levels;
+
+ /* Calculate the actual size of our pgd (without concatenation) */
+ data->pgd_bits = va_bits - (data->bits_per_level * (levels - 1));
+ return 0;
+}
+
+int arm_lpae_init_pgtable_s1(struct io_pgtable_cfg *cfg,
+ struct arm_lpae_io_pgtable *data,
+ void *cookie)
+{
+ u64 reg;
+ int ret;
+ typeof(&cfg->arm_lpae_s1_cfg.tcr) tcr = &cfg->arm_lpae_s1_cfg.tcr;
+ bool tg1;
+
+ if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS |
+ IO_PGTABLE_QUIRK_ARM_TTBR1 |
+ IO_PGTABLE_QUIRK_ARM_OUTER_WBWA |
+ IO_PGTABLE_QUIRK_ARM_HD |
+ IO_PGTABLE_QUIRK_NO_WARN))
+ return -EINVAL;
+
+ ret = arm_lpae_init_pgtable(cfg, data);
+ if (ret)
+ return ret;
+
+ /* TCR */
+ if (cfg->coherent_walk) {
+ tcr->sh = ARM_LPAE_TCR_SH_IS;
+ tcr->irgn = ARM_LPAE_TCR_RGN_WBWA;
+ tcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
+ if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_OUTER_WBWA)
+ return -EINVAL;
+ } else {
+ tcr->sh = ARM_LPAE_TCR_SH_OS;
+ tcr->irgn = ARM_LPAE_TCR_RGN_NC;
+ if (!(cfg->quirks & IO_PGTABLE_QUIRK_ARM_OUTER_WBWA))
+ tcr->orgn = ARM_LPAE_TCR_RGN_NC;
+ else
+ tcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
+ }
+
+ tg1 = cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1;
+ switch (ARM_LPAE_GRANULE(data)) {
+ case SZ_4K:
+ tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_4K : ARM_LPAE_TCR_TG0_4K;
+ break;
+ case SZ_16K:
+ tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_16K : ARM_LPAE_TCR_TG0_16K;
+ break;
+ case SZ_64K:
+ tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_64K : ARM_LPAE_TCR_TG0_64K;
+ break;
+ }
+
+ switch (cfg->oas) {
+ case 32:
+ tcr->ips = ARM_LPAE_TCR_PS_32_BIT;
+ break;
+ case 36:
+ tcr->ips = ARM_LPAE_TCR_PS_36_BIT;
+ break;
+ case 40:
+ tcr->ips = ARM_LPAE_TCR_PS_40_BIT;
+ break;
+ case 42:
+ tcr->ips = ARM_LPAE_TCR_PS_42_BIT;
+ break;
+ case 44:
+ tcr->ips = ARM_LPAE_TCR_PS_44_BIT;
+ break;
+ case 48:
+ tcr->ips = ARM_LPAE_TCR_PS_48_BIT;
+ break;
+ case 52:
+ tcr->ips = ARM_LPAE_TCR_PS_52_BIT;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ tcr->tsz = 64ULL - cfg->ias;
+
+ /* MAIRs */
+ reg = (ARM_LPAE_MAIR_ATTR_NC
+ << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_NC)) |
+ (ARM_LPAE_MAIR_ATTR_WBRWA
+ << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_CACHE)) |
+ (ARM_LPAE_MAIR_ATTR_DEVICE
+ << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_DEV)) |
+ (ARM_LPAE_MAIR_ATTR_INC_OWBRWA
+ << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_INC_OCACHE));
+
+ cfg->arm_lpae_s1_cfg.mair = reg;
+
+ /* Looking good; allocate a pgd */
+ data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data),
+ GFP_KERNEL, cfg, cookie);
+ if (!data->pgd)
+ return -ENOMEM;
+
+ /* Ensure the empty pgd is visible before any actual TTBR write */
+ wmb();
+
+ /* TTBR */
+ cfg->arm_lpae_s1_cfg.ttbr = __arm_lpae_virt_to_phys(data->pgd);
+ return 0;
+}
+
+int arm_lpae_init_pgtable_s2(struct io_pgtable_cfg *cfg,
+ struct arm_lpae_io_pgtable *data,
+ void *cookie)
+{
+ u64 sl;
+ int ret;
+ typeof(&cfg->arm_lpae_s2_cfg.vtcr) vtcr = &cfg->arm_lpae_s2_cfg.vtcr;
+
+ if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_S2FWB |
+ IO_PGTABLE_QUIRK_NO_WARN))
+ return -EINVAL;
+
+ ret = arm_lpae_init_pgtable(cfg, data);
+ if (ret)
+ return ret;
+
+ if (arm_lpae_concat_mandatory(cfg, data)) {
+ if (WARN_ON((ARM_LPAE_PGD_SIZE(data) / sizeof(arm_lpae_iopte)) >
+ ARM_LPAE_S2_MAX_CONCAT_PAGES))
+ return -EINVAL;
+ data->pgd_bits += data->bits_per_level;
+ data->start_level++;
+ }
+
+ /* VTCR */
+ if (cfg->coherent_walk) {
+ vtcr->sh = ARM_LPAE_TCR_SH_IS;
+ vtcr->irgn = ARM_LPAE_TCR_RGN_WBWA;
+ vtcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
+ } else {
+ vtcr->sh = ARM_LPAE_TCR_SH_OS;
+ vtcr->irgn = ARM_LPAE_TCR_RGN_NC;
+ vtcr->orgn = ARM_LPAE_TCR_RGN_NC;
+ }
+
+ sl = data->start_level;
+
+ switch (ARM_LPAE_GRANULE(data)) {
+ case SZ_4K:
+ vtcr->tg = ARM_LPAE_TCR_TG0_4K;
+ sl++; /* SL0 format is different for 4K granule size */
+ break;
+ case SZ_16K:
+ vtcr->tg = ARM_LPAE_TCR_TG0_16K;
+ break;
+ case SZ_64K:
+ vtcr->tg = ARM_LPAE_TCR_TG0_64K;
+ break;
+ }
+
+ switch (cfg->oas) {
+ case 32:
+ vtcr->ps = ARM_LPAE_TCR_PS_32_BIT;
+ break;
+ case 36:
+ vtcr->ps = ARM_LPAE_TCR_PS_36_BIT;
+ break;
+ case 40:
+ vtcr->ps = ARM_LPAE_TCR_PS_40_BIT;
+ break;
+ case 42:
+ vtcr->ps = ARM_LPAE_TCR_PS_42_BIT;
+ break;
+ case 44:
+ vtcr->ps = ARM_LPAE_TCR_PS_44_BIT;
+ break;
+ case 48:
+ vtcr->ps = ARM_LPAE_TCR_PS_48_BIT;
+ break;
+ case 52:
+ vtcr->ps = ARM_LPAE_TCR_PS_52_BIT;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ vtcr->tsz = 64ULL - cfg->ias;
+ vtcr->sl = ~sl & ARM_LPAE_VTCR_SL0_MASK;
+
+ /* Allocate pgd pages */
+ data->pgd = __arm_lpae_alloc_pages(PAGE_ALIGN(ARM_LPAE_PGD_SIZE(data)),
+ GFP_KERNEL, cfg, cookie);
+ if (!data->pgd)
+ return -ENOMEM;
+
+ /* Ensure the empty pgd is visible before any actual TTBR write */
+ wmb();
+
+ /* VTTBR */
+ cfg->arm_lpae_s2_cfg.vttbr = __arm_lpae_virt_to_phys(data->pgd);
+ return 0;
+}
+
/*
* Convert an index returned by ARM_LPAE_PGD_IDX(), which can point into
* a concatenated PGD, into the maximum number of entries that can be
diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c
index ca4467ad3c40..dad6964f462a 100644
--- a/drivers/iommu/io-pgtable-arm.c
+++ b/drivers/iommu/io-pgtable-arm.c
@@ -36,34 +36,6 @@ void arm_lpae_split_blk(void)
WARN_ONCE(true, "Unmap of a partial large IOPTE is not allowed");
}
-/*
- * Check if concatenated PGDs are mandatory according to Arm DDI0487 (K.a)
- * 1) R_DXBSH: For 16KB, and 48-bit input size, use level 1 instead of 0.
- * 2) R_SRKBC: After de-ciphering the table for PA size and valid initial lookup
- * a) 40 bits PA size with 4K: use level 1 instead of level 0 (2 tables for ias = oas)
- * b) 40 bits PA size with 16K: use level 2 instead of level 1 (16 tables for ias = oas)
- * c) 42 bits PA size with 4K: use level 1 instead of level 0 (8 tables for ias = oas)
- * d) 48 bits PA size with 16K: use level 1 instead of level 0 (2 tables for ias = oas)
- */
-static inline bool arm_lpae_concat_mandatory(struct io_pgtable_cfg *cfg,
- struct arm_lpae_io_pgtable *data)
-{
- unsigned int ias = cfg->ias;
- unsigned int oas = cfg->oas;
-
- /* Covers 1 and 2.d */
- if ((ARM_LPAE_GRANULE(data) == SZ_16K) && (data->start_level == 0))
- return (oas == 48) || (ias == 48);
-
- /* Covers 2.a and 2.c */
- if ((ARM_LPAE_GRANULE(data) == SZ_4K) && (data->start_level == 0))
- return (oas == 40) || (oas == 42);
-
- /* Case 2.b */
- return (ARM_LPAE_GRANULE(data) == SZ_16K) &&
- (data->start_level == 1) && (oas == 40);
-}
-
static dma_addr_t __arm_lpae_dma_addr(void *pages)
{
return (dma_addr_t)virt_to_phys(pages);
@@ -317,80 +289,15 @@ static int arm_lpae_read_and_clear_dirty(struct io_pgtable_ops *ops,
return __arm_lpae_iopte_walk(data, &walk_data, ptep, lvl);
}
-static void arm_lpae_restrict_pgsizes(struct io_pgtable_cfg *cfg)
-{
- unsigned long granule, page_sizes;
- unsigned int max_addr_bits = 48;
-
- /*
- * We need to restrict the supported page sizes to match the
- * translation regime for a particular granule. Aim to match
- * the CPU page size if possible, otherwise prefer smaller sizes.
- * While we're at it, restrict the block sizes to match the
- * chosen granule.
- */
- if (cfg->pgsize_bitmap & PAGE_SIZE)
- granule = PAGE_SIZE;
- else if (cfg->pgsize_bitmap & ~PAGE_MASK)
- granule = 1UL << __fls(cfg->pgsize_bitmap & ~PAGE_MASK);
- else if (cfg->pgsize_bitmap & PAGE_MASK)
- granule = 1UL << __ffs(cfg->pgsize_bitmap & PAGE_MASK);
- else
- granule = 0;
-
- switch (granule) {
- case SZ_4K:
- page_sizes = (SZ_4K | SZ_2M | SZ_1G);
- break;
- case SZ_16K:
- page_sizes = (SZ_16K | SZ_32M);
- break;
- case SZ_64K:
- max_addr_bits = 52;
- page_sizes = (SZ_64K | SZ_512M);
- if (cfg->oas > 48)
- page_sizes |= 1ULL << 42; /* 4TB */
- break;
- default:
- page_sizes = 0;
- }
-
- cfg->pgsize_bitmap &= page_sizes;
- cfg->ias = min(cfg->ias, max_addr_bits);
- cfg->oas = min(cfg->oas, max_addr_bits);
-}
-
static struct arm_lpae_io_pgtable *
arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg)
{
struct arm_lpae_io_pgtable *data;
- int levels, va_bits, pg_shift;
-
- arm_lpae_restrict_pgsizes(cfg);
-
- if (!(cfg->pgsize_bitmap & (SZ_4K | SZ_16K | SZ_64K)))
- return NULL;
-
- if (cfg->ias > ARM_LPAE_MAX_ADDR_BITS)
- return NULL;
-
- if (cfg->oas > ARM_LPAE_MAX_ADDR_BITS)
- return NULL;
data = kmalloc(sizeof(*data), GFP_KERNEL);
if (!data)
return NULL;
- pg_shift = __ffs(cfg->pgsize_bitmap);
- data->bits_per_level = pg_shift - ilog2(sizeof(arm_lpae_iopte));
-
- va_bits = cfg->ias - pg_shift;
- levels = DIV_ROUND_UP(va_bits, data->bits_per_level);
- data->start_level = ARM_LPAE_MAX_LEVELS - levels;
-
- /* Calculate the actual size of our pgd (without concatenation) */
- data->pgd_bits = va_bits - (data->bits_per_level * (levels - 1));
-
data->iop.ops = (struct io_pgtable_ops) {
.map_pages = arm_lpae_map_pages,
.unmap_pages = arm_lpae_unmap_pages,
@@ -405,203 +312,31 @@ arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg)
static struct io_pgtable *
arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg, void *cookie)
{
- u64 reg;
struct arm_lpae_io_pgtable *data;
- typeof(&cfg->arm_lpae_s1_cfg.tcr) tcr = &cfg->arm_lpae_s1_cfg.tcr;
- bool tg1;
-
- if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS |
- IO_PGTABLE_QUIRK_ARM_TTBR1 |
- IO_PGTABLE_QUIRK_ARM_OUTER_WBWA |
- IO_PGTABLE_QUIRK_ARM_HD |
- IO_PGTABLE_QUIRK_NO_WARN))
- return NULL;
data = arm_lpae_alloc_pgtable(cfg);
if (!data)
return NULL;
-
- /* TCR */
- if (cfg->coherent_walk) {
- tcr->sh = ARM_LPAE_TCR_SH_IS;
- tcr->irgn = ARM_LPAE_TCR_RGN_WBWA;
- tcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
- if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_OUTER_WBWA)
- goto out_free_data;
- } else {
- tcr->sh = ARM_LPAE_TCR_SH_OS;
- tcr->irgn = ARM_LPAE_TCR_RGN_NC;
- if (!(cfg->quirks & IO_PGTABLE_QUIRK_ARM_OUTER_WBWA))
- tcr->orgn = ARM_LPAE_TCR_RGN_NC;
- else
- tcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
- }
-
- tg1 = cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1;
- switch (ARM_LPAE_GRANULE(data)) {
- case SZ_4K:
- tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_4K : ARM_LPAE_TCR_TG0_4K;
- break;
- case SZ_16K:
- tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_16K : ARM_LPAE_TCR_TG0_16K;
- break;
- case SZ_64K:
- tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_64K : ARM_LPAE_TCR_TG0_64K;
- break;
- }
-
- switch (cfg->oas) {
- case 32:
- tcr->ips = ARM_LPAE_TCR_PS_32_BIT;
- break;
- case 36:
- tcr->ips = ARM_LPAE_TCR_PS_36_BIT;
- break;
- case 40:
- tcr->ips = ARM_LPAE_TCR_PS_40_BIT;
- break;
- case 42:
- tcr->ips = ARM_LPAE_TCR_PS_42_BIT;
- break;
- case 44:
- tcr->ips = ARM_LPAE_TCR_PS_44_BIT;
- break;
- case 48:
- tcr->ips = ARM_LPAE_TCR_PS_48_BIT;
- break;
- case 52:
- tcr->ips = ARM_LPAE_TCR_PS_52_BIT;
- break;
- default:
- goto out_free_data;
+ if (arm_lpae_init_pgtable_s1(cfg, data, cookie)) {
+ kfree(data);
+ return NULL;
}
-
- tcr->tsz = 64ULL - cfg->ias;
-
- /* MAIRs */
- reg = (ARM_LPAE_MAIR_ATTR_NC
- << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_NC)) |
- (ARM_LPAE_MAIR_ATTR_WBRWA
- << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_CACHE)) |
- (ARM_LPAE_MAIR_ATTR_DEVICE
- << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_DEV)) |
- (ARM_LPAE_MAIR_ATTR_INC_OWBRWA
- << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_INC_OCACHE));
-
- cfg->arm_lpae_s1_cfg.mair = reg;
-
- /* Looking good; allocate a pgd */
- data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data),
- GFP_KERNEL, cfg, cookie);
- if (!data->pgd)
- goto out_free_data;
-
- /* Ensure the empty pgd is visible before any actual TTBR write */
- wmb();
-
- /* TTBR */
- cfg->arm_lpae_s1_cfg.ttbr = virt_to_phys(data->pgd);
return &data->iop;
-
-out_free_data:
- kfree(data);
- return NULL;
}
static struct io_pgtable *
arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie)
{
- u64 sl;
struct arm_lpae_io_pgtable *data;
- typeof(&cfg->arm_lpae_s2_cfg.vtcr) vtcr = &cfg->arm_lpae_s2_cfg.vtcr;
-
- if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_S2FWB |
- IO_PGTABLE_QUIRK_NO_WARN))
- return NULL;
data = arm_lpae_alloc_pgtable(cfg);
if (!data)
return NULL;
-
- if (arm_lpae_concat_mandatory(cfg, data)) {
- if (WARN_ON((ARM_LPAE_PGD_SIZE(data) / sizeof(arm_lpae_iopte)) >
- ARM_LPAE_S2_MAX_CONCAT_PAGES))
- return NULL;
- data->pgd_bits += data->bits_per_level;
- data->start_level++;
- }
-
- /* VTCR */
- if (cfg->coherent_walk) {
- vtcr->sh = ARM_LPAE_TCR_SH_IS;
- vtcr->irgn = ARM_LPAE_TCR_RGN_WBWA;
- vtcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
- } else {
- vtcr->sh = ARM_LPAE_TCR_SH_OS;
- vtcr->irgn = ARM_LPAE_TCR_RGN_NC;
- vtcr->orgn = ARM_LPAE_TCR_RGN_NC;
- }
-
- sl = data->start_level;
-
- switch (ARM_LPAE_GRANULE(data)) {
- case SZ_4K:
- vtcr->tg = ARM_LPAE_TCR_TG0_4K;
- sl++; /* SL0 format is different for 4K granule size */
- break;
- case SZ_16K:
- vtcr->tg = ARM_LPAE_TCR_TG0_16K;
- break;
- case SZ_64K:
- vtcr->tg = ARM_LPAE_TCR_TG0_64K;
- break;
- }
-
- switch (cfg->oas) {
- case 32:
- vtcr->ps = ARM_LPAE_TCR_PS_32_BIT;
- break;
- case 36:
- vtcr->ps = ARM_LPAE_TCR_PS_36_BIT;
- break;
- case 40:
- vtcr->ps = ARM_LPAE_TCR_PS_40_BIT;
- break;
- case 42:
- vtcr->ps = ARM_LPAE_TCR_PS_42_BIT;
- break;
- case 44:
- vtcr->ps = ARM_LPAE_TCR_PS_44_BIT;
- break;
- case 48:
- vtcr->ps = ARM_LPAE_TCR_PS_48_BIT;
- break;
- case 52:
- vtcr->ps = ARM_LPAE_TCR_PS_52_BIT;
- break;
- default:
- goto out_free_data;
+ if (arm_lpae_init_pgtable_s2(cfg, data, cookie)) {
+ kfree(data);
+ return NULL;
}
-
- vtcr->tsz = 64ULL - cfg->ias;
- vtcr->sl = ~sl & ARM_LPAE_VTCR_SL0_MASK;
-
- /* Allocate pgd pages */
- data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data),
- GFP_KERNEL, cfg, cookie);
- if (!data->pgd)
- goto out_free_data;
-
- /* Ensure the empty pgd is visible before any actual TTBR write */
- wmb();
-
- /* VTTBR */
- cfg->arm_lpae_s2_cfg.vttbr = virt_to_phys(data->pgd);
return &data->iop;
-
-out_free_data:
- kfree(data);
- return NULL;
}
static struct io_pgtable *
@@ -642,6 +377,9 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)
if (!data)
return NULL;
+ if (arm_lpae_init_pgtable(cfg, data))
+ goto out_free_data;
+
/* Mali seems to need a full 4-level table regardless of IAS */
if (data->start_level > 0) {
data->start_level = 0;
diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h
index c3a3b4fd44c3..2807cf563f11 100644
--- a/drivers/iommu/io-pgtable-arm.h
+++ b/drivers/iommu/io-pgtable-arm.h
@@ -230,4 +230,12 @@ void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries,
void __arm_lpae_free_pgtable(struct arm_lpae_io_pgtable *data, int lvl,
arm_lpae_iopte *ptep);
void arm_lpae_split_blk(void);
+int arm_lpae_init_pgtable(struct io_pgtable_cfg *cfg,
+ struct arm_lpae_io_pgtable *data);
+int arm_lpae_init_pgtable_s1(struct io_pgtable_cfg *cfg,
+ struct arm_lpae_io_pgtable *data,
+ void *cookie);
+int arm_lpae_init_pgtable_s2(struct io_pgtable_cfg *cfg,
+ struct arm_lpae_io_pgtable *data,
+ void *cookie);
#endif /* IO_PGTABLE_H_ */
\ No newline at end of file
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 06/29] iommu/arm-smmu-v3: Move some definitions to a new common file
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (4 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 05/29] iommu/io-pgtable-arm: Split initialization Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 07/29] iommu/arm-smmu-v3: Extract driver-specific bits from probe function Mostafa Saleh
` (22 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
So that the KVM SMMUv3 driver can re-use architectural definitions,
command structures and feature bits, move them to a new common
header that can be included by both the kernel and the hypervisor.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../arm/arm-smmu-v3/arm-smmu-v3-common.h | 462 ++++++++++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 456 +----------------
2 files changed, 464 insertions(+), 454 deletions(-)
create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
new file mode 100644
index 000000000000..0f9ae032680d
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
@@ -0,0 +1,462 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _ARM_SMMU_V3_COMMON_H
+#define _ARM_SMMU_V3_COMMON_H
+
+#include <linux/bitfield.h>
+
+/* MMIO registers */
+#define ARM_SMMU_IDR0 0x0
+#define IDR0_ST_LVL GENMASK(28, 27)
+#define IDR0_ST_LVL_2LVL 1
+#define IDR0_STALL_MODEL GENMASK(25, 24)
+#define IDR0_STALL_MODEL_STALL 0
+#define IDR0_STALL_MODEL_FORCE 2
+#define IDR0_TTENDIAN GENMASK(22, 21)
+#define IDR0_TTENDIAN_MIXED 0
+#define IDR0_TTENDIAN_LE 2
+#define IDR0_TTENDIAN_BE 3
+#define IDR0_CD2L (1 << 19)
+#define IDR0_VMID16 (1 << 18)
+#define IDR0_PRI (1 << 16)
+#define IDR0_SEV (1 << 14)
+#define IDR0_MSI (1 << 13)
+#define IDR0_ASID16 (1 << 12)
+#define IDR0_ATS (1 << 10)
+#define IDR0_HYP (1 << 9)
+#define IDR0_HTTU GENMASK(7, 6)
+#define IDR0_HTTU_ACCESS 1
+#define IDR0_HTTU_ACCESS_DIRTY 2
+#define IDR0_COHACC (1 << 4)
+#define IDR0_TTF GENMASK(3, 2)
+#define IDR0_TTF_AARCH64 2
+#define IDR0_TTF_AARCH32_64 3
+#define IDR0_S1P (1 << 1)
+#define IDR0_S2P (1 << 0)
+
+#define ARM_SMMU_IDR1 0x4
+#define IDR1_TABLES_PRESET (1 << 30)
+#define IDR1_QUEUES_PRESET (1 << 29)
+#define IDR1_REL (1 << 28)
+#define IDR1_ATTR_TYPES_OVR (1 << 27)
+#define IDR1_CMDQS GENMASK(25, 21)
+#define IDR1_EVTQS GENMASK(20, 16)
+#define IDR1_PRIQS GENMASK(15, 11)
+#define IDR1_SSIDSIZE GENMASK(10, 6)
+#define IDR1_SIDSIZE GENMASK(5, 0)
+
+#define ARM_SMMU_IDR3 0xc
+#define IDR3_FWB (1 << 8)
+#define IDR3_RIL (1 << 10)
+
+#define ARM_SMMU_IDR5 0x14
+#define IDR5_STALL_MAX GENMASK(31, 16)
+#define IDR5_GRAN64K (1 << 6)
+#define IDR5_GRAN16K (1 << 5)
+#define IDR5_GRAN4K (1 << 4)
+#define IDR5_OAS GENMASK(2, 0)
+#define IDR5_OAS_32_BIT 0
+#define IDR5_OAS_36_BIT 1
+#define IDR5_OAS_40_BIT 2
+#define IDR5_OAS_42_BIT 3
+#define IDR5_OAS_44_BIT 4
+#define IDR5_OAS_48_BIT 5
+#define IDR5_OAS_52_BIT 6
+#define IDR5_VAX GENMASK(11, 10)
+#define IDR5_VAX_52_BIT 1
+
+#define ARM_SMMU_IIDR 0x18
+#define IIDR_PRODUCTID GENMASK(31, 20)
+#define IIDR_VARIANT GENMASK(19, 16)
+#define IIDR_REVISION GENMASK(15, 12)
+#define IIDR_IMPLEMENTER GENMASK(11, 0)
+
+#define ARM_SMMU_AIDR 0x1C
+
+#define ARM_SMMU_CR0 0x20
+#define CR0_ATSCHK (1 << 4)
+#define CR0_CMDQEN (1 << 3)
+#define CR0_EVTQEN (1 << 2)
+#define CR0_PRIQEN (1 << 1)
+#define CR0_SMMUEN (1 << 0)
+
+#define ARM_SMMU_CR0ACK 0x24
+
+#define ARM_SMMU_CR1 0x28
+#define CR1_TABLE_SH GENMASK(11, 10)
+#define CR1_TABLE_OC GENMASK(9, 8)
+#define CR1_TABLE_IC GENMASK(7, 6)
+#define CR1_QUEUE_SH GENMASK(5, 4)
+#define CR1_QUEUE_OC GENMASK(3, 2)
+#define CR1_QUEUE_IC GENMASK(1, 0)
+/* CR1 cacheability fields don't quite follow the usual TCR-style encoding */
+#define CR1_CACHE_NC 0
+#define CR1_CACHE_WB 1
+#define CR1_CACHE_WT 2
+
+#define ARM_SMMU_CR2 0x2c
+#define CR2_PTM (1 << 2)
+#define CR2_RECINVSID (1 << 1)
+#define CR2_E2H (1 << 0)
+
+#define ARM_SMMU_GBPA 0x44
+#define GBPA_UPDATE (1 << 31)
+#define GBPA_ABORT (1 << 20)
+
+#define ARM_SMMU_IRQ_CTRL 0x50
+#define IRQ_CTRL_EVTQ_IRQEN (1 << 2)
+#define IRQ_CTRL_PRIQ_IRQEN (1 << 1)
+#define IRQ_CTRL_GERROR_IRQEN (1 << 0)
+
+#define ARM_SMMU_IRQ_CTRLACK 0x54
+
+#define ARM_SMMU_GERROR 0x60
+#define GERROR_SFM_ERR (1 << 8)
+#define GERROR_MSI_GERROR_ABT_ERR (1 << 7)
+#define GERROR_MSI_PRIQ_ABT_ERR (1 << 6)
+#define GERROR_MSI_EVTQ_ABT_ERR (1 << 5)
+#define GERROR_MSI_CMDQ_ABT_ERR (1 << 4)
+#define GERROR_PRIQ_ABT_ERR (1 << 3)
+#define GERROR_EVTQ_ABT_ERR (1 << 2)
+#define GERROR_CMDQ_ERR (1 << 0)
+#define GERROR_ERR_MASK 0x1fd
+
+#define ARM_SMMU_GERRORN 0x64
+
+#define ARM_SMMU_GERROR_IRQ_CFG0 0x68
+#define ARM_SMMU_GERROR_IRQ_CFG1 0x70
+#define ARM_SMMU_GERROR_IRQ_CFG2 0x74
+
+#define ARM_SMMU_STRTAB_BASE 0x80
+#define STRTAB_BASE_RA (1UL << 62)
+#define STRTAB_BASE_ADDR_MASK GENMASK_ULL(51, 6)
+
+#define ARM_SMMU_STRTAB_BASE_CFG 0x88
+#define STRTAB_BASE_CFG_FMT GENMASK(17, 16)
+#define STRTAB_BASE_CFG_FMT_LINEAR 0
+#define STRTAB_BASE_CFG_FMT_2LVL 1
+#define STRTAB_BASE_CFG_SPLIT GENMASK(10, 6)
+#define STRTAB_BASE_CFG_LOG2SIZE GENMASK(5, 0)
+
+#define ARM_SMMU_CMDQ_BASE 0x90
+#define ARM_SMMU_CMDQ_PROD 0x98
+#define ARM_SMMU_CMDQ_CONS 0x9c
+
+#define ARM_SMMU_EVTQ_BASE 0xa0
+#define ARM_SMMU_EVTQ_PROD 0xa8
+#define ARM_SMMU_EVTQ_CONS 0xac
+#define ARM_SMMU_EVTQ_IRQ_CFG0 0xb0
+#define ARM_SMMU_EVTQ_IRQ_CFG1 0xb8
+#define ARM_SMMU_EVTQ_IRQ_CFG2 0xbc
+
+#define ARM_SMMU_PRIQ_BASE 0xc0
+#define ARM_SMMU_PRIQ_PROD 0xc8
+#define ARM_SMMU_PRIQ_CONS 0xcc
+#define ARM_SMMU_PRIQ_IRQ_CFG0 0xd0
+#define ARM_SMMU_PRIQ_IRQ_CFG1 0xd8
+#define ARM_SMMU_PRIQ_IRQ_CFG2 0xdc
+
+#define ARM_SMMU_REG_SZ 0xe00
+
+/* Common MSI config fields */
+#define MSI_CFG0_ADDR_MASK GENMASK_ULL(51, 2)
+#define MSI_CFG2_SH GENMASK(5, 4)
+#define MSI_CFG2_MEMATTR GENMASK(3, 0)
+
+/* Common memory attribute values */
+#define ARM_SMMU_SH_NSH 0
+#define ARM_SMMU_SH_OSH 2
+#define ARM_SMMU_SH_ISH 3
+#define ARM_SMMU_MEMATTR_DEVICE_nGnRE 0x1
+#define ARM_SMMU_MEMATTR_OIWB 0xf
+
+#define Q_BASE_RWA (1UL << 62)
+#define Q_BASE_ADDR_MASK GENMASK_ULL(51, 5)
+#define Q_BASE_LOG2SIZE GENMASK(4, 0)
+
+/*
+ * Stream table.
+ *
+ * Linear: Enough to cover 1 << IDR1.SIDSIZE entries
+ * 2lvl: 128k L1 entries,
+ * 256 lazy entries per table (each table covers a PCI bus)
+ */
+#define STRTAB_SPLIT 8
+
+#define STRTAB_L1_DESC_SPAN GENMASK_ULL(4, 0)
+#define STRTAB_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 6)
+
+#define STRTAB_STE_DWORDS 8
+
+struct arm_smmu_ste {
+ __le64 data[STRTAB_STE_DWORDS];
+};
+
+#define STRTAB_NUM_L2_STES (1 << STRTAB_SPLIT)
+struct arm_smmu_strtab_l2 {
+ struct arm_smmu_ste stes[STRTAB_NUM_L2_STES];
+};
+
+struct arm_smmu_strtab_l1 {
+ __le64 l2ptr;
+};
+#define STRTAB_MAX_L1_ENTRIES (1 << 17)
+
+static inline u32 arm_smmu_strtab_l1_idx(u32 sid)
+{
+ return sid / STRTAB_NUM_L2_STES;
+}
+
+static inline u32 arm_smmu_strtab_l2_idx(u32 sid)
+{
+ return sid % STRTAB_NUM_L2_STES;
+}
+
+#define STRTAB_STE_0_V (1UL << 0)
+#define STRTAB_STE_0_CFG GENMASK_ULL(3, 1)
+#define STRTAB_STE_0_CFG_ABORT 0
+#define STRTAB_STE_0_CFG_BYPASS 4
+#define STRTAB_STE_0_CFG_S1_TRANS 5
+#define STRTAB_STE_0_CFG_S2_TRANS 6
+#define STRTAB_STE_0_CFG_NESTED 7
+
+#define STRTAB_STE_0_S1FMT GENMASK_ULL(5, 4)
+#define STRTAB_STE_0_S1FMT_LINEAR 0
+#define STRTAB_STE_0_S1FMT_64K_L2 2
+#define STRTAB_STE_0_S1CTXPTR_MASK GENMASK_ULL(51, 6)
+#define STRTAB_STE_0_S1CDMAX GENMASK_ULL(63, 59)
+
+#define STRTAB_STE_1_S1DSS GENMASK_ULL(1, 0)
+#define STRTAB_STE_1_S1DSS_TERMINATE 0x0
+#define STRTAB_STE_1_S1DSS_BYPASS 0x1
+#define STRTAB_STE_1_S1DSS_SSID0 0x2
+
+#define STRTAB_STE_1_S1C_CACHE_NC 0UL
+#define STRTAB_STE_1_S1C_CACHE_WBRA 1UL
+#define STRTAB_STE_1_S1C_CACHE_WT 2UL
+#define STRTAB_STE_1_S1C_CACHE_WB 3UL
+#define STRTAB_STE_1_S1CIR GENMASK_ULL(3, 2)
+#define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4)
+#define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6)
+
+#define STRTAB_STE_1_MEV (1UL << 19)
+#define STRTAB_STE_1_S2FWB (1UL << 25)
+#define STRTAB_STE_1_S1STALLD (1UL << 27)
+
+#define STRTAB_STE_1_EATS GENMASK_ULL(29, 28)
+#define STRTAB_STE_1_EATS_ABT 0UL
+#define STRTAB_STE_1_EATS_TRANS 1UL
+#define STRTAB_STE_1_EATS_S1CHK 2UL
+
+#define STRTAB_STE_1_STRW GENMASK_ULL(31, 30)
+#define STRTAB_STE_1_STRW_NSEL1 0UL
+#define STRTAB_STE_1_STRW_EL2 2UL
+
+#define STRTAB_STE_1_SHCFG GENMASK_ULL(45, 44)
+#define STRTAB_STE_1_SHCFG_INCOMING 1UL
+
+#define STRTAB_STE_2_S2VMID GENMASK_ULL(15, 0)
+#define STRTAB_STE_2_VTCR GENMASK_ULL(50, 32)
+#define STRTAB_STE_2_VTCR_S2T0SZ GENMASK_ULL(5, 0)
+#define STRTAB_STE_2_VTCR_S2SL0 GENMASK_ULL(7, 6)
+#define STRTAB_STE_2_VTCR_S2IR0 GENMASK_ULL(9, 8)
+#define STRTAB_STE_2_VTCR_S2OR0 GENMASK_ULL(11, 10)
+#define STRTAB_STE_2_VTCR_S2SH0 GENMASK_ULL(13, 12)
+#define STRTAB_STE_2_VTCR_S2TG GENMASK_ULL(15, 14)
+#define STRTAB_STE_2_VTCR_S2PS GENMASK_ULL(18, 16)
+#define STRTAB_STE_2_S2AA64 (1UL << 51)
+#define STRTAB_STE_2_S2ENDI (1UL << 52)
+#define STRTAB_STE_2_S2PTW (1UL << 54)
+#define STRTAB_STE_2_S2S (1UL << 57)
+#define STRTAB_STE_2_S2R (1UL << 58)
+
+#define STRTAB_STE_3_S2TTB_MASK GENMASK_ULL(51, 4)
+
+/* These bits can be controlled by userspace for STRTAB_STE_0_CFG_NESTED */
+#define STRTAB_STE_0_NESTING_ALLOWED \
+ cpu_to_le64(STRTAB_STE_0_V | STRTAB_STE_0_CFG | STRTAB_STE_0_S1FMT | \
+ STRTAB_STE_0_S1CTXPTR_MASK | STRTAB_STE_0_S1CDMAX)
+#define STRTAB_STE_1_NESTING_ALLOWED \
+ cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | \
+ STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | \
+ STRTAB_STE_1_S1STALLD | STRTAB_STE_1_EATS)
+
+/* Command queue */
+#define CMDQ_ENT_SZ_SHIFT 4
+#define CMDQ_ENT_DWORDS ((1 << CMDQ_ENT_SZ_SHIFT) >> 3)
+#define CMDQ_MAX_SZ_SHIFT (Q_MAX_SZ_SHIFT - CMDQ_ENT_SZ_SHIFT)
+
+#define CMDQ_CONS_ERR GENMASK(30, 24)
+#define CMDQ_ERR_CERROR_NONE_IDX 0
+#define CMDQ_ERR_CERROR_ILL_IDX 1
+#define CMDQ_ERR_CERROR_ABT_IDX 2
+#define CMDQ_ERR_CERROR_ATC_INV_IDX 3
+
+#define CMDQ_PROD_OWNED_FLAG Q_OVERFLOW_FLAG
+
+/*
+ * This is used to size the command queue and therefore must be at least
+ * BITS_PER_LONG so that the valid_map works correctly (it relies on the
+ * total number of queue entries being a multiple of BITS_PER_LONG).
+ */
+#define CMDQ_BATCH_ENTRIES BITS_PER_LONG
+
+#define CMDQ_0_OP GENMASK_ULL(7, 0)
+#define CMDQ_0_SSV (1UL << 11)
+
+#define CMDQ_PREFETCH_0_SID GENMASK_ULL(63, 32)
+#define CMDQ_PREFETCH_1_SIZE GENMASK_ULL(4, 0)
+#define CMDQ_PREFETCH_1_ADDR_MASK GENMASK_ULL(63, 12)
+
+#define CMDQ_CFGI_0_SSID GENMASK_ULL(31, 12)
+#define CMDQ_CFGI_0_SID GENMASK_ULL(63, 32)
+#define CMDQ_CFGI_1_LEAF (1UL << 0)
+#define CMDQ_CFGI_1_RANGE GENMASK_ULL(4, 0)
+
+#define CMDQ_TLBI_0_NUM GENMASK_ULL(16, 12)
+#define CMDQ_TLBI_RANGE_NUM_MAX 31
+#define CMDQ_TLBI_0_SCALE GENMASK_ULL(24, 20)
+#define CMDQ_TLBI_0_VMID GENMASK_ULL(47, 32)
+#define CMDQ_TLBI_0_ASID GENMASK_ULL(63, 48)
+#define CMDQ_TLBI_1_LEAF (1UL << 0)
+#define CMDQ_TLBI_1_TTL GENMASK_ULL(9, 8)
+#define CMDQ_TLBI_1_TG GENMASK_ULL(11, 10)
+#define CMDQ_TLBI_1_VA_MASK GENMASK_ULL(63, 12)
+#define CMDQ_TLBI_1_IPA_MASK GENMASK_ULL(51, 12)
+
+#define CMDQ_ATC_0_SSID GENMASK_ULL(31, 12)
+#define CMDQ_ATC_0_SID GENMASK_ULL(63, 32)
+#define CMDQ_ATC_0_GLOBAL (1UL << 9)
+#define CMDQ_ATC_1_SIZE GENMASK_ULL(5, 0)
+#define CMDQ_ATC_1_ADDR_MASK GENMASK_ULL(63, 12)
+
+#define CMDQ_PRI_0_SSID GENMASK_ULL(31, 12)
+#define CMDQ_PRI_0_SID GENMASK_ULL(63, 32)
+#define CMDQ_PRI_1_GRPID GENMASK_ULL(8, 0)
+#define CMDQ_PRI_1_RESP GENMASK_ULL(13, 12)
+
+#define CMDQ_RESUME_0_RESP_TERM 0UL
+#define CMDQ_RESUME_0_RESP_RETRY 1UL
+#define CMDQ_RESUME_0_RESP_ABORT 2UL
+#define CMDQ_RESUME_0_RESP GENMASK_ULL(13, 12)
+#define CMDQ_RESUME_0_SID GENMASK_ULL(63, 32)
+#define CMDQ_RESUME_1_STAG GENMASK_ULL(15, 0)
+
+#define CMDQ_SYNC_0_CS GENMASK_ULL(13, 12)
+#define CMDQ_SYNC_0_CS_NONE 0
+#define CMDQ_SYNC_0_CS_IRQ 1
+#define CMDQ_SYNC_0_CS_SEV 2
+#define CMDQ_SYNC_0_MSH GENMASK_ULL(23, 22)
+#define CMDQ_SYNC_0_MSIATTR GENMASK_ULL(27, 24)
+#define CMDQ_SYNC_0_MSIDATA GENMASK_ULL(63, 32)
+#define CMDQ_SYNC_1_MSIADDR_MASK GENMASK_ULL(51, 2)
+
+enum pri_resp {
+ PRI_RESP_DENY = 0,
+ PRI_RESP_FAIL = 1,
+ PRI_RESP_SUCC = 2,
+};
+
+struct arm_smmu_cmdq_ent {
+ /* Common fields */
+ u8 opcode;
+ bool substream_valid;
+
+ /* Command-specific fields */
+ union {
+ #define CMDQ_OP_PREFETCH_CFG 0x1
+ struct {
+ u32 sid;
+ } prefetch;
+
+ #define CMDQ_OP_CFGI_STE 0x3
+ #define CMDQ_OP_CFGI_ALL 0x4
+ #define CMDQ_OP_CFGI_CD 0x5
+ #define CMDQ_OP_CFGI_CD_ALL 0x6
+ struct {
+ u32 sid;
+ u32 ssid;
+ union {
+ bool leaf;
+ u8 span;
+ };
+ } cfgi;
+
+ #define CMDQ_OP_TLBI_NH_ALL 0x10
+ #define CMDQ_OP_TLBI_NH_ASID 0x11
+ #define CMDQ_OP_TLBI_NH_VA 0x12
+ #define CMDQ_OP_TLBI_NH_VAA 0x13
+ #define CMDQ_OP_TLBI_EL2_ALL 0x20
+ #define CMDQ_OP_TLBI_EL2_ASID 0x21
+ #define CMDQ_OP_TLBI_EL2_VA 0x22
+ #define CMDQ_OP_TLBI_S12_VMALL 0x28
+ #define CMDQ_OP_TLBI_S2_IPA 0x2a
+ #define CMDQ_OP_TLBI_NSNH_ALL 0x30
+ struct {
+ u8 num;
+ u8 scale;
+ u16 asid;
+ u16 vmid;
+ bool leaf;
+ u8 ttl;
+ u8 tg;
+ u64 addr;
+ } tlbi;
+
+ #define CMDQ_OP_ATC_INV 0x40
+ #define ATC_INV_SIZE_ALL 52
+ struct {
+ u32 sid;
+ u32 ssid;
+ u64 addr;
+ u8 size;
+ bool global;
+ } atc;
+
+ #define CMDQ_OP_PRI_RESP 0x41
+ struct {
+ u32 sid;
+ u32 ssid;
+ u16 grpid;
+ enum pri_resp resp;
+ } pri;
+
+ #define CMDQ_OP_RESUME 0x44
+ struct {
+ u32 sid;
+ u16 stag;
+ u8 resp;
+ } resume;
+
+ #define CMDQ_OP_CMD_SYNC 0x46
+ struct {
+ u64 msiaddr;
+ } sync;
+ };
+};
+
+#define ARM_SMMU_FEAT_2_LVL_STRTAB (1 << 0)
+#define ARM_SMMU_FEAT_2_LVL_CDTAB (1 << 1)
+#define ARM_SMMU_FEAT_TT_LE (1 << 2)
+#define ARM_SMMU_FEAT_TT_BE (1 << 3)
+#define ARM_SMMU_FEAT_PRI (1 << 4)
+#define ARM_SMMU_FEAT_ATS (1 << 5)
+#define ARM_SMMU_FEAT_SEV (1 << 6)
+#define ARM_SMMU_FEAT_MSI (1 << 7)
+#define ARM_SMMU_FEAT_COHERENCY (1 << 8)
+#define ARM_SMMU_FEAT_TRANS_S1 (1 << 9)
+#define ARM_SMMU_FEAT_TRANS_S2 (1 << 10)
+#define ARM_SMMU_FEAT_STALLS (1 << 11)
+#define ARM_SMMU_FEAT_HYP (1 << 12)
+#define ARM_SMMU_FEAT_STALL_FORCE (1 << 13)
+#define ARM_SMMU_FEAT_VAX (1 << 14)
+#define ARM_SMMU_FEAT_RANGE_INV (1 << 15)
+#define ARM_SMMU_FEAT_BTM (1 << 16)
+#define ARM_SMMU_FEAT_SVA (1 << 17)
+#define ARM_SMMU_FEAT_E2H (1 << 18)
+#define ARM_SMMU_FEAT_NESTING (1 << 19)
+#define ARM_SMMU_FEAT_ATTR_TYPES_OVR (1 << 20)
+#define ARM_SMMU_FEAT_HA (1 << 21)
+#define ARM_SMMU_FEAT_HD (1 << 22)
+#define ARM_SMMU_FEAT_S2FWB (1 << 23)
+
+#endif /* _ARM_SMMU_V3_COMMON_H */
\ No newline at end of file
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index ea41d790463e..3b88f19fcefc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -8,7 +8,6 @@
#ifndef _ARM_SMMU_V3_H
#define _ARM_SMMU_V3_H
-#include <linux/bitfield.h>
#include <linux/iommu.h>
#include <linux/iommufd.h>
#include <linux/kernel.h>
@@ -17,170 +16,7 @@
struct arm_smmu_device;
-/* MMIO registers */
-#define ARM_SMMU_IDR0 0x0
-#define IDR0_ST_LVL GENMASK(28, 27)
-#define IDR0_ST_LVL_2LVL 1
-#define IDR0_STALL_MODEL GENMASK(25, 24)
-#define IDR0_STALL_MODEL_STALL 0
-#define IDR0_STALL_MODEL_FORCE 2
-#define IDR0_TTENDIAN GENMASK(22, 21)
-#define IDR0_TTENDIAN_MIXED 0
-#define IDR0_TTENDIAN_LE 2
-#define IDR0_TTENDIAN_BE 3
-#define IDR0_CD2L (1 << 19)
-#define IDR0_VMID16 (1 << 18)
-#define IDR0_PRI (1 << 16)
-#define IDR0_SEV (1 << 14)
-#define IDR0_MSI (1 << 13)
-#define IDR0_ASID16 (1 << 12)
-#define IDR0_ATS (1 << 10)
-#define IDR0_HYP (1 << 9)
-#define IDR0_HTTU GENMASK(7, 6)
-#define IDR0_HTTU_ACCESS 1
-#define IDR0_HTTU_ACCESS_DIRTY 2
-#define IDR0_COHACC (1 << 4)
-#define IDR0_TTF GENMASK(3, 2)
-#define IDR0_TTF_AARCH64 2
-#define IDR0_TTF_AARCH32_64 3
-#define IDR0_S1P (1 << 1)
-#define IDR0_S2P (1 << 0)
-
-#define ARM_SMMU_IDR1 0x4
-#define IDR1_TABLES_PRESET (1 << 30)
-#define IDR1_QUEUES_PRESET (1 << 29)
-#define IDR1_REL (1 << 28)
-#define IDR1_ATTR_TYPES_OVR (1 << 27)
-#define IDR1_CMDQS GENMASK(25, 21)
-#define IDR1_EVTQS GENMASK(20, 16)
-#define IDR1_PRIQS GENMASK(15, 11)
-#define IDR1_SSIDSIZE GENMASK(10, 6)
-#define IDR1_SIDSIZE GENMASK(5, 0)
-
-#define ARM_SMMU_IDR3 0xc
-#define IDR3_FWB (1 << 8)
-#define IDR3_RIL (1 << 10)
-
-#define ARM_SMMU_IDR5 0x14
-#define IDR5_STALL_MAX GENMASK(31, 16)
-#define IDR5_GRAN64K (1 << 6)
-#define IDR5_GRAN16K (1 << 5)
-#define IDR5_GRAN4K (1 << 4)
-#define IDR5_OAS GENMASK(2, 0)
-#define IDR5_OAS_32_BIT 0
-#define IDR5_OAS_36_BIT 1
-#define IDR5_OAS_40_BIT 2
-#define IDR5_OAS_42_BIT 3
-#define IDR5_OAS_44_BIT 4
-#define IDR5_OAS_48_BIT 5
-#define IDR5_OAS_52_BIT 6
-#define IDR5_VAX GENMASK(11, 10)
-#define IDR5_VAX_52_BIT 1
-
-#define ARM_SMMU_IIDR 0x18
-#define IIDR_PRODUCTID GENMASK(31, 20)
-#define IIDR_VARIANT GENMASK(19, 16)
-#define IIDR_REVISION GENMASK(15, 12)
-#define IIDR_IMPLEMENTER GENMASK(11, 0)
-
-#define ARM_SMMU_AIDR 0x1C
-
-#define ARM_SMMU_CR0 0x20
-#define CR0_ATSCHK (1 << 4)
-#define CR0_CMDQEN (1 << 3)
-#define CR0_EVTQEN (1 << 2)
-#define CR0_PRIQEN (1 << 1)
-#define CR0_SMMUEN (1 << 0)
-
-#define ARM_SMMU_CR0ACK 0x24
-
-#define ARM_SMMU_CR1 0x28
-#define CR1_TABLE_SH GENMASK(11, 10)
-#define CR1_TABLE_OC GENMASK(9, 8)
-#define CR1_TABLE_IC GENMASK(7, 6)
-#define CR1_QUEUE_SH GENMASK(5, 4)
-#define CR1_QUEUE_OC GENMASK(3, 2)
-#define CR1_QUEUE_IC GENMASK(1, 0)
-/* CR1 cacheability fields don't quite follow the usual TCR-style encoding */
-#define CR1_CACHE_NC 0
-#define CR1_CACHE_WB 1
-#define CR1_CACHE_WT 2
-
-#define ARM_SMMU_CR2 0x2c
-#define CR2_PTM (1 << 2)
-#define CR2_RECINVSID (1 << 1)
-#define CR2_E2H (1 << 0)
-
-#define ARM_SMMU_GBPA 0x44
-#define GBPA_UPDATE (1 << 31)
-#define GBPA_ABORT (1 << 20)
-
-#define ARM_SMMU_IRQ_CTRL 0x50
-#define IRQ_CTRL_EVTQ_IRQEN (1 << 2)
-#define IRQ_CTRL_PRIQ_IRQEN (1 << 1)
-#define IRQ_CTRL_GERROR_IRQEN (1 << 0)
-
-#define ARM_SMMU_IRQ_CTRLACK 0x54
-
-#define ARM_SMMU_GERROR 0x60
-#define GERROR_SFM_ERR (1 << 8)
-#define GERROR_MSI_GERROR_ABT_ERR (1 << 7)
-#define GERROR_MSI_PRIQ_ABT_ERR (1 << 6)
-#define GERROR_MSI_EVTQ_ABT_ERR (1 << 5)
-#define GERROR_MSI_CMDQ_ABT_ERR (1 << 4)
-#define GERROR_PRIQ_ABT_ERR (1 << 3)
-#define GERROR_EVTQ_ABT_ERR (1 << 2)
-#define GERROR_CMDQ_ERR (1 << 0)
-#define GERROR_ERR_MASK 0x1fd
-
-#define ARM_SMMU_GERRORN 0x64
-
-#define ARM_SMMU_GERROR_IRQ_CFG0 0x68
-#define ARM_SMMU_GERROR_IRQ_CFG1 0x70
-#define ARM_SMMU_GERROR_IRQ_CFG2 0x74
-
-#define ARM_SMMU_STRTAB_BASE 0x80
-#define STRTAB_BASE_RA (1UL << 62)
-#define STRTAB_BASE_ADDR_MASK GENMASK_ULL(51, 6)
-
-#define ARM_SMMU_STRTAB_BASE_CFG 0x88
-#define STRTAB_BASE_CFG_FMT GENMASK(17, 16)
-#define STRTAB_BASE_CFG_FMT_LINEAR 0
-#define STRTAB_BASE_CFG_FMT_2LVL 1
-#define STRTAB_BASE_CFG_SPLIT GENMASK(10, 6)
-#define STRTAB_BASE_CFG_LOG2SIZE GENMASK(5, 0)
-
-#define ARM_SMMU_CMDQ_BASE 0x90
-#define ARM_SMMU_CMDQ_PROD 0x98
-#define ARM_SMMU_CMDQ_CONS 0x9c
-
-#define ARM_SMMU_EVTQ_BASE 0xa0
-#define ARM_SMMU_EVTQ_PROD 0xa8
-#define ARM_SMMU_EVTQ_CONS 0xac
-#define ARM_SMMU_EVTQ_IRQ_CFG0 0xb0
-#define ARM_SMMU_EVTQ_IRQ_CFG1 0xb8
-#define ARM_SMMU_EVTQ_IRQ_CFG2 0xbc
-
-#define ARM_SMMU_PRIQ_BASE 0xc0
-#define ARM_SMMU_PRIQ_PROD 0xc8
-#define ARM_SMMU_PRIQ_CONS 0xcc
-#define ARM_SMMU_PRIQ_IRQ_CFG0 0xd0
-#define ARM_SMMU_PRIQ_IRQ_CFG1 0xd8
-#define ARM_SMMU_PRIQ_IRQ_CFG2 0xdc
-
-#define ARM_SMMU_REG_SZ 0xe00
-
-/* Common MSI config fields */
-#define MSI_CFG0_ADDR_MASK GENMASK_ULL(51, 2)
-#define MSI_CFG2_SH GENMASK(5, 4)
-#define MSI_CFG2_MEMATTR GENMASK(3, 0)
-
-/* Common memory attribute values */
-#define ARM_SMMU_SH_NSH 0
-#define ARM_SMMU_SH_OSH 2
-#define ARM_SMMU_SH_ISH 3
-#define ARM_SMMU_MEMATTR_DEVICE_nGnRE 0x1
-#define ARM_SMMU_MEMATTR_OIWB 0xf
+#include "arm-smmu-v3-common.h"
#define Q_IDX(llq, p) ((p) & ((1 << (llq)->max_n_shift) - 1))
#define Q_WRP(llq, p) ((p) & (1 << (llq)->max_n_shift))
@@ -190,10 +26,6 @@ struct arm_smmu_device;
Q_IDX(&((q)->llq), p) * \
(q)->ent_dwords)
-#define Q_BASE_RWA (1UL << 62)
-#define Q_BASE_ADDR_MASK GENMASK_ULL(51, 5)
-#define Q_BASE_LOG2SIZE GENMASK(4, 0)
-
/* Ensure DMA allocations are naturally aligned */
#ifdef CONFIG_CMA_ALIGNMENT
#define Q_MAX_SZ_SHIFT (PAGE_SHIFT + CONFIG_CMA_ALIGNMENT)
@@ -201,113 +33,6 @@ struct arm_smmu_device;
#define Q_MAX_SZ_SHIFT (PAGE_SHIFT + MAX_PAGE_ORDER)
#endif
-/*
- * Stream table.
- *
- * Linear: Enough to cover 1 << IDR1.SIDSIZE entries
- * 2lvl: 128k L1 entries,
- * 256 lazy entries per table (each table covers a PCI bus)
- */
-#define STRTAB_SPLIT 8
-
-#define STRTAB_L1_DESC_SPAN GENMASK_ULL(4, 0)
-#define STRTAB_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 6)
-
-#define STRTAB_STE_DWORDS 8
-
-struct arm_smmu_ste {
- __le64 data[STRTAB_STE_DWORDS];
-};
-
-#define STRTAB_NUM_L2_STES (1 << STRTAB_SPLIT)
-struct arm_smmu_strtab_l2 {
- struct arm_smmu_ste stes[STRTAB_NUM_L2_STES];
-};
-
-struct arm_smmu_strtab_l1 {
- __le64 l2ptr;
-};
-#define STRTAB_MAX_L1_ENTRIES (1 << 17)
-
-static inline u32 arm_smmu_strtab_l1_idx(u32 sid)
-{
- return sid / STRTAB_NUM_L2_STES;
-}
-
-static inline u32 arm_smmu_strtab_l2_idx(u32 sid)
-{
- return sid % STRTAB_NUM_L2_STES;
-}
-
-#define STRTAB_STE_0_V (1UL << 0)
-#define STRTAB_STE_0_CFG GENMASK_ULL(3, 1)
-#define STRTAB_STE_0_CFG_ABORT 0
-#define STRTAB_STE_0_CFG_BYPASS 4
-#define STRTAB_STE_0_CFG_S1_TRANS 5
-#define STRTAB_STE_0_CFG_S2_TRANS 6
-#define STRTAB_STE_0_CFG_NESTED 7
-
-#define STRTAB_STE_0_S1FMT GENMASK_ULL(5, 4)
-#define STRTAB_STE_0_S1FMT_LINEAR 0
-#define STRTAB_STE_0_S1FMT_64K_L2 2
-#define STRTAB_STE_0_S1CTXPTR_MASK GENMASK_ULL(51, 6)
-#define STRTAB_STE_0_S1CDMAX GENMASK_ULL(63, 59)
-
-#define STRTAB_STE_1_S1DSS GENMASK_ULL(1, 0)
-#define STRTAB_STE_1_S1DSS_TERMINATE 0x0
-#define STRTAB_STE_1_S1DSS_BYPASS 0x1
-#define STRTAB_STE_1_S1DSS_SSID0 0x2
-
-#define STRTAB_STE_1_S1C_CACHE_NC 0UL
-#define STRTAB_STE_1_S1C_CACHE_WBRA 1UL
-#define STRTAB_STE_1_S1C_CACHE_WT 2UL
-#define STRTAB_STE_1_S1C_CACHE_WB 3UL
-#define STRTAB_STE_1_S1CIR GENMASK_ULL(3, 2)
-#define STRTAB_STE_1_S1COR GENMASK_ULL(5, 4)
-#define STRTAB_STE_1_S1CSH GENMASK_ULL(7, 6)
-
-#define STRTAB_STE_1_MEV (1UL << 19)
-#define STRTAB_STE_1_S2FWB (1UL << 25)
-#define STRTAB_STE_1_S1STALLD (1UL << 27)
-
-#define STRTAB_STE_1_EATS GENMASK_ULL(29, 28)
-#define STRTAB_STE_1_EATS_ABT 0UL
-#define STRTAB_STE_1_EATS_TRANS 1UL
-#define STRTAB_STE_1_EATS_S1CHK 2UL
-
-#define STRTAB_STE_1_STRW GENMASK_ULL(31, 30)
-#define STRTAB_STE_1_STRW_NSEL1 0UL
-#define STRTAB_STE_1_STRW_EL2 2UL
-
-#define STRTAB_STE_1_SHCFG GENMASK_ULL(45, 44)
-#define STRTAB_STE_1_SHCFG_INCOMING 1UL
-
-#define STRTAB_STE_2_S2VMID GENMASK_ULL(15, 0)
-#define STRTAB_STE_2_VTCR GENMASK_ULL(50, 32)
-#define STRTAB_STE_2_VTCR_S2T0SZ GENMASK_ULL(5, 0)
-#define STRTAB_STE_2_VTCR_S2SL0 GENMASK_ULL(7, 6)
-#define STRTAB_STE_2_VTCR_S2IR0 GENMASK_ULL(9, 8)
-#define STRTAB_STE_2_VTCR_S2OR0 GENMASK_ULL(11, 10)
-#define STRTAB_STE_2_VTCR_S2SH0 GENMASK_ULL(13, 12)
-#define STRTAB_STE_2_VTCR_S2TG GENMASK_ULL(15, 14)
-#define STRTAB_STE_2_VTCR_S2PS GENMASK_ULL(18, 16)
-#define STRTAB_STE_2_S2AA64 (1UL << 51)
-#define STRTAB_STE_2_S2ENDI (1UL << 52)
-#define STRTAB_STE_2_S2PTW (1UL << 54)
-#define STRTAB_STE_2_S2S (1UL << 57)
-#define STRTAB_STE_2_S2R (1UL << 58)
-
-#define STRTAB_STE_3_S2TTB_MASK GENMASK_ULL(51, 4)
-
-/* These bits can be controlled by userspace for STRTAB_STE_0_CFG_NESTED */
-#define STRTAB_STE_0_NESTING_ALLOWED \
- cpu_to_le64(STRTAB_STE_0_V | STRTAB_STE_0_CFG | STRTAB_STE_0_S1FMT | \
- STRTAB_STE_0_S1CTXPTR_MASK | STRTAB_STE_0_S1CDMAX)
-#define STRTAB_STE_1_NESTING_ALLOWED \
- cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | \
- STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | \
- STRTAB_STE_1_S1STALLD | STRTAB_STE_1_EATS)
-
/*
* Context descriptors.
*
@@ -376,76 +101,6 @@ static inline unsigned int arm_smmu_cdtab_l2_idx(unsigned int ssid)
*/
#define CTXDESC_LINEAR_CDMAX ilog2(SZ_64K / sizeof(struct arm_smmu_cd))
-/* Command queue */
-#define CMDQ_ENT_SZ_SHIFT 4
-#define CMDQ_ENT_DWORDS ((1 << CMDQ_ENT_SZ_SHIFT) >> 3)
-#define CMDQ_MAX_SZ_SHIFT (Q_MAX_SZ_SHIFT - CMDQ_ENT_SZ_SHIFT)
-
-#define CMDQ_CONS_ERR GENMASK(30, 24)
-#define CMDQ_ERR_CERROR_NONE_IDX 0
-#define CMDQ_ERR_CERROR_ILL_IDX 1
-#define CMDQ_ERR_CERROR_ABT_IDX 2
-#define CMDQ_ERR_CERROR_ATC_INV_IDX 3
-
-#define CMDQ_PROD_OWNED_FLAG Q_OVERFLOW_FLAG
-
-/*
- * This is used to size the command queue and therefore must be at least
- * BITS_PER_LONG so that the valid_map works correctly (it relies on the
- * total number of queue entries being a multiple of BITS_PER_LONG).
- */
-#define CMDQ_BATCH_ENTRIES BITS_PER_LONG
-
-#define CMDQ_0_OP GENMASK_ULL(7, 0)
-#define CMDQ_0_SSV (1UL << 11)
-
-#define CMDQ_PREFETCH_0_SID GENMASK_ULL(63, 32)
-#define CMDQ_PREFETCH_1_SIZE GENMASK_ULL(4, 0)
-#define CMDQ_PREFETCH_1_ADDR_MASK GENMASK_ULL(63, 12)
-
-#define CMDQ_CFGI_0_SSID GENMASK_ULL(31, 12)
-#define CMDQ_CFGI_0_SID GENMASK_ULL(63, 32)
-#define CMDQ_CFGI_1_LEAF (1UL << 0)
-#define CMDQ_CFGI_1_RANGE GENMASK_ULL(4, 0)
-
-#define CMDQ_TLBI_0_NUM GENMASK_ULL(16, 12)
-#define CMDQ_TLBI_RANGE_NUM_MAX 31
-#define CMDQ_TLBI_0_SCALE GENMASK_ULL(24, 20)
-#define CMDQ_TLBI_0_VMID GENMASK_ULL(47, 32)
-#define CMDQ_TLBI_0_ASID GENMASK_ULL(63, 48)
-#define CMDQ_TLBI_1_LEAF (1UL << 0)
-#define CMDQ_TLBI_1_TTL GENMASK_ULL(9, 8)
-#define CMDQ_TLBI_1_TG GENMASK_ULL(11, 10)
-#define CMDQ_TLBI_1_VA_MASK GENMASK_ULL(63, 12)
-#define CMDQ_TLBI_1_IPA_MASK GENMASK_ULL(51, 12)
-
-#define CMDQ_ATC_0_SSID GENMASK_ULL(31, 12)
-#define CMDQ_ATC_0_SID GENMASK_ULL(63, 32)
-#define CMDQ_ATC_0_GLOBAL (1UL << 9)
-#define CMDQ_ATC_1_SIZE GENMASK_ULL(5, 0)
-#define CMDQ_ATC_1_ADDR_MASK GENMASK_ULL(63, 12)
-
-#define CMDQ_PRI_0_SSID GENMASK_ULL(31, 12)
-#define CMDQ_PRI_0_SID GENMASK_ULL(63, 32)
-#define CMDQ_PRI_1_GRPID GENMASK_ULL(8, 0)
-#define CMDQ_PRI_1_RESP GENMASK_ULL(13, 12)
-
-#define CMDQ_RESUME_0_RESP_TERM 0UL
-#define CMDQ_RESUME_0_RESP_RETRY 1UL
-#define CMDQ_RESUME_0_RESP_ABORT 2UL
-#define CMDQ_RESUME_0_RESP GENMASK_ULL(13, 12)
-#define CMDQ_RESUME_0_SID GENMASK_ULL(63, 32)
-#define CMDQ_RESUME_1_STAG GENMASK_ULL(15, 0)
-
-#define CMDQ_SYNC_0_CS GENMASK_ULL(13, 12)
-#define CMDQ_SYNC_0_CS_NONE 0
-#define CMDQ_SYNC_0_CS_IRQ 1
-#define CMDQ_SYNC_0_CS_SEV 2
-#define CMDQ_SYNC_0_MSH GENMASK_ULL(23, 22)
-#define CMDQ_SYNC_0_MSIATTR GENMASK_ULL(27, 24)
-#define CMDQ_SYNC_0_MSIDATA GENMASK_ULL(63, 32)
-#define CMDQ_SYNC_1_MSIADDR_MASK GENMASK_ULL(51, 2)
-
/* Event queue */
#define EVTQ_ENT_SZ_SHIFT 5
#define EVTQ_ENT_DWORDS ((1 << EVTQ_ENT_SZ_SHIFT) >> 3)
@@ -506,90 +161,6 @@ static inline unsigned int arm_smmu_cdtab_l2_idx(unsigned int ssid)
#define MSI_IOVA_BASE 0x8000000
#define MSI_IOVA_LENGTH 0x100000
-enum pri_resp {
- PRI_RESP_DENY = 0,
- PRI_RESP_FAIL = 1,
- PRI_RESP_SUCC = 2,
-};
-
-struct arm_smmu_cmdq_ent {
- /* Common fields */
- u8 opcode;
- bool substream_valid;
-
- /* Command-specific fields */
- union {
- #define CMDQ_OP_PREFETCH_CFG 0x1
- struct {
- u32 sid;
- } prefetch;
-
- #define CMDQ_OP_CFGI_STE 0x3
- #define CMDQ_OP_CFGI_ALL 0x4
- #define CMDQ_OP_CFGI_CD 0x5
- #define CMDQ_OP_CFGI_CD_ALL 0x6
- struct {
- u32 sid;
- u32 ssid;
- union {
- bool leaf;
- u8 span;
- };
- } cfgi;
-
- #define CMDQ_OP_TLBI_NH_ALL 0x10
- #define CMDQ_OP_TLBI_NH_ASID 0x11
- #define CMDQ_OP_TLBI_NH_VA 0x12
- #define CMDQ_OP_TLBI_NH_VAA 0x13
- #define CMDQ_OP_TLBI_EL2_ALL 0x20
- #define CMDQ_OP_TLBI_EL2_ASID 0x21
- #define CMDQ_OP_TLBI_EL2_VA 0x22
- #define CMDQ_OP_TLBI_S12_VMALL 0x28
- #define CMDQ_OP_TLBI_S2_IPA 0x2a
- #define CMDQ_OP_TLBI_NSNH_ALL 0x30
- struct {
- u8 num;
- u8 scale;
- u16 asid;
- u16 vmid;
- bool leaf;
- u8 ttl;
- u8 tg;
- u64 addr;
- } tlbi;
-
- #define CMDQ_OP_ATC_INV 0x40
- #define ATC_INV_SIZE_ALL 52
- struct {
- u32 sid;
- u32 ssid;
- u64 addr;
- u8 size;
- bool global;
- } atc;
-
- #define CMDQ_OP_PRI_RESP 0x41
- struct {
- u32 sid;
- u32 ssid;
- u16 grpid;
- enum pri_resp resp;
- } pri;
-
- #define CMDQ_OP_RESUME 0x44
- struct {
- u32 sid;
- u16 stag;
- u8 resp;
- } resume;
-
- #define CMDQ_OP_CMD_SYNC 0x46
- struct {
- u64 msiaddr;
- } sync;
- };
-};
-
struct arm_smmu_ll_queue {
union {
u64 val;
@@ -731,30 +302,7 @@ struct arm_smmu_device {
void __iomem *base;
void __iomem *page1;
-#define ARM_SMMU_FEAT_2_LVL_STRTAB (1 << 0)
-#define ARM_SMMU_FEAT_2_LVL_CDTAB (1 << 1)
-#define ARM_SMMU_FEAT_TT_LE (1 << 2)
-#define ARM_SMMU_FEAT_TT_BE (1 << 3)
-#define ARM_SMMU_FEAT_PRI (1 << 4)
-#define ARM_SMMU_FEAT_ATS (1 << 5)
-#define ARM_SMMU_FEAT_SEV (1 << 6)
-#define ARM_SMMU_FEAT_MSI (1 << 7)
-#define ARM_SMMU_FEAT_COHERENCY (1 << 8)
-#define ARM_SMMU_FEAT_TRANS_S1 (1 << 9)
-#define ARM_SMMU_FEAT_TRANS_S2 (1 << 10)
-#define ARM_SMMU_FEAT_STALLS (1 << 11)
-#define ARM_SMMU_FEAT_HYP (1 << 12)
-#define ARM_SMMU_FEAT_STALL_FORCE (1 << 13)
-#define ARM_SMMU_FEAT_VAX (1 << 14)
-#define ARM_SMMU_FEAT_RANGE_INV (1 << 15)
-#define ARM_SMMU_FEAT_BTM (1 << 16)
-#define ARM_SMMU_FEAT_SVA (1 << 17)
-#define ARM_SMMU_FEAT_E2H (1 << 18)
-#define ARM_SMMU_FEAT_NESTING (1 << 19)
-#define ARM_SMMU_FEAT_ATTR_TYPES_OVR (1 << 20)
-#define ARM_SMMU_FEAT_HA (1 << 21)
-#define ARM_SMMU_FEAT_HD (1 << 22)
-#define ARM_SMMU_FEAT_S2FWB (1 << 23)
+ /* See ARM_SMMU_FEAT_* in arm-smmu-v3-common.h*/
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 07/29] iommu/arm-smmu-v3: Extract driver-specific bits from probe function
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (5 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 06/29] iommu/arm-smmu-v3: Move some definitions to a new common file Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 08/29] iommu/arm-smmu-v3: Move some functions to arm-smmu-v3-common.c Mostafa Saleh
` (21 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
As we're about to share the arm_smmu_device_hw_probe() function with the
KVM driver, extract bits that are specific to the normal driver.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 10cc6dc26b7b..801b792dda36 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -4353,7 +4353,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
if (reg & IDR0_MSI) {
smmu->features |= ARM_SMMU_FEAT_MSI;
- if (coherent && !disable_msipolling)
+ if (coherent)
smmu->options |= ARM_SMMU_OPT_MSIPOLL;
}
@@ -4504,11 +4504,6 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
smmu->oas = 48;
}
- if (arm_smmu_ops.pgsize_bitmap == -1UL)
- arm_smmu_ops.pgsize_bitmap = smmu->pgsize_bitmap;
- else
- arm_smmu_ops.pgsize_bitmap |= smmu->pgsize_bitmap;
-
/* Set the DMA mask for our table walker */
if (dma_set_mask_and_coherent(smmu->dev, DMA_BIT_MASK(smmu->oas)))
dev_warn(smmu->dev,
@@ -4522,9 +4517,6 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
arm_smmu_device_iidr_probe(smmu);
- if (arm_smmu_sva_supported(smmu))
- smmu->features |= ARM_SMMU_FEAT_SVA;
-
dev_info(smmu->dev, "ias %lu-bit, oas %lu-bit (features 0x%08x)\n",
smmu->ias, smmu->oas, smmu->features);
return 0;
@@ -4794,6 +4786,17 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
if (ret)
return ret;
+ if (arm_smmu_sva_supported(smmu))
+ smmu->features |= ARM_SMMU_FEAT_SVA;
+
+ if (disable_msipolling)
+ smmu->options &= ~ARM_SMMU_OPT_MSIPOLL;
+
+ if (arm_smmu_ops.pgsize_bitmap == -1UL)
+ arm_smmu_ops.pgsize_bitmap = smmu->pgsize_bitmap;
+ else
+ arm_smmu_ops.pgsize_bitmap |= smmu->pgsize_bitmap;
+
/* Initialise in-memory data structures */
ret = arm_smmu_init_structures(smmu);
if (ret)
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 08/29] iommu/arm-smmu-v3: Move some functions to arm-smmu-v3-common.c
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (6 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 07/29] iommu/arm-smmu-v3: Extract driver-specific bits from probe function Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 09/29] iommu/arm-smmu-v3: Move queue and table allocation " Mostafa Saleh
` (20 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Move functions that can be shared between normal and KVM drivers to
arm-smmu-v3-common.c
Only straightforward moves here. More subtle factoring will be done in
then next patches.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
drivers/iommu/arm/arm-smmu-v3/Makefile | 1 +
.../arm/arm-smmu-v3/arm-smmu-v3-common.c | 367 ++++++++++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 366 -----------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +
4 files changed, 376 insertions(+), 366 deletions(-)
create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm-smmu-v3/Makefile
index 493a659cc66b..0bc0c116b851 100644
--- a/drivers/iommu/arm/arm-smmu-v3/Makefile
+++ b/drivers/iommu/arm/arm-smmu-v3/Makefile
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_ARM_SMMU_V3) += arm_smmu_v3.o
arm_smmu_v3-y := arm-smmu-v3.o
+arm_smmu_v3-y += arm-smmu-v3-common.o
arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) += arm-smmu-v3-iommufd.o
arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) += arm-smmu-v3-sva.o
arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) += tegra241-cmdqv.o
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
new file mode 100644
index 000000000000..c2b14b0c9f53
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
@@ -0,0 +1,367 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/dma-mapping.h>
+#include <linux/iopoll.h>
+#include <linux/pci.h>
+#include <linux/string_choices.h>
+
+#include "arm-smmu-v3.h"
+#include "../../dma-iommu.h"
+
+#define IIDR_IMPLEMENTER_ARM 0x43b
+#define IIDR_PRODUCTID_ARM_MMU_600 0x483
+#define IIDR_PRODUCTID_ARM_MMU_700 0x487
+
+static void arm_smmu_device_iidr_probe(struct arm_smmu_device *smmu)
+{
+ u32 reg;
+ unsigned int implementer, productid, variant, revision;
+
+ reg = readl_relaxed(smmu->base + ARM_SMMU_IIDR);
+ implementer = FIELD_GET(IIDR_IMPLEMENTER, reg);
+ productid = FIELD_GET(IIDR_PRODUCTID, reg);
+ variant = FIELD_GET(IIDR_VARIANT, reg);
+ revision = FIELD_GET(IIDR_REVISION, reg);
+
+ switch (implementer) {
+ case IIDR_IMPLEMENTER_ARM:
+ switch (productid) {
+ case IIDR_PRODUCTID_ARM_MMU_600:
+ /* Arm erratum 1076982 */
+ if (variant == 0 && revision <= 2)
+ smmu->features &= ~ARM_SMMU_FEAT_SEV;
+ /* Arm erratum 1209401 */
+ if (variant < 2)
+ smmu->features &= ~ARM_SMMU_FEAT_NESTING;
+ break;
+ case IIDR_PRODUCTID_ARM_MMU_700:
+ /* Arm erratum 2812531 */
+ smmu->features &= ~ARM_SMMU_FEAT_BTM;
+ smmu->options |= ARM_SMMU_OPT_CMDQ_FORCE_SYNC;
+ /* Arm errata 2268618, 2812531 */
+ smmu->features &= ~ARM_SMMU_FEAT_NESTING;
+ break;
+ }
+ break;
+ }
+}
+
+static void arm_smmu_get_httu(struct arm_smmu_device *smmu, u32 reg)
+{
+ u32 fw_features = smmu->features & (ARM_SMMU_FEAT_HA | ARM_SMMU_FEAT_HD);
+ u32 hw_features = 0;
+
+ switch (FIELD_GET(IDR0_HTTU, reg)) {
+ case IDR0_HTTU_ACCESS_DIRTY:
+ hw_features |= ARM_SMMU_FEAT_HD;
+ fallthrough;
+ case IDR0_HTTU_ACCESS:
+ hw_features |= ARM_SMMU_FEAT_HA;
+ }
+
+ if (smmu->dev->of_node)
+ smmu->features |= hw_features;
+ else if (hw_features != fw_features)
+ /* ACPI IORT sets the HTTU bits */
+ dev_warn(smmu->dev,
+ "IDR0.HTTU features(0x%x) overridden by FW configuration (0x%x)\n",
+ hw_features, fw_features);
+}
+
+int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
+{
+ u32 reg;
+ bool coherent = smmu->features & ARM_SMMU_FEAT_COHERENCY;
+
+ /* IDR0 */
+ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR0);
+
+ /* 2-level structures */
+ if (FIELD_GET(IDR0_ST_LVL, reg) == IDR0_ST_LVL_2LVL)
+ smmu->features |= ARM_SMMU_FEAT_2_LVL_STRTAB;
+
+ if (reg & IDR0_CD2L)
+ smmu->features |= ARM_SMMU_FEAT_2_LVL_CDTAB;
+
+ /*
+ * Translation table endianness.
+ * We currently require the same endianness as the CPU, but this
+ * could be changed later by adding a new IO_PGTABLE_QUIRK.
+ */
+ switch (FIELD_GET(IDR0_TTENDIAN, reg)) {
+ case IDR0_TTENDIAN_MIXED:
+ smmu->features |= ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE;
+ break;
+#ifdef __BIG_ENDIAN
+ case IDR0_TTENDIAN_BE:
+ smmu->features |= ARM_SMMU_FEAT_TT_BE;
+ break;
+#else
+ case IDR0_TTENDIAN_LE:
+ smmu->features |= ARM_SMMU_FEAT_TT_LE;
+ break;
+#endif
+ default:
+ dev_err(smmu->dev, "unknown/unsupported TT endianness!\n");
+ return -ENXIO;
+ }
+
+ /* Boolean feature flags */
+ if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI)
+ smmu->features |= ARM_SMMU_FEAT_PRI;
+
+ if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS)
+ smmu->features |= ARM_SMMU_FEAT_ATS;
+
+ if (reg & IDR0_SEV)
+ smmu->features |= ARM_SMMU_FEAT_SEV;
+
+ if (reg & IDR0_MSI) {
+ smmu->features |= ARM_SMMU_FEAT_MSI;
+ if (coherent)
+ smmu->options |= ARM_SMMU_OPT_MSIPOLL;
+ }
+
+ if (reg & IDR0_HYP) {
+ smmu->features |= ARM_SMMU_FEAT_HYP;
+ if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))
+ smmu->features |= ARM_SMMU_FEAT_E2H;
+ }
+
+ arm_smmu_get_httu(smmu, reg);
+
+ /*
+ * The coherency feature as set by FW is used in preference to the ID
+ * register, but warn on mismatch.
+ */
+ if (!!(reg & IDR0_COHACC) != coherent)
+ dev_warn(smmu->dev, "IDR0.COHACC overridden by FW configuration (%s)\n",
+ str_true_false(coherent));
+
+ switch (FIELD_GET(IDR0_STALL_MODEL, reg)) {
+ case IDR0_STALL_MODEL_FORCE:
+ smmu->features |= ARM_SMMU_FEAT_STALL_FORCE;
+ fallthrough;
+ case IDR0_STALL_MODEL_STALL:
+ smmu->features |= ARM_SMMU_FEAT_STALLS;
+ }
+
+ if (reg & IDR0_S1P)
+ smmu->features |= ARM_SMMU_FEAT_TRANS_S1;
+
+ if (reg & IDR0_S2P)
+ smmu->features |= ARM_SMMU_FEAT_TRANS_S2;
+
+ if (!(reg & (IDR0_S1P | IDR0_S2P))) {
+ dev_err(smmu->dev, "no translation support!\n");
+ return -ENXIO;
+ }
+
+ /* We only support the AArch64 table format at present */
+ switch (FIELD_GET(IDR0_TTF, reg)) {
+ case IDR0_TTF_AARCH32_64:
+ smmu->ias = 40;
+ fallthrough;
+ case IDR0_TTF_AARCH64:
+ break;
+ default:
+ dev_err(smmu->dev, "AArch64 table format not supported!\n");
+ return -ENXIO;
+ }
+
+ /* ASID/VMID sizes */
+ smmu->asid_bits = reg & IDR0_ASID16 ? 16 : 8;
+ smmu->vmid_bits = reg & IDR0_VMID16 ? 16 : 8;
+
+ /* IDR1 */
+ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR1);
+ if (reg & (IDR1_TABLES_PRESET | IDR1_QUEUES_PRESET | IDR1_REL)) {
+ dev_err(smmu->dev, "embedded implementation not supported\n");
+ return -ENXIO;
+ }
+
+ if (reg & IDR1_ATTR_TYPES_OVR)
+ smmu->features |= ARM_SMMU_FEAT_ATTR_TYPES_OVR;
+
+ /* Queue sizes, capped to ensure natural alignment */
+ smmu->cmdq.q.llq.max_n_shift = min_t(u32, CMDQ_MAX_SZ_SHIFT,
+ FIELD_GET(IDR1_CMDQS, reg));
+ if (smmu->cmdq.q.llq.max_n_shift <= ilog2(CMDQ_BATCH_ENTRIES)) {
+ /*
+ * We don't support splitting up batches, so one batch of
+ * commands plus an extra sync needs to fit inside the command
+ * queue. There's also no way we can handle the weird alignment
+ * restrictions on the base pointer for a unit-length queue.
+ */
+ dev_err(smmu->dev, "command queue size <= %d entries not supported\n",
+ CMDQ_BATCH_ENTRIES);
+ return -ENXIO;
+ }
+
+ smmu->evtq.q.llq.max_n_shift = min_t(u32, EVTQ_MAX_SZ_SHIFT,
+ FIELD_GET(IDR1_EVTQS, reg));
+ smmu->priq.q.llq.max_n_shift = min_t(u32, PRIQ_MAX_SZ_SHIFT,
+ FIELD_GET(IDR1_PRIQS, reg));
+
+ /* SID/SSID sizes */
+ smmu->ssid_bits = FIELD_GET(IDR1_SSIDSIZE, reg);
+ smmu->sid_bits = FIELD_GET(IDR1_SIDSIZE, reg);
+ smmu->iommu.max_pasids = 1UL << smmu->ssid_bits;
+
+ /*
+ * If the SMMU supports fewer bits than would fill a single L2 stream
+ * table, use a linear table instead.
+ */
+ if (smmu->sid_bits <= STRTAB_SPLIT)
+ smmu->features &= ~ARM_SMMU_FEAT_2_LVL_STRTAB;
+
+ /* IDR3 */
+ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR3);
+ if (FIELD_GET(IDR3_RIL, reg))
+ smmu->features |= ARM_SMMU_FEAT_RANGE_INV;
+ if (FIELD_GET(IDR3_FWB, reg))
+ smmu->features |= ARM_SMMU_FEAT_S2FWB;
+
+ /* IDR5 */
+ reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5);
+
+ /* Maximum number of outstanding stalls */
+ smmu->evtq.max_stalls = FIELD_GET(IDR5_STALL_MAX, reg);
+
+ /* Page sizes */
+ if (reg & IDR5_GRAN64K)
+ smmu->pgsize_bitmap |= SZ_64K | SZ_512M;
+ if (reg & IDR5_GRAN16K)
+ smmu->pgsize_bitmap |= SZ_16K | SZ_32M;
+ if (reg & IDR5_GRAN4K)
+ smmu->pgsize_bitmap |= SZ_4K | SZ_2M | SZ_1G;
+
+ /* Input address size */
+ if (FIELD_GET(IDR5_VAX, reg) == IDR5_VAX_52_BIT)
+ smmu->features |= ARM_SMMU_FEAT_VAX;
+
+ /* Output address size */
+ switch (FIELD_GET(IDR5_OAS, reg)) {
+ case IDR5_OAS_32_BIT:
+ smmu->oas = 32;
+ break;
+ case IDR5_OAS_36_BIT:
+ smmu->oas = 36;
+ break;
+ case IDR5_OAS_40_BIT:
+ smmu->oas = 40;
+ break;
+ case IDR5_OAS_42_BIT:
+ smmu->oas = 42;
+ break;
+ case IDR5_OAS_44_BIT:
+ smmu->oas = 44;
+ break;
+ case IDR5_OAS_52_BIT:
+ smmu->oas = 52;
+ smmu->pgsize_bitmap |= 1ULL << 42; /* 4TB */
+ break;
+ default:
+ dev_info(smmu->dev,
+ "unknown output address size. Truncating to 48-bit\n");
+ fallthrough;
+ case IDR5_OAS_48_BIT:
+ smmu->oas = 48;
+ }
+
+ /* Set the DMA mask for our table walker */
+ if (dma_set_mask_and_coherent(smmu->dev, DMA_BIT_MASK(smmu->oas)))
+ dev_warn(smmu->dev,
+ "failed to set DMA mask for table walker\n");
+
+ smmu->ias = max(smmu->ias, smmu->oas);
+
+ if ((smmu->features & ARM_SMMU_FEAT_TRANS_S1) &&
+ (smmu->features & ARM_SMMU_FEAT_TRANS_S2))
+ smmu->features |= ARM_SMMU_FEAT_NESTING;
+
+ arm_smmu_device_iidr_probe(smmu);
+
+ dev_info(smmu->dev, "ias %lu-bit, oas %lu-bit (features 0x%08x)\n",
+ smmu->ias, smmu->oas, smmu->features);
+ return 0;
+}
+
+int arm_smmu_write_reg_sync(struct arm_smmu_device *smmu, u32 val,
+ unsigned int reg_off, unsigned int ack_off)
+{
+ u32 reg;
+
+ writel_relaxed(val, smmu->base + reg_off);
+ return readl_relaxed_poll_timeout(smmu->base + ack_off, reg, reg == val,
+ 1, ARM_SMMU_POLL_TIMEOUT_US);
+}
+
+/* GBPA is "special" */
+int arm_smmu_update_gbpa(struct arm_smmu_device *smmu, u32 set, u32 clr)
+{
+ int ret;
+ u32 reg, __iomem *gbpa = smmu->base + ARM_SMMU_GBPA;
+
+ ret = readl_relaxed_poll_timeout(gbpa, reg, !(reg & GBPA_UPDATE),
+ 1, ARM_SMMU_POLL_TIMEOUT_US);
+ if (ret)
+ return ret;
+
+ reg &= ~clr;
+ reg |= set;
+ writel_relaxed(reg | GBPA_UPDATE, gbpa);
+ ret = readl_relaxed_poll_timeout(gbpa, reg, !(reg & GBPA_UPDATE),
+ 1, ARM_SMMU_POLL_TIMEOUT_US);
+
+ if (ret)
+ dev_err(smmu->dev, "GBPA not responding to update\n");
+ return ret;
+}
+
+int arm_smmu_device_disable(struct arm_smmu_device *smmu)
+{
+ int ret;
+
+ ret = arm_smmu_write_reg_sync(smmu, 0, ARM_SMMU_CR0, ARM_SMMU_CR0ACK);
+ if (ret)
+ dev_err(smmu->dev, "failed to clear cr0\n");
+
+ return ret;
+}
+
+struct iommu_group *arm_smmu_device_group(struct device *dev)
+{
+ struct iommu_group *group;
+
+ /*
+ * We don't support devices sharing stream IDs other than PCI RID
+ * aliases, since the necessary ID-to-device lookup becomes rather
+ * impractical given a potential sparse 32-bit stream ID space.
+ */
+ if (dev_is_pci(dev))
+ group = pci_device_group(dev);
+ else
+ group = generic_device_group(dev);
+
+ return group;
+}
+
+int arm_smmu_of_xlate(struct device *dev, const struct of_phandle_args *args)
+{
+ return iommu_fwspec_add_ids(dev, args->args, 1);
+}
+
+void arm_smmu_get_resv_regions(struct device *dev, struct list_head *head)
+{
+ struct iommu_resv_region *region;
+ int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
+
+ region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
+ prot, IOMMU_RESV_SW_MSI, GFP_KERNEL);
+ if (!region)
+ return;
+
+ list_add_tail(®ion->list, head);
+
+ iommu_dma_get_resv_regions(dev, head);
+}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 801b792dda36..fc4bf270c365 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -17,7 +17,6 @@
#include <linux/err.h>
#include <linux/interrupt.h>
#include <linux/io-pgtable.h>
-#include <linux/iopoll.h>
#include <linux/module.h>
#include <linux/msi.h>
#include <linux/of.h>
@@ -26,12 +25,10 @@
#include <linux/pci.h>
#include <linux/pci-ats.h>
#include <linux/platform_device.h>
-#include <linux/string_choices.h>
#include <kunit/visibility.h>
#include <uapi/linux/iommufd.h>
#include "arm-smmu-v3.h"
-#include "../../dma-iommu.h"
static bool disable_msipolling;
module_param(disable_msipolling, bool, 0444);
@@ -2030,8 +2027,6 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev)
return IRQ_HANDLED;
}
-static int arm_smmu_device_disable(struct arm_smmu_device *smmu);
-
static irqreturn_t arm_smmu_gerror_handler(int irq, void *dev)
{
u32 gerror, gerrorn, active;
@@ -3615,45 +3610,6 @@ static int arm_smmu_set_dirty_tracking(struct iommu_domain *domain,
return 0;
}
-static struct iommu_group *arm_smmu_device_group(struct device *dev)
-{
- struct iommu_group *group;
-
- /*
- * We don't support devices sharing stream IDs other than PCI RID
- * aliases, since the necessary ID-to-device lookup becomes rather
- * impractical given a potential sparse 32-bit stream ID space.
- */
- if (dev_is_pci(dev))
- group = pci_device_group(dev);
- else
- group = generic_device_group(dev);
-
- return group;
-}
-
-static int arm_smmu_of_xlate(struct device *dev,
- const struct of_phandle_args *args)
-{
- return iommu_fwspec_add_ids(dev, args->args, 1);
-}
-
-static void arm_smmu_get_resv_regions(struct device *dev,
- struct list_head *head)
-{
- struct iommu_resv_region *region;
- int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
-
- region = iommu_alloc_resv_region(MSI_IOVA_BASE, MSI_IOVA_LENGTH,
- prot, IOMMU_RESV_SW_MSI, GFP_KERNEL);
- if (!region)
- return;
-
- list_add_tail(®ion->list, head);
-
- iommu_dma_get_resv_regions(dev, head);
-}
-
/*
* HiSilicon PCIe tune and trace device can be used to trace TLP headers on the
* PCIe link and save the data to memory by DMA. The hardware is restricted to
@@ -3897,38 +3853,6 @@ static int arm_smmu_init_structures(struct arm_smmu_device *smmu)
return 0;
}
-static int arm_smmu_write_reg_sync(struct arm_smmu_device *smmu, u32 val,
- unsigned int reg_off, unsigned int ack_off)
-{
- u32 reg;
-
- writel_relaxed(val, smmu->base + reg_off);
- return readl_relaxed_poll_timeout(smmu->base + ack_off, reg, reg == val,
- 1, ARM_SMMU_POLL_TIMEOUT_US);
-}
-
-/* GBPA is "special" */
-static int arm_smmu_update_gbpa(struct arm_smmu_device *smmu, u32 set, u32 clr)
-{
- int ret;
- u32 reg, __iomem *gbpa = smmu->base + ARM_SMMU_GBPA;
-
- ret = readl_relaxed_poll_timeout(gbpa, reg, !(reg & GBPA_UPDATE),
- 1, ARM_SMMU_POLL_TIMEOUT_US);
- if (ret)
- return ret;
-
- reg &= ~clr;
- reg |= set;
- writel_relaxed(reg | GBPA_UPDATE, gbpa);
- ret = readl_relaxed_poll_timeout(gbpa, reg, !(reg & GBPA_UPDATE),
- 1, ARM_SMMU_POLL_TIMEOUT_US);
-
- if (ret)
- dev_err(smmu->dev, "GBPA not responding to update\n");
- return ret;
-}
-
static void arm_smmu_free_msis(void *data)
{
struct device *dev = data;
@@ -4075,17 +3999,6 @@ static int arm_smmu_setup_irqs(struct arm_smmu_device *smmu)
return 0;
}
-static int arm_smmu_device_disable(struct arm_smmu_device *smmu)
-{
- int ret;
-
- ret = arm_smmu_write_reg_sync(smmu, 0, ARM_SMMU_CR0, ARM_SMMU_CR0ACK);
- if (ret)
- dev_err(smmu->dev, "failed to clear cr0\n");
-
- return ret;
-}
-
static void arm_smmu_write_strtab(struct arm_smmu_device *smmu)
{
struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
@@ -4243,285 +4156,6 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu)
return 0;
}
-#define IIDR_IMPLEMENTER_ARM 0x43b
-#define IIDR_PRODUCTID_ARM_MMU_600 0x483
-#define IIDR_PRODUCTID_ARM_MMU_700 0x487
-
-static void arm_smmu_device_iidr_probe(struct arm_smmu_device *smmu)
-{
- u32 reg;
- unsigned int implementer, productid, variant, revision;
-
- reg = readl_relaxed(smmu->base + ARM_SMMU_IIDR);
- implementer = FIELD_GET(IIDR_IMPLEMENTER, reg);
- productid = FIELD_GET(IIDR_PRODUCTID, reg);
- variant = FIELD_GET(IIDR_VARIANT, reg);
- revision = FIELD_GET(IIDR_REVISION, reg);
-
- switch (implementer) {
- case IIDR_IMPLEMENTER_ARM:
- switch (productid) {
- case IIDR_PRODUCTID_ARM_MMU_600:
- /* Arm erratum 1076982 */
- if (variant == 0 && revision <= 2)
- smmu->features &= ~ARM_SMMU_FEAT_SEV;
- /* Arm erratum 1209401 */
- if (variant < 2)
- smmu->features &= ~ARM_SMMU_FEAT_NESTING;
- break;
- case IIDR_PRODUCTID_ARM_MMU_700:
- /* Arm erratum 2812531 */
- smmu->features &= ~ARM_SMMU_FEAT_BTM;
- smmu->options |= ARM_SMMU_OPT_CMDQ_FORCE_SYNC;
- /* Arm errata 2268618, 2812531 */
- smmu->features &= ~ARM_SMMU_FEAT_NESTING;
- break;
- }
- break;
- }
-}
-
-static void arm_smmu_get_httu(struct arm_smmu_device *smmu, u32 reg)
-{
- u32 fw_features = smmu->features & (ARM_SMMU_FEAT_HA | ARM_SMMU_FEAT_HD);
- u32 hw_features = 0;
-
- switch (FIELD_GET(IDR0_HTTU, reg)) {
- case IDR0_HTTU_ACCESS_DIRTY:
- hw_features |= ARM_SMMU_FEAT_HD;
- fallthrough;
- case IDR0_HTTU_ACCESS:
- hw_features |= ARM_SMMU_FEAT_HA;
- }
-
- if (smmu->dev->of_node)
- smmu->features |= hw_features;
- else if (hw_features != fw_features)
- /* ACPI IORT sets the HTTU bits */
- dev_warn(smmu->dev,
- "IDR0.HTTU features(0x%x) overridden by FW configuration (0x%x)\n",
- hw_features, fw_features);
-}
-
-static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
-{
- u32 reg;
- bool coherent = smmu->features & ARM_SMMU_FEAT_COHERENCY;
-
- /* IDR0 */
- reg = readl_relaxed(smmu->base + ARM_SMMU_IDR0);
-
- /* 2-level structures */
- if (FIELD_GET(IDR0_ST_LVL, reg) == IDR0_ST_LVL_2LVL)
- smmu->features |= ARM_SMMU_FEAT_2_LVL_STRTAB;
-
- if (reg & IDR0_CD2L)
- smmu->features |= ARM_SMMU_FEAT_2_LVL_CDTAB;
-
- /*
- * Translation table endianness.
- * We currently require the same endianness as the CPU, but this
- * could be changed later by adding a new IO_PGTABLE_QUIRK.
- */
- switch (FIELD_GET(IDR0_TTENDIAN, reg)) {
- case IDR0_TTENDIAN_MIXED:
- smmu->features |= ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE;
- break;
-#ifdef __BIG_ENDIAN
- case IDR0_TTENDIAN_BE:
- smmu->features |= ARM_SMMU_FEAT_TT_BE;
- break;
-#else
- case IDR0_TTENDIAN_LE:
- smmu->features |= ARM_SMMU_FEAT_TT_LE;
- break;
-#endif
- default:
- dev_err(smmu->dev, "unknown/unsupported TT endianness!\n");
- return -ENXIO;
- }
-
- /* Boolean feature flags */
- if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI)
- smmu->features |= ARM_SMMU_FEAT_PRI;
-
- if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS)
- smmu->features |= ARM_SMMU_FEAT_ATS;
-
- if (reg & IDR0_SEV)
- smmu->features |= ARM_SMMU_FEAT_SEV;
-
- if (reg & IDR0_MSI) {
- smmu->features |= ARM_SMMU_FEAT_MSI;
- if (coherent)
- smmu->options |= ARM_SMMU_OPT_MSIPOLL;
- }
-
- if (reg & IDR0_HYP) {
- smmu->features |= ARM_SMMU_FEAT_HYP;
- if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))
- smmu->features |= ARM_SMMU_FEAT_E2H;
- }
-
- arm_smmu_get_httu(smmu, reg);
-
- /*
- * The coherency feature as set by FW is used in preference to the ID
- * register, but warn on mismatch.
- */
- if (!!(reg & IDR0_COHACC) != coherent)
- dev_warn(smmu->dev, "IDR0.COHACC overridden by FW configuration (%s)\n",
- str_true_false(coherent));
-
- switch (FIELD_GET(IDR0_STALL_MODEL, reg)) {
- case IDR0_STALL_MODEL_FORCE:
- smmu->features |= ARM_SMMU_FEAT_STALL_FORCE;
- fallthrough;
- case IDR0_STALL_MODEL_STALL:
- smmu->features |= ARM_SMMU_FEAT_STALLS;
- }
-
- if (reg & IDR0_S1P)
- smmu->features |= ARM_SMMU_FEAT_TRANS_S1;
-
- if (reg & IDR0_S2P)
- smmu->features |= ARM_SMMU_FEAT_TRANS_S2;
-
- if (!(reg & (IDR0_S1P | IDR0_S2P))) {
- dev_err(smmu->dev, "no translation support!\n");
- return -ENXIO;
- }
-
- /* We only support the AArch64 table format at present */
- switch (FIELD_GET(IDR0_TTF, reg)) {
- case IDR0_TTF_AARCH32_64:
- smmu->ias = 40;
- fallthrough;
- case IDR0_TTF_AARCH64:
- break;
- default:
- dev_err(smmu->dev, "AArch64 table format not supported!\n");
- return -ENXIO;
- }
-
- /* ASID/VMID sizes */
- smmu->asid_bits = reg & IDR0_ASID16 ? 16 : 8;
- smmu->vmid_bits = reg & IDR0_VMID16 ? 16 : 8;
-
- /* IDR1 */
- reg = readl_relaxed(smmu->base + ARM_SMMU_IDR1);
- if (reg & (IDR1_TABLES_PRESET | IDR1_QUEUES_PRESET | IDR1_REL)) {
- dev_err(smmu->dev, "embedded implementation not supported\n");
- return -ENXIO;
- }
-
- if (reg & IDR1_ATTR_TYPES_OVR)
- smmu->features |= ARM_SMMU_FEAT_ATTR_TYPES_OVR;
-
- /* Queue sizes, capped to ensure natural alignment */
- smmu->cmdq.q.llq.max_n_shift = min_t(u32, CMDQ_MAX_SZ_SHIFT,
- FIELD_GET(IDR1_CMDQS, reg));
- if (smmu->cmdq.q.llq.max_n_shift <= ilog2(CMDQ_BATCH_ENTRIES)) {
- /*
- * We don't support splitting up batches, so one batch of
- * commands plus an extra sync needs to fit inside the command
- * queue. There's also no way we can handle the weird alignment
- * restrictions on the base pointer for a unit-length queue.
- */
- dev_err(smmu->dev, "command queue size <= %d entries not supported\n",
- CMDQ_BATCH_ENTRIES);
- return -ENXIO;
- }
-
- smmu->evtq.q.llq.max_n_shift = min_t(u32, EVTQ_MAX_SZ_SHIFT,
- FIELD_GET(IDR1_EVTQS, reg));
- smmu->priq.q.llq.max_n_shift = min_t(u32, PRIQ_MAX_SZ_SHIFT,
- FIELD_GET(IDR1_PRIQS, reg));
-
- /* SID/SSID sizes */
- smmu->ssid_bits = FIELD_GET(IDR1_SSIDSIZE, reg);
- smmu->sid_bits = FIELD_GET(IDR1_SIDSIZE, reg);
- smmu->iommu.max_pasids = 1UL << smmu->ssid_bits;
-
- /*
- * If the SMMU supports fewer bits than would fill a single L2 stream
- * table, use a linear table instead.
- */
- if (smmu->sid_bits <= STRTAB_SPLIT)
- smmu->features &= ~ARM_SMMU_FEAT_2_LVL_STRTAB;
-
- /* IDR3 */
- reg = readl_relaxed(smmu->base + ARM_SMMU_IDR3);
- if (FIELD_GET(IDR3_RIL, reg))
- smmu->features |= ARM_SMMU_FEAT_RANGE_INV;
- if (FIELD_GET(IDR3_FWB, reg))
- smmu->features |= ARM_SMMU_FEAT_S2FWB;
-
- /* IDR5 */
- reg = readl_relaxed(smmu->base + ARM_SMMU_IDR5);
-
- /* Maximum number of outstanding stalls */
- smmu->evtq.max_stalls = FIELD_GET(IDR5_STALL_MAX, reg);
-
- /* Page sizes */
- if (reg & IDR5_GRAN64K)
- smmu->pgsize_bitmap |= SZ_64K | SZ_512M;
- if (reg & IDR5_GRAN16K)
- smmu->pgsize_bitmap |= SZ_16K | SZ_32M;
- if (reg & IDR5_GRAN4K)
- smmu->pgsize_bitmap |= SZ_4K | SZ_2M | SZ_1G;
-
- /* Input address size */
- if (FIELD_GET(IDR5_VAX, reg) == IDR5_VAX_52_BIT)
- smmu->features |= ARM_SMMU_FEAT_VAX;
-
- /* Output address size */
- switch (FIELD_GET(IDR5_OAS, reg)) {
- case IDR5_OAS_32_BIT:
- smmu->oas = 32;
- break;
- case IDR5_OAS_36_BIT:
- smmu->oas = 36;
- break;
- case IDR5_OAS_40_BIT:
- smmu->oas = 40;
- break;
- case IDR5_OAS_42_BIT:
- smmu->oas = 42;
- break;
- case IDR5_OAS_44_BIT:
- smmu->oas = 44;
- break;
- case IDR5_OAS_52_BIT:
- smmu->oas = 52;
- smmu->pgsize_bitmap |= 1ULL << 42; /* 4TB */
- break;
- default:
- dev_info(smmu->dev,
- "unknown output address size. Truncating to 48-bit\n");
- fallthrough;
- case IDR5_OAS_48_BIT:
- smmu->oas = 48;
- }
-
- /* Set the DMA mask for our table walker */
- if (dma_set_mask_and_coherent(smmu->dev, DMA_BIT_MASK(smmu->oas)))
- dev_warn(smmu->dev,
- "failed to set DMA mask for table walker\n");
-
- smmu->ias = max(smmu->ias, smmu->oas);
-
- if ((smmu->features & ARM_SMMU_FEAT_TRANS_S1) &&
- (smmu->features & ARM_SMMU_FEAT_TRANS_S2))
- smmu->features |= ARM_SMMU_FEAT_NESTING;
-
- arm_smmu_device_iidr_probe(smmu);
-
- dev_info(smmu->dev, "ias %lu-bit, oas %lu-bit (features 0x%08x)\n",
- smmu->ias, smmu->oas, smmu->features);
- return 0;
-}
-
#ifdef CONFIG_ACPI
#ifdef CONFIG_TEGRA241_CMDQV
static void acpi_smmu_dsdt_probe_tegra241_cmdqv(struct acpi_iort_node *node,
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 3b88f19fcefc..367529538eb8 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -545,6 +545,14 @@ void arm_smmu_install_ste_for_dev(struct arm_smmu_master *master,
int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu,
struct arm_smmu_cmdq *cmdq, u64 *cmds, int n,
bool sync);
+int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu);
+int arm_smmu_write_reg_sync(struct arm_smmu_device *smmu, u32 val,
+ unsigned int reg_off, unsigned int ack_off);
+int arm_smmu_update_gbpa(struct arm_smmu_device *smmu, u32 set, u32 clr);
+int arm_smmu_device_disable(struct arm_smmu_device *smmu);
+struct iommu_group *arm_smmu_device_group(struct device *dev);
+int arm_smmu_of_xlate(struct device *dev, const struct of_phandle_args *args);
+void arm_smmu_get_resv_regions(struct device *dev, struct list_head *head);
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 09/29] iommu/arm-smmu-v3: Move queue and table allocation to arm-smmu-v3-common.c
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (7 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 08/29] iommu/arm-smmu-v3: Move some functions to arm-smmu-v3-common.c Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 10/29] iommu/arm-smmu-v3: Move firmware probe to arm-smmu-v3-common Mostafa Saleh
` (19 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Move more code to arm-smmu-v3-common.c, so that the KVM driver can reuse
it.
Also, make sure that allocated memory is aligned as it its going
to protected by the hypervisor stage-2.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../arm/arm-smmu-v3/arm-smmu-v3-common.c | 160 ++++++++++++++++
.../arm/arm-smmu-v3/arm-smmu-v3-common.h | 12 ++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 171 +-----------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 8 +
4 files changed, 181 insertions(+), 170 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
index c2b14b0c9f53..6cd5187d62a1 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
@@ -3,6 +3,7 @@
#include <linux/iopoll.h>
#include <linux/pci.h>
#include <linux/string_choices.h>
+#include <kunit/visibility.h>
#include "arm-smmu-v3.h"
#include "../../dma-iommu.h"
@@ -365,3 +366,162 @@ void arm_smmu_get_resv_regions(struct device *dev, struct list_head *head)
iommu_dma_get_resv_regions(dev, head);
}
+
+int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
+ struct arm_smmu_queue *q, void __iomem *page,
+ unsigned long prod_off, unsigned long cons_off,
+ size_t dwords, const char *name)
+{
+ size_t qsz;
+
+ do {
+ qsz = ((1 << q->llq.max_n_shift) * dwords) << 3;
+ q->base = dmam_alloc_coherent(smmu->dev, qsz, &q->base_dma,
+ GFP_KERNEL);
+ if (q->base || qsz < PAGE_SIZE)
+ break;
+
+ q->llq.max_n_shift--;
+ } while (1);
+
+ if (!q->base) {
+ dev_err(smmu->dev,
+ "failed to allocate queue (0x%zx bytes) for %s\n",
+ qsz, name);
+ return -ENOMEM;
+ }
+
+ if (!WARN_ON(q->base_dma & (qsz - 1))) {
+ dev_info(smmu->dev, "allocated %u entries for %s\n",
+ 1 << q->llq.max_n_shift, name);
+ }
+
+ q->prod_reg = page + prod_off;
+ q->cons_reg = page + cons_off;
+ q->ent_dwords = dwords;
+
+ q->q_base = Q_BASE_RWA;
+ q->q_base |= q->base_dma & Q_BASE_ADDR_MASK;
+ q->q_base |= FIELD_PREP(Q_BASE_LOG2SIZE, q->llq.max_n_shift);
+
+ q->llq.prod = q->llq.cons = 0;
+ return 0;
+}
+
+/* Stream table manipulation functions */
+static int arm_smmu_init_strtab_2lvl(struct arm_smmu_device *smmu)
+{
+ u32 l1size;
+ struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+ unsigned int last_sid_idx =
+ arm_smmu_strtab_l1_idx((1ULL << smmu->sid_bits) - 1);
+
+ /* Calculate the L1 size, capped to the SIDSIZE. */
+ cfg->l2.num_l1_ents = min(last_sid_idx + 1, STRTAB_MAX_L1_ENTRIES);
+ if (cfg->l2.num_l1_ents <= last_sid_idx)
+ dev_warn(smmu->dev,
+ "2-level strtab only covers %u/%u bits of SID\n",
+ ilog2(cfg->l2.num_l1_ents * STRTAB_NUM_L2_STES),
+ smmu->sid_bits);
+
+ l1size = cfg->l2.num_l1_ents * sizeof(struct arm_smmu_strtab_l1);
+ cfg->l2.l1tab = dmam_alloc_coherent(smmu->dev, l1size, &cfg->l2.l1_dma,
+ GFP_KERNEL);
+ if (!cfg->l2.l1tab) {
+ dev_err(smmu->dev,
+ "failed to allocate l1 stream table (%u bytes)\n",
+ l1size);
+ return -ENOMEM;
+ }
+
+ cfg->l2.l2ptrs = devm_kcalloc(smmu->dev, cfg->l2.num_l1_ents,
+ sizeof(*cfg->l2.l2ptrs), GFP_KERNEL);
+ if (!cfg->l2.l2ptrs)
+ return -ENOMEM;
+
+ return 0;
+}
+
+void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)
+{
+ memset(target, 0, sizeof(*target));
+ target->data[0] = cpu_to_le64(
+ STRTAB_STE_0_V |
+ FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_ABORT));
+}
+
+/*
+ * This can safely directly manipulate the STE memory without a sync sequence
+ * because the STE table has not been installed in the SMMU yet.
+ */
+void arm_smmu_init_initial_stes(struct arm_smmu_ste *strtab,
+ unsigned int nent)
+{
+ unsigned int i;
+
+ for (i = 0; i < nent; ++i) {
+ arm_smmu_make_abort_ste(strtab);
+ strtab++;
+ }
+}
+
+static int arm_smmu_init_strtab_linear(struct arm_smmu_device *smmu)
+{
+ u32 size;
+ struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+
+ size = (1 << smmu->sid_bits) * sizeof(struct arm_smmu_ste);
+ cfg->linear.table = dmam_alloc_coherent(smmu->dev, size,
+ &cfg->linear.ste_dma,
+ GFP_KERNEL);
+ if (!cfg->linear.table) {
+ dev_err(smmu->dev,
+ "failed to allocate linear stream table (%u bytes)\n",
+ size);
+ return -ENOMEM;
+ }
+ cfg->linear.num_ents = 1 << smmu->sid_bits;
+
+ arm_smmu_init_initial_stes(cfg->linear.table, cfg->linear.num_ents);
+ return 0;
+}
+
+int arm_smmu_init_strtab(struct arm_smmu_device *smmu)
+{
+ int ret;
+
+ if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB)
+ ret = arm_smmu_init_strtab_2lvl(smmu);
+ else
+ ret = arm_smmu_init_strtab_linear(smmu);
+ if (ret)
+ return ret;
+
+ ida_init(&smmu->vmid_map);
+
+ return 0;
+}
+
+void arm_smmu_write_strtab(struct arm_smmu_device *smmu)
+{
+ struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+ dma_addr_t dma;
+ u32 reg;
+
+ if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) {
+ reg = FIELD_PREP(STRTAB_BASE_CFG_FMT,
+ STRTAB_BASE_CFG_FMT_2LVL) |
+ FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE,
+ ilog2(cfg->l2.num_l1_ents) + STRTAB_SPLIT) |
+ FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT);
+ dma = cfg->l2.l1_dma;
+ } else {
+ reg = FIELD_PREP(STRTAB_BASE_CFG_FMT,
+ STRTAB_BASE_CFG_FMT_LINEAR) |
+ FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits);
+ dma = cfg->linear.ste_dma;
+ }
+ writeq_relaxed((dma & STRTAB_BASE_ADDR_MASK) | STRTAB_BASE_RA,
+ smmu->base + ARM_SMMU_STRTAB_BASE);
+ writel_relaxed(reg, smmu->base + ARM_SMMU_STRTAB_BASE_CFG);
+}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
index 0f9ae032680d..3bcb5d94f21a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
@@ -2,6 +2,7 @@
#ifndef _ARM_SMMU_V3_COMMON_H
#define _ARM_SMMU_V3_COMMON_H
+#include <linux/bits.h>
#include <linux/bitfield.h>
/* MMIO registers */
@@ -459,4 +460,15 @@ struct arm_smmu_cmdq_ent {
#define ARM_SMMU_FEAT_HD (1 << 22)
#define ARM_SMMU_FEAT_S2FWB (1 << 23)
+static inline void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst,
+ dma_addr_t l2ptr_dma)
+{
+ u64 val = 0;
+
+ val |= FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1);
+ val |= l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK;
+
+ /* The HW has 64 bit atomicity with stores to the L2 STE table */
+ WRITE_ONCE(dst->l2ptr, cpu_to_le64(val));
+}
#endif /* _ARM_SMMU_V3_COMMON_H */
\ No newline at end of file
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index fc4bf270c365..6e6ac64a5399 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -1504,19 +1504,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_master *master)
}
}
-/* Stream table manipulation functions */
-static void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst,
- dma_addr_t l2ptr_dma)
-{
- u64 val = 0;
-
- val |= FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1);
- val |= l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK;
-
- /* The HW has 64 bit atomicity with stores to the L2 STE table */
- WRITE_ONCE(dst->l2ptr, cpu_to_le64(val));
-}
-
struct arm_smmu_ste_writer {
struct arm_smmu_entry_writer writer;
u32 sid;
@@ -1569,13 +1556,7 @@ static void arm_smmu_write_ste(struct arm_smmu_master *master, u32 sid,
}
}
-void arm_smmu_make_abort_ste(struct arm_smmu_ste *target)
-{
- memset(target, 0, sizeof(*target));
- target->data[0] = cpu_to_le64(
- STRTAB_STE_0_V |
- FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_ABORT));
-}
+/* Defined in arm-smmu-v3-common.c */
EXPORT_SYMBOL_IF_KUNIT(arm_smmu_make_abort_ste);
VISIBLE_IF_KUNIT
@@ -1702,21 +1683,6 @@ void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste *target,
}
EXPORT_SYMBOL_IF_KUNIT(arm_smmu_make_s2_domain_ste);
-/*
- * This can safely directly manipulate the STE memory without a sync sequence
- * because the STE table has not been installed in the SMMU yet.
- */
-static void arm_smmu_init_initial_stes(struct arm_smmu_ste *strtab,
- unsigned int nent)
-{
- unsigned int i;
-
- for (i = 0; i < nent; ++i) {
- arm_smmu_make_abort_ste(strtab);
- strtab++;
- }
-}
-
static int arm_smmu_init_l2_strtab(struct arm_smmu_device *smmu, u32 sid)
{
dma_addr_t l2ptr_dma;
@@ -3667,47 +3633,6 @@ static struct iommu_dirty_ops arm_smmu_dirty_ops = {
};
/* Probing and initialisation functions */
-int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
- struct arm_smmu_queue *q, void __iomem *page,
- unsigned long prod_off, unsigned long cons_off,
- size_t dwords, const char *name)
-{
- size_t qsz;
-
- do {
- qsz = ((1 << q->llq.max_n_shift) * dwords) << 3;
- q->base = dmam_alloc_coherent(smmu->dev, qsz, &q->base_dma,
- GFP_KERNEL);
- if (q->base || qsz < PAGE_SIZE)
- break;
-
- q->llq.max_n_shift--;
- } while (1);
-
- if (!q->base) {
- dev_err(smmu->dev,
- "failed to allocate queue (0x%zx bytes) for %s\n",
- qsz, name);
- return -ENOMEM;
- }
-
- if (!WARN_ON(q->base_dma & (qsz - 1))) {
- dev_info(smmu->dev, "allocated %u entries for %s\n",
- 1 << q->llq.max_n_shift, name);
- }
-
- q->prod_reg = page + prod_off;
- q->cons_reg = page + cons_off;
- q->ent_dwords = dwords;
-
- q->q_base = Q_BASE_RWA;
- q->q_base |= q->base_dma & Q_BASE_ADDR_MASK;
- q->q_base |= FIELD_PREP(Q_BASE_LOG2SIZE, q->llq.max_n_shift);
-
- q->llq.prod = q->llq.cons = 0;
- return 0;
-}
-
int arm_smmu_cmdq_init(struct arm_smmu_device *smmu,
struct arm_smmu_cmdq *cmdq)
{
@@ -3762,76 +3687,6 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu)
PRIQ_ENT_DWORDS, "priq");
}
-static int arm_smmu_init_strtab_2lvl(struct arm_smmu_device *smmu)
-{
- u32 l1size;
- struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
- unsigned int last_sid_idx =
- arm_smmu_strtab_l1_idx((1ULL << smmu->sid_bits) - 1);
-
- /* Calculate the L1 size, capped to the SIDSIZE. */
- cfg->l2.num_l1_ents = min(last_sid_idx + 1, STRTAB_MAX_L1_ENTRIES);
- if (cfg->l2.num_l1_ents <= last_sid_idx)
- dev_warn(smmu->dev,
- "2-level strtab only covers %u/%u bits of SID\n",
- ilog2(cfg->l2.num_l1_ents * STRTAB_NUM_L2_STES),
- smmu->sid_bits);
-
- l1size = cfg->l2.num_l1_ents * sizeof(struct arm_smmu_strtab_l1);
- cfg->l2.l1tab = dmam_alloc_coherent(smmu->dev, l1size, &cfg->l2.l1_dma,
- GFP_KERNEL);
- if (!cfg->l2.l1tab) {
- dev_err(smmu->dev,
- "failed to allocate l1 stream table (%u bytes)\n",
- l1size);
- return -ENOMEM;
- }
-
- cfg->l2.l2ptrs = devm_kcalloc(smmu->dev, cfg->l2.num_l1_ents,
- sizeof(*cfg->l2.l2ptrs), GFP_KERNEL);
- if (!cfg->l2.l2ptrs)
- return -ENOMEM;
-
- return 0;
-}
-
-static int arm_smmu_init_strtab_linear(struct arm_smmu_device *smmu)
-{
- u32 size;
- struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
-
- size = (1 << smmu->sid_bits) * sizeof(struct arm_smmu_ste);
- cfg->linear.table = dmam_alloc_coherent(smmu->dev, size,
- &cfg->linear.ste_dma,
- GFP_KERNEL);
- if (!cfg->linear.table) {
- dev_err(smmu->dev,
- "failed to allocate linear stream table (%u bytes)\n",
- size);
- return -ENOMEM;
- }
- cfg->linear.num_ents = 1 << smmu->sid_bits;
-
- arm_smmu_init_initial_stes(cfg->linear.table, cfg->linear.num_ents);
- return 0;
-}
-
-static int arm_smmu_init_strtab(struct arm_smmu_device *smmu)
-{
- int ret;
-
- if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB)
- ret = arm_smmu_init_strtab_2lvl(smmu);
- else
- ret = arm_smmu_init_strtab_linear(smmu);
- if (ret)
- return ret;
-
- ida_init(&smmu->vmid_map);
-
- return 0;
-}
-
static int arm_smmu_init_structures(struct arm_smmu_device *smmu)
{
int ret;
@@ -3999,30 +3854,6 @@ static int arm_smmu_setup_irqs(struct arm_smmu_device *smmu)
return 0;
}
-static void arm_smmu_write_strtab(struct arm_smmu_device *smmu)
-{
- struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
- dma_addr_t dma;
- u32 reg;
-
- if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) {
- reg = FIELD_PREP(STRTAB_BASE_CFG_FMT,
- STRTAB_BASE_CFG_FMT_2LVL) |
- FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE,
- ilog2(cfg->l2.num_l1_ents) + STRTAB_SPLIT) |
- FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT);
- dma = cfg->l2.l1_dma;
- } else {
- reg = FIELD_PREP(STRTAB_BASE_CFG_FMT,
- STRTAB_BASE_CFG_FMT_LINEAR) |
- FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits);
- dma = cfg->linear.ste_dma;
- }
- writeq_relaxed((dma & STRTAB_BASE_ADDR_MASK) | STRTAB_BASE_RA,
- smmu->base + ARM_SMMU_STRTAB_BASE);
- writel_relaxed(reg, smmu->base + ARM_SMMU_STRTAB_BASE_CFG);
-}
-
static int arm_smmu_device_reset(struct arm_smmu_device *smmu)
{
int ret;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 367529538eb8..09c1cf85cccc 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -553,6 +553,14 @@ int arm_smmu_device_disable(struct arm_smmu_device *smmu);
struct iommu_group *arm_smmu_device_group(struct device *dev);
int arm_smmu_of_xlate(struct device *dev, const struct of_phandle_args *args);
void arm_smmu_get_resv_regions(struct device *dev, struct list_head *head);
+int arm_smmu_init_strtab(struct arm_smmu_device *smmu);
+void arm_smmu_write_strtab(struct arm_smmu_device *smmu);
+void arm_smmu_init_initial_stes(struct arm_smmu_ste *strtab,
+ unsigned int nent);
+int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
+ struct arm_smmu_queue *q, void __iomem *page,
+ unsigned long prod_off, unsigned long cons_off,
+ size_t dwords, const char *name);
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 10/29] iommu/arm-smmu-v3: Move firmware probe to arm-smmu-v3-common
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (8 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 09/29] iommu/arm-smmu-v3: Move queue and table allocation " Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 11/29] iommu/arm-smmu-v3: Move IOMMU registration to arm-smmu-v3-common.c Mostafa Saleh
` (18 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Move the FW probe functions to the common source.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../arm/arm-smmu-v3/arm-smmu-v3-common.c | 146 ++++++++++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 143 +----------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 +
3 files changed, 150 insertions(+), 142 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
index 6cd5187d62a1..835eeae3bf5e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
@@ -1,7 +1,12 @@
// SPDX-License-Identifier: GPL-2.0
+#include <linux/acpi.h>
#include <linux/dma-mapping.h>
#include <linux/iopoll.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_platform.h>
#include <linux/pci.h>
+#include <linux/platform_device.h>
#include <linux/string_choices.h>
#include <kunit/visibility.h>
@@ -12,6 +17,11 @@
#define IIDR_PRODUCTID_ARM_MMU_600 0x483
#define IIDR_PRODUCTID_ARM_MMU_700 0x487
+struct arm_smmu_option_prop {
+ u32 opt;
+ const char *prop;
+};
+
static void arm_smmu_device_iidr_probe(struct arm_smmu_device *smmu)
{
u32 reg;
@@ -525,3 +535,139 @@ void arm_smmu_write_strtab(struct arm_smmu_device *smmu)
smmu->base + ARM_SMMU_STRTAB_BASE);
writel_relaxed(reg, smmu->base + ARM_SMMU_STRTAB_BASE_CFG);
}
+
+static struct arm_smmu_option_prop arm_smmu_options[] = {
+ { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
+ { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"},
+ { 0, NULL},
+};
+
+static void parse_driver_options(struct arm_smmu_device *smmu)
+{
+ int i = 0;
+
+ do {
+ if (of_property_read_bool(smmu->dev->of_node,
+ arm_smmu_options[i].prop)) {
+ smmu->options |= arm_smmu_options[i].opt;
+ dev_notice(smmu->dev, "option %s\n",
+ arm_smmu_options[i].prop);
+ }
+ } while (arm_smmu_options[++i].opt);
+}
+
+#ifdef CONFIG_ACPI
+#ifdef CONFIG_TEGRA241_CMDQV
+static void acpi_smmu_dsdt_probe_tegra241_cmdqv(struct acpi_iort_node *node,
+ struct arm_smmu_device *smmu)
+{
+ const char *uid = kasprintf(GFP_KERNEL, "%u", node->identifier);
+ struct acpi_device *adev;
+
+ /* Look for an NVDA200C node whose _UID matches the SMMU node ID */
+ adev = acpi_dev_get_first_match_dev("NVDA200C", uid, -1);
+ if (adev) {
+ /* Tegra241 CMDQV driver is responsible for put_device() */
+ smmu->impl_dev = &adev->dev;
+ smmu->options |= ARM_SMMU_OPT_TEGRA241_CMDQV;
+ dev_info(smmu->dev, "found companion CMDQV device: %s\n",
+ dev_name(smmu->impl_dev));
+ }
+ kfree(uid);
+}
+#else
+static void acpi_smmu_dsdt_probe_tegra241_cmdqv(struct acpi_iort_node *node,
+ struct arm_smmu_device *smmu)
+{
+}
+#endif
+
+static int acpi_smmu_iort_probe_model(struct acpi_iort_node *node,
+ struct arm_smmu_device *smmu)
+{
+ struct acpi_iort_smmu_v3 *iort_smmu =
+ (struct acpi_iort_smmu_v3 *)node->node_data;
+
+ switch (iort_smmu->model) {
+ case ACPI_IORT_SMMU_V3_CAVIUM_CN99XX:
+ smmu->options |= ARM_SMMU_OPT_PAGE0_REGS_ONLY;
+ break;
+ case ACPI_IORT_SMMU_V3_HISILICON_HI161X:
+ smmu->options |= ARM_SMMU_OPT_SKIP_PREFETCH;
+ break;
+ case ACPI_IORT_SMMU_V3_GENERIC:
+ /*
+ * Tegra241 implementation stores its SMMU options and impl_dev
+ * in DSDT. Thus, go through the ACPI tables unconditionally.
+ */
+ acpi_smmu_dsdt_probe_tegra241_cmdqv(node, smmu);
+ break;
+ }
+
+ dev_notice(smmu->dev, "option mask 0x%x\n", smmu->options);
+ return 0;
+}
+
+static int arm_smmu_device_acpi_probe(struct platform_device *pdev,
+ struct arm_smmu_device *smmu)
+{
+ struct acpi_iort_smmu_v3 *iort_smmu;
+ struct device *dev = smmu->dev;
+ struct acpi_iort_node *node;
+
+ node = *(struct acpi_iort_node **)dev_get_platdata(dev);
+
+ /* Retrieve SMMUv3 specific data */
+ iort_smmu = (struct acpi_iort_smmu_v3 *)node->node_data;
+
+ if (iort_smmu->flags & ACPI_IORT_SMMU_V3_COHACC_OVERRIDE)
+ smmu->features |= ARM_SMMU_FEAT_COHERENCY;
+
+ switch (FIELD_GET(ACPI_IORT_SMMU_V3_HTTU_OVERRIDE, iort_smmu->flags)) {
+ case IDR0_HTTU_ACCESS_DIRTY:
+ smmu->features |= ARM_SMMU_FEAT_HD;
+ fallthrough;
+ case IDR0_HTTU_ACCESS:
+ smmu->features |= ARM_SMMU_FEAT_HA;
+ }
+
+ return acpi_smmu_iort_probe_model(node, smmu);
+}
+#else
+static inline int arm_smmu_device_acpi_probe(struct platform_device *pdev,
+ struct arm_smmu_device *smmu)
+{
+ return -ENODEV;
+}
+#endif
+
+static int arm_smmu_device_dt_probe(struct platform_device *pdev,
+ struct arm_smmu_device *smmu)
+{
+ struct device *dev = &pdev->dev;
+ u32 cells;
+ int ret = -EINVAL;
+
+ if (of_property_read_u32(dev->of_node, "#iommu-cells", &cells))
+ dev_err(dev, "missing #iommu-cells property\n");
+ else if (cells != 1)
+ dev_err(dev, "invalid #iommu-cells value (%d)\n", cells);
+ else
+ ret = 0;
+
+ parse_driver_options(smmu);
+
+ if (of_dma_is_coherent(dev->of_node))
+ smmu->features |= ARM_SMMU_FEAT_COHERENCY;
+
+ return ret;
+}
+
+int arm_smmu_fw_probe(struct platform_device *pdev,
+ struct arm_smmu_device *smmu)
+{
+ if (smmu->dev->of_node)
+ return arm_smmu_device_dt_probe(pdev, smmu);
+ else
+ return arm_smmu_device_acpi_probe(pdev, smmu);
+}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 6e6ac64a5399..4cd0a115fc0e 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -9,7 +9,6 @@
* This driver is powered by bad coffee and bombay mix.
*/
-#include <linux/acpi.h>
#include <linux/acpi_iort.h>
#include <linux/bitops.h>
#include <linux/crash_dump.h>
@@ -19,12 +18,8 @@
#include <linux/io-pgtable.h>
#include <linux/module.h>
#include <linux/msi.h>
-#include <linux/of.h>
-#include <linux/of_address.h>
-#include <linux/of_platform.h>
#include <linux/pci.h>
#include <linux/pci-ats.h>
-#include <linux/platform_device.h>
#include <kunit/visibility.h>
#include <uapi/linux/iommufd.h>
@@ -67,20 +62,9 @@ static phys_addr_t arm_smmu_msi_cfg[ARM_SMMU_MAX_MSIS][3] = {
},
};
-struct arm_smmu_option_prop {
- u32 opt;
- const char *prop;
-};
-
DEFINE_XARRAY_ALLOC1(arm_smmu_asid_xa);
DEFINE_MUTEX(arm_smmu_asid_lock);
-static struct arm_smmu_option_prop arm_smmu_options[] = {
- { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" },
- { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"},
- { 0, NULL},
-};
-
static const char * const event_str[] = {
[EVT_ID_BAD_STREAMID_CONFIG] = "C_BAD_STREAMID",
[EVT_ID_STE_FETCH_FAULT] = "F_STE_FETCH",
@@ -105,20 +89,6 @@ static const char * const event_class_str[] = {
static int arm_smmu_alloc_cd_tables(struct arm_smmu_master *master);
-static void parse_driver_options(struct arm_smmu_device *smmu)
-{
- int i = 0;
-
- do {
- if (of_property_read_bool(smmu->dev->of_node,
- arm_smmu_options[i].prop)) {
- smmu->options |= arm_smmu_options[i].opt;
- dev_notice(smmu->dev, "option %s\n",
- arm_smmu_options[i].prop);
- }
- } while (arm_smmu_options[++i].opt);
-}
-
/* Low-level queue manipulation functions */
static bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n)
{
@@ -3987,113 +3957,6 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu)
return 0;
}
-#ifdef CONFIG_ACPI
-#ifdef CONFIG_TEGRA241_CMDQV
-static void acpi_smmu_dsdt_probe_tegra241_cmdqv(struct acpi_iort_node *node,
- struct arm_smmu_device *smmu)
-{
- const char *uid = kasprintf(GFP_KERNEL, "%u", node->identifier);
- struct acpi_device *adev;
-
- /* Look for an NVDA200C node whose _UID matches the SMMU node ID */
- adev = acpi_dev_get_first_match_dev("NVDA200C", uid, -1);
- if (adev) {
- /* Tegra241 CMDQV driver is responsible for put_device() */
- smmu->impl_dev = &adev->dev;
- smmu->options |= ARM_SMMU_OPT_TEGRA241_CMDQV;
- dev_info(smmu->dev, "found companion CMDQV device: %s\n",
- dev_name(smmu->impl_dev));
- }
- kfree(uid);
-}
-#else
-static void acpi_smmu_dsdt_probe_tegra241_cmdqv(struct acpi_iort_node *node,
- struct arm_smmu_device *smmu)
-{
-}
-#endif
-
-static int acpi_smmu_iort_probe_model(struct acpi_iort_node *node,
- struct arm_smmu_device *smmu)
-{
- struct acpi_iort_smmu_v3 *iort_smmu =
- (struct acpi_iort_smmu_v3 *)node->node_data;
-
- switch (iort_smmu->model) {
- case ACPI_IORT_SMMU_V3_CAVIUM_CN99XX:
- smmu->options |= ARM_SMMU_OPT_PAGE0_REGS_ONLY;
- break;
- case ACPI_IORT_SMMU_V3_HISILICON_HI161X:
- smmu->options |= ARM_SMMU_OPT_SKIP_PREFETCH;
- break;
- case ACPI_IORT_SMMU_V3_GENERIC:
- /*
- * Tegra241 implementation stores its SMMU options and impl_dev
- * in DSDT. Thus, go through the ACPI tables unconditionally.
- */
- acpi_smmu_dsdt_probe_tegra241_cmdqv(node, smmu);
- break;
- }
-
- dev_notice(smmu->dev, "option mask 0x%x\n", smmu->options);
- return 0;
-}
-
-static int arm_smmu_device_acpi_probe(struct platform_device *pdev,
- struct arm_smmu_device *smmu)
-{
- struct acpi_iort_smmu_v3 *iort_smmu;
- struct device *dev = smmu->dev;
- struct acpi_iort_node *node;
-
- node = *(struct acpi_iort_node **)dev_get_platdata(dev);
-
- /* Retrieve SMMUv3 specific data */
- iort_smmu = (struct acpi_iort_smmu_v3 *)node->node_data;
-
- if (iort_smmu->flags & ACPI_IORT_SMMU_V3_COHACC_OVERRIDE)
- smmu->features |= ARM_SMMU_FEAT_COHERENCY;
-
- switch (FIELD_GET(ACPI_IORT_SMMU_V3_HTTU_OVERRIDE, iort_smmu->flags)) {
- case IDR0_HTTU_ACCESS_DIRTY:
- smmu->features |= ARM_SMMU_FEAT_HD;
- fallthrough;
- case IDR0_HTTU_ACCESS:
- smmu->features |= ARM_SMMU_FEAT_HA;
- }
-
- return acpi_smmu_iort_probe_model(node, smmu);
-}
-#else
-static inline int arm_smmu_device_acpi_probe(struct platform_device *pdev,
- struct arm_smmu_device *smmu)
-{
- return -ENODEV;
-}
-#endif
-
-static int arm_smmu_device_dt_probe(struct platform_device *pdev,
- struct arm_smmu_device *smmu)
-{
- struct device *dev = &pdev->dev;
- u32 cells;
- int ret = -EINVAL;
-
- if (of_property_read_u32(dev->of_node, "#iommu-cells", &cells))
- dev_err(dev, "missing #iommu-cells property\n");
- else if (cells != 1)
- dev_err(dev, "invalid #iommu-cells value (%d)\n", cells);
- else
- ret = 0;
-
- parse_driver_options(smmu);
-
- if (of_dma_is_coherent(dev->of_node))
- smmu->features |= ARM_SMMU_FEAT_COHERENCY;
-
- return ret;
-}
-
static unsigned long arm_smmu_resource_size(struct arm_smmu_device *smmu)
{
if (smmu->options & ARM_SMMU_OPT_PAGE0_REGS_ONLY)
@@ -4189,11 +4052,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
return -ENOMEM;
smmu->dev = dev;
- if (dev->of_node) {
- ret = arm_smmu_device_dt_probe(pdev, smmu);
- } else {
- ret = arm_smmu_device_acpi_probe(pdev, smmu);
- }
+ ret = arm_smmu_fw_probe(pdev, smmu);
if (ret)
return ret;
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 09c1cf85cccc..787296a2f0c2 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -12,6 +12,7 @@
#include <linux/iommufd.h>
#include <linux/kernel.h>
#include <linux/mmzone.h>
+#include <linux/platform_device.h>
#include <linux/sizes.h>
struct arm_smmu_device;
@@ -561,6 +562,8 @@ int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
struct arm_smmu_queue *q, void __iomem *page,
unsigned long prod_off, unsigned long cons_off,
size_t dwords, const char *name);
+int arm_smmu_fw_probe(struct platform_device *pdev,
+ struct arm_smmu_device *smmu);
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 11/29] iommu/arm-smmu-v3: Move IOMMU registration to arm-smmu-v3-common.c
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (9 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 10/29] iommu/arm-smmu-v3: Move firmware probe to arm-smmu-v3-common Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:52 ` [PATCH v3 12/29] iommu/arm-smmu-v3: Split cmdq code with hyp Mostafa Saleh
` (17 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
The KVM driver will need to implement a few IOMMU ops, so move the
helpers to arm-smmu-v3-common.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../arm/arm-smmu-v3/arm-smmu-v3-common.c | 27 +++++++++++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 18 ++-----------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 3 +++
3 files changed, 32 insertions(+), 16 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
index 835eeae3bf5e..7453d9894ca3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.c
@@ -671,3 +671,30 @@ int arm_smmu_fw_probe(struct platform_device *pdev,
else
return arm_smmu_device_acpi_probe(pdev, smmu);
}
+
+int arm_smmu_register_iommu(struct arm_smmu_device *smmu,
+ struct iommu_ops *ops, phys_addr_t ioaddr)
+{
+ int ret;
+ struct device *dev = smmu->dev;
+
+ ret = iommu_device_sysfs_add(&smmu->iommu, dev, NULL,
+ "smmu3.%pa", &ioaddr);
+ if (ret)
+ return ret;
+
+ ret = iommu_device_register(&smmu->iommu, ops, dev);
+ if (ret) {
+ dev_err(dev, "Failed to register iommu\n");
+ iommu_device_sysfs_remove(&smmu->iommu);
+ return ret;
+ }
+
+ return 0;
+}
+
+void arm_smmu_unregister_iommu(struct arm_smmu_device *smmu)
+{
+ iommu_device_unregister(&smmu->iommu);
+ iommu_device_sysfs_remove(&smmu->iommu);
+}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 4cd0a115fc0e..8fbc7dd2a2db 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -4138,21 +4138,8 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
goto err_disable;
/* And we're up. Go go go! */
- ret = iommu_device_sysfs_add(&smmu->iommu, dev, NULL,
- "smmu3.%pa", &ioaddr);
- if (ret)
- goto err_disable;
-
- ret = iommu_device_register(&smmu->iommu, &arm_smmu_ops, dev);
- if (ret) {
- dev_err(dev, "Failed to register iommu\n");
- goto err_free_sysfs;
- }
-
- return 0;
+ return arm_smmu_register_iommu(smmu, &arm_smmu_ops, ioaddr);
-err_free_sysfs:
- iommu_device_sysfs_remove(&smmu->iommu);
err_disable:
arm_smmu_device_disable(smmu);
err_free_iopf:
@@ -4164,8 +4151,7 @@ static void arm_smmu_device_remove(struct platform_device *pdev)
{
struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
- iommu_device_unregister(&smmu->iommu);
- iommu_device_sysfs_remove(&smmu->iommu);
+ arm_smmu_unregister_iommu(smmu);
arm_smmu_device_disable(smmu);
iopf_queue_free(smmu->evtq.iopf);
ida_destroy(&smmu->vmid_map);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index 787296a2f0c2..e3c46b3344db 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -564,6 +564,9 @@ int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
size_t dwords, const char *name);
int arm_smmu_fw_probe(struct platform_device *pdev,
struct arm_smmu_device *smmu);
+int arm_smmu_register_iommu(struct arm_smmu_device *smmu,
+ struct iommu_ops *ops, phys_addr_t ioaddr);
+void arm_smmu_unregister_iommu(struct arm_smmu_device *smmu);
#ifdef CONFIG_ARM_SMMU_V3_SVA
bool arm_smmu_sva_supported(struct arm_smmu_device *smmu);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 12/29] iommu/arm-smmu-v3: Split cmdq code with hyp
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (10 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 11/29] iommu/arm-smmu-v3: Move IOMMU registration to arm-smmu-v3-common.c Mostafa Saleh
@ 2025-07-28 17:52 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 13/29] iommu/arm-smmu-v3: Move TLB range invalidation into a macro Mostafa Saleh
` (16 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:52 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
The KVM SMMUv3 driver would re-use some of the cmdq code inside
the hypervisor, move these functions to a new common c file that
is shared between the host kernel and the hypervisor.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
drivers/iommu/arm/arm-smmu-v3/Makefile | 2 +-
.../arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c | 114 ++++++++++++++++
.../arm/arm-smmu-v3/arm-smmu-v3-common.h | 69 ++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 128 ------------------
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 38 ------
5 files changed, 184 insertions(+), 167 deletions(-)
create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c
diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm-smmu-v3/Makefile
index 0bc0c116b851..16be1249000a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/Makefile
+++ b/drivers/iommu/arm/arm-smmu-v3/Makefile
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_ARM_SMMU_V3) += arm_smmu_v3.o
arm_smmu_v3-y := arm-smmu-v3.o
-arm_smmu_v3-y += arm-smmu-v3-common.o
+arm_smmu_v3-y += arm-smmu-v3-common.o arm-smmu-v3-common-hyp.o
arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) += arm-smmu-v3-iommufd.o
arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) += arm-smmu-v3-sva.o
arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) += tegra241-cmdqv.o
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c
new file mode 100644
index 000000000000..da41c5024a1c
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c
@@ -0,0 +1,114 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2015 ARM Limited
+ *
+ * Author: Will Deacon <will.deacon@arm.com>
+ * Arm SMMUv3 driver functions shared with hypervisor.
+ */
+
+#include "arm-smmu-v3-common.h"
+#include <asm-generic/errno-base.h>
+
+#include <linux/string.h>
+
+int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
+{
+ memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT);
+ cmd[0] |= FIELD_PREP(CMDQ_0_OP, ent->opcode);
+
+ switch (ent->opcode) {
+ case CMDQ_OP_TLBI_EL2_ALL:
+ case CMDQ_OP_TLBI_NSNH_ALL:
+ break;
+ case CMDQ_OP_PREFETCH_CFG:
+ cmd[0] |= FIELD_PREP(CMDQ_PREFETCH_0_SID, ent->prefetch.sid);
+ break;
+ case CMDQ_OP_CFGI_CD:
+ cmd[0] |= FIELD_PREP(CMDQ_CFGI_0_SSID, ent->cfgi.ssid);
+ fallthrough;
+ case CMDQ_OP_CFGI_STE:
+ cmd[0] |= FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid);
+ cmd[1] |= FIELD_PREP(CMDQ_CFGI_1_LEAF, ent->cfgi.leaf);
+ break;
+ case CMDQ_OP_CFGI_CD_ALL:
+ cmd[0] |= FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid);
+ break;
+ case CMDQ_OP_CFGI_ALL:
+ /* Cover the entire SID range */
+ cmd[1] |= FIELD_PREP(CMDQ_CFGI_1_RANGE, 31);
+ break;
+ case CMDQ_OP_TLBI_NH_VA:
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
+ fallthrough;
+ case CMDQ_OP_TLBI_EL2_VA:
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num);
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale);
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
+ cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf);
+ cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl);
+ cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg);
+ cmd[1] |= ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK;
+ break;
+ case CMDQ_OP_TLBI_S2_IPA:
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num);
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale);
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
+ cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf);
+ cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl);
+ cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg);
+ cmd[1] |= ent->tlbi.addr & CMDQ_TLBI_1_IPA_MASK;
+ break;
+ case CMDQ_OP_TLBI_NH_ASID:
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
+ fallthrough;
+ case CMDQ_OP_TLBI_NH_ALL:
+ case CMDQ_OP_TLBI_S12_VMALL:
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
+ break;
+ case CMDQ_OP_TLBI_EL2_ASID:
+ cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
+ break;
+ case CMDQ_OP_ATC_INV:
+ cmd[0] |= FIELD_PREP(CMDQ_0_SSV, ent->substream_valid);
+ cmd[0] |= FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global);
+ cmd[0] |= FIELD_PREP(CMDQ_ATC_0_SSID, ent->atc.ssid);
+ cmd[0] |= FIELD_PREP(CMDQ_ATC_0_SID, ent->atc.sid);
+ cmd[1] |= FIELD_PREP(CMDQ_ATC_1_SIZE, ent->atc.size);
+ cmd[1] |= ent->atc.addr & CMDQ_ATC_1_ADDR_MASK;
+ break;
+ case CMDQ_OP_PRI_RESP:
+ cmd[0] |= FIELD_PREP(CMDQ_0_SSV, ent->substream_valid);
+ cmd[0] |= FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid);
+ cmd[0] |= FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid);
+ cmd[1] |= FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid);
+ switch (ent->pri.resp) {
+ case PRI_RESP_DENY:
+ case PRI_RESP_FAIL:
+ case PRI_RESP_SUCC:
+ break;
+ default:
+ return -EINVAL;
+ }
+ cmd[1] |= FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp);
+ break;
+ case CMDQ_OP_RESUME:
+ cmd[0] |= FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid);
+ cmd[0] |= FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp);
+ cmd[1] |= FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag);
+ break;
+ case CMDQ_OP_CMD_SYNC:
+ if (ent->sync.msiaddr) {
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ);
+ cmd[1] |= ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK;
+ } else {
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV);
+ }
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH);
+ cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB);
+ break;
+ default:
+ return -ENOENT;
+ }
+
+ return 0;
+}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
index 3bcb5d94f21a..f115ff9a547f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
@@ -4,6 +4,7 @@
#include <linux/bits.h>
#include <linux/bitfield.h>
+#include <linux/cache.h>
/* MMIO registers */
#define ARM_SMMU_IDR0 0x0
@@ -460,6 +461,44 @@ struct arm_smmu_cmdq_ent {
#define ARM_SMMU_FEAT_HD (1 << 22)
#define ARM_SMMU_FEAT_S2FWB (1 << 23)
+struct arm_smmu_ll_queue {
+ union {
+ u64 val;
+ struct {
+ u32 prod;
+ u32 cons;
+ };
+ struct {
+ atomic_t prod;
+ atomic_t cons;
+ } atomic;
+ u8 __pad[SMP_CACHE_BYTES];
+ } ____cacheline_aligned_in_smp;
+ u32 max_n_shift;
+};
+
+struct arm_smmu_queue {
+ struct arm_smmu_ll_queue llq;
+ int irq; /* Wired interrupt */
+
+ __le64 *base;
+ dma_addr_t base_dma;
+ u64 q_base;
+
+ size_t ent_dwords;
+
+ u32 __iomem *prod_reg;
+ u32 __iomem *cons_reg;
+};
+
+#define Q_IDX(llq, p) ((p) & ((1 << (llq)->max_n_shift) - 1))
+#define Q_WRP(llq, p) ((p) & (1 << (llq)->max_n_shift))
+#define Q_OVERFLOW_FLAG (1U << 31)
+#define Q_OVF(p) ((p) & Q_OVERFLOW_FLAG)
+#define Q_ENT(q, p) ((q)->base + \
+ Q_IDX(&((q)->llq), p) * \
+ (q)->ent_dwords)
+
static inline void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst,
dma_addr_t l2ptr_dma)
{
@@ -471,4 +510,34 @@ static inline void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst,
/* The HW has 64 bit atomicity with stores to the L2 STE table */
WRITE_ONCE(dst->l2ptr, cpu_to_le64(val));
}
+
+int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent);
+
+
+/* Queue functions shared between kernel and hyp. */
+static inline bool queue_full(struct arm_smmu_ll_queue *q)
+{
+ return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
+ Q_WRP(q, q->prod) != Q_WRP(q, q->cons);
+}
+
+static inline bool queue_empty(struct arm_smmu_ll_queue *q)
+{
+ return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
+ Q_WRP(q, q->prod) == Q_WRP(q, q->cons);
+}
+
+static inline u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n)
+{
+ u32 prod = (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + n;
+ return Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod);
+}
+
+static inline void queue_write(__le64 *dst, u64 *src, size_t n_dwords)
+{
+ int i;
+
+ for (i = 0; i < n_dwords; ++i)
+ *dst++ = cpu_to_le64(*src++);
+}
#endif /* _ARM_SMMU_V3_COMMON_H */
\ No newline at end of file
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 8fbc7dd2a2db..066d9d9e8088 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -105,18 +105,6 @@ static bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n)
return space >= n;
}
-static bool queue_full(struct arm_smmu_ll_queue *q)
-{
- return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
- Q_WRP(q, q->prod) != Q_WRP(q, q->cons);
-}
-
-static bool queue_empty(struct arm_smmu_ll_queue *q)
-{
- return Q_IDX(q, q->prod) == Q_IDX(q, q->cons) &&
- Q_WRP(q, q->prod) == Q_WRP(q, q->cons);
-}
-
static bool queue_consumed(struct arm_smmu_ll_queue *q, u32 prod)
{
return ((Q_WRP(q, q->cons) == Q_WRP(q, prod)) &&
@@ -172,12 +160,6 @@ static int queue_sync_prod_in(struct arm_smmu_queue *q)
return ret;
}
-static u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n)
-{
- u32 prod = (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + n;
- return Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod);
-}
-
static void queue_poll_init(struct arm_smmu_device *smmu,
struct arm_smmu_queue_poll *qp)
{
@@ -205,14 +187,6 @@ static int queue_poll(struct arm_smmu_queue_poll *qp)
return 0;
}
-static void queue_write(__le64 *dst, u64 *src, size_t n_dwords)
-{
- int i;
-
- for (i = 0; i < n_dwords; ++i)
- *dst++ = cpu_to_le64(*src++);
-}
-
static void queue_read(u64 *dst, __le64 *src, size_t n_dwords)
{
int i;
@@ -233,108 +207,6 @@ static int queue_remove_raw(struct arm_smmu_queue *q, u64 *ent)
}
/* High-level queue accessors */
-static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
-{
- memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT);
- cmd[0] |= FIELD_PREP(CMDQ_0_OP, ent->opcode);
-
- switch (ent->opcode) {
- case CMDQ_OP_TLBI_EL2_ALL:
- case CMDQ_OP_TLBI_NSNH_ALL:
- break;
- case CMDQ_OP_PREFETCH_CFG:
- cmd[0] |= FIELD_PREP(CMDQ_PREFETCH_0_SID, ent->prefetch.sid);
- break;
- case CMDQ_OP_CFGI_CD:
- cmd[0] |= FIELD_PREP(CMDQ_CFGI_0_SSID, ent->cfgi.ssid);
- fallthrough;
- case CMDQ_OP_CFGI_STE:
- cmd[0] |= FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid);
- cmd[1] |= FIELD_PREP(CMDQ_CFGI_1_LEAF, ent->cfgi.leaf);
- break;
- case CMDQ_OP_CFGI_CD_ALL:
- cmd[0] |= FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid);
- break;
- case CMDQ_OP_CFGI_ALL:
- /* Cover the entire SID range */
- cmd[1] |= FIELD_PREP(CMDQ_CFGI_1_RANGE, 31);
- break;
- case CMDQ_OP_TLBI_NH_VA:
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
- fallthrough;
- case CMDQ_OP_TLBI_EL2_VA:
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num);
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale);
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
- cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf);
- cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl);
- cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg);
- cmd[1] |= ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK;
- break;
- case CMDQ_OP_TLBI_S2_IPA:
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num);
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale);
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
- cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf);
- cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl);
- cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg);
- cmd[1] |= ent->tlbi.addr & CMDQ_TLBI_1_IPA_MASK;
- break;
- case CMDQ_OP_TLBI_NH_ASID:
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
- fallthrough;
- case CMDQ_OP_TLBI_NH_ALL:
- case CMDQ_OP_TLBI_S12_VMALL:
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
- break;
- case CMDQ_OP_TLBI_EL2_ASID:
- cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
- break;
- case CMDQ_OP_ATC_INV:
- cmd[0] |= FIELD_PREP(CMDQ_0_SSV, ent->substream_valid);
- cmd[0] |= FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global);
- cmd[0] |= FIELD_PREP(CMDQ_ATC_0_SSID, ent->atc.ssid);
- cmd[0] |= FIELD_PREP(CMDQ_ATC_0_SID, ent->atc.sid);
- cmd[1] |= FIELD_PREP(CMDQ_ATC_1_SIZE, ent->atc.size);
- cmd[1] |= ent->atc.addr & CMDQ_ATC_1_ADDR_MASK;
- break;
- case CMDQ_OP_PRI_RESP:
- cmd[0] |= FIELD_PREP(CMDQ_0_SSV, ent->substream_valid);
- cmd[0] |= FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid);
- cmd[0] |= FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid);
- cmd[1] |= FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid);
- switch (ent->pri.resp) {
- case PRI_RESP_DENY:
- case PRI_RESP_FAIL:
- case PRI_RESP_SUCC:
- break;
- default:
- return -EINVAL;
- }
- cmd[1] |= FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp);
- break;
- case CMDQ_OP_RESUME:
- cmd[0] |= FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid);
- cmd[0] |= FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp);
- cmd[1] |= FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag);
- break;
- case CMDQ_OP_CMD_SYNC:
- if (ent->sync.msiaddr) {
- cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ);
- cmd[1] |= ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK;
- } else {
- cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV);
- }
- cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH);
- cmd[0] |= FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB);
- break;
- default:
- return -ENOENT;
- }
-
- return 0;
-}
-
static struct arm_smmu_cmdq *arm_smmu_get_cmdq(struct arm_smmu_device *smmu,
struct arm_smmu_cmdq_ent *ent)
{
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index e3c46b3344db..f9a496e0bd94 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -19,14 +19,6 @@ struct arm_smmu_device;
#include "arm-smmu-v3-common.h"
-#define Q_IDX(llq, p) ((p) & ((1 << (llq)->max_n_shift) - 1))
-#define Q_WRP(llq, p) ((p) & (1 << (llq)->max_n_shift))
-#define Q_OVERFLOW_FLAG (1U << 31)
-#define Q_OVF(p) ((p) & Q_OVERFLOW_FLAG)
-#define Q_ENT(q, p) ((q)->base + \
- Q_IDX(&((q)->llq), p) * \
- (q)->ent_dwords)
-
/* Ensure DMA allocations are naturally aligned */
#ifdef CONFIG_CMA_ALIGNMENT
#define Q_MAX_SZ_SHIFT (PAGE_SHIFT + CONFIG_CMA_ALIGNMENT)
@@ -162,36 +154,6 @@ static inline unsigned int arm_smmu_cdtab_l2_idx(unsigned int ssid)
#define MSI_IOVA_BASE 0x8000000
#define MSI_IOVA_LENGTH 0x100000
-struct arm_smmu_ll_queue {
- union {
- u64 val;
- struct {
- u32 prod;
- u32 cons;
- };
- struct {
- atomic_t prod;
- atomic_t cons;
- } atomic;
- u8 __pad[SMP_CACHE_BYTES];
- } ____cacheline_aligned_in_smp;
- u32 max_n_shift;
-};
-
-struct arm_smmu_queue {
- struct arm_smmu_ll_queue llq;
- int irq; /* Wired interrupt */
-
- __le64 *base;
- dma_addr_t base_dma;
- u64 q_base;
-
- size_t ent_dwords;
-
- u32 __iomem *prod_reg;
- u32 __iomem *cons_reg;
-};
-
struct arm_smmu_queue_poll {
ktime_t timeout;
unsigned int delay;
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 13/29] iommu/arm-smmu-v3: Move TLB range invalidation into a macro
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (11 preceding siblings ...)
2025-07-28 17:52 ` [PATCH v3 12/29] iommu/arm-smmu-v3: Split cmdq code with hyp Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 14/29] KVM: arm64: iommu: Introduce IOMMU driver infrastructure Mostafa Saleh
` (15 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Range TLB invalidation has a very specific algorithm, instead of
re-writing it for the hypervisor, put it in a macro so it can be
re-used.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../arm/arm-smmu-v3/arm-smmu-v3-common.h | 65 +++++++++++++++++++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 59 +----------------
2 files changed, 68 insertions(+), 56 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
index f115ff9a547f..ab6db8b4b9f2 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
@@ -540,4 +540,69 @@ static inline void queue_write(__le64 *dst, u64 *src, size_t n_dwords)
for (i = 0; i < n_dwords; ++i)
*dst++ = cpu_to_le64(*src++);
}
+
+/**
+ * arm_smmu_tlb_inv_build - Create a range invalidation command
+ * @cmd: Base command initialized with OPCODE (S1, S2..), vmid and asid.
+ * @iova: Start IOVA to invalidate
+ * @size: Size of range
+ * @granule: Granule of invalidation
+ * @pgsize_bitmap: Page size bit map of the page table.
+ * @smmu: Struct for the smmu, must have ::features
+ * @add_cmd: Function to send/batch the invalidation command
+ * @cmds: Incase of batching, it includes the pointer to the batch
+ */
+#define arm_smmu_tlb_inv_build(cmd, iova, size, granule, pgsize_bitmap, smmu, add_cmd, cmds) \
+{ \
+ unsigned long _iova = (iova); \
+ size_t _size = (size); \
+ size_t _granule = (granule); \
+ unsigned long end = _iova + _size, num_pages = 0, tg = 0; \
+ size_t inv_range = _granule; \
+ if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { \
+ /* Get the leaf page size */ \
+ tg = __ffs(pgsize_bitmap); \
+ num_pages = _size >> tg; \
+ /* Convert page size of 12,14,16 (log2) to 1,2,3 */ \
+ cmd->tlbi.tg = (tg - 10) / 2; \
+ /*
+ * Determine what level the granule is at. For non-leaf, both
+ * io-pgtable and SVA pass a nominal last-level granule because
+ * they don't know what level(s) actually apply, so ignore that
+ * and leave TTL=0. However for various errata reasons we still
+ * want to use a range command, so avoid the SVA corner case
+ * where both scale and num could be 0 as well.
+ */ \
+ if (cmd->tlbi.leaf) \
+ cmd->tlbi.ttl = 4 - ((ilog2(_granule) - 3) / (tg - 3)); \
+ else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) == 1) \
+ num_pages++; \
+ } \
+ while (_iova < end) { \
+ if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { \
+ /*
+ * On each iteration of the loop, the range is 5 bits
+ * worth of the aligned size remaining.
+ * The range in pages is:
+ *
+ * range = (num_pages & (0x1f << __ffs(num_pages)))
+ */ \
+ unsigned long scale, num; \
+ /* Determine the power of 2 multiple number of pages */ \
+ scale = __ffs(num_pages); \
+ cmd->tlbi.scale = scale; \
+ /* Determine how many chunks of 2^scale size we have */ \
+ num = (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; \
+ cmd->tlbi.num = num - 1; \
+ /* range is num * 2^scale * pgsize */ \
+ inv_range = num << (scale + tg); \
+ /* Clear out the lower order bits for the next iteration */ \
+ num_pages -= num << scale; \
+ } \
+ cmd->tlbi.addr = iova; \
+ add_cmd(smmu, cmds, cmd); \
+ _iova += inv_range; \
+ } \
+} \
+
#endif /* _ARM_SMMU_V3_COMMON_H */
\ No newline at end of file
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 066d9d9e8088..245b39ddef66 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2075,68 +2075,15 @@ static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd,
struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_device *smmu = smmu_domain->smmu;
- unsigned long end = iova + size, num_pages = 0, tg = 0;
- size_t inv_range = granule;
struct arm_smmu_cmdq_batch cmds;
if (!size)
return;
- if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
- /* Get the leaf page size */
- tg = __ffs(smmu_domain->domain.pgsize_bitmap);
-
- num_pages = size >> tg;
-
- /* Convert page size of 12,14,16 (log2) to 1,2,3 */
- cmd->tlbi.tg = (tg - 10) / 2;
-
- /*
- * Determine what level the granule is at. For non-leaf, both
- * io-pgtable and SVA pass a nominal last-level granule because
- * they don't know what level(s) actually apply, so ignore that
- * and leave TTL=0. However for various errata reasons we still
- * want to use a range command, so avoid the SVA corner case
- * where both scale and num could be 0 as well.
- */
- if (cmd->tlbi.leaf)
- cmd->tlbi.ttl = 4 - ((ilog2(granule) - 3) / (tg - 3));
- else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) == 1)
- num_pages++;
- }
-
arm_smmu_cmdq_batch_init(smmu, &cmds, cmd);
-
- while (iova < end) {
- if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
- /*
- * On each iteration of the loop, the range is 5 bits
- * worth of the aligned size remaining.
- * The range in pages is:
- *
- * range = (num_pages & (0x1f << __ffs(num_pages)))
- */
- unsigned long scale, num;
-
- /* Determine the power of 2 multiple number of pages */
- scale = __ffs(num_pages);
- cmd->tlbi.scale = scale;
-
- /* Determine how many chunks of 2^scale size we have */
- num = (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX;
- cmd->tlbi.num = num - 1;
-
- /* range is num * 2^scale * pgsize */
- inv_range = num << (scale + tg);
-
- /* Clear out the lower order bits for the next iteration */
- num_pages -= num << scale;
- }
-
- cmd->tlbi.addr = iova;
- arm_smmu_cmdq_batch_add(smmu, &cmds, cmd);
- iova += inv_range;
- }
+ arm_smmu_tlb_inv_build(cmd, iova, size, granule,
+ smmu_domain->domain.pgsize_bitmap,
+ smmu, arm_smmu_cmdq_batch_add, &cmds);
arm_smmu_cmdq_batch_submit(smmu, &cmds);
}
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 14/29] KVM: arm64: iommu: Introduce IOMMU driver infrastructure
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (12 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 13/29] iommu/arm-smmu-v3: Move TLB range invalidation into a macro Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 15/29] KVM: arm64: iommu: Shadow host stage-2 page table Mostafa Saleh
` (14 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
To establish DMA isolation, KVM needs an IOMMU driver which provide
certain ops, these ops are defined outside of the iommu_ops,
and has 2 components:
- kvm_iommu_driver (kernel): Implements simple interaction with
the kernel (init, remove,...)
- kvm_iommu_ops (hypervisor): Implements identity mapping for
the IOMMU page tables
Only one driver can be used and is registered with
kvm_iommu_register_driver() by passing pointers to both ops.
The flow of the KVM IOMMU drivers works as follows:
- The KVM IOMMU driver in the kernel after being loaded using an
initcall (before module_init) it would register with KVM both
the kernel and hypervisor ops.
- KVM will initialise the driver after it initialises and before the
de-privilege point, which is a suitable point to establish trusted
interaction between the host and the hypervisor.
In this call the IOMMU kernel driver typically starts probing
the IOMMUs, parsing firmware tables and initialising hardware.
- Then the hypervisor init will be called, to take over the IOMMUs
and initialise its data structures.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
arch/arm64/include/asm/kvm_host.h | 9 +++++
arch/arm64/kvm/Makefile | 3 +-
arch/arm64/kvm/arm.c | 8 +++-
arch/arm64/kvm/hyp/include/nvhe/iommu.h | 13 +++++++
arch/arm64/kvm/hyp/nvhe/Makefile | 3 +-
arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 18 +++++++++
arch/arm64/kvm/hyp/nvhe/setup.c | 5 +++
arch/arm64/kvm/iommu.c | 49 +++++++++++++++++++++++++
8 files changed, 105 insertions(+), 3 deletions(-)
create mode 100644 arch/arm64/kvm/hyp/include/nvhe/iommu.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
create mode 100644 arch/arm64/kvm/iommu.c
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 3e41a880b062..43f5a64bbd1d 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1674,5 +1674,14 @@ void compute_fgu(struct kvm *kvm, enum fgt_group_id fgt);
void get_reg_fixed_bits(struct kvm *kvm, enum vcpu_sysreg reg, u64 *res0, u64 *res1);
void check_feature_map(void);
+struct kvm_iommu_driver {
+ int (*init_driver)(void);
+ void (*remove_driver)(void);
+};
+
+struct kvm_iommu_ops;
+int kvm_iommu_register_driver(struct kvm_iommu_driver *kern_ops, struct kvm_iommu_ops *hyp_ops);
+int kvm_iommu_init_driver(void);
+void kvm_iommu_remove_driver(void);
#endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 7c329e01c557..5528704bfd72 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -23,7 +23,8 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
vgic/vgic-v3.o vgic/vgic-v4.o \
vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \
vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \
- vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o
+ vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o \
+ iommu.o
kvm-$(CONFIG_HW_PERF_EVENTS) += pmu-emul.o pmu.o
kvm-$(CONFIG_ARM64_PTR_AUTH) += pauth.o
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 23dd3f3fc3eb..2f494f402c48 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -2451,10 +2451,16 @@ static int __init kvm_hyp_init_protection(u32 hyp_va_bits)
if (ret)
return ret;
- ret = do_pkvm_init(hyp_va_bits);
+ ret = kvm_iommu_init_driver();
if (ret)
return ret;
+ ret = do_pkvm_init(hyp_va_bits);
+ if (ret) {
+ kvm_iommu_remove_driver();
+ return ret;
+ }
+
free_hyp_pgds();
return 0;
diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/include/nvhe/iommu.h
new file mode 100644
index 000000000000..1ac70cc28a9e
--- /dev/null
+++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ARM64_KVM_NVHE_IOMMU_H__
+#define __ARM64_KVM_NVHE_IOMMU_H__
+
+#include <asm/kvm_host.h>
+
+struct kvm_iommu_ops {
+ int (*init)(void);
+};
+
+int kvm_iommu_init(void);
+
+#endif /* __ARM64_KVM_NVHE_IOMMU_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index a76522d63c3e..393ff143f0be 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -24,7 +24,8 @@ CFLAGS_switch.nvhe.o += -Wno-override-init
hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \
hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \
- cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o
+ cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o \
+ iommu/iommu.o
hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o
hyp-obj-$(CONFIG_LIST_HARDENED) += list_debug.o
diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
new file mode 100644
index 000000000000..a01c036c55be
--- /dev/null
+++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * IOMMU operations for pKVM
+ *
+ * Copyright (C) 2022 Linaro Ltd.
+ */
+#include <nvhe/iommu.h>
+
+/* Only one set of ops supported */
+struct kvm_iommu_ops *kvm_iommu_ops;
+
+int kvm_iommu_init(void)
+{
+ if (!kvm_iommu_ops || !kvm_iommu_ops->init)
+ return -ENODEV;
+
+ return kvm_iommu_ops->init();
+}
diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
index ee6435473204..bdbc77395e03 100644
--- a/arch/arm64/kvm/hyp/nvhe/setup.c
+++ b/arch/arm64/kvm/hyp/nvhe/setup.c
@@ -13,6 +13,7 @@
#include <nvhe/early_alloc.h>
#include <nvhe/ffa.h>
#include <nvhe/gfp.h>
+#include <nvhe/iommu.h>
#include <nvhe/memory.h>
#include <nvhe/mem_protect.h>
#include <nvhe/mm.h>
@@ -320,6 +321,10 @@ void __noreturn __pkvm_init_finalise(void)
if (ret)
goto out;
+ ret = kvm_iommu_init();
+ if (ret)
+ goto out;
+
ret = hyp_ffa_init(ffa_proxy_pages);
if (ret)
goto out;
diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c
new file mode 100644
index 000000000000..39465101074a
--- /dev/null
+++ b/arch/arm64/kvm/iommu.c
@@ -0,0 +1,49 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2023 Google LLC
+ * Author: Mostafa Saleh <smostafa@google.com>
+ */
+
+#include <asm/kvm_mmu.h>
+#include <linux/kvm_host.h>
+
+struct kvm_iommu_driver *iommu_driver;
+extern struct kvm_iommu_ops *kvm_nvhe_sym(kvm_iommu_ops);
+
+int kvm_iommu_register_driver(struct kvm_iommu_driver *kern_ops, struct kvm_iommu_ops *el2_ops)
+{
+ int ret;
+
+ if (WARN_ON(!kern_ops || !el2_ops))
+ return -EINVAL;
+
+ /*
+ * Paired with smp_load_acquire(&iommu_driver)
+ * Ensure memory stores happening during a driver
+ * init are observed before executing kvm iommu callbacks.
+ */
+ ret = cmpxchg_release(&iommu_driver, NULL, kern_ops) ? -EBUSY : 0;
+ if (ret)
+ return ret;
+
+ kvm_nvhe_sym(kvm_iommu_ops) = el2_ops;
+ return 0;
+}
+
+int kvm_iommu_init_driver(void)
+{
+ /* See kvm_iommu_register_driver() */
+ if (WARN_ON(!smp_load_acquire(&iommu_driver))) {
+ kvm_err("pKVM enabled without an IOMMU driver, do not run confidential workload in virtual machines\n");
+ return -ENODEV;
+ }
+
+ return iommu_driver->init_driver();
+}
+
+void kvm_iommu_remove_driver(void)
+{
+ /* See kvm_iommu_register_driver() */
+ if (smp_load_acquire(&iommu_driver))
+ iommu_driver->remove_driver();
+}
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 15/29] KVM: arm64: iommu: Shadow host stage-2 page table
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (13 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 14/29] KVM: arm64: iommu: Introduce IOMMU driver infrastructure Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 16/29] KVM: arm64: iommu: Add a memory pool Mostafa Saleh
` (13 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Create a shadow page table for the IOMMU that shadows the
host CPU stage-2 into the IOMMUs to establish DMA isolation.
An initial snapshot is created after the driver init, then
on every permission change a callback would be called for
the IOMMU driver to update the page table.
For some cases, an SMMUv3 may be able to share the same page
table used with the host CPU stage-2 directly.
However, this is too strict and requires changes to the core hypervisor
page table code, plus it would require the hypervisor to handle IOMMU
page faults. This can be added later as an optimization for SMMUV3.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
arch/arm64/kvm/hyp/include/nvhe/iommu.h | 4 ++
arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 83 ++++++++++++++++++++++++-
arch/arm64/kvm/hyp/nvhe/mem_protect.c | 5 ++
3 files changed, 90 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/include/nvhe/iommu.h
index 1ac70cc28a9e..219363045b1c 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h
@@ -3,11 +3,15 @@
#define __ARM64_KVM_NVHE_IOMMU_H__
#include <asm/kvm_host.h>
+#include <asm/kvm_pgtable.h>
struct kvm_iommu_ops {
int (*init)(void);
+ void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot);
};
int kvm_iommu_init(void);
+void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end,
+ enum kvm_pgtable_prot prot);
#endif /* __ARM64_KVM_NVHE_IOMMU_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
index a01c036c55be..f7d1c8feb358 100644
--- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
+++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
@@ -4,15 +4,94 @@
*
* Copyright (C) 2022 Linaro Ltd.
*/
+#include <linux/iommu.h>
+
#include <nvhe/iommu.h>
+#include <nvhe/mem_protect.h>
+#include <nvhe/spinlock.h>
/* Only one set of ops supported */
struct kvm_iommu_ops *kvm_iommu_ops;
+/* Protected by host_mmu.lock */
+static bool kvm_idmap_initialized;
+
+static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot)
+{
+ int iommu_prot = 0;
+
+ if (prot & KVM_PGTABLE_PROT_R)
+ iommu_prot |= IOMMU_READ;
+ if (prot & KVM_PGTABLE_PROT_W)
+ iommu_prot |= IOMMU_WRITE;
+ if (prot == PKVM_HOST_MMIO_PROT)
+ iommu_prot |= IOMMU_MMIO;
+
+ /* We don't understand that, might be dangerous. */
+ WARN_ON(prot & ~PKVM_HOST_MEM_PROT);
+ return iommu_prot;
+}
+
+static int __snapshot_host_stage2(const struct kvm_pgtable_visit_ctx *ctx,
+ enum kvm_pgtable_walk_flags visit)
+{
+ u64 start = ctx->addr;
+ kvm_pte_t pte = *ctx->ptep;
+ u32 level = ctx->level;
+ u64 end = start + kvm_granule_size(level);
+ int prot = IOMMU_READ | IOMMU_WRITE;
+
+ /* Keep unmapped. */
+ if (pte && !kvm_pte_valid(pte))
+ return 0;
+
+ if (kvm_pte_valid(pte))
+ prot = pkvm_to_iommu_prot(kvm_pgtable_stage2_pte_prot(pte));
+ else if (!addr_is_memory(start))
+ prot |= IOMMU_MMIO;
+
+ kvm_iommu_ops->host_stage2_idmap(start, end, prot);
+ return 0;
+}
+
+static int kvm_iommu_snapshot_host_stage2(void)
+{
+ int ret;
+ struct kvm_pgtable_walker walker = {
+ .cb = __snapshot_host_stage2,
+ .flags = KVM_PGTABLE_WALK_LEAF,
+ };
+ struct kvm_pgtable *pgt = &host_mmu.pgt;
+
+ hyp_spin_lock(&host_mmu.lock);
+ ret = kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker);
+ /* Start receiving calls to host_stage2_idmap. */
+ kvm_idmap_initialized = !!ret;
+ hyp_spin_unlock(&host_mmu.lock);
+
+ return ret;
+}
+
int kvm_iommu_init(void)
{
- if (!kvm_iommu_ops || !kvm_iommu_ops->init)
+ int ret;
+
+ if (!kvm_iommu_ops || !kvm_iommu_ops->init ||
+ !kvm_iommu_ops->host_stage2_idmap)
return -ENODEV;
- return kvm_iommu_ops->init();
+ ret = kvm_iommu_ops->init();
+ if (ret)
+ return ret;
+ return kvm_iommu_snapshot_host_stage2();
+}
+
+void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end,
+ enum kvm_pgtable_prot prot)
+{
+ hyp_assert_lock_held(&host_mmu.lock);
+
+ if (!kvm_idmap_initialized)
+ return;
+ kvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(prot));
}
diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
index c9a15ef6b18d..bce6690f29c0 100644
--- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c
+++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c
@@ -15,6 +15,7 @@
#include <hyp/fault.h>
#include <nvhe/gfp.h>
+#include <nvhe/iommu.h>
#include <nvhe/memory.h>
#include <nvhe/mem_protect.h>
#include <nvhe/mm.h>
@@ -529,6 +530,7 @@ static void __host_update_page_state(phys_addr_t addr, u64 size, enum pkvm_page_
int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id)
{
int ret;
+ enum kvm_pgtable_prot prot;
if (!range_is_memory(addr, addr + size))
return -EPERM;
@@ -538,6 +540,9 @@ int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id)
if (ret)
return ret;
+ prot = owner_id == PKVM_ID_HOST ? PKVM_HOST_MEM_PROT : 0;
+ kvm_iommu_host_stage2_idmap(addr, addr + size, prot);
+
/* Don't forget to update the vmemmap tracking for the host */
if (owner_id == PKVM_ID_HOST)
__host_update_page_state(addr, size, PKVM_PAGE_OWNED);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 16/29] KVM: arm64: iommu: Add a memory pool
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (14 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 15/29] KVM: arm64: iommu: Shadow host stage-2 page table Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 17/29] KVM: arm64: iommu: Add enable/disable hypercalls Mostafa Saleh
` (12 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
IOMMU drivers would require to allocate memory for the shadow page
table. Similar to the host stage-2 CPU page table, the IOMMU pool
is allocated early from the carveout and it's memory is added in
a pool which the IOMMU driver can allocate from and reclaim at
run time.
At this point the nr_pages is 0 as there are no driver, in the next
patches when the SMMUv3 driver is added, it will add it's own function
to return the number of pages needed in kvm/iommu.c.
Unfortunately, this part has 2 leak into kvm/iommu as this happens too
early before drivers can have any init calls.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
arch/arm64/include/asm/kvm_host.h | 2 +-
arch/arm64/kvm/hyp/include/nvhe/iommu.h | 5 ++++-
arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 20 +++++++++++++++++++-
arch/arm64/kvm/hyp/nvhe/setup.c | 10 +++++++++-
arch/arm64/kvm/iommu.c | 11 +++++++++++
arch/arm64/kvm/pkvm.c | 1 +
6 files changed, 45 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 43f5a64bbd1d..b42e32ff2a84 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1683,5 +1683,5 @@ struct kvm_iommu_ops;
int kvm_iommu_register_driver(struct kvm_iommu_driver *kern_ops, struct kvm_iommu_ops *hyp_ops);
int kvm_iommu_init_driver(void);
void kvm_iommu_remove_driver(void);
-
+size_t kvm_iommu_pages(void);
#endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/include/nvhe/iommu.h
index 219363045b1c..9f4906c6dcc9 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h
@@ -10,8 +10,11 @@ struct kvm_iommu_ops {
void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot);
};
-int kvm_iommu_init(void);
+int kvm_iommu_init(void *pool_base, size_t nr_pages);
void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end,
enum kvm_pgtable_prot prot);
+void *kvm_iommu_donate_pages(u8 order);
+void kvm_iommu_reclaim_pages(void *ptr);
+
#endif /* __ARM64_KVM_NVHE_IOMMU_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
index f7d1c8feb358..1673165c7330 100644
--- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
+++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
@@ -15,6 +15,7 @@ struct kvm_iommu_ops *kvm_iommu_ops;
/* Protected by host_mmu.lock */
static bool kvm_idmap_initialized;
+static struct hyp_pool iommu_pages_pool;
static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot)
{
@@ -72,7 +73,7 @@ static int kvm_iommu_snapshot_host_stage2(void)
return ret;
}
-int kvm_iommu_init(void)
+int kvm_iommu_init(void *pool_base, size_t nr_pages)
{
int ret;
@@ -80,6 +81,13 @@ int kvm_iommu_init(void)
!kvm_iommu_ops->host_stage2_idmap)
return -ENODEV;
+ if (nr_pages) {
+ ret = hyp_pool_init(&iommu_pages_pool, hyp_virt_to_pfn(pool_base),
+ nr_pages, 0);
+ if (ret)
+ return ret;
+ }
+
ret = kvm_iommu_ops->init();
if (ret)
return ret;
@@ -95,3 +103,13 @@ void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end,
return;
kvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(prot));
}
+
+void *kvm_iommu_donate_pages(u8 order)
+{
+ return hyp_alloc_pages(&iommu_pages_pool, order);
+}
+
+void kvm_iommu_reclaim_pages(void *ptr)
+{
+ hyp_put_page(&iommu_pages_pool, ptr);
+}
diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c
index bdbc77395e03..09ecee2cd864 100644
--- a/arch/arm64/kvm/hyp/nvhe/setup.c
+++ b/arch/arm64/kvm/hyp/nvhe/setup.c
@@ -21,6 +21,7 @@
#include <nvhe/trap_handler.h>
unsigned long hyp_nr_cpus;
+size_t hyp_kvm_iommu_pages;
#define hyp_percpu_size ((unsigned long)__per_cpu_end - \
(unsigned long)__per_cpu_start)
@@ -33,6 +34,7 @@ static void *selftest_base;
static void *ffa_proxy_pages;
static struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops;
static struct hyp_pool hpool;
+static void *iommu_base;
static int divide_memory_pool(void *virt, unsigned long size)
{
@@ -70,6 +72,12 @@ static int divide_memory_pool(void *virt, unsigned long size)
if (!ffa_proxy_pages)
return -ENOMEM;
+ if (hyp_kvm_iommu_pages) {
+ iommu_base = hyp_early_alloc_contig(hyp_kvm_iommu_pages);
+ if (!iommu_base)
+ return -ENOMEM;
+ }
+
return 0;
}
@@ -321,7 +329,7 @@ void __noreturn __pkvm_init_finalise(void)
if (ret)
goto out;
- ret = kvm_iommu_init();
+ ret = kvm_iommu_init(iommu_base, hyp_kvm_iommu_pages);
if (ret)
goto out;
diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c
index 39465101074a..0e52cc0e18de 100644
--- a/arch/arm64/kvm/iommu.c
+++ b/arch/arm64/kvm/iommu.c
@@ -9,6 +9,7 @@
struct kvm_iommu_driver *iommu_driver;
extern struct kvm_iommu_ops *kvm_nvhe_sym(kvm_iommu_ops);
+extern size_t kvm_nvhe_sym(hyp_kvm_iommu_pages);
int kvm_iommu_register_driver(struct kvm_iommu_driver *kern_ops, struct kvm_iommu_ops *el2_ops)
{
@@ -47,3 +48,13 @@ void kvm_iommu_remove_driver(void)
if (smp_load_acquire(&iommu_driver))
iommu_driver->remove_driver();
}
+
+size_t kvm_iommu_pages(void)
+{
+ /*
+ * This is called very early during setup_arch() where no initcalls,
+ * so this has to call specific functions per each KVM driver.
+ */
+ kvm_nvhe_sym(hyp_kvm_iommu_pages) = 0;
+ return 0;
+}
diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c
index fcd70bfe44fb..6098beda36fa 100644
--- a/arch/arm64/kvm/pkvm.c
+++ b/arch/arm64/kvm/pkvm.c
@@ -63,6 +63,7 @@ void __init kvm_hyp_reserve(void)
hyp_mem_pages += hyp_vmemmap_pages(STRUCT_HYP_PAGE_SIZE);
hyp_mem_pages += pkvm_selftest_pages();
hyp_mem_pages += hyp_ffa_proxy_pages();
+ hyp_mem_pages += kvm_iommu_pages();
/*
* Try to allocate a PMD-aligned region to reduce TLB pressure once
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 17/29] KVM: arm64: iommu: Add enable/disable hypercalls
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (15 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 16/29] KVM: arm64: iommu: Add a memory pool Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 18/29] iommu/arm-smmu-v3-kvm: Add SMMUv3 driver Mostafa Saleh
` (11 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Add hypercalls to enable/disable devices, this is needed as
IOMMUs as SMMUv3 can have massive ID space, and it would be
inefficient to enable translation for all of them, as most of them
are not used.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 2 ++
arch/arm64/kvm/hyp/include/nvhe/iommu.h | 5 ++++-
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 19 +++++++++++++++++++
arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 14 ++++++++++++++
4 files changed, 39 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index bec227f9500a..51cedc405e77 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -87,6 +87,8 @@ enum __kvm_host_smccc_func {
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load,
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
+ __KVM_HOST_SMCCC_FUNC___pkvm_iommu_enable_dev,
+ __KVM_HOST_SMCCC_FUNC___pkvm_iommu_disable_dev,
};
#define DECLARE_KVM_VHE_SYM(sym) extern char sym[]
diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/include/nvhe/iommu.h
index 9f4906c6dcc9..82e00cb37f82 100644
--- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h
+++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h
@@ -8,6 +8,8 @@
struct kvm_iommu_ops {
int (*init)(void);
void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot);
+ int (*enable_dev)(pkvm_handle_t iommu, pkvm_handle_t dev);
+ int (*disable_dev)(pkvm_handle_t iommu, pkvm_handle_t dev);
};
int kvm_iommu_init(void *pool_base, size_t nr_pages);
@@ -16,5 +18,6 @@ void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end,
enum kvm_pgtable_prot prot);
void *kvm_iommu_donate_pages(u8 order);
void kvm_iommu_reclaim_pages(void *ptr);
-
+int kvm_iommu_enable_dev(pkvm_handle_t iommu, pkvm_handle_t dev);
+int kvm_iommu_disable_dev(pkvm_handle_t iommu, pkvm_handle_t dev);
#endif /* __ARM64_KVM_NVHE_IOMMU_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 3206b2c07f82..8739895ee22b 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -15,6 +15,7 @@
#include <asm/kvm_mmu.h>
#include <nvhe/ffa.h>
+#include <nvhe/iommu.h>
#include <nvhe/mem_protect.h>
#include <nvhe/mm.h>
#include <nvhe/pkvm.h>
@@ -573,6 +574,22 @@ static void handle___pkvm_teardown_vm(struct kvm_cpu_context *host_ctxt)
cpu_reg(host_ctxt, 1) = __pkvm_teardown_vm(handle);
}
+static void handle___pkvm_iommu_enable_dev(struct kvm_cpu_context *host_ctxt)
+{
+ DECLARE_REG(pkvm_handle_t, iommu, host_ctxt, 1);
+ DECLARE_REG(pkvm_handle_t, dev, host_ctxt, 2);
+
+ cpu_reg(host_ctxt, 1) = kvm_iommu_enable_dev(iommu, dev);
+}
+
+static void handle___pkvm_iommu_disable_dev(struct kvm_cpu_context *host_ctxt)
+{
+ DECLARE_REG(pkvm_handle_t, iommu, host_ctxt, 1);
+ DECLARE_REG(pkvm_handle_t, dev, host_ctxt, 2);
+
+ cpu_reg(host_ctxt, 1) = kvm_iommu_disable_dev(iommu, dev);
+}
+
typedef void (*hcall_t)(struct kvm_cpu_context *);
#define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = (hcall_t)handle_##x
@@ -612,6 +629,8 @@ static const hcall_t host_hcall[] = {
HANDLE_FUNC(__pkvm_vcpu_load),
HANDLE_FUNC(__pkvm_vcpu_put),
HANDLE_FUNC(__pkvm_tlb_flush_vmid),
+ HANDLE_FUNC(__pkvm_iommu_enable_dev),
+ HANDLE_FUNC(__pkvm_iommu_disable_dev),
};
static void handle_host_hcall(struct kvm_cpu_context *host_ctxt)
diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
index 1673165c7330..c1ba7e042a7a 100644
--- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
+++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c
@@ -113,3 +113,17 @@ void kvm_iommu_reclaim_pages(void *ptr)
{
hyp_put_page(&iommu_pages_pool, ptr);
}
+
+int kvm_iommu_enable_dev(pkvm_handle_t iommu, pkvm_handle_t dev)
+{
+ if (kvm_iommu_ops && kvm_iommu_ops->enable_dev)
+ return kvm_iommu_ops->enable_dev(iommu, dev);
+ return -ENODEV;
+}
+
+int kvm_iommu_disable_dev(pkvm_handle_t iommu, pkvm_handle_t dev)
+{
+ if (kvm_iommu_ops && kvm_iommu_ops->disable_dev)
+ return kvm_iommu_ops->disable_dev(iommu, dev);
+ return -ENODEV;
+}
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 18/29] iommu/arm-smmu-v3-kvm: Add SMMUv3 driver
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (16 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 17/29] KVM: arm64: iommu: Add enable/disable hypercalls Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 19/29] iommu/arm-smmu-v3-kvm: Initialize registers Mostafa Saleh
` (10 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Add the skeleton for an Arm SMMUv3 driver at EL2.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
arch/arm64/kvm/hyp/nvhe/Makefile | 5 ++++
drivers/iommu/arm/Kconfig | 9 +++++++
.../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 24 +++++++++++++++++++
.../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 16 +++++++++++++
4 files changed, 54 insertions(+)
create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index 393ff143f0be..c71c96262378 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -31,6 +31,11 @@ hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
hyp-obj-$(CONFIG_LIST_HARDENED) += list_debug.o
hyp-obj-y += $(lib-objs)
+HYP_SMMU_V3_DRV_PATH = ../../../../../drivers/iommu/arm/arm-smmu-v3
+
+hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) += $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-smmu-v3.o \
+ $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-hyp.o
+
##
## Build rules for compiling nVHE hyp code
## Output of this folder is `kvm_nvhe.o`, a partially linked object
diff --git a/drivers/iommu/arm/Kconfig b/drivers/iommu/arm/Kconfig
index ef42bbe07dbe..bcf4001521ad 100644
--- a/drivers/iommu/arm/Kconfig
+++ b/drivers/iommu/arm/Kconfig
@@ -142,3 +142,12 @@ config QCOM_IOMMU
select ARM_DMA_USE_IOMMU
help
Support for IOMMU on certain Qualcomm SoCs.
+
+config ARM_SMMU_V3_PKVM
+ bool "ARM SMMUv3 support for protected Virtual Machines"
+ depends on KVM && ARM64
+ help
+ Enable a SMMUv3 driver in the KVM hypervisor, to protect VMs against
+ memory accesses from devices owned by the host.
+
+ Say Y here if you intend to enable KVM in protected mode.
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
new file mode 100644
index 000000000000..2f43804e08e0
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pKVM hyp driver for the Arm SMMUv3
+ *
+ * Copyright (C) 2022 Linaro Ltd.
+ */
+#include <asm/kvm_hyp.h>
+
+#include <nvhe/iommu.h>
+
+#include "arm_smmu_v3.h"
+
+size_t __ro_after_init kvm_hyp_arm_smmu_v3_count;
+struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus;
+
+static int smmu_init(void)
+{
+ return -ENOSYS;
+}
+
+/* Shared with the kernel driver in EL1 */
+struct kvm_iommu_ops smmu_ops = {
+ .init = smmu_init,
+};
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
new file mode 100644
index 000000000000..f6ad91d3fb85
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_ARM_SMMU_V3_H
+#define __KVM_ARM_SMMU_V3_H
+
+#include <asm/kvm_asm.h>
+
+struct hyp_arm_smmu_v3_device {
+};
+
+extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count);
+#define kvm_hyp_arm_smmu_v3_count kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count)
+
+extern struct hyp_arm_smmu_v3_device *kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smmus);
+#define kvm_hyp_arm_smmu_v3_smmus kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smmus)
+
+#endif /* __KVM_ARM_SMMU_V3_H */
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 19/29] iommu/arm-smmu-v3-kvm: Initialize registers
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (17 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 18/29] iommu/arm-smmu-v3-kvm: Add SMMUv3 driver Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 20/29] iommu/arm-smmu-v3-kvm: Setup command queue Mostafa Saleh
` (9 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Ensure all writable registers are properly initialized. We do not touch
registers that will not be read by the SMMU due to disabled features.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 149 +++++++++++++++++-
.../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 12 ++
2 files changed, 160 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
index 2f43804e08e0..e9bc35e019b6 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
@@ -7,15 +7,162 @@
#include <asm/kvm_hyp.h>
#include <nvhe/iommu.h>
+#include <nvhe/mem_protect.h>
#include "arm_smmu_v3.h"
+#define ARM_SMMU_POLL_TIMEOUT_US 100000 /* 100ms arbitrary timeout */
+
size_t __ro_after_init kvm_hyp_arm_smmu_v3_count;
struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus;
+#define for_each_smmu(smmu) \
+ for ((smmu) = kvm_hyp_arm_smmu_v3_smmus; \
+ (smmu) != &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \
+ (smmu)++)
+
+/*
+ * Wait until @cond is true.
+ * Return 0 on success, or -ETIMEDOUT
+ */
+#define smmu_wait(_cond) \
+({ \
+ int __ret = 0; \
+ u64 delay = pkvm_time_get() + ARM_SMMU_POLL_TIMEOUT_US; \
+ \
+ while (!(_cond)) { \
+ if (pkvm_time_get() >= delay) { \
+ __ret = -ETIMEDOUT; \
+ break; \
+ } \
+ } \
+ __ret; \
+})
+
+static int smmu_write_cr0(struct hyp_arm_smmu_v3_device *smmu, u32 val)
+{
+ writel_relaxed(val, smmu->base + ARM_SMMU_CR0);
+ return smmu_wait(readl_relaxed(smmu->base + ARM_SMMU_CR0ACK) == val);
+}
+
+/* Transfer ownership of structures from host to hyp */
+static int smmu_take_pages(u64 phys, size_t size)
+{
+ WARN_ON(!PAGE_ALIGNED(phys) || !PAGE_ALIGNED(size));
+ return __pkvm_host_donate_hyp(phys >> PAGE_SHIFT, size >> PAGE_SHIFT);
+}
+
+static void smmu_reclaim_pages(u64 phys, size_t size)
+{
+ WARN_ON(!PAGE_ALIGNED(phys) || !PAGE_ALIGNED(size));
+ WARN_ON(__pkvm_hyp_donate_host(phys >> PAGE_SHIFT, size >> PAGE_SHIFT));
+}
+
+static int smmu_init_registers(struct hyp_arm_smmu_v3_device *smmu)
+{
+ u64 val, old;
+ int ret;
+
+ if (!(readl_relaxed(smmu->base + ARM_SMMU_GBPA) & GBPA_ABORT))
+ return -EINVAL;
+
+ /* Initialize all RW registers that will be read by the SMMU */
+ ret = smmu_write_cr0(smmu, 0);
+ if (ret)
+ return ret;
+
+ val = FIELD_PREP(CR1_TABLE_SH, ARM_SMMU_SH_ISH) |
+ FIELD_PREP(CR1_TABLE_OC, CR1_CACHE_WB) |
+ FIELD_PREP(CR1_TABLE_IC, CR1_CACHE_WB) |
+ FIELD_PREP(CR1_QUEUE_SH, ARM_SMMU_SH_ISH) |
+ FIELD_PREP(CR1_QUEUE_OC, CR1_CACHE_WB) |
+ FIELD_PREP(CR1_QUEUE_IC, CR1_CACHE_WB);
+ writel_relaxed(val, smmu->base + ARM_SMMU_CR1);
+ writel_relaxed(CR2_PTM, smmu->base + ARM_SMMU_CR2);
+ writel_relaxed(0, smmu->base + ARM_SMMU_IRQ_CTRL);
+
+ val = readl_relaxed(smmu->base + ARM_SMMU_GERROR);
+ old = readl_relaxed(smmu->base + ARM_SMMU_GERRORN);
+ /* Service Failure Mode is fatal */
+ if ((val ^ old) & GERROR_SFM_ERR)
+ return -EIO;
+ /* Clear pending errors */
+ writel_relaxed(val, smmu->base + ARM_SMMU_GERRORN);
+
+ return 0;
+}
+
+/* Put the device in a state that can be probed by the host driver. */
+static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu)
+{
+ int i;
+ size_t nr_pages = smmu->mmio_size >> PAGE_SHIFT;
+
+ for (i = 0 ; i < nr_pages ; ++i) {
+ u64 pfn = (smmu->mmio_addr >> PAGE_SHIFT) + i;
+
+ WARN_ON(__pkvm_hyp_donate_host_mmio(pfn));
+ }
+}
+
+static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
+{
+ int i;
+ size_t nr_pages;
+ int ret;
+
+ if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size))
+ return -EINVAL;
+
+ nr_pages = smmu->mmio_size >> PAGE_SHIFT;
+ for (i = 0 ; i < nr_pages ; ++i) {
+ u64 pfn = (smmu->mmio_addr >> PAGE_SHIFT) + i;
+
+ /*
+ * This should never happen, so it's fine to be strict to avoid
+ * complicated error handling.
+ */
+ WARN_ON(__pkvm_host_donate_hyp_mmio(pfn));
+ }
+ smmu->base = hyp_phys_to_virt(smmu->mmio_addr);
+
+ ret = smmu_init_registers(smmu);
+ if (ret)
+ goto out_err;
+ return ret;
+
+out_err:
+ smmu_deinit_device(smmu);
+ return ret;
+}
+
static int smmu_init(void)
{
- return -ENOSYS;
+ int ret;
+ struct hyp_arm_smmu_v3_device *smmu;
+ size_t smmu_arr_size = PAGE_ALIGN(sizeof(*kvm_hyp_arm_smmu_v3_smmus) *
+ kvm_hyp_arm_smmu_v3_count);
+ phys_addr_t smmu_arr_phys;
+
+ kvm_hyp_arm_smmu_v3_smmus = kern_hyp_va(kvm_hyp_arm_smmu_v3_smmus);
+ smmu_arr_phys = hyp_virt_to_phys(kvm_hyp_arm_smmu_v3_smmus);
+
+ ret = smmu_take_pages(smmu_arr_phys, smmu_arr_size);
+ if (ret)
+ return ret;
+
+ for_each_smmu(smmu) {
+ ret = smmu_init_device(smmu);
+ if (ret)
+ goto out_reclaim_smmu;
+ }
+
+ return 0;
+out_reclaim_smmu:
+ while (smmu != kvm_hyp_arm_smmu_v3_smmus)
+ smmu_deinit_device(--smmu);
+ smmu_reclaim_pages(smmu_arr_phys, smmu_arr_size);
+ return ret;
}
/* Shared with the kernel driver in EL1 */
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
index f6ad91d3fb85..9b1d021ada63 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
@@ -4,7 +4,19 @@
#include <asm/kvm_asm.h>
+#include "../arm-smmu-v3-common.h"
+
+/*
+ * Parameters from the trusted host:
+ * @mmio_addr base address of the SMMU registers
+ * @mmio_size size of the registers resource
+ * @base Virtual address of SMMU registers
+ * Other members are filled and used at runtime by the SMMU driver.
+ */
struct hyp_arm_smmu_v3_device {
+ phys_addr_t mmio_addr;
+ size_t mmio_size;
+ void __iomem *base;
};
extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 20/29] iommu/arm-smmu-v3-kvm: Setup command queue
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (18 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 19/29] iommu/arm-smmu-v3-kvm: Initialize registers Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-29 6:44 ` Krzysztof Kozlowski
2025-07-28 17:53 ` [PATCH v3 21/29] iommu/arm-smmu-v3-kvm: Setup stream table Mostafa Saleh
` (8 subsequent siblings)
28 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Map the command queue allocated by the host into the hypervisor address
space. When the host mappings are finalized, the queue is unmapped from
the host.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
.../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 104 ++++++++++++++++++
.../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 4 +
2 files changed, 108 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
index e9bc35e019b6..c3d67c603619 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
@@ -39,6 +39,15 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus;
__ret; \
})
+#define smmu_wait_event(_smmu, _cond) \
+({ \
+ if ((_smmu)->features & ARM_SMMU_FEAT_SEV) { \
+ while (!(_cond)) \
+ wfe(); \
+ } \
+ smmu_wait(_cond); \
+})
+
static int smmu_write_cr0(struct hyp_arm_smmu_v3_device *smmu, u32 val)
{
writel_relaxed(val, smmu->base + ARM_SMMU_CR0);
@@ -58,6 +67,70 @@ static void smmu_reclaim_pages(u64 phys, size_t size)
WARN_ON(__pkvm_hyp_donate_host(phys >> PAGE_SHIFT, size >> PAGE_SHIFT));
}
+static bool smmu_cmdq_full(struct arm_smmu_queue *cmdq)
+{
+ struct arm_smmu_ll_queue *llq = &cmdq->llq;
+
+ WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg));
+ return queue_full(llq);
+}
+
+static bool smmu_cmdq_empty(struct arm_smmu_queue *cmdq)
+{
+ struct arm_smmu_ll_queue *llq = &cmdq->llq;
+
+ WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg));
+ return queue_empty(llq);
+}
+
+static int smmu_add_cmd(struct hyp_arm_smmu_v3_device *smmu,
+ struct arm_smmu_cmdq_ent *ent)
+{
+ int ret;
+ u64 cmd[CMDQ_ENT_DWORDS];
+ struct arm_smmu_queue *q = &smmu->cmdq;
+ struct arm_smmu_ll_queue *llq = &q->llq;
+
+ ret = smmu_wait_event(smmu, !smmu_cmdq_full(&smmu->cmdq));
+ if (ret)
+ return ret;
+
+ ret = arm_smmu_cmdq_build_cmd(cmd, ent);
+ if (ret)
+ return ret;
+
+ queue_write(Q_ENT(q, llq->prod), cmd, CMDQ_ENT_DWORDS);
+ llq->prod = queue_inc_prod_n(llq, 1);
+ writel_relaxed(llq->prod, q->prod_reg);
+ return 0;
+}
+
+static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device *smmu)
+{
+ int ret;
+ struct arm_smmu_cmdq_ent cmd = {
+ .opcode = CMDQ_OP_CMD_SYNC,
+ };
+
+ ret = smmu_add_cmd(smmu, &cmd);
+ if (ret)
+ return ret;
+
+ return smmu_wait_event(smmu, smmu_cmdq_empty(&smmu->cmdq));
+}
+
+__maybe_unused
+static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu,
+ struct arm_smmu_cmdq_ent *cmd)
+{
+ int ret = smmu_add_cmd(smmu, cmd);
+
+ if (ret)
+ return ret;
+
+ return smmu_sync_cmd(smmu);
+}
+
static int smmu_init_registers(struct hyp_arm_smmu_v3_device *smmu)
{
u64 val, old;
@@ -105,6 +178,33 @@ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu)
}
}
+static int smmu_init_cmdq(struct hyp_arm_smmu_v3_device *smmu)
+{
+ size_t cmdq_size;
+ int ret;
+ enum kvm_pgtable_prot prot = PAGE_HYP;
+
+ cmdq_size = (1 << (smmu->cmdq.llq.max_n_shift)) *
+ CMDQ_ENT_DWORDS * 8;
+
+ if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY))
+ prot |= KVM_PGTABLE_PROT_NORMAL_NC;
+
+ ret = ___pkvm_host_donate_hyp(smmu->cmdq.base_dma >> PAGE_SHIFT,
+ PAGE_ALIGN(cmdq_size) >> PAGE_SHIFT, prot);
+ if (ret)
+ return ret;
+
+ smmu->cmdq.base = hyp_phys_to_virt(smmu->cmdq.base_dma);
+ smmu->cmdq.prod_reg = smmu->base + ARM_SMMU_CMDQ_PROD;
+ smmu->cmdq.cons_reg = smmu->base + ARM_SMMU_CMDQ_CONS;
+ memset(smmu->cmdq.base, 0, cmdq_size);
+ writel_relaxed(0, smmu->cmdq.prod_reg);
+ writel_relaxed(0, smmu->cmdq.cons_reg);
+
+ return 0;
+}
+
static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
{
int i;
@@ -129,6 +229,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
ret = smmu_init_registers(smmu);
if (ret)
goto out_err;
+ ret = smmu_init_cmdq(smmu);
+ if (ret)
+ goto out_err;
+
return ret;
out_err:
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
index 9b1d021ada63..f639312cf295 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
@@ -11,12 +11,16 @@
* @mmio_addr base address of the SMMU registers
* @mmio_size size of the registers resource
* @base Virtual address of SMMU registers
+ * @features SMMUv3 features as defined in arm-smmu-v3-common.h
+ * @cmdq CMDQ queue struct
* Other members are filled and used at runtime by the SMMU driver.
*/
struct hyp_arm_smmu_v3_device {
phys_addr_t mmio_addr;
size_t mmio_size;
void __iomem *base;
+ unsigned long features;
+ struct arm_smmu_queue cmdq;
};
extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH v3 20/29] iommu/arm-smmu-v3-kvm: Setup command queue
2025-07-28 17:53 ` [PATCH v3 20/29] iommu/arm-smmu-v3-kvm: Setup command queue Mostafa Saleh
@ 2025-07-29 6:44 ` Krzysztof Kozlowski
2025-07-29 9:55 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Krzysztof Kozlowski @ 2025-07-29 6:44 UTC (permalink / raw)
To: Mostafa Saleh, linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan
On 28/07/2025 19:53, Mostafa Saleh wrote:
> Map the command queue allocated by the host into the hypervisor address
> space. When the host mappings are finalized, the queue is unmapped from
> the host.
>
> Signed-off-by: Mostafa Saleh <smostafa@google.com>
> Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Your SoB must be the last one.
Best regards,
Krzysztof
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 20/29] iommu/arm-smmu-v3-kvm: Setup command queue
2025-07-29 6:44 ` Krzysztof Kozlowski
@ 2025-07-29 9:55 ` Mostafa Saleh
0 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-29 9:55 UTC (permalink / raw)
To: Krzysztof Kozlowski
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, jgg, mark.rutland,
praan
On Tue, Jul 29, 2025 at 08:44:03AM +0200, Krzysztof Kozlowski wrote:
> On 28/07/2025 19:53, Mostafa Saleh wrote:
> > Map the command queue allocated by the host into the hypervisor address
> > space. When the host mappings are finalized, the queue is unmapped from
> > the host.
> >
> > Signed-off-by: Mostafa Saleh <smostafa@google.com>
> > Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
>
>
> Your SoB must be the last one.
I see, thanks for pointing it out.
Thanks,
Mostafa
>
>
> Best regards,
> Krzysztof
^ permalink raw reply [flat|nested] 48+ messages in thread
* [PATCH v3 21/29] iommu/arm-smmu-v3-kvm: Setup stream table
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (19 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 20/29] iommu/arm-smmu-v3-kvm: Setup command queue Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 22/29] iommu/arm-smmu-v3-kvm: Reset the device Mostafa Saleh
` (7 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Map the stream table allocated by the host into the hypervisor address
space. When the host mappings are finalized, the table is unmapped from
the host. Depending on the host configuration, the stream table may have
one or two levels. Populate the level-2 stream table lazily.
Also, add accessors for STEs.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
.../arm/arm-smmu-v3/arm-smmu-v3-common.h | 16 +++
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 16 ---
.../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 102 +++++++++++++++++-
.../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 2 +
4 files changed, 119 insertions(+), 17 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
index ab6db8b4b9f2..a24fdbdaeb74 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h
@@ -499,6 +499,22 @@ struct arm_smmu_queue {
Q_IDX(&((q)->llq), p) * \
(q)->ent_dwords)
+struct arm_smmu_strtab_cfg {
+ union {
+ struct {
+ struct arm_smmu_ste *table;
+ dma_addr_t ste_dma;
+ unsigned int num_ents;
+ } linear;
+ struct {
+ struct arm_smmu_strtab_l1 *l1tab;
+ struct arm_smmu_strtab_l2 **l2ptrs;
+ dma_addr_t l1_dma;
+ unsigned int num_l1_ents;
+ } l2;
+ };
+};
+
static inline void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst,
dma_addr_t l2ptr_dma)
{
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
index f9a496e0bd94..480463ec118a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
@@ -232,22 +232,6 @@ struct arm_smmu_s2_cfg {
u16 vmid;
};
-struct arm_smmu_strtab_cfg {
- union {
- struct {
- struct arm_smmu_ste *table;
- dma_addr_t ste_dma;
- unsigned int num_ents;
- } linear;
- struct {
- struct arm_smmu_strtab_l1 *l1tab;
- struct arm_smmu_strtab_l2 **l2ptrs;
- dma_addr_t l1_dma;
- unsigned int num_l1_ents;
- } l2;
- };
-};
-
struct arm_smmu_impl_ops {
int (*device_reset)(struct arm_smmu_device *smmu);
void (*device_remove)(struct arm_smmu_device *smmu);
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
index c3d67c603619..74e6dfb53528 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
@@ -119,7 +119,6 @@ static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device *smmu)
return smmu_wait_event(smmu, smmu_cmdq_empty(&smmu->cmdq));
}
-__maybe_unused
static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu,
struct arm_smmu_cmdq_ent *cmd)
{
@@ -131,6 +130,80 @@ static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu,
return smmu_sync_cmd(smmu);
}
+__maybe_unused
+static int smmu_sync_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid, unsigned long ste)
+{
+ struct arm_smmu_cmdq_ent cmd = {
+ .opcode = CMDQ_OP_CFGI_STE,
+ .cfgi.sid = sid,
+ .cfgi.leaf = true,
+ };
+
+ /*
+ * In case of 2 level STEs, L2 is allocated as cacheable, so flush it everytime
+ * we update the STE.
+ */
+ if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY) &&
+ (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB))
+ kvm_flush_dcache_to_poc(ste, sizeof(struct arm_smmu_ste));
+ return smmu_send_cmd(smmu, &cmd);
+}
+
+static int smmu_alloc_l2_strtab(struct hyp_arm_smmu_v3_device *smmu, u32 sid)
+{
+ struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+ struct arm_smmu_strtab_l1 *l1_desc;
+ struct arm_smmu_strtab_l2 *l2table;
+
+ l1_desc = &cfg->l2.l1tab[arm_smmu_strtab_l1_idx(sid)];
+ if (l1_desc->l2ptr)
+ return 0;
+
+ l2table = kvm_iommu_donate_pages(get_order(sizeof(*l2table)));
+ if (!l2table)
+ return -ENOMEM;
+
+ arm_smmu_write_strtab_l1_desc(l1_desc, hyp_virt_to_phys(l2table));
+ if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY))
+ kvm_flush_dcache_to_poc(l1_desc, sizeof(*l1_desc));
+ return 0;
+}
+
+static struct arm_smmu_ste *
+smmu_get_ste_ptr(struct hyp_arm_smmu_v3_device *smmu, u32 sid)
+{
+ struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+
+ if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) {
+ u32 l1_idx = arm_smmu_strtab_l1_idx(sid);
+ struct arm_smmu_strtab_l2 *l2ptr;
+
+ if (l1_idx >= cfg->l2.num_l1_ents)
+ return NULL;
+ l2ptr = hyp_phys_to_virt(cfg->l2.l1tab[l1_idx].l2ptr & STRTAB_L1_DESC_L2PTR_MASK);
+ /* Two-level walk */
+ return &l2ptr->stes[arm_smmu_strtab_l2_idx(sid)];
+ }
+
+ if (sid >= cfg->linear.num_ents)
+ return NULL;
+ /* Simple linear lookup */
+ return &cfg->linear.table[sid];
+}
+
+__maybe_unused
+static struct arm_smmu_ste *
+smmu_get_alloc_ste_ptr(struct hyp_arm_smmu_v3_device *smmu, u32 sid)
+{
+ if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) {
+ int ret = smmu_alloc_l2_strtab(smmu, sid);
+
+ if (ret)
+ return NULL;
+ }
+ return smmu_get_ste_ptr(smmu, sid);
+}
+
static int smmu_init_registers(struct hyp_arm_smmu_v3_device *smmu)
{
u64 val, old;
@@ -205,6 +278,29 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_device *smmu)
return 0;
}
+static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu)
+{
+ size_t strtab_size;
+ u64 strtab_base;
+ struct arm_smmu_strtab_cfg *cfg = &smmu->strtab_cfg;
+ enum kvm_pgtable_prot prot = PAGE_HYP;
+
+ if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY))
+ prot |= KVM_PGTABLE_PROT_NORMAL_NC;
+
+ if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) {
+ strtab_size = PAGE_ALIGN(cfg->l2.num_l1_ents * sizeof(struct arm_smmu_strtab_l1));
+ strtab_base = (u64)cfg->l2.l1_dma;
+ cfg->linear.table = hyp_phys_to_virt(strtab_base);
+ } else {
+ strtab_size = PAGE_ALIGN(cfg->linear.num_ents * sizeof(struct arm_smmu_ste));
+ strtab_base = (u64)cfg->linear.ste_dma;
+ cfg->l2.l1tab = hyp_phys_to_virt(strtab_base);
+ }
+ return ___pkvm_host_donate_hyp(hyp_phys_to_pfn(strtab_base),
+ strtab_size >> PAGE_SHIFT, prot);
+}
+
static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
{
int i;
@@ -233,6 +329,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
if (ret)
goto out_err;
+ ret = smmu_init_strtab(smmu);
+ if (ret)
+ goto out_err;
+
return ret;
out_err:
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
index f639312cf295..d188537545b1 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
@@ -13,6 +13,7 @@
* @base Virtual address of SMMU registers
* @features SMMUv3 features as defined in arm-smmu-v3-common.h
* @cmdq CMDQ queue struct
+ * @strtab_cfg stream table config, strtab_cfg.l2.l2ptrs is not used
* Other members are filled and used at runtime by the SMMU driver.
*/
struct hyp_arm_smmu_v3_device {
@@ -21,6 +22,7 @@ struct hyp_arm_smmu_v3_device {
void __iomem *base;
unsigned long features;
struct arm_smmu_queue cmdq;
+ struct arm_smmu_strtab_cfg strtab_cfg;
};
extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 22/29] iommu/arm-smmu-v3-kvm: Reset the device
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (20 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 21/29] iommu/arm-smmu-v3-kvm: Setup stream table Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 23/29] iommu/arm-smmu-v3-kvm: Support io-pgtable Mostafa Saleh
` (6 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Now that all structures are initialized, send global invalidations and
reset the SMMUv3 device.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 37 +++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
index 74e6dfb53528..5e988ffede92 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
@@ -301,6 +301,40 @@ static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu)
strtab_size >> PAGE_SHIFT, prot);
}
+static int smmu_reset_device(struct hyp_arm_smmu_v3_device *smmu)
+{
+ int ret;
+ struct arm_smmu_cmdq_ent cfgi_cmd = {
+ .opcode = CMDQ_OP_CFGI_ALL,
+ };
+ struct arm_smmu_cmdq_ent tlbi_cmd = {
+ .opcode = CMDQ_OP_TLBI_NSNH_ALL,
+ };
+
+ /* Invalidate all cached configs and TLBs */
+ ret = smmu_write_cr0(smmu, CR0_CMDQEN);
+ if (ret)
+ return ret;
+
+ ret = smmu_add_cmd(smmu, &cfgi_cmd);
+ if (ret)
+ goto err_disable_cmdq;
+
+ ret = smmu_add_cmd(smmu, &tlbi_cmd);
+ if (ret)
+ goto err_disable_cmdq;
+
+ ret = smmu_sync_cmd(smmu);
+ if (ret)
+ goto err_disable_cmdq;
+
+ /* Enable translation */
+ return smmu_write_cr0(smmu, CR0_SMMUEN | CR0_CMDQEN | CR0_ATSCHK);
+
+err_disable_cmdq:
+ return smmu_write_cr0(smmu, 0);
+}
+
static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
{
int i;
@@ -333,6 +367,9 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
if (ret)
goto out_err;
+ ret = smmu_reset_device(smmu);
+ if (ret)
+ goto out_err;
return ret;
out_err:
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 23/29] iommu/arm-smmu-v3-kvm: Support io-pgtable
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (21 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 22/29] iommu/arm-smmu-v3-kvm: Reset the device Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 24/29] iommu/arm-smmu-v3-kvm: Shadow the CPU stage-2 page table Mostafa Saleh
` (5 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Implement the hypervisor version of io-pgtable allocation functions,
mirroring drivers/iommu/io-pgtable-arm.c. Page allocation uses the
IOMMU pool filled by the host.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
arch/arm64/kvm/hyp/nvhe/Makefile | 4 +-
.../arm/arm-smmu-v3/pkvm/io-pgtable-arm.c | 115 ++++++++++++++++++
drivers/iommu/io-pgtable-arm.h | 15 ++-
3 files changed, 131 insertions(+), 3 deletions(-)
create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm.c
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index c71c96262378..d641a9987152 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -34,7 +34,9 @@ hyp-obj-y += $(lib-objs)
HYP_SMMU_V3_DRV_PATH = ../../../../../drivers/iommu/arm/arm-smmu-v3
hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) += $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-smmu-v3.o \
- $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-hyp.o
+ $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-hyp.o \
+ $(HYP_SMMU_V3_DRV_PATH)/pkvm/io-pgtable-arm.o \
+ $(HYP_SMMU_V3_DRV_PATH)/../../io-pgtable-arm-common.o
##
## Build rules for compiling nVHE hyp code
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm.c b/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm.c
new file mode 100644
index 000000000000..ce17f21238c8
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm.c
@@ -0,0 +1,115 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2022 Arm Ltd.
+ */
+#include <nvhe/iommu.h>
+
+#include "../../../io-pgtable-arm.h"
+
+void arm_lpae_split_blk(void)
+{
+ WARN_ON(1);
+}
+
+void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp,
+ struct io_pgtable_cfg *cfg, void *cookie)
+{
+ void *addr;
+
+ if (!PAGE_ALIGNED(size))
+ return NULL;
+
+ addr = kvm_iommu_donate_pages(get_order(size));
+
+ if (addr && !cfg->coherent_walk)
+ kvm_flush_dcache_to_poc(addr, size);
+
+ return addr;
+}
+
+void __arm_lpae_free_pages(void *addr, size_t size, struct io_pgtable_cfg *cfg,
+ void *cookie)
+{
+ if (!cfg->coherent_walk)
+ kvm_flush_dcache_to_poc(addr, size);
+
+ kvm_iommu_reclaim_pages(addr);
+}
+
+void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries,
+ struct io_pgtable_cfg *cfg)
+{
+ if (!cfg->coherent_walk)
+ kvm_flush_dcache_to_poc(ptep, sizeof(*ptep) * num_entries);
+}
+
+static int kvm_arm_io_pgtable_init(struct io_pgtable_cfg *cfg,
+ enum io_pgtable_fmt fmt,
+ struct arm_lpae_io_pgtable *data,
+ void *cookie)
+{
+ int ret = -EINVAL;
+
+ if (fmt == ARM_64_LPAE_S2)
+ ret = arm_lpae_init_pgtable_s2(cfg, data, cookie);
+ else if (fmt == ARM_64_LPAE_S1)
+ ret = arm_lpae_init_pgtable_s1(cfg, data, cookie);
+
+ if (ret)
+ return ret;
+
+ data->iop.cfg = *cfg;
+ data->iop.fmt = fmt;
+ return 0;
+}
+
+struct io_pgtable *kvm_arm_io_pgtable_alloc(struct io_pgtable_cfg *cfg,
+ void *cookie,
+ enum io_pgtable_fmt fmt,
+ int *out_ret)
+{
+ size_t pgd_size;
+ struct arm_lpae_io_pgtable *data;
+ int ret;
+
+ data = kvm_iommu_donate_pages(get_order(sizeof(*data)));
+ if (!data) {
+ *out_ret = -ENOMEM;
+ return NULL;
+ }
+
+ data->iop.ops = (struct io_pgtable_ops) {
+ .map_pages = arm_lpae_map_pages,
+ .unmap_pages = arm_lpae_unmap_pages,
+ };
+
+ ret = kvm_arm_io_pgtable_init(cfg, fmt, data, cookie);
+ if (ret) {
+ *out_ret = ret;
+ goto out_free;
+ }
+ pgd_size = PAGE_ALIGN(ARM_LPAE_PGD_SIZE(data));
+ data->pgd = __arm_lpae_alloc_pages(pgd_size, 0, &data->iop.cfg, cookie);
+ if (!data->pgd) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+
+ if (fmt == ARM_64_LPAE_S2)
+ data->iop.cfg.arm_lpae_s2_cfg.vttbr = __arm_lpae_virt_to_phys(data->pgd);
+ else if (fmt == ARM_64_LPAE_S1)
+ data->iop.cfg.arm_lpae_s1_cfg.ttbr = __arm_lpae_virt_to_phys(data->pgd);
+
+ if (!data->iop.cfg.coherent_walk)
+ kvm_flush_dcache_to_poc(data->pgd, pgd_size);
+
+ /* Ensure the empty pgd is visible before any actual TTBR write */
+ wmb();
+
+ *out_ret = 0;
+ return &data->iop;
+out_free:
+ kvm_iommu_reclaim_pages(data);
+ *out_ret = ret;
+ return NULL;
+}
diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h
index 2807cf563f11..c1450eca934f 100644
--- a/drivers/iommu/io-pgtable-arm.h
+++ b/drivers/iommu/io-pgtable-arm.h
@@ -188,8 +188,19 @@ static inline bool iopte_table(arm_lpae_iopte pte, int lvl)
return iopte_type(pte) == ARM_LPAE_PTE_TYPE_TABLE;
}
-#define __arm_lpae_virt_to_phys __pa
-#define __arm_lpae_phys_to_virt __va
+#ifdef __KVM_NVHE_HYPERVISOR__
+#include <nvhe/memory.h>
+#define __arm_lpae_virt_to_phys hyp_virt_to_phys
+#define __arm_lpae_phys_to_virt hyp_phys_to_virt
+
+struct io_pgtable *kvm_arm_io_pgtable_alloc(struct io_pgtable_cfg *cfg,
+ void *cookie,
+ enum io_pgtable_fmt fmt,
+ int *out_ret);
+#else
+#define __arm_lpae_virt_to_phys __pa
+#define __arm_lpae_phys_to_virt __va
+#endif
static inline phys_addr_t iopte_to_paddr(arm_lpae_iopte pte,
struct arm_lpae_io_pgtable *data)
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 24/29] iommu/arm-smmu-v3-kvm: Shadow the CPU stage-2 page table
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (22 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 23/29] iommu/arm-smmu-v3-kvm: Support io-pgtable Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 25/29] iommu/arm-smmu-v3-kvm: Add enable/disable device HVCs Mostafa Saleh
` (4 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Based on the callbacks from the hypervisor, update the SMMUv3
Identity mapped page table.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 161 +++++++++++++++++-
.../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 6 +
2 files changed, 166 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
index 5e988ffede92..38d81cd6d24a 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
@@ -10,6 +10,7 @@
#include <nvhe/mem_protect.h>
#include "arm_smmu_v3.h"
+#include "../../../io-pgtable-arm.h"
#define ARM_SMMU_POLL_TIMEOUT_US 100000 /* 100ms arbitrary timeout */
@@ -48,6 +49,8 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus;
smmu_wait(_cond); \
})
+static struct io_pgtable *idmap_pgtable;
+
static int smmu_write_cr0(struct hyp_arm_smmu_v3_device *smmu, u32 val)
{
writel_relaxed(val, smmu->base + ARM_SMMU_CR0);
@@ -130,6 +133,56 @@ static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu,
return smmu_sync_cmd(smmu);
}
+static void __smmu_add_cmd(struct hyp_arm_smmu_v3_device *smmu, void *unused,
+ struct arm_smmu_cmdq_ent *cmd)
+{
+ WARN_ON(smmu_add_cmd(smmu, cmd));
+}
+
+static int smmu_tlb_inv_range_smmu(struct hyp_arm_smmu_v3_device *smmu,
+ struct arm_smmu_cmdq_ent *cmd,
+ unsigned long iova, size_t size, size_t granule)
+{
+ arm_smmu_tlb_inv_build(cmd, iova, size, granule,
+ idmap_pgtable->cfg.pgsize_bitmap, smmu,
+ __smmu_add_cmd, NULL);
+ return smmu_sync_cmd(smmu);
+}
+
+static void smmu_tlb_inv_range(unsigned long iova, size_t size, size_t granule,
+ bool leaf)
+{
+ struct arm_smmu_cmdq_ent cmd = {
+ .opcode = CMDQ_OP_TLBI_S2_IPA,
+ .tlbi = {
+ .leaf = leaf,
+ .vmid = 0,
+ },
+ };
+ struct hyp_arm_smmu_v3_device *smmu;
+
+ for_each_smmu(smmu)
+ WARN_ON(smmu_tlb_inv_range_smmu(smmu, &cmd, iova, size, granule));
+}
+
+static void smmu_tlb_flush_walk(unsigned long iova, size_t size,
+ size_t granule, void *cookie)
+{
+ smmu_tlb_inv_range(iova, size, granule, false);
+}
+
+static void smmu_tlb_add_page(struct iommu_iotlb_gather *gather,
+ unsigned long iova, size_t granule,
+ void *cookie)
+{
+ smmu_tlb_inv_range(iova, granule, granule, true);
+}
+
+static const struct iommu_flush_ops smmu_tlb_ops = {
+ .tlb_flush_walk = smmu_tlb_flush_walk,
+ .tlb_add_page = smmu_tlb_add_page,
+};
+
__maybe_unused
static int smmu_sync_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid, unsigned long ste)
{
@@ -377,6 +430,34 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
return ret;
}
+static int smmu_init_pgt(void)
+{
+ /* Default values overridden based on SMMUs common features. */
+ struct io_pgtable_cfg cfg = (struct io_pgtable_cfg) {
+ .tlb = &smmu_tlb_ops,
+ .pgsize_bitmap = -1,
+ .ias = 48,
+ .oas = 48,
+ .coherent_walk = true,
+ };
+ int ret = 0;
+ struct hyp_arm_smmu_v3_device *smmu;
+
+ for_each_smmu(smmu) {
+ cfg.ias = min(cfg.ias, smmu->ias);
+ cfg.oas = min(cfg.oas, smmu->oas);
+ cfg.pgsize_bitmap &= smmu->pgsize_bitmap;
+ cfg.coherent_walk &= !!(smmu->features & ARM_SMMU_FEAT_COHERENCY);
+ }
+
+ /* At least PAGE_SIZE must be supported by all SMMUs*/
+ if ((cfg.pgsize_bitmap & PAGE_SIZE) == 0)
+ return -EINVAL;
+
+ idmap_pgtable = kvm_arm_io_pgtable_alloc(&cfg, NULL, ARM_64_LPAE_S2, &ret);
+ return ret;
+}
+
static int smmu_init(void)
{
int ret;
@@ -398,7 +479,7 @@ static int smmu_init(void)
goto out_reclaim_smmu;
}
- return 0;
+ return smmu_init_pgt();
out_reclaim_smmu:
while (smmu != kvm_hyp_arm_smmu_v3_smmus)
smmu_deinit_device(--smmu);
@@ -406,7 +487,85 @@ static int smmu_init(void)
return ret;
}
+static size_t smmu_pgsize_idmap(size_t size, u64 paddr, size_t pgsize_bitmap)
+{
+ size_t pgsizes;
+
+ /* Remove page sizes that are larger than the current size */
+ pgsizes = pgsize_bitmap & GENMASK_ULL(__fls(size), 0);
+
+ /* Remove page sizes that the address is not aligned to. */
+ if (likely(paddr))
+ pgsizes &= GENMASK_ULL(__ffs(paddr), 0);
+
+ WARN_ON(!pgsizes);
+
+ /* Return the larget page size that fits. */
+ return BIT(__fls(pgsizes));
+}
+
+static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int prot)
+{
+ size_t size = end - start;
+ size_t pgsize = PAGE_SIZE, pgcount;
+ size_t mapped, unmapped;
+ int ret;
+ struct io_pgtable *pgtable = idmap_pgtable;
+
+ end = min(end, BIT(pgtable->cfg.oas));
+ if (start >= end)
+ return;
+
+ if (prot) {
+ if (!(prot & IOMMU_MMIO))
+ prot |= IOMMU_CACHE;
+
+ while (size) {
+ mapped = 0;
+ /*
+ * We handle pages size for memory and MMIO differently:
+ * - memory: Map everything with PAGE_SIZE, that is guaranteed to
+ * find memory as we allocated enough pages to cover the entire
+ * memory, we do that as io-pgtable-arm doesn't support
+ * split_blk_unmap logic any more, so we can't break blocks once
+ * mapped to tables.
+ * - MMIO: Unlike memory, pKVM allocate 1G to for all MMIO, while
+ * the MMIO space can be large, as it is assumed to cover the
+ * whole IAS that is not memory, we have to use block mappings,
+ * that is fine for MMIO as it is never donated at the moment,
+ * so we never need to unmap MMIO at the run time triggereing
+ * split block logic.
+ */
+ if (prot & IOMMU_MMIO)
+ pgsize = smmu_pgsize_idmap(size, start, pgtable->cfg.pgsize_bitmap);
+
+ pgcount = size / pgsize;
+ ret = pgtable->ops.map_pages(&pgtable->ops, start, start,
+ pgsize, pgcount, prot, 0, &mapped);
+ size -= mapped;
+ start += mapped;
+ if (!mapped || ret)
+ return;
+ }
+ } else {
+ /* Shouldn't happen. */
+ WARN_ON(prot & IOMMU_MMIO);
+ while (size) {
+ pgcount = size / pgsize;
+ unmapped = pgtable->ops.unmap_pages(&pgtable->ops, start,
+ pgsize, pgcount, NULL);
+ size -= unmapped;
+ start += unmapped;
+ if (!unmapped)
+ return;
+ }
+ /* Some memory were not unmapped. */
+ WARN_ON(size);
+ }
+}
+
/* Shared with the kernel driver in EL1 */
struct kvm_iommu_ops smmu_ops = {
.init = smmu_init,
+ .host_stage2_idmap = smmu_host_stage2_idmap,
};
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
index d188537545b1..5c2f121837ad 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
@@ -14,6 +14,9 @@
* @features SMMUv3 features as defined in arm-smmu-v3-common.h
* @cmdq CMDQ queue struct
* @strtab_cfg stream table config, strtab_cfg.l2.l2ptrs is not used
+ * @ias IAS of the SMMUv3
+ * @oas OAS of the SMMUv3
+ * @pgsize_bitmap Pages sizes supported by the SMMUv3
* Other members are filled and used at runtime by the SMMU driver.
*/
struct hyp_arm_smmu_v3_device {
@@ -23,6 +26,9 @@ struct hyp_arm_smmu_v3_device {
unsigned long features;
struct arm_smmu_queue cmdq;
struct arm_smmu_strtab_cfg strtab_cfg;
+ unsigned int ias;
+ unsigned int oas;
+ size_t pgsize_bitmap;
};
extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 25/29] iommu/arm-smmu-v3-kvm: Add enable/disable device HVCs
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (23 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 24/29] iommu/arm-smmu-v3-kvm: Shadow the CPU stage-2 page table Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 26/29] iommu/arm-smmu-v3-kvm: Add host driver for pKVM Mostafa Saleh
` (3 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Add implementation of enable/disable HVCs, for enable, the STE of
the device would be configured to use the idmap page table, while
for disable it would zero the STE.
As now STEs can be configured from kernel, add a lock to protect
against concurrent access.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 109 +++++++++++++++++-
.../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 11 ++
2 files changed, 118 insertions(+), 2 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
index 38d81cd6d24a..85131962094f 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c
@@ -143,10 +143,15 @@ static int smmu_tlb_inv_range_smmu(struct hyp_arm_smmu_v3_device *smmu,
struct arm_smmu_cmdq_ent *cmd,
unsigned long iova, size_t size, size_t granule)
{
+ int ret;
+
+ hyp_spin_lock(&smmu->lock);
arm_smmu_tlb_inv_build(cmd, iova, size, granule,
idmap_pgtable->cfg.pgsize_bitmap, smmu,
__smmu_add_cmd, NULL);
- return smmu_sync_cmd(smmu);
+ ret = smmu_sync_cmd(smmu);
+ hyp_spin_unlock(&smmu->lock);
+ return ret;
}
static void smmu_tlb_inv_range(unsigned long iova, size_t size, size_t granule,
@@ -183,7 +188,6 @@ static const struct iommu_flush_ops smmu_tlb_ops = {
.tlb_add_page = smmu_tlb_add_page,
};
-__maybe_unused
static int smmu_sync_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid, unsigned long ste)
{
struct arm_smmu_cmdq_ent cmd = {
@@ -409,6 +413,9 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
}
smmu->base = hyp_phys_to_virt(smmu->mmio_addr);
+ hyp_spin_lock_init(&smmu->lock);
+ BUILD_BUG_ON(sizeof(smmu->lock) != sizeof(hyp_spinlock_t));
+
ret = smmu_init_registers(smmu);
if (ret)
goto out_err;
@@ -430,6 +437,102 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu)
return ret;
}
+static struct hyp_arm_smmu_v3_device *smmu_id_to_ptr(pkvm_handle_t smmu_id)
+{
+ if (smmu_id >= kvm_hyp_arm_smmu_v3_count)
+ return NULL;
+ smmu_id = array_index_nospec(smmu_id, kvm_hyp_arm_smmu_v3_count);
+
+ return &kvm_hyp_arm_smmu_v3_smmus[smmu_id];
+}
+
+static void smmu_init_s2_ste(struct arm_smmu_ste *ste)
+{
+ struct io_pgtable_cfg *cfg;
+ u64 ts, sl, ic, oc, sh, tg, ps;
+
+ cfg = &idmap_pgtable->cfg;
+ ps = cfg->arm_lpae_s2_cfg.vtcr.ps;
+ tg = cfg->arm_lpae_s2_cfg.vtcr.tg;
+ sh = cfg->arm_lpae_s2_cfg.vtcr.sh;
+ oc = cfg->arm_lpae_s2_cfg.vtcr.orgn;
+ ic = cfg->arm_lpae_s2_cfg.vtcr.irgn;
+ sl = cfg->arm_lpae_s2_cfg.vtcr.sl;
+ ts = cfg->arm_lpae_s2_cfg.vtcr.tsz;
+
+ ste->data[0] = STRTAB_STE_0_V |
+ FIELD_PREP(STRTAB_STE_0_CFG, STRTAB_STE_0_CFG_S2_TRANS);
+ ste->data[1] = FIELD_PREP(STRTAB_STE_1_SHCFG, STRTAB_STE_1_SHCFG_INCOMING);
+ ste->data[2] = FIELD_PREP(STRTAB_STE_2_VTCR,
+ FIELD_PREP(STRTAB_STE_2_VTCR_S2PS, ps) |
+ FIELD_PREP(STRTAB_STE_2_VTCR_S2TG, tg) |
+ FIELD_PREP(STRTAB_STE_2_VTCR_S2SH0, sh) |
+ FIELD_PREP(STRTAB_STE_2_VTCR_S2OR0, oc) |
+ FIELD_PREP(STRTAB_STE_2_VTCR_S2IR0, ic) |
+ FIELD_PREP(STRTAB_STE_2_VTCR_S2SL0, sl) |
+ FIELD_PREP(STRTAB_STE_2_VTCR_S2T0SZ, ts)) |
+ FIELD_PREP(STRTAB_STE_2_S2VMID, 0) |
+ STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2R;
+ ste->data[3] = cfg->arm_lpae_s2_cfg.vttbr & STRTAB_STE_3_S2TTB_MASK;
+}
+
+static int smmu_enable_dev(pkvm_handle_t iommu, pkvm_handle_t dev)
+{
+ static struct arm_smmu_ste *ste, target;
+ struct hyp_arm_smmu_v3_device *smmu = smmu_id_to_ptr(iommu);
+ int ret;
+
+ if (!smmu)
+ return -ENODEV;
+
+ hyp_spin_lock(&smmu->lock);
+ ste = smmu_get_alloc_ste_ptr(smmu, dev);
+ if (!ste) {
+ ret = - EINVAL;
+ goto out_ret;
+ }
+
+ smmu_init_s2_ste(&target);
+ WRITE_ONCE(ste->data[1], target.data[1]);
+ WRITE_ONCE(ste->data[2], target.data[2]);
+ WRITE_ONCE(ste->data[3], target.data[3]);
+ smmu_sync_ste(smmu, dev, (unsigned long)ste);
+ WRITE_ONCE(ste->data[0], target.data[0]);
+ ret = smmu_sync_ste(smmu, dev, (unsigned long)ste);
+
+out_ret:
+ hyp_spin_unlock(&smmu->lock);
+ return ret;
+}
+
+static int smmu_disable_dev(pkvm_handle_t iommu, pkvm_handle_t dev)
+{
+ static struct arm_smmu_ste *ste;
+ struct hyp_arm_smmu_v3_device *smmu = smmu_id_to_ptr(iommu);
+ int ret;
+
+ if (!smmu)
+ return -ENODEV;
+
+ hyp_spin_lock(&smmu->lock);
+ ste = smmu_get_alloc_ste_ptr(smmu, dev);
+ if (!ste) {
+ ret = -EINVAL;
+ goto out_ret;
+ }
+
+ WRITE_ONCE(ste->data[0], 0);
+ smmu_sync_ste(smmu, dev, (unsigned long)ste);
+ WRITE_ONCE(ste->data[1], 0);
+ WRITE_ONCE(ste->data[2], 0);
+ WRITE_ONCE(ste->data[3], 0);
+ ret = smmu_sync_ste(smmu, dev, (unsigned long)ste);
+
+out_ret:
+ hyp_spin_unlock(&smmu->lock);
+ return ret;
+}
+
static int smmu_init_pgt(void)
{
/* Default values overridden based on SMMUs common features. */
@@ -568,4 +671,6 @@ static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int prot)
struct kvm_iommu_ops smmu_ops = {
.init = smmu_init,
.host_stage2_idmap = smmu_host_stage2_idmap,
+ .enable_dev = smmu_enable_dev,
+ .disable_dev = smmu_disable_dev,
};
diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
index 5c2f121837ad..89faa622af24 100644
--- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
+++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h
@@ -4,6 +4,10 @@
#include <asm/kvm_asm.h>
+#ifdef __KVM_NVHE_HYPERVISOR__
+#include <nvhe/spinlock.h>
+#endif
+
#include "../arm-smmu-v3-common.h"
/*
@@ -18,6 +22,7 @@
* @oas OAS of the SMMUv3
* @pgsize_bitmap Pages sizes supported by the SMMUv3
* Other members are filled and used at runtime by the SMMU driver.
+ * @lock Lock to protect the SMMU resources (STE/CMDQ)
*/
struct hyp_arm_smmu_v3_device {
phys_addr_t mmio_addr;
@@ -29,6 +34,12 @@ struct hyp_arm_smmu_v3_device {
unsigned int ias;
unsigned int oas;
size_t pgsize_bitmap;
+ /* nvhe/spinlock.h not exposed to EL1. */
+#ifdef __KVM_NVHE_HYPERVISOR__
+ hyp_spinlock_t lock;
+#else
+ u32 lock;
+#endif
};
extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 26/29] iommu/arm-smmu-v3-kvm: Add host driver for pKVM
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (24 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 25/29] iommu/arm-smmu-v3-kvm: Add enable/disable device HVCs Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 27/29] iommu/arm-smmu-v3-kvm: Pass a list of SMMU devices to the hypervisor Mostafa Saleh
` (2 subsequent siblings)
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Under protected KVM (pKVM), the host does not have access to guest or
hypervisor memory. This means that devices owned by the host must be
isolated by the SMMU, and the hypervisor is in charge of the SMMU.
Introduce the host component that replaces the normal SMMUv3 driver when
pKVM is enabled, and sends configuration and requests to the actual
driver running in the hypervisor (EL2).
Rather than rely on regular driver probe, pKVM directly calls
kvm_arm_smmu_v3_init(), which synchronously finds all SMMUs and hands
them to the hypervisor. If the regular driver is enabled, it will not
find any free SMMU to drive once it gets probed.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
drivers/iommu/arm/arm-smmu-v3/Makefile | 5 ++
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 62 +++++++++++++++++++
2 files changed, 67 insertions(+)
create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm-smmu-v3/Makefile
index 16be1249000a..6829860f2ce3 100644
--- a/drivers/iommu/arm/arm-smmu-v3/Makefile
+++ b/drivers/iommu/arm/arm-smmu-v3/Makefile
@@ -7,3 +7,8 @@ arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) += arm-smmu-v3-sva.o
arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) += tegra241-cmdqv.o
obj-$(CONFIG_ARM_SMMU_V3_KUNIT_TEST) += arm-smmu-v3-test.o
+
+obj-$(CONFIG_ARM_SMMU_V3_PKVM) += arm_smmu_v3_kvm.o
+arm_smmu_v3_kvm-objs-y += arm-smmu-v3-kvm.o
+arm_smmu_v3_kvm-objs-y += arm-smmu-v3-common.o
+arm_smmu_v3_kvm-objs := $(arm_smmu_v3_kvm-objs-y)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
new file mode 100644
index 000000000000..73c0150bdc81
--- /dev/null
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pKVM host driver for the Arm SMMUv3
+ *
+ * Copyright (C) 2022 Linaro Ltd.
+ */
+#include <asm/kvm_mmu.h>
+
+#include <linux/of_platform.h>
+
+#include "arm-smmu-v3.h"
+#include "pkvm/arm_smmu_v3.h"
+
+extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops);
+
+static int kvm_arm_smmu_probe(struct platform_device *pdev)
+{
+ return -ENOSYS;
+}
+
+static void kvm_arm_smmu_remove(struct platform_device *pdev)
+{
+}
+
+static const struct of_device_id arm_smmu_of_match[] = {
+ { .compatible = "arm,smmu-v3", },
+ { },
+};
+
+static struct platform_driver kvm_arm_smmu_driver = {
+ .driver = {
+ .name = "kvm-arm-smmu-v3",
+ .of_match_table = arm_smmu_of_match,
+ },
+ .remove = kvm_arm_smmu_remove,
+};
+
+static int kvm_arm_smmu_v3_init_drv(void)
+{
+ return platform_driver_probe(&kvm_arm_smmu_driver, kvm_arm_smmu_probe);
+}
+
+static void kvm_arm_smmu_v3_remove_drv(void)
+{
+ platform_driver_unregister(&kvm_arm_smmu_driver);
+}
+
+struct kvm_iommu_driver kvm_smmu_v3_ops = {
+ .init_driver = kvm_arm_smmu_v3_init_drv,
+ .remove_driver = kvm_arm_smmu_v3_remove_drv,
+};
+
+static int kvm_arm_smmu_v3_register(void)
+{
+ if (!is_protected_kvm_enabled())
+ return 0;
+
+ return kvm_iommu_register_driver(&kvm_smmu_v3_ops,
+ kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu_ops))));
+};
+
+core_initcall(kvm_arm_smmu_v3_register);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 27/29] iommu/arm-smmu-v3-kvm: Pass a list of SMMU devices to the hypervisor
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (25 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 26/29] iommu/arm-smmu-v3-kvm: Add host driver for pKVM Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 28/29] iommu/arm-smmu-v3-kvm: Allocate structures and reset device Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops Mostafa Saleh
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Build a list of devices and donate the page to the hypervisor. At this
point the host is trusted and this would be a good opportunity to
provide more information about the system.
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 182 +++++++++++++++++-
1 file changed, 179 insertions(+), 3 deletions(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
index 73c0150bdc81..2fa56ddfbdab 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
@@ -13,9 +13,119 @@
extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops);
+struct host_arm_smmu_device {
+ struct arm_smmu_device smmu;
+ pkvm_handle_t id;
+};
+
+#define smmu_to_host(_smmu) \
+ container_of(_smmu, struct host_arm_smmu_device, smmu);
+
+static size_t kvm_arm_smmu_cur;
+static size_t kvm_arm_smmu_count;
+static struct hyp_arm_smmu_v3_device *kvm_arm_smmu_array;
+
+static bool kvm_arm_smmu_validate_features(struct arm_smmu_device *smmu)
+{
+ unsigned int required_features =
+ ARM_SMMU_FEAT_TT_LE |
+ ARM_SMMU_FEAT_TRANS_S2;
+ unsigned int forbidden_features =
+ ARM_SMMU_FEAT_STALL_FORCE;
+ unsigned int keep_features =
+ ARM_SMMU_FEAT_2_LVL_STRTAB |
+ ARM_SMMU_FEAT_2_LVL_CDTAB |
+ ARM_SMMU_FEAT_TT_LE |
+ ARM_SMMU_FEAT_SEV |
+ ARM_SMMU_FEAT_COHERENCY |
+ ARM_SMMU_FEAT_TRANS_S1 |
+ ARM_SMMU_FEAT_TRANS_S2 |
+ ARM_SMMU_FEAT_VAX |
+ ARM_SMMU_FEAT_RANGE_INV;
+
+ if (smmu->options & ARM_SMMU_OPT_PAGE0_REGS_ONLY) {
+ dev_err(smmu->dev, "unsupported layout\n");
+ return false;
+ }
+
+ if ((smmu->features & required_features) != required_features) {
+ dev_err(smmu->dev, "missing features 0x%x\n",
+ required_features & ~smmu->features);
+ return false;
+ }
+
+ if (smmu->features & forbidden_features) {
+ dev_err(smmu->dev, "features 0x%x forbidden\n",
+ smmu->features & forbidden_features);
+ return false;
+ }
+
+ smmu->features &= keep_features;
+
+ return true;
+}
+
static int kvm_arm_smmu_probe(struct platform_device *pdev)
{
- return -ENOSYS;
+ int ret;
+ size_t size;
+ phys_addr_t ioaddr;
+ struct resource *res;
+ struct arm_smmu_device *smmu;
+ struct device *dev = &pdev->dev;
+ struct host_arm_smmu_device *host_smmu;
+ struct hyp_arm_smmu_v3_device *hyp_smmu;
+
+ if (kvm_arm_smmu_cur >= kvm_arm_smmu_count)
+ return -ENOSPC;
+
+ hyp_smmu = &kvm_arm_smmu_array[kvm_arm_smmu_cur];
+
+ host_smmu = devm_kzalloc(dev, sizeof(*host_smmu), GFP_KERNEL);
+ if (!host_smmu)
+ return -ENOMEM;
+
+ smmu = &host_smmu->smmu;
+ smmu->dev = dev;
+
+ ret = arm_smmu_fw_probe(pdev, smmu);
+ if (ret)
+ return ret;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ size = resource_size(res);
+ if (size < SZ_128K) {
+ dev_err(dev, "unsupported MMIO region size (%pr)\n", res);
+ return -EINVAL;
+ }
+ ioaddr = res->start;
+ host_smmu->id = kvm_arm_smmu_cur;
+
+ smmu->base = devm_ioremap_resource(dev, res);
+ if (IS_ERR(smmu->base))
+ return PTR_ERR(smmu->base);
+
+ ret = arm_smmu_device_hw_probe(smmu);
+ if (ret)
+ return ret;
+
+ if (!kvm_arm_smmu_validate_features(smmu))
+ return -ENODEV;
+
+ platform_set_drvdata(pdev, smmu);
+
+ /* Hypervisor parameters */
+ hyp_smmu->cmdq = smmu->cmdq.q;
+ hyp_smmu->strtab_cfg = smmu->strtab_cfg;
+ hyp_smmu->pgsize_bitmap = smmu->pgsize_bitmap;
+ hyp_smmu->oas = smmu->oas;
+ hyp_smmu->ias = smmu->ias;
+ hyp_smmu->mmio_addr = ioaddr;
+ hyp_smmu->mmio_size = size;
+ hyp_smmu->features = smmu->features;
+ kvm_arm_smmu_cur++;
+
+ return 0;
}
static void kvm_arm_smmu_remove(struct platform_device *pdev)
@@ -35,9 +145,62 @@ static struct platform_driver kvm_arm_smmu_driver = {
.remove = kvm_arm_smmu_remove,
};
+static int kvm_arm_smmu_array_alloc(void)
+{
+ int smmu_order;
+ struct device_node *np;
+
+ kvm_arm_smmu_count = 0;
+ for_each_compatible_node(np, NULL, "arm,smmu-v3")
+ kvm_arm_smmu_count++;
+
+ if (!kvm_arm_smmu_count)
+ return 0;
+
+ /* Allocate the parameter list shared with the hypervisor */
+ smmu_order = get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array));
+ kvm_arm_smmu_array = (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO,
+ smmu_order);
+ if (!kvm_arm_smmu_array)
+ return -ENOMEM;
+
+ return 0;
+}
+
+static void kvm_arm_smmu_array_free(void)
+{
+ int order;
+
+ order = get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array));
+ free_pages((unsigned long)kvm_arm_smmu_array, order);
+}
+
static int kvm_arm_smmu_v3_init_drv(void)
{
- return platform_driver_probe(&kvm_arm_smmu_driver, kvm_arm_smmu_probe);
+ int ret;
+
+ ret = platform_driver_probe(&kvm_arm_smmu_driver, kvm_arm_smmu_probe);
+ if (ret)
+ goto err_free;
+
+ if (kvm_arm_smmu_cur != kvm_arm_smmu_count) {
+ /* A device exists but failed to probe */
+ ret = -EUNATCH;
+ goto err_free;
+ }
+
+ /*
+ * These variables are stored in the nVHE image, and won't be accessible
+ * after KVM initialization. Ownership of kvm_arm_smmu_array will be
+ * transferred to the hypervisor as well.
+ */
+ kvm_hyp_arm_smmu_v3_smmus = kvm_arm_smmu_array;
+ kvm_hyp_arm_smmu_v3_count = kvm_arm_smmu_count;
+ return 0;
+
+err_free:
+ kvm_arm_smmu_array_free();
+ return ret;
}
static void kvm_arm_smmu_v3_remove_drv(void)
@@ -52,11 +215,24 @@ struct kvm_iommu_driver kvm_smmu_v3_ops = {
static int kvm_arm_smmu_v3_register(void)
{
+ int ret;
+
if (!is_protected_kvm_enabled())
return 0;
- return kvm_iommu_register_driver(&kvm_smmu_v3_ops,
+ /*
+ * Only one KVM IOMMU driver can be registered, so only call the
+ * register function if any SMMUv3 exists on the platform.
+ */
+ ret = kvm_arm_smmu_array_alloc();
+ if (ret || !kvm_arm_smmu_count)
+ return ret;
+
+ ret = kvm_iommu_register_driver(&kvm_smmu_v3_ops,
kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu_ops))));
+ if (ret)
+ kvm_arm_smmu_array_free();
+ return ret;
};
core_initcall(kvm_arm_smmu_v3_register);
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 28/29] iommu/arm-smmu-v3-kvm: Allocate structures and reset device
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (26 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 27/29] iommu/arm-smmu-v3-kvm: Pass a list of SMMU devices to the hypervisor Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-28 17:53 ` [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops Mostafa Saleh
28 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Allocate the structures that will be shared between hypervisor and SMMU:
command queue, page table and stream table. Install them in the MMIO
registers, along with some configuration bits. After hyp initialization,
the host won't have access to those pages anymore.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
arch/arm64/include/asm/kvm_host.h | 5 ++
arch/arm64/kvm/iommu.c | 9 ++-
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 65 +++++++++++++++++++
3 files changed, 77 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index b42e32ff2a84..37b34a9f7135 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1684,4 +1684,9 @@ int kvm_iommu_register_driver(struct kvm_iommu_driver *kern_ops, struct kvm_iomm
int kvm_iommu_init_driver(void);
void kvm_iommu_remove_driver(void);
size_t kvm_iommu_pages(void);
+
+#ifdef CONFIG_ARM_SMMU_V3_PKVM
+size_t smmu_hyp_pgt_pages(void);
+#endif
+
#endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c
index 0e52cc0e18de..c4605e8001ca 100644
--- a/arch/arm64/kvm/iommu.c
+++ b/arch/arm64/kvm/iommu.c
@@ -51,10 +51,15 @@ void kvm_iommu_remove_driver(void)
size_t kvm_iommu_pages(void)
{
+ size_t nr_pages = 0;
/*
* This is called very early during setup_arch() where no initcalls,
* so this has to call specific functions per each KVM driver.
*/
- kvm_nvhe_sym(hyp_kvm_iommu_pages) = 0;
- return 0;
+#ifdef CONFIG_ARM_SMMU_V3_PKVM
+ nr_pages = smmu_hyp_pgt_pages();
+#endif
+
+ kvm_nvhe_sym(hyp_kvm_iommu_pages) = nr_pages;
+ return nr_pages;
}
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
index 2fa56ddfbdab..2e51e211250d 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
@@ -5,6 +5,7 @@
* Copyright (C) 2022 Linaro Ltd.
*/
#include <asm/kvm_mmu.h>
+#include <asm/kvm_pkvm.h>
#include <linux/of_platform.h>
@@ -16,6 +17,7 @@ extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops);
struct host_arm_smmu_device {
struct arm_smmu_device smmu;
pkvm_handle_t id;
+ u32 boot_gbpa;
};
#define smmu_to_host(_smmu) \
@@ -65,6 +67,35 @@ static bool kvm_arm_smmu_validate_features(struct arm_smmu_device *smmu)
return true;
}
+static int kvm_arm_smmu_device_reset(struct host_arm_smmu_device *host_smmu)
+{
+ int ret;
+ u32 reg;
+ struct arm_smmu_device *smmu = &host_smmu->smmu;
+
+ reg = readl_relaxed(smmu->base + ARM_SMMU_CR0);
+ if (reg & CR0_SMMUEN)
+ dev_warn(smmu->dev, "SMMU currently enabled! Resetting...\n");
+
+ /* Disable bypass */
+ host_smmu->boot_gbpa = readl_relaxed(smmu->base + ARM_SMMU_GBPA);
+ ret = arm_smmu_update_gbpa(smmu, GBPA_ABORT, 0);
+ if (ret)
+ return ret;
+
+ ret = arm_smmu_device_disable(smmu);
+ if (ret)
+ return ret;
+
+ /* Stream table */
+ arm_smmu_write_strtab(smmu);
+
+ /* Command queue */
+ writeq_relaxed(smmu->cmdq.q.q_base, smmu->base + ARM_SMMU_CMDQ_BASE);
+
+ return 0;
+}
+
static int kvm_arm_smmu_probe(struct platform_device *pdev)
{
int ret;
@@ -112,6 +143,20 @@ static int kvm_arm_smmu_probe(struct platform_device *pdev)
if (!kvm_arm_smmu_validate_features(smmu))
return -ENODEV;
+ ret = arm_smmu_init_one_queue(smmu, &smmu->cmdq.q, smmu->base,
+ ARM_SMMU_CMDQ_PROD, ARM_SMMU_CMDQ_CONS,
+ CMDQ_ENT_DWORDS, "cmdq");
+ if (ret)
+ return ret;
+
+ ret = arm_smmu_init_strtab(smmu);
+ if (ret)
+ return ret;
+
+ ret = kvm_arm_smmu_device_reset(host_smmu);
+ if (ret)
+ return ret;
+
platform_set_drvdata(pdev, smmu);
/* Hypervisor parameters */
@@ -130,6 +175,15 @@ static int kvm_arm_smmu_probe(struct platform_device *pdev)
static void kvm_arm_smmu_remove(struct platform_device *pdev)
{
+ struct arm_smmu_device *smmu = platform_get_drvdata(pdev);
+ struct host_arm_smmu_device *host_smmu = smmu_to_host(smmu);
+
+ /*
+ * There was an error during hypervisor setup. The hyp driver may
+ * have already enabled the device, so disable it.
+ */
+ arm_smmu_device_disable(smmu);
+ arm_smmu_update_gbpa(smmu, host_smmu->boot_gbpa, GBPA_ABORT);
}
static const struct of_device_id arm_smmu_of_match[] = {
@@ -208,6 +262,17 @@ static void kvm_arm_smmu_v3_remove_drv(void)
platform_driver_unregister(&kvm_arm_smmu_driver);
}
+size_t smmu_hyp_pgt_pages(void)
+{
+ /*
+ * SMMUv3 uses the same format as stage-2 and hence have the same memory
+ * requirements, we add extra 100 pages for L2 ste.
+ */
+ if (of_find_compatible_node(NULL, NULL, "arm,smmu-v3"))
+ return host_s2_pgtable_pages() + 100;
+ return 0;
+}
+
struct kvm_iommu_driver kvm_smmu_v3_ops = {
.init_driver = kvm_arm_smmu_v3_init_drv,
.remove_driver = kvm_arm_smmu_v3_remove_drv,
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-07-28 17:52 [PATCH v3 00/29] KVM: arm64: SMMUv3 driver for pKVM Mostafa Saleh
` (27 preceding siblings ...)
2025-07-28 17:53 ` [PATCH v3 28/29] iommu/arm-smmu-v3-kvm: Allocate structures and reset device Mostafa Saleh
@ 2025-07-28 17:53 ` Mostafa Saleh
2025-07-30 14:42 ` Jason Gunthorpe
28 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-28 17:53 UTC (permalink / raw)
To: linux-kernel, kvmarm, linux-arm-kernel, iommu
Cc: maz, oliver.upton, joey.gouly, suzuki.poulose, yuzenghui,
catalin.marinas, will, robin.murphy, jean-philippe, qperret,
tabba, jgg, mark.rutland, praan, Mostafa Saleh
Register the SMMUv3 through IOMMU ops, that only support identity
domains. This allows the driver to know which device are currently used
to properly enable/disable then.
Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
.../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 92 ++++++++++++++++++-
1 file changed, 91 insertions(+), 1 deletion(-)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
index 2e51e211250d..d7b2a50a4cb2 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c
@@ -96,6 +96,95 @@ static int kvm_arm_smmu_device_reset(struct host_arm_smmu_device *host_smmu)
return 0;
}
+static struct platform_driver kvm_arm_smmu_driver;
+static struct arm_smmu_device *
+kvm_arm_smmu_get_by_fwnode(struct fwnode_handle *fwnode)
+{
+ struct device *dev;
+
+ dev = driver_find_device_by_fwnode(&kvm_arm_smmu_driver.driver, fwnode);
+ put_device(dev);
+ return dev ? dev_get_drvdata(dev) : NULL;
+}
+
+static struct iommu_device *kvm_arm_smmu_probe_device(struct device *dev)
+{
+ struct arm_smmu_device *smmu;
+ struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+
+ if (WARN_ON_ONCE(dev_iommu_priv_get(dev)))
+ return ERR_PTR(-EBUSY);
+
+ smmu = kvm_arm_smmu_get_by_fwnode(fwspec->iommu_fwnode);
+ if (!smmu)
+ return ERR_PTR(-ENODEV);
+
+ dev_iommu_priv_set(dev, smmu);
+ return &smmu->iommu;
+}
+
+static void kvm_arm_smmu_release_device(struct device *dev)
+{
+ int i;
+ struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+ struct arm_smmu_device *smmu = dev_iommu_priv_get(dev);
+ struct host_arm_smmu_device *host_smmu = smmu_to_host(smmu);
+
+ for (i = 0; i < fwspec->num_ids; i++) {
+ int sid = fwspec->ids[i];
+
+ kvm_call_hyp_nvhe(__pkvm_iommu_disable_dev, host_smmu->id, sid);
+ }
+}
+
+static phys_addr_t kvm_arm_smmu_iova_to_phys(struct iommu_domain *domain,
+ dma_addr_t iova)
+{
+ return iova;
+}
+
+static int kvm_arm_smmu_attach_dev(struct iommu_domain *domain,
+ struct device *dev)
+{
+ int i, ret = 0;
+ struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+ struct arm_smmu_device *smmu = dev_iommu_priv_get(dev);
+ struct host_arm_smmu_device *host_smmu = smmu_to_host(smmu);
+
+ for (i = 0; i < fwspec->num_ids; i++) {
+ int sid = fwspec->ids[i];
+
+ ret = kvm_call_hyp_nvhe(__pkvm_iommu_enable_dev, host_smmu->id, sid);
+ if (ret)
+ goto out_err;
+ }
+ return ret;
+out_err:
+ while (i--)
+ kvm_call_hyp_nvhe(__pkvm_iommu_disable_dev, host_smmu->id, fwspec->ids[i]);
+
+ return ret;
+}
+
+static struct iommu_domain kvm_arm_smmu_def_domain = {
+ .type = IOMMU_DOMAIN_IDENTITY,
+ .ops = &(const struct iommu_domain_ops) {
+ .attach_dev = kvm_arm_smmu_attach_dev,
+ .iova_to_phys = kvm_arm_smmu_iova_to_phys,
+ }
+};
+
+static struct iommu_ops kvm_arm_smmu_ops = {
+ .device_group = arm_smmu_device_group,
+ .of_xlate = arm_smmu_of_xlate,
+ .get_resv_regions = arm_smmu_get_resv_regions,
+ .probe_device = kvm_arm_smmu_probe_device,
+ .release_device = kvm_arm_smmu_release_device,
+ .pgsize_bitmap = -1UL,
+ .owner = THIS_MODULE,
+ .default_domain = &kvm_arm_smmu_def_domain,
+};
+
static int kvm_arm_smmu_probe(struct platform_device *pdev)
{
int ret;
@@ -170,7 +259,7 @@ static int kvm_arm_smmu_probe(struct platform_device *pdev)
hyp_smmu->features = smmu->features;
kvm_arm_smmu_cur++;
- return 0;
+ return arm_smmu_register_iommu(smmu, &kvm_arm_smmu_ops, ioaddr);
}
static void kvm_arm_smmu_remove(struct platform_device *pdev)
@@ -184,6 +273,7 @@ static void kvm_arm_smmu_remove(struct platform_device *pdev)
*/
arm_smmu_device_disable(smmu);
arm_smmu_update_gbpa(smmu, host_smmu->boot_gbpa, GBPA_ABORT);
+ arm_smmu_unregister_iommu(smmu);
}
static const struct of_device_id arm_smmu_of_match[] = {
--
2.50.1.552.g942d659e1b-goog
^ permalink raw reply related [flat|nested] 48+ messages in thread* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-07-28 17:53 ` [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops Mostafa Saleh
@ 2025-07-30 14:42 ` Jason Gunthorpe
2025-07-30 15:07 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Jason Gunthorpe @ 2025-07-30 14:42 UTC (permalink / raw)
To: Mostafa Saleh
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Mon, Jul 28, 2025 at 05:53:16PM +0000, Mostafa Saleh wrote:
> Register the SMMUv3 through IOMMU ops, that only support identity
> domains. This allows the driver to know which device are currently used
> to properly enable/disable then.
>
> Signed-off-by: Mostafa Saleh <smostafa@google.com>
> ---
> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 92 ++++++++++++++++++-
> 1 file changed, 91 insertions(+), 1 deletion(-)
Can you split the new iommu subysstem driver out please? I think I
asked this before.
This series is big, reviewing a new iommu driver should be done separately.
Please review all the comments for the verisilicon driver, I think
many of the remarks apply here too:
- Domain attachment looks questionable. Please do not have
attach/detach language at all in the hypervisor facing API.
- Get the ordering and APIs right so replace works. You need this to support RMRs
- Use a blocking domain not some unclear detatch idea
- Use a blocking domain for release
- Use the smmu-v3 approach for the fwspec, don't store things in the drvdata.
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-07-30 14:42 ` Jason Gunthorpe
@ 2025-07-30 15:07 ` Mostafa Saleh
2025-07-30 16:47 ` Jason Gunthorpe
0 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-30 15:07 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Wed, Jul 30, 2025 at 11:42:53AM -0300, Jason Gunthorpe wrote:
> On Mon, Jul 28, 2025 at 05:53:16PM +0000, Mostafa Saleh wrote:
> > Register the SMMUv3 through IOMMU ops, that only support identity
> > domains. This allows the driver to know which device are currently used
> > to properly enable/disable then.
> >
> > Signed-off-by: Mostafa Saleh <smostafa@google.com>
> > ---
> > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 92 ++++++++++++++++++-
> > 1 file changed, 91 insertions(+), 1 deletion(-)
>
> Can you split the new iommu subysstem driver out please? I think I
> asked this before.
Sorry, maybe I misunderstood, do you mean split this patch into multiple
patches or split all KVM SMMUv3 driver out of this series?
>
> This series is big, reviewing a new iommu driver should be done separately.
>
> Please review all the comments for the verisilicon driver, I think
> many of the remarks apply here too:
Thanks, I will go through them.
>
> - Domain attachment looks questionable. Please do not have
> attach/detach language at all in the hypervisor facing API.
I am not sure I understand this one, the hypervisor API has no
attach/detach APIs?
We only notify the hypervisor via “enable/disable” hypercalls when
devices are attached or released (as only IDENTITY DOMAIN is supported)
so it can enable or disable translation.
> - Get the ordering and APIs right so replace works. You need this to support RMRs
I see, we can’t support bypass for security reasons, but we can enable
the identity map for such devices.
> - Use a blocking domain not some unclear detatch idea
There is not a single detach in this driver, we only support attach and release
operation, which just goes to the hypervisor as enable/disable identity.
> - Use a blocking domain for release
I see, I can add “blocked_domain” similar to “arm_smmu_blocked_domain”
which just disables the device translation, I will look into, I am just
concerned it's more code as this driver only supports a single domain
type.
> - Use the smmu-v3 approach for the fwspec, don't store things in the drvdata.
This is using “dev_iommu_fwspec_get” similar to the SMMUv3 driver,
am I missing something?
The driver only save the "struct arm_smmu_device" in drvdata, which is
exactly what the other driver does (which allows us to re-use a lot of
the code)
Thanks,
Mostafa
>
> Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-07-30 15:07 ` Mostafa Saleh
@ 2025-07-30 16:47 ` Jason Gunthorpe
2025-07-31 14:17 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Jason Gunthorpe @ 2025-07-30 16:47 UTC (permalink / raw)
To: Mostafa Saleh
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Wed, Jul 30, 2025 at 03:07:14PM +0000, Mostafa Saleh wrote:
> On Wed, Jul 30, 2025 at 11:42:53AM -0300, Jason Gunthorpe wrote:
> > On Mon, Jul 28, 2025 at 05:53:16PM +0000, Mostafa Saleh wrote:
> > > Register the SMMUv3 through IOMMU ops, that only support identity
> > > domains. This allows the driver to know which device are currently used
> > > to properly enable/disable then.
> > >
> > > Signed-off-by: Mostafa Saleh <smostafa@google.com>
> > > ---
> > > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 92 ++++++++++++++++++-
> > > 1 file changed, 91 insertions(+), 1 deletion(-)
> >
> > Can you split the new iommu subysstem driver out please? I think I
> > asked this before.
>
> Sorry, maybe I misunderstood, do you mean split this patch into multiple
> patches or split all KVM SMMUv3 driver out of this series?
Yes the latter, the iommu driver introduction is best as its own
series
> > - Domain attachment looks questionable. Please do not have
> > attach/detach language at all in the hypervisor facing API.
>
> I am not sure I understand this one, the hypervisor API has no
> attach/detach APIs?
> We only notify the hypervisor via “enable/disable” hypercalls when
> devices are attached or released (as only IDENTITY DOMAIN is supported)
> so it can enable or disable translation.
Same difference, different words.. We've had trouble with this kind of
ambiguous language.
attach identity
attach blocking
Keep it clean and simple
> > - Get the ordering and APIs right so replace works. You need this to support RMRs
>
> I see, we can’t support bypass for security reasons, but we can enable
> the identity map for such devices.
You will have a small issue if the bootup has the pkvm side put the
device into a blocking translation then later switches to an
"identity".
As far as I understand it display scan out buffers have to be
continuously hitlessly mapped. So you want all the transitions from
bootup bypass, pkvm isolation, "identity", etc to continuously
hitlessly map the RMRs containing the buffers.
I think :)
> > - Use a blocking domain not some unclear detatch idea
>
> There is not a single detach in this driver, we only support attach and release
> operation, which just goes to the hypervisor as enable/disable
> identity.
'disable' then.
> > - Use a blocking domain for release
>
> I see, I can add “blocked_domain” similar to “arm_smmu_blocked_domain”
> which just disables the device translation, I will look into, I am just
> concerned it's more code as this driver only supports a single domain
> type.
Yes, this is the right thing. There should not be operations in the
driver that change the underyling translation that are anything other
than attaching domains. It becomes too hard to maintain.
> > - Use the smmu-v3 approach for the fwspec, don't store things in the drvdata.
>
> This is using “dev_iommu_fwspec_get” similar to the SMMUv3 driver,
> am I missing something?
Oh I got it confused, you are just calling
+ .device_group = arm_smmu_device_group,
+ .of_xlate = arm_smmu_of_xlate,
+ .get_resv_regions = arm_smmu_get_resv_regions,
Which is also weird, why does an entirely new driver call 4 random functions
in arm smmu?
Maybe don't, they are all trivial functions, or you don't need
them. For example you don't need MSI_IOVA_BASE if the driver doesn't
support paging.
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-07-30 16:47 ` Jason Gunthorpe
@ 2025-07-31 14:17 ` Mostafa Saleh
2025-07-31 16:57 ` Jason Gunthorpe
0 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-31 14:17 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Wed, Jul 30, 2025 at 01:47:52PM -0300, Jason Gunthorpe wrote:
> On Wed, Jul 30, 2025 at 03:07:14PM +0000, Mostafa Saleh wrote:
> > On Wed, Jul 30, 2025 at 11:42:53AM -0300, Jason Gunthorpe wrote:
> > > On Mon, Jul 28, 2025 at 05:53:16PM +0000, Mostafa Saleh wrote:
> > > > Register the SMMUv3 through IOMMU ops, that only support identity
> > > > domains. This allows the driver to know which device are currently used
> > > > to properly enable/disable then.
> > > >
> > > > Signed-off-by: Mostafa Saleh <smostafa@google.com>
> > > > ---
> > > > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 92 ++++++++++++++++++-
> > > > 1 file changed, 91 insertions(+), 1 deletion(-)
> > >
> > > Can you split the new iommu subysstem driver out please? I think I
> > > asked this before.
> >
> > Sorry, maybe I misunderstood, do you mean split this patch into multiple
> > patches or split all KVM SMMUv3 driver out of this series?
>
> Yes the latter, the iommu driver introduction is best as its own
> series
I thought about that but I was worried the maintainers wouldn't like
introducing the infrastructure first in the hypervisor without a user.
I am open to split this, but let’s see what they think.
>
> > > - Domain attachment looks questionable. Please do not have
> > > attach/detach language at all in the hypervisor facing API.
> >
> > I am not sure I understand this one, the hypervisor API has no
> > attach/detach APIs?
> > We only notify the hypervisor via “enable/disable” hypercalls when
> > devices are attached or released (as only IDENTITY DOMAIN is supported)
> > so it can enable or disable translation.
>
> Same difference, different words.. We've had trouble with this kind of
> ambiguous language.
>
> attach identity
> attach blocking
>
> Keep it clean and simple
Makes sense, from the kernel point of view it will be attached to
identity/blocking domains, but the hypervisor api is just enable/disable HVC
as it doesn’t know what is a domain. If terminology is really a problem,
I can make it one hypercall as “set_state” with on/off or identity/blocking
>
> > > - Get the ordering and APIs right so replace works. You need this to support RMRs
> >
> > I see, we can’t support bypass for security reasons, but we can enable
> > the identity map for such devices.
>
> You will have a small issue if the bootup has the pkvm side put the
> device into a blocking translation then later switches to an
> "identity".
>
> As far as I understand it display scan out buffers have to be
> continuously hitlessly mapped. So you want all the transitions from
> bootup bypass, pkvm isolation, "identity", etc to continuously
> hitlessly map the RMRs containing the buffers.
>
> I think :)
I think that would be hard with pKVM, maybe it’s possible to achieve
that seamingless, we would need to export such devices to pKVM at
boot before de-privilege.
TBH, I am not sure what hardware does that. So, another option is to fail
gracefully if RMR exists (which falls back to the current driver) and then
pKVM would run with DMA isolation, which is the status quo.
And then RMR support for pKVM can be added later if there is interset.
>
> > > - Use a blocking domain not some unclear detatch idea
> >
> > There is not a single detach in this driver, we only support attach and release
> > operation, which just goes to the hypervisor as enable/disable
> > identity.
>
> 'disable' then.
>
> > > - Use a blocking domain for release
> >
> > I see, I can add “blocked_domain” similar to “arm_smmu_blocked_domain”
> > which just disables the device translation, I will look into, I am just
> > concerned it's more code as this driver only supports a single domain
> > type.
>
> Yes, this is the right thing. There should not be operations in the
> driver that change the underyling translation that are anything other
> than attaching domains. It becomes too hard to maintain.
Will do as mentioned above.
>
>
> > > - Use the smmu-v3 approach for the fwspec, don't store things in the drvdata.
> >
> > This is using “dev_iommu_fwspec_get” similar to the SMMUv3 driver,
> > am I missing something?
>
> Oh I got it confused, you are just calling
>
> + .device_group = arm_smmu_device_group,
> + .of_xlate = arm_smmu_of_xlate,
> + .get_resv_regions = arm_smmu_get_resv_regions,
>
> Which is also weird, why does an entirely new driver call 4 random functions
> in arm smmu?
>
> Maybe don't, they are all trivial functions, or you don't need
> them. For example you don't need MSI_IOVA_BASE if the driver doesn't
> support paging.
>
They are not random, as part of this series the SMMUv3 driver is split
where some of the code goes to “arm-smmu-v3-common.c” which is used by
both drivers, this reduces a lot of duplication.
I am not sure if we need get_resv_regions, maybe it's useful for sysfs
"/sys/kernel/iommu_groups/reserved_regions"? I will double check.
Thanks,
Mostafa
> Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-07-31 14:17 ` Mostafa Saleh
@ 2025-07-31 16:57 ` Jason Gunthorpe
2025-07-31 17:44 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Jason Gunthorpe @ 2025-07-31 16:57 UTC (permalink / raw)
To: Mostafa Saleh
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Thu, Jul 31, 2025 at 02:17:17PM +0000, Mostafa Saleh wrote:
> On Wed, Jul 30, 2025 at 01:47:52PM -0300, Jason Gunthorpe wrote:
> > On Wed, Jul 30, 2025 at 03:07:14PM +0000, Mostafa Saleh wrote:
> > > On Wed, Jul 30, 2025 at 11:42:53AM -0300, Jason Gunthorpe wrote:
> > > > On Mon, Jul 28, 2025 at 05:53:16PM +0000, Mostafa Saleh wrote:
> > > > > Register the SMMUv3 through IOMMU ops, that only support identity
> > > > > domains. This allows the driver to know which device are currently used
> > > > > to properly enable/disable then.
> > > > >
> > > > > Signed-off-by: Mostafa Saleh <smostafa@google.com>
> > > > > ---
> > > > > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 92 ++++++++++++++++++-
> > > > > 1 file changed, 91 insertions(+), 1 deletion(-)
> > > >
> > > > Can you split the new iommu subysstem driver out please? I think I
> > > > asked this before.
> > >
> > > Sorry, maybe I misunderstood, do you mean split this patch into multiple
> > > patches or split all KVM SMMUv3 driver out of this series?
> >
> > Yes the latter, the iommu driver introduction is best as its own
> > series
>
> I thought about that but I was worried the maintainers wouldn't like
> introducing the infrastructure first in the hypervisor without a user.
> I am open to split this, but let’s see what they think.
You can merge both series at the same time
> Makes sense, from the kernel point of view it will be attached to
> identity/blocking domains, but the hypervisor api is just enable/disable HVC
> as it doesn’t know what is a domain. If terminology is really a problem,
> I can make it one hypercall as “set_state” with on/off or identity/blocking
I would call it set_state with states IDENTITY/BLOCKING. That is
clear. enable/disable is ambiguous.
> TBH, I am not sure what hardware does that. So, another option is to fail
> gracefully if RMR exists (which falls back to the current driver) and then
> pKVM would run with DMA isolation, which is the status quo.
iGPUs either access the DRAM through the iommu or they use some OS
invisible side band channel.
The ones that use the iommu have this quirk.
> They are not random, as part of this series the SMMUv3 driver is split
> where some of the code goes to “arm-smmu-v3-common.c” which is used by
> both drivers, this reduces a lot of duplication.
I find it very confusing.
It made sense to factor some of the code out so that pKVM can have
it's own smmv3 HW driver, sure.
But I don't understand why a paravirtualized iommu driver for pKVM has
any relation to smmuv3. Shouldn't it just be calling some hypercalls
to set IDENTITY/BLOCKING?
> I am not sure if we need get_resv_regions, maybe it's useful for sysfs
> "/sys/kernel/iommu_groups/reserved_regions"? I will double check.
It is important to get this info from the FW..
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-07-31 16:57 ` Jason Gunthorpe
@ 2025-07-31 17:44 ` Mostafa Saleh
2025-08-01 18:59 ` Jason Gunthorpe
0 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-07-31 17:44 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Thu, Jul 31, 2025 at 01:57:57PM -0300, Jason Gunthorpe wrote:
> On Thu, Jul 31, 2025 at 02:17:17PM +0000, Mostafa Saleh wrote:
> > On Wed, Jul 30, 2025 at 01:47:52PM -0300, Jason Gunthorpe wrote:
> > > On Wed, Jul 30, 2025 at 03:07:14PM +0000, Mostafa Saleh wrote:
> > > > On Wed, Jul 30, 2025 at 11:42:53AM -0300, Jason Gunthorpe wrote:
> > > > > On Mon, Jul 28, 2025 at 05:53:16PM +0000, Mostafa Saleh wrote:
> > > > > > Register the SMMUv3 through IOMMU ops, that only support identity
> > > > > > domains. This allows the driver to know which device are currently used
> > > > > > to properly enable/disable then.
> > > > > >
> > > > > > Signed-off-by: Mostafa Saleh <smostafa@google.com>
> > > > > > ---
> > > > > > .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 92 ++++++++++++++++++-
> > > > > > 1 file changed, 91 insertions(+), 1 deletion(-)
> > > > >
> > > > > Can you split the new iommu subysstem driver out please? I think I
> > > > > asked this before.
> > > >
> > > > Sorry, maybe I misunderstood, do you mean split this patch into multiple
> > > > patches or split all KVM SMMUv3 driver out of this series?
> > >
> > > Yes the latter, the iommu driver introduction is best as its own
> > > series
> >
> > I thought about that but I was worried the maintainers wouldn't like
> > introducing the infrastructure first in the hypervisor without a user.
> > I am open to split this, but let’s see what they think.
>
> You can merge both series at the same time
Ok, I can split the last 12 patches which are the SMMUv3 driver, if the
maintainers are ok with that.
>
> > Makes sense, from the kernel point of view it will be attached to
> > identity/blocking domains, but the hypervisor api is just enable/disable HVC
> > as it doesn’t know what is a domain. If terminology is really a problem,
> > I can make it one hypercall as “set_state” with on/off or identity/blocking
>
> I would call it set_state with states IDENTITY/BLOCKING. That is
> clear. enable/disable is ambiguous.
Ok, will do that.
>
> > TBH, I am not sure what hardware does that. So, another option is to fail
> > gracefully if RMR exists (which falls back to the current driver) and then
> > pKVM would run with DMA isolation, which is the status quo.
>
> iGPUs either access the DRAM through the iommu or they use some OS
> invisible side band channel.
>
> The ones that use the iommu have this quirk.
I see, I think that can be added later, and these devices can keep using the
current SMMU_V3 driver as it, I can add a check to elide this
registeration this driver if the platform have this quirk so the other
driver can probe the SMMUs after.
>
> > They are not random, as part of this series the SMMUv3 driver is split
> > where some of the code goes to “arm-smmu-v3-common.c” which is used by
> > both drivers, this reduces a lot of duplication.
>
> I find it very confusing.
>
> It made sense to factor some of the code out so that pKVM can have
> it's own smmv3 HW driver, sure.
>
> But I don't understand why a paravirtualized iommu driver for pKVM has
> any relation to smmuv3. Shouldn't it just be calling some hypercalls
> to set IDENTITY/BLOCKING?
Well it’s not really “paravirtualized” as virtio-iommu, this is an SMMUv3
driver (it uses the same binding a the smmu-v3)
It re-use the same probe code, fw/hw parsing and so on (inside the kernel),
also re-use the same structs to make that possible. The only difference is
that the page tables and STEs are managed by the hypervisor.
In part-2[1] I add event q parsing, which reuses 90% of the irq/evtq,
insert/remove_master logic, otherwise we have to duplicate all of that logic.
So, I think it makes sense to re-use as much logic as possible, as both drivers
are smmu-v3 drivers with one caveat about HW table management
As mentioned in the cover letter, we can also still build nesting on top of
this driver, and I plan to post an RFC for that, once this one is sorted.
[1] https://android-kvm.googlesource.com/linux/+log/refs/heads/for-upstream/pkvm-smmu-v3-part-2
>
> > I am not sure if we need get_resv_regions, maybe it's useful for sysfs
> > "/sys/kernel/iommu_groups/reserved_regions"? I will double check.
>
> It is important to get this info from the FW..
>
Yes, I think that should remain.
Thanks,
Mostafa
> Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-07-31 17:44 ` Mostafa Saleh
@ 2025-08-01 18:59 ` Jason Gunthorpe
2025-08-04 14:41 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Jason Gunthorpe @ 2025-08-01 18:59 UTC (permalink / raw)
To: Mostafa Saleh
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Thu, Jul 31, 2025 at 05:44:55PM +0000, Mostafa Saleh wrote:
> > > They are not random, as part of this series the SMMUv3 driver is split
> > > where some of the code goes to “arm-smmu-v3-common.c” which is used by
> > > both drivers, this reduces a lot of duplication.
> >
> > I find it very confusing.
> >
> > It made sense to factor some of the code out so that pKVM can have
> > it's own smmv3 HW driver, sure.
> >
> > But I don't understand why a paravirtualized iommu driver for pKVM has
> > any relation to smmuv3. Shouldn't it just be calling some hypercalls
> > to set IDENTITY/BLOCKING?
>
> Well it’s not really “paravirtualized” as virtio-iommu, this is an SMMUv3
> driver (it uses the same binding a the smmu-v3)
> It re-use the same probe code, fw/hw parsing and so on (inside the kernel),
> also re-use the same structs to make that possible.
I think this is not quite true, I think you have some part of the smmu driver
bootstrap the pkvm protected driver.
But then the pkvm takes over all the registers and the command queue.
Are you saying the event queue is left behind for the kernel? How does
that work if it doesn't have access to the registers?
So what is left of the actual *iommu subsystem* driver is just some
pkvm hypercalls?
It seems more sensible to me to have a pkvm HW driver for SMMUv3 that
is split between pkvm and kernel, that operates the HW - but is NOT an
iommu subsystem driver
Then an iommu subsystem driver that does the hypercalls, that is NOT
connected to SMMUv3 at all.
In other words you have two cleanly seperate concerns here, an "pkvm
iommu subsystem" that lets pkvm control iommu HW - and the current
"iommu subsystem" that lets the kernel control iommu HW. The same
driver should not register to both.
> As mentioned in the cover letter, we can also still build nesting on top of
> this driver, and I plan to post an RFC for that, once this one is sorted.
I would expect nesting to present an actual paravirtualized SMMUv3
though, with a proper normal SMMUv3 IOMMU subystem driver. This is how
ARM architecture is built to work, why mess it up?
So my advice above seems cleaner, you have the pkvm iommu HW driver
that turns around and presents a completely normal SMMUv3 HW API which
is bound by the ordinately SMMUv3 iommu subsystem driver.
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-01 18:59 ` Jason Gunthorpe
@ 2025-08-04 14:41 ` Mostafa Saleh
2025-08-05 17:57 ` Jason Gunthorpe
0 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-08-04 14:41 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Fri, Aug 01, 2025 at 03:59:30PM -0300, Jason Gunthorpe wrote:
> On Thu, Jul 31, 2025 at 05:44:55PM +0000, Mostafa Saleh wrote:
> > > > They are not random, as part of this series the SMMUv3 driver is split
> > > > where some of the code goes to “arm-smmu-v3-common.c” which is used by
> > > > both drivers, this reduces a lot of duplication.
> > >
> > > I find it very confusing.
> > >
> > > It made sense to factor some of the code out so that pKVM can have
> > > it's own smmv3 HW driver, sure.
> > >
> > > But I don't understand why a paravirtualized iommu driver for pKVM has
> > > any relation to smmuv3. Shouldn't it just be calling some hypercalls
> > > to set IDENTITY/BLOCKING?
> >
> > Well it’s not really “paravirtualized” as virtio-iommu, this is an SMMUv3
> > driver (it uses the same binding a the smmu-v3)
>
> > It re-use the same probe code, fw/hw parsing and so on (inside the kernel),
> > also re-use the same structs to make that possible.
>
> I think this is not quite true, I think you have some part of the smmu driver
> bootstrap the pkvm protected driver.
>
> But then the pkvm takes over all the registers and the command queue.
>
> Are you saying the event queue is left behind for the kernel? How does
> that work if it doesn't have access to the registers?
The evtq itself will be owned by the kernel, However, MMIO access would be
trapped and emulated, here the PoC for part-2 of this series (as mentioned in
the cover letter) This is very close to how nesting will work.
https://android-kvm.googlesource.com/linux/+/refs/heads/for-upstream/pkvm-smmu-v3-part-2/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c#744
>
> So what is left of the actual *iommu subsystem* driver is just some
> pkvm hypercalls?
Yes at the moment there are only 2 hypercalls and one hypervisor callback
to shadow the page table, when we go to nesting, the hypercalls will be
removed and there will be only data abort callback for MMIO emulation.
>
> It seems more sensible to me to have a pkvm HW driver for SMMUv3 that
> is split between pkvm and kernel, that operates the HW - but is NOT an
> iommu subsystem driver
>
> Then an iommu subsystem driver that does the hypercalls, that is NOT
> connected to SMMUv3 at all.
>
> In other words you have two cleanly seperate concerns here, an "pkvm
> iommu subsystem" that lets pkvm control iommu HW - and the current
> "iommu subsystem" that lets the kernel control iommu HW. The same
> driver should not register to both.
>
I am not sure how that would work exactly, for example how would probe_device
work, xlate... in a generic way? same for other ops. We can make some of these
functions (hypercalls wrappers) in a separate file. Also I am not sure how that
looks from the kernel perspective (do we have 2 struct devices per SMMU?)
But, tbh, i’d prefer to drop iommu_ops at all, check my answer below.
> > As mentioned in the cover letter, we can also still build nesting on top of
> > this driver, and I plan to post an RFC for that, once this one is sorted.
>
> I would expect nesting to present an actual paravirtualized SMMUv3
> though, with a proper normal SMMUv3 IOMMU subystem driver. This is how
> ARM architecture is built to work, why mess it up?
>
> So my advice above seems cleaner, you have the pkvm iommu HW driver
> that turns around and presents a completely normal SMMUv3 HW API which
> is bound by the ordinately SMMUv3 iommu subsystem driver.
>
I think we are on the same page about how that will look at the end.
For nesting there will be a pKVM driver (as mentioned in the cover letter)
to probe register the SMMUs, then it will unbind itself to let the current
(ARM_SMMU_V3) driver probe the SMMUs and it can run unmodified. Which
will be full transparent.
Then the hypervisor driver will use trap and emulate to handle SMMU access
to the MMIO, providing an architecturally accurate SMMUv3 emualation, and
it will not register iommu_ops.
Nor will it use any hypercalls, as the main reason I added those is to tell
the hypervisor what SIDs are used in identity while others remain blocked, as
enabling all the SID space doesn’t only require a lot of memory but also
doesn't feel secure.
With nesting, we don’t need those, as the hypervisor will trap CFGI and will
know what SID to shadow.
However, based on the feedback on my v2 series, it was better to split pKVM
support, so the initial series only establishes DMA isolation. Then when we
can enable full translating domains (either nesting or pv which is another
discussion)
So, I will happily drop the hypercalls and the iommu_ops from this series,
if there is a better way to enlighten the hypervisor about the SIDs to be
in identity.
Otherwise I can’t see any other way to move forward other than going back to
posting large serieses.
Thanks,
Mostafa
> Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-04 14:41 ` Mostafa Saleh
@ 2025-08-05 17:57 ` Jason Gunthorpe
2025-08-06 14:10 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Jason Gunthorpe @ 2025-08-05 17:57 UTC (permalink / raw)
To: Mostafa Saleh
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Mon, Aug 04, 2025 at 02:41:31PM +0000, Mostafa Saleh wrote:
> > Are you saying the event queue is left behind for the kernel? How does
> > that work if it doesn't have access to the registers?
>
> The evtq itself will be owned by the kernel, However, MMIO access would be
> trapped and emulated, here the PoC for part-2 of this series (as mentioned in
> the cover letter) This is very close to how nesting will work.
> https://android-kvm.googlesource.com/linux/+/refs/heads/for-upstream/pkvm-smmu-v3-part-2/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c#744
Oh weird, but Ok.
> > In other words you have two cleanly seperate concerns here, an "pkvm
> > iommu subsystem" that lets pkvm control iommu HW - and the current
> > "iommu subsystem" that lets the kernel control iommu HW. The same
> > driver should not register to both.
> >
>
> I am not sure how that would work exactly, for example how would probe_device
> work, xlate... in a generic way?
Well, I think it is not so bad actually.
You just need to call iommu_device_register
ret = iommu_device_register(&smmu->iommu, &arm_smmu_ops, dev);
Where 'dev' is always the smmu struct device, even if your current
probing driver is not the smmu device. That will capture all the iommu
activity the ACPI/DT says is linked to that dev.
From there you just make a normal small iommu driver with no
connection back to the SMMUv3.
Eg you could spawn an aux device from smmuv3 to do this with a far
cleaner separation.
xlate is just calling iommu_fwspec_add_ids() it is one line.
> same for other ops. We can make some of these
> functions (hypercalls wrappers) in a separate file.
What other ops are you worried about?
static struct iommu_ops arm_smmu_ops = {
.identity_domain = &arm_smmu_identity_domain,
.blocked_domain = &arm_smmu_blocked_domain,
.release_device = arm_smmu_release_device,
.domain_alloc_paging_flags = arm_smmu_domain_alloc_paging_flags,
^^ Those are all your new domain type that does the hypercalls
.probe_device = arm_smmu_probe_device,
.get_resv_regions = arm_smmu_get_resv_regions,
^^ These are pretty empty
.device_group = arm_smmu_device_group,
.of_xlate = arm_smmu_of_xlate,
^^ Common code
Don't need these:
.capable = arm_smmu_capable,
.hw_info = arm_smmu_hw_info,
.domain_alloc_sva = arm_smmu_sva_domain_alloc,
.page_response = arm_smmu_page_response,
.def_domain_type = arm_smmu_def_domain_type,
.get_viommu_size = arm_smmu_get_viommu_size,
.viommu_init = arm_vsmmu_init,
.user_pasid_table = 1,
.pgsize_bitmap = -1UL, /* Restricted during device attach */
.owner = THIS_MODULE,
> Also I am not sure how that
> looks from the kernel perspective (do we have 2 struct devices per SMMU?)
I think you'd want to have pkvm bound to the physical struct device
and then spawn new faux, aux or something devices for the virtualized
IOMMUs that probes the new paravirt driver. This driver would be fully
self contained.
> I think we are on the same page about how that will look at the end.
> For nesting there will be a pKVM driver (as mentioned in the cover letter)
> to probe register the SMMUs, then it will unbind itself to let the
> current
That sounds a little freaky to me. I guess it can be made to work, and
is maybe the best option, but if this is the longterm plan then
starting out with a non-iommu subsystem pkvm driver seems like a good
idea.
Today the pkvm driver would do enough to boot up pkvm in this limited
mode, and maybe you have some small code duplications with the iommu
driver. It forks off a aux device to create the para-virt pkvm iommu
subsystem driver.
Tomorrow you further shrink down the pkvm driver and remove the
para-virt driver, then just somehow bind/unbind the pkvm one at early
boot.
Minimal changes to the existing smmu driver in either case.
> Then the hypervisor driver will use trap and emulate to handle SMMU access
> to the MMIO, providing an architecturally accurate SMMUv3 emualation, and
> it will not register iommu_ops.
Well, it registers normal smmuv3 ops only, I think you mean.
> So, I will happily drop the hypercalls and the iommu_ops from this series,
> if there is a better way to enlighten the hypervisor about the SIDs to be
> in identity.
The iommu ops are a reasonable and appropriate way to do this.
But since pkvm is all so special anyhow maybe you could hook
pci_enable_device() and do your identity hypercall there? Do you
already trap the config write to catch Bus Master Enable? Can pkvm
just set identity when BME=1 and ABORT when BME=0?
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-05 17:57 ` Jason Gunthorpe
@ 2025-08-06 14:10 ` Mostafa Saleh
2025-08-11 18:55 ` Jason Gunthorpe
0 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-08-06 14:10 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Tue, Aug 05, 2025 at 02:57:53PM -0300, Jason Gunthorpe wrote:
> On Mon, Aug 04, 2025 at 02:41:31PM +0000, Mostafa Saleh wrote:
> > > Are you saying the event queue is left behind for the kernel? How does
> > > that work if it doesn't have access to the registers?
> >
> > The evtq itself will be owned by the kernel, However, MMIO access would be
> > trapped and emulated, here the PoC for part-2 of this series (as mentioned in
> > the cover letter) This is very close to how nesting will work.
> > https://android-kvm.googlesource.com/linux/+/refs/heads/for-upstream/pkvm-smmu-v3-part-2/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c#744
>
> Oh weird, but Ok.
>
> > > In other words you have two cleanly seperate concerns here, an "pkvm
> > > iommu subsystem" that lets pkvm control iommu HW - and the current
> > > "iommu subsystem" that lets the kernel control iommu HW. The same
> > > driver should not register to both.
> > >
> >
> > I am not sure how that would work exactly, for example how would probe_device
> > work, xlate... in a generic way?
>
> Well, I think it is not so bad actually.
>
> You just need to call iommu_device_register
>
> ret = iommu_device_register(&smmu->iommu, &arm_smmu_ops, dev);
>
> Where 'dev' is always the smmu struct device, even if your current
> probing driver is not the smmu device. That will capture all the iommu
> activity the ACPI/DT says is linked to that dev.
>
> From there you just make a normal small iommu driver with no
> connection back to the SMMUv3.
>
> Eg you could spawn an aux device from smmuv3 to do this with a far
> cleaner separation.
>
> xlate is just calling iommu_fwspec_add_ids() it is one line.
I am not sure I understand, the SMMU driver will register its IOMMU
ops to probe the devices, then faux devices would also register
IOMMU ops for the KVM HVCs?
But that means that all DMA operations would still go through the
SMMU one?
>
> > same for other ops. We can make some of these
> > functions (hypercalls wrappers) in a separate file.
>
> What other ops are you worried about?
>
> static struct iommu_ops arm_smmu_ops = {
> .identity_domain = &arm_smmu_identity_domain,
> .blocked_domain = &arm_smmu_blocked_domain,
> .release_device = arm_smmu_release_device,
> .domain_alloc_paging_flags = arm_smmu_domain_alloc_paging_flags,
>
> ^^ Those are all your new domain type that does the hypercalls
>
> .probe_device = arm_smmu_probe_device,
> .get_resv_regions = arm_smmu_get_resv_regions,
>
> ^^ These are pretty empty
>
> .device_group = arm_smmu_device_group,
> .of_xlate = arm_smmu_of_xlate,
>
> ^^ Common code
>
> Don't need these:
>
> .capable = arm_smmu_capable,
> .hw_info = arm_smmu_hw_info,
> .domain_alloc_sva = arm_smmu_sva_domain_alloc,
> .page_response = arm_smmu_page_response,
> .def_domain_type = arm_smmu_def_domain_type,
> .get_viommu_size = arm_smmu_get_viommu_size,
> .viommu_init = arm_vsmmu_init,
> .user_pasid_table = 1,
> .pgsize_bitmap = -1UL, /* Restricted during device attach */
> .owner = THIS_MODULE,
Makes sense, thanks for the detailed explanation.
>
> > Also I am not sure how that
> > looks from the kernel perspective (do we have 2 struct devices per SMMU?)
>
> I think you'd want to have pkvm bound to the physical struct device
> and then spawn new faux, aux or something devices for the virtualized
> IOMMUs that probes the new paravirt driver. This driver would be fully
> self contained.
I think it’s hard to reason about this as 2 devices, from my pov it seems
that the pKVM HVCs are a library that can be part of separate common file,
then called from drivers. (with common ops)
Instead of having extra complexity of 2 drivers (KVM and IOMMU PV).
However, I can see the value of that as it totally abstracts the iommu ops
outside the device specific code, I will give it more thought.
But it feels that might be more suitable for a full fledged PV
implementation (as in RFC v1 and v2).
>
> > I think we are on the same page about how that will look at the end.
> > For nesting there will be a pKVM driver (as mentioned in the cover letter)
> > to probe register the SMMUs, then it will unbind itself to let the
> > current
>
> That sounds a little freaky to me. I guess it can be made to work, and
> is maybe the best option, but if this is the longterm plan then
> starting out with a non-iommu subsystem pkvm driver seems like a good
> idea.
>
> Today the pkvm driver would do enough to boot up pkvm in this limited
> mode, and maybe you have some small code duplications with the iommu
> driver. It forks off a aux device to create the para-virt pkvm iommu
> subsystem driver.
>
> Tomorrow you further shrink down the pkvm driver and remove the
> para-virt driver, then just somehow bind/unbind the pkvm one at early
> boot.
>
> Minimal changes to the existing smmu driver in either case.
>
> > Then the hypervisor driver will use trap and emulate to handle SMMU access
> > to the MMIO, providing an architecturally accurate SMMUv3 emualation, and
> > it will not register iommu_ops.
>
> Well, it registers normal smmuv3 ops only, I think you mean.
I mean when nesting is added, the arm-smmu-v3-kvm, will not call
“iommu_device_register”
>
> > So, I will happily drop the hypercalls and the iommu_ops from this series,
> > if there is a better way to enlighten the hypervisor about the SIDs to be
> > in identity.
>
> The iommu ops are a reasonable and appropriate way to do this.
>
> But since pkvm is all so special anyhow maybe you could hook
> pci_enable_device() and do your identity hypercall there? Do you
> already trap the config write to catch Bus Master Enable? Can pkvm
> just set identity when BME=1 and ABORT when BME=0?
pKVM doesn’t trap actual device access, also we have to support
platform devices, so it would be better to rely on existing
abstractions as “probe_device”
I had an offline discussion with Will and Robin and they believe it might
be better if we get rid of the kernel KVM SMMUv3 driver at all, and just
rely on ARM_SMMU_V3 + extra hooks, so there is a single driver managing
the SMMUs in the system.
This way we don’t need to split current SMMUv3 or have different IOMMU ops,
and reduces some of the duplication, also that avoids the need for a fake device.
Then we have an extra file for KVM with some of the hooks (similar to the
hooks in arm_smmu_impl_ops we have for tegra)
And that might be more suitable for nesting also, to avoid the bind/unbind flow.
I will investigate that and if feasible I will send v4 (hopefully
shortly) based on this idea, otherwise I will see if we can separate
KVM code and SMMU bootstrap code.
Thanks,
Mostafa
>
> Jason
^ permalink raw reply [flat|nested] 48+ messages in thread* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-06 14:10 ` Mostafa Saleh
@ 2025-08-11 18:55 ` Jason Gunthorpe
2025-08-12 10:29 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Jason Gunthorpe @ 2025-08-11 18:55 UTC (permalink / raw)
To: Mostafa Saleh
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Wed, Aug 06, 2025 at 02:10:35PM +0000, Mostafa Saleh wrote:
> I am not sure I understand, the SMMU driver will register its IOMMU
> ops to probe the devices
You couldn't do this. But why do you need the iommu subsystem to help
you do probing for the pKVM driver? Today SMMU starts all devices in
ABORT mode except for some it scans manually from the fw tables.
They switch to identity when the iommu subsystem attaches devices, you
can continue to do that by having the paravirt driver tell pkvm when
it attaches.
What is wrong with this approach?
> > > Also I am not sure how that
> > > looks from the kernel perspective (do we have 2 struct devices per SMMU?)
> >
> > I think you'd want to have pkvm bound to the physical struct device
> > and then spawn new faux, aux or something devices for the virtualized
> > IOMMUs that probes the new paravirt driver. This driver would be fully
> > self contained.
>
> I think it’s hard to reason about this as 2 devices, from my pov it seems
> that the pKVM HVCs are a library that can be part of separate common file,
> then called from drivers. (with common ops)
> Instead of having extra complexity of 2 drivers (KVM and IOMMU PV).
> However, I can see the value of that as it totally abstracts the iommu ops
> outside the device specific code, I will give it more thought.
> But it feels that might be more suitable for a full fledged PV
> implementation (as in RFC v1 and v2).
Maybe, but I'm feeling sensitive here to not mess up the ARM SMMU
driver with this stuff that is honestly looking harder and harder to
understand what it is trying to do...
If you can keep the pkvm enablement to three drivers:
- A pKVM SMMU driver sharing some header files
- A the untrusted half of the above driver
- A para virt IOMMU driver
And not further change the smmu driver beyond making some code
sharable it sure would be nice from a maintenance perspective.
> I had an offline discussion with Will and Robin and they believe it might
> be better if we get rid of the kernel KVM SMMUv3 driver at all, and just
> rely on ARM_SMMU_V3 + extra hooks, so there is a single driver managing
> the SMMUs in the system.
> This way we don’t need to split current SMMUv3 or have different IOMMU ops,
> and reduces some of the duplication, also that avoids the need for a fake device.
>
> Then we have an extra file for KVM with some of the hooks (similar to the
> hooks in arm_smmu_impl_ops we have for tegra)
>
> And that might be more suitable for nesting also, to avoid the bind/unbind flow.
>
> I will investigate that and if feasible I will send v4 (hopefully
> shortly) based on this idea, otherwise I will see if we can separate
> KVM code and SMMU bootstrap code.
Maybe, not sure what exactly you imagine here.. You still have your
para virt driver, yes?
This especially is what bothers me, I don't think you should have a
para virt driver for pkvm hidden inside the smmu driver at all.
And if we have a smmu driver that optionally doesn't register with the
iommu subsystem at all - that seems unwise..
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-11 18:55 ` Jason Gunthorpe
@ 2025-08-12 10:29 ` Mostafa Saleh
2025-08-12 12:10 ` Jason Gunthorpe
0 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-08-12 10:29 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Mon, Aug 11, 2025 at 03:55:23PM -0300, Jason Gunthorpe wrote:
> On Wed, Aug 06, 2025 at 02:10:35PM +0000, Mostafa Saleh wrote:
> > I am not sure I understand, the SMMU driver will register its IOMMU
> > ops to probe the devices
>
> You couldn't do this. But why do you need the iommu subsystem to help
> you do probing for the pKVM driver? Today SMMU starts all devices in
> ABORT mode except for some it scans manually from the fw tables.
>
> They switch to identity when the iommu subsystem attaches devices, you
> can continue to do that by having the paravirt driver tell pkvm when
> it attaches.
>
> What is wrong with this approach?
>
My confusion is that in this proposal we have 2 drivers:
- arm-smmu-v3-kvm: Register arm_smmu_ops? binds to the SMMUs
- pkvm-iommu: Register pkvm_iommu_ops, binds to faux devices.
So how does attach/detach... (rest of iommu_ops) work? In that case we need
the pkvm driver to handle those. So, why do we need to have iommu_ops for the
kvm one?
> > > > Also I am not sure how that
> > > > looks from the kernel perspective (do we have 2 struct devices per SMMU?)
> > >
> > > I think you'd want to have pkvm bound to the physical struct device
> > > and then spawn new faux, aux or something devices for the virtualized
> > > IOMMUs that probes the new paravirt driver. This driver would be fully
> > > self contained.
> >
> > I think it’s hard to reason about this as 2 devices, from my pov it seems
> > that the pKVM HVCs are a library that can be part of separate common file,
> > then called from drivers. (with common ops)
> > Instead of having extra complexity of 2 drivers (KVM and IOMMU PV).
> > However, I can see the value of that as it totally abstracts the iommu ops
> > outside the device specific code, I will give it more thought.
> > But it feels that might be more suitable for a full fledged PV
> > implementation (as in RFC v1 and v2).
>
> Maybe, but I'm feeling sensitive here to not mess up the ARM SMMU
> driver with this stuff that is honestly looking harder and harder to
> understand what it is trying to do...
>
> If you can keep the pkvm enablement to three drivers:
> - A pKVM SMMU driver sharing some header files
> - A the untrusted half of the above driver
> - A para virt IOMMU driver
>
> And not further change the smmu driver beyond making some code
> sharable it sure would be nice from a maintenance perspective.
I am almost done with v4, which relies on a single driver, I don’t think
it’s that complicated, it adds a few impl_ops + some few re-works.
I think that is much simpler than having 3 drivers.
Also better for the current SMMUv3 driver maintainability to have the KVM driver
as mode, where all the KVM logic is implemented in a new file which relies on few
ops, similar to “tegra241-cmdqv.c”
I will post this version, and then it would be easier to compare both approaches.
>
> > I had an offline discussion with Will and Robin and they believe it might
> > be better if we get rid of the kernel KVM SMMUv3 driver at all, and just
> > rely on ARM_SMMU_V3 + extra hooks, so there is a single driver managing
> > the SMMUs in the system.
>
> > This way we don’t need to split current SMMUv3 or have different IOMMU ops,
> > and reduces some of the duplication, also that avoids the need for a fake device.
> >
> > Then we have an extra file for KVM with some of the hooks (similar to the
> > hooks in arm_smmu_impl_ops we have for tegra)
> >
> > And that might be more suitable for nesting also, to avoid the bind/unbind flow.
> >
> > I will investigate that and if feasible I will send v4 (hopefully
> > shortly) based on this idea, otherwise I will see if we can separate
> > KVM code and SMMU bootstrap code.
>
> Maybe, not sure what exactly you imagine here.. You still have your
> para virt driver, yes?
>
> This especially is what bothers me, I don't think you should have a
> para virt driver for pkvm hidden inside the smmu driver at all.
>
> And if we have a smmu driver that optionally doesn't register with the
> iommu subsystem at all - that seems unwise..
I was imagining just splitting all the KVM specific code outside of the
SMMU code, but not as a driver, it would be a library which “arm-smmu-v3-kvm”
calls into.
Thanks,
Mostafa
>
> Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-12 10:29 ` Mostafa Saleh
@ 2025-08-12 12:10 ` Jason Gunthorpe
2025-08-12 12:37 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Jason Gunthorpe @ 2025-08-12 12:10 UTC (permalink / raw)
To: Mostafa Saleh
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Tue, Aug 12, 2025 at 10:29:38AM +0000, Mostafa Saleh wrote:
> On Mon, Aug 11, 2025 at 03:55:23PM -0300, Jason Gunthorpe wrote:
> > On Wed, Aug 06, 2025 at 02:10:35PM +0000, Mostafa Saleh wrote:
> > > I am not sure I understand, the SMMU driver will register its IOMMU
> > > ops to probe the devices
> >
> > You couldn't do this. But why do you need the iommu subsystem to help
> > you do probing for the pKVM driver? Today SMMU starts all devices in
> > ABORT mode except for some it scans manually from the fw tables.
> >
> > They switch to identity when the iommu subsystem attaches devices, you
> > can continue to do that by having the paravirt driver tell pkvm when
> > it attaches.
> >
> > What is wrong with this approach?
> >
>
> My confusion is that in this proposal we have 2 drivers:
> - arm-smmu-v3-kvm: Register arm_smmu_ops? binds to the SMMUs
No, I don't mean two iommu subsystem drivers. You have only the
pkvm-iommu driver. Whatever you bind to the arm-smmu does not register
with the iommu subsystem.
> I am almost done with v4, which relies on a single driver, I don’t think
> it’s that complicated, it adds a few impl_ops + some few re-works.
>
> I think that is much simpler than having 3 drivers.
> Also better for the current SMMUv3 driver maintainability to have the KVM driver
> as mode, where all the KVM logic is implemented in a new file which relies on few
> ops, similar to “tegra241-cmdqv.c”
I don't understand how you can do this, it is fundamentally not an
iommu subsystem driver if pkvm is in control of the HW.
And I still strongly do not want to see a para virt iommu driver hidden
inside the smmu driver. It should be its own thing.
The tegra ops were for customizing the iommu subsystem behavior of the
arm iommu driver, not to turn it into a wrapper for a different
paravirt driver!!
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-12 12:10 ` Jason Gunthorpe
@ 2025-08-12 12:37 ` Mostafa Saleh
2025-08-12 13:48 ` Jason Gunthorpe
0 siblings, 1 reply; 48+ messages in thread
From: Mostafa Saleh @ 2025-08-12 12:37 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Tue, Aug 12, 2025 at 09:10:56AM -0300, Jason Gunthorpe wrote:
> On Tue, Aug 12, 2025 at 10:29:38AM +0000, Mostafa Saleh wrote:
> > On Mon, Aug 11, 2025 at 03:55:23PM -0300, Jason Gunthorpe wrote:
> > > On Wed, Aug 06, 2025 at 02:10:35PM +0000, Mostafa Saleh wrote:
> > > > I am not sure I understand, the SMMU driver will register its IOMMU
> > > > ops to probe the devices
> > >
> > > You couldn't do this. But why do you need the iommu subsystem to help
> > > you do probing for the pKVM driver? Today SMMU starts all devices in
> > > ABORT mode except for some it scans manually from the fw tables.
> > >
> > > They switch to identity when the iommu subsystem attaches devices, you
> > > can continue to do that by having the paravirt driver tell pkvm when
> > > it attaches.
> > >
> > > What is wrong with this approach?
> > >
> >
> > My confusion is that in this proposal we have 2 drivers:
> > - arm-smmu-v3-kvm: Register arm_smmu_ops? binds to the SMMUs
>
> No, I don't mean two iommu subsystem drivers. You have only the
> pkvm-iommu driver. Whatever you bind to the arm-smmu does not register
> with the iommu subsystem.
I see.
>
> > I am almost done with v4, which relies on a single driver, I don’t think
> > it’s that complicated, it adds a few impl_ops + some few re-works.
> >
> > I think that is much simpler than having 3 drivers.
> > Also better for the current SMMUv3 driver maintainability to have the KVM driver
> > as mode, where all the KVM logic is implemented in a new file which relies on few
> > ops, similar to “tegra241-cmdqv.c”
>
> I don't understand how you can do this, it is fundamentally not an
> iommu subsystem driver if pkvm is in control of the HW.
>
> And I still strongly do not want to see a para virt iommu driver hidden
> inside the smmu driver. It should be its own thing.
>
> The tegra ops were for customizing the iommu subsystem behavior of the
> arm iommu driver, not to turn it into a wrapper for a different
> paravirt driver!!
I see, but most of the code in KVM mode is exactly the same as in the
current driver, as the driver is not HW agnostic (unlike virtio-iommu).
In fact it does almost everything, and just delegates
attach_identity/blocked to the hypervisor.
IMO, having a full fledged KVM IOMMU driver + faux devices + moving
all shared SMMUv3 code, just for this driver to implement a handful
lines of code, is an over-kill, especially since most of this logic
won’t be needed in the future.
In addition, with no standard iommu-binding, this driver has to be
enlightened somehow about how to deal with device operations.
As mentioned before, when nesting is added, many of the hooks will be
removed anyway as KVM would rely on trap and emulate instead of HVCs.
Otherwise, we can skip this series and I can post nesting directly
(which would be a relatively bigger one), that probably would rely
on the same concept of the driver bootstrapping the hypervisor driver.
I am generally open to any path to move this forward, as Robin and
Will originally suggested the KVM mode in the upstream driver approach,
what do you think?
Thank,
Mostafa
>
> Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-12 12:37 ` Mostafa Saleh
@ 2025-08-12 13:48 ` Jason Gunthorpe
2025-08-13 13:52 ` Mostafa Saleh
0 siblings, 1 reply; 48+ messages in thread
From: Jason Gunthorpe @ 2025-08-12 13:48 UTC (permalink / raw)
To: Mostafa Saleh
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Tue, Aug 12, 2025 at 12:37:08PM +0000, Mostafa Saleh wrote:
> I see, but most of the code in KVM mode is exactly the same as in the
> current driver, as the driver is not HW agnostic (unlike virtio-iommu).
> In fact it does almost everything, and just delegates
> attach_identity/blocked to the hypervisor.
How is that possible? the kvm driver for smmuv3 should be very
different than the iommu subsystem driver. That seemed to be what this
series was showing.. Even the memory allocations and page table code
were all modified?
This delta seems to only get bigger as you move on toward having full
emulation in the hypervisor.
So, I'm confused what is being shared here and why are we trying so
hard to force two different things to share some unclear amount of
code?
> In addition, with no standard iommu-binding, this driver has to be
> enlightened somehow about how to deal with device operations.
>
> As mentioned before, when nesting is added, many of the hooks will be
> removed anyway as KVM would rely on trap and emulate instead of HVCs.
>
> Otherwise, we can skip this series and I can post nesting directly
> (which would be a relatively bigger one), that probably would rely
> on the same concept of the driver bootstrapping the hypervisor driver.
I think you end up with the same design I am suggesting here, it is
nonsense to have one smmu driver when it is actually split into two
instances where one is running inside the protected world and one is
the normal iommu subsystem driver. That's not just bootstrapping, that
is two full instances running in parallel that are really very
different things.
> I am generally open to any path to move this forward, as Robin and
> Will originally suggested the KVM mode in the upstream driver approach,
> what do you think?
Well, I'd like to see you succeed here too, but it these series are
very big and seem a pretty invasive. I'm very keen we are still able
to support the smmuv3 driver in all the non-pkvm configurations
without hassle, and I don't want to become blocked on inscrutible pkvm
problems because we have decided to share a few lines of code..
Are you sure there isn't some inbetween you can do that allows getting
to the full emulation of smmuv3 without so much churn to existing
code?
This is why I prefer adding new stuff that you then just erase vs
mangling the existing drivers potentially forever..
Jason
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [PATCH v3 29/29] iommu/arm-smmu-v3-kvm: Add IOMMU ops
2025-08-12 13:48 ` Jason Gunthorpe
@ 2025-08-13 13:52 ` Mostafa Saleh
0 siblings, 0 replies; 48+ messages in thread
From: Mostafa Saleh @ 2025-08-13 13:52 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-kernel, kvmarm, linux-arm-kernel, iommu, maz, oliver.upton,
joey.gouly, suzuki.poulose, yuzenghui, catalin.marinas, will,
robin.murphy, jean-philippe, qperret, tabba, mark.rutland, praan
On Tue, Aug 12, 2025 at 10:48:08AM -0300, Jason Gunthorpe wrote:
> On Tue, Aug 12, 2025 at 12:37:08PM +0000, Mostafa Saleh wrote:
> > I see, but most of the code in KVM mode is exactly the same as in the
> > current driver, as the driver is not HW agnostic (unlike virtio-iommu).
> > In fact it does almost everything, and just delegates
> > attach_identity/blocked to the hypervisor.
>
> How is that possible? the kvm driver for smmuv3 should be very
> different than the iommu subsystem driver. That seemed to be what this
> series was showing.. Even the memory allocations and page table code
> were all modified?
>
> This delta seems to only get bigger as you move on toward having full
> emulation in the hypervisor.
>
> So, I'm confused what is being shared here and why are we trying so
> hard to force two different things to share some unclear amount of
> code?
>
Originally, this series from v1-v3(currently), just split the common
code (SMMUv3 and io-pgtable) to separate files and added a new driver
so it was a clear boundary.
But, I started re-working the series as I mentioned based on the feedback
to have KVM as mode in the driver.
IMO, there is a value in either approaches (single or 2 drivers), a
single driver is less intrusive to the current driver as it doesn't do
any splitting, and just add few hooks.
As I mentioned I open to both, not trying to force anything :)
> > In addition, with no standard iommu-binding, this driver has to be
> > enlightened somehow about how to deal with device operations.
> >
> > As mentioned before, when nesting is added, many of the hooks will be
> > removed anyway as KVM would rely on trap and emulate instead of HVCs.
> >
> > Otherwise, we can skip this series and I can post nesting directly
> > (which would be a relatively bigger one), that probably would rely
> > on the same concept of the driver bootstrapping the hypervisor driver.
>
> I think you end up with the same design I am suggesting here, it is
> nonsense to have one smmu driver when it is actually split into two
> instances where one is running inside the protected world and one is
> the normal iommu subsystem driver. That's not just bootstrapping, that
> is two full instances running in parallel that are really very
> different things.
But they don’t do the same thing? they collaborate to configure the IOMMUs.
That’s how the kernel boots, it starts as one object then splits between
EL1 and EL2, which runs in parallel. At run time KVM will decide if it
runs fully in EL2 or in split mode and either do function calls or
hypercalls. It makes sense for KVM drivers to follow the same model.
See kvm_call_* for example.
>
> > I am generally open to any path to move this forward, as Robin and
> > Will originally suggested the KVM mode in the upstream driver approach,
> > what do you think?
>
> Well, I'd like to see you succeed here too, but it these series are
> very big and seem a pretty invasive. I'm very keen we are still able
> to support the smmuv3 driver in all the non-pkvm configurations
> without hassle, and I don't want to become blocked on inscrutible pkvm
> problems because we have decided to share a few lines of code..
Makes sense, but making changes is part of evolving the driver, for
example attaching devices looks nothing at the moment as 3 years ago
when I started working on this.
Many of the cmdq code has been reworked for tegra which is not part of
the SMMUv3 architecture.
IMHO, we can’t just reject changes because it’s invasive, or some people
doesn't care about it.
Instead based on maintainability of new changes and whether it makes
sense or not.
>
> Are you sure there isn't some inbetween you can do that allows getting
> to the full emulation of smmuv3 without so much churn to existing
> code?
>
> This is why I prefer adding new stuff that you then just erase vs
> mangling the existing drivers potentially forever..
>
What I can do is to hold back v4, and I can revive my nesting patches,
it doesn’t need any hooks in iommu_ops nor hypercalls.
However, the tricky part would be the probe, as the hypervisor has to know
about the SMMUs before the kernel doesn’t any anything significant with,
but the same time we don't to repeat the probe path.
If that goes well, I can post something next week with nesting instead
of the current approach.
Thanks,
Mostafa
> Jason
^ permalink raw reply [flat|nested] 48+ messages in thread